Run a backup-and-restore fire drill

A backup that has never been restored is a hope, not a control. This drill rehearses the whole restore path on a production cluster with near-zero data rewind, times it, and produces the evidence pack auditors ask for. It also demonstrates something the docs can only assert: after a single-node restore, the cluster heals the node back to current state on its own, and you will measure how fast.

Scenario

An admin at Aldertree Financial (fictional), three-node cluster, scheduled backups already running per backup and restore. Compliance requires a quarterly rehearsal with measured timings, matching the quarterly floor the docs recommend. Budget about 45 minutes, including one node restart. API steps send an admin bearer token as $DTM_TOKEN (how to capture one).

Success criteria, agreed before you start: the snapshot pipeline proven end to end (integrity, staging, swap, verification), production data rewound by approximately nothing, and two stopwatch numbers recorded: time-to-restore and replication convergence.

1. Take a fresh snapshot, right now

Drilling against a fresh snapshot is what makes the rewind near-zero:

curl -sk https://10.0.250.4:8443/api/v1/backup \
  -H "Authorization: Bearer $DTM_TOKEN" -H "X-DTM-CSRF: 1" \
  -o drill.db -D headers.txt
sha256sum drill.db > drill.db.sha256
az storage blob upload --account-name <account> --container-name dtm-backups --auth-mode login --file drill.db

The response also carries the digest in an X-DTM-Backup-SHA256 trailer; keep headers.txt for the evidence pack.

Tip

Two independent mechanisms keep this drill safe on production: the snapshot is minutes old, and the cluster converges the restored node back to current state regardless (step 5 measures exactly that). You are rehearsing with a net under the net.

2. Verify integrity before you trust it

sha256sum -c drill.db.sha256

Staging adds its own guard: the restore endpoint verifies the snapshot’s signed envelope against the cluster encryption key and confirms the file is structurally complete, so a truncated, tampered, or wrong-cluster file is rejected before the node’s local data store is touched.

3. Plant the canary

Create a record the snapshot does not contain, so the rewind is observable: zone drill.example.internal., record canary with any value, created after step 1’s snapshot. Note the time.

4. Stage and apply the restore on one node

curl -sk -X POST https://10.0.250.4:8443/api/v1/restore \
  -H "Authorization: Bearer $DTM_TOKEN" -H "X-DTM-CSRF: 1" \
  --data-binary @drill.db
# 202: {"status": "staged", "next_action": "systemctl restart dtm-server", ...}

Danger

Stage while the server is running, always. The API only listens while the service is up, so “stop the service, then POST the restore” dead-ends with a connection refused, and during a real incident that surprise costs you minutes you do not have.

Apply the swap by restarting that node (az vm restart, or systemctl restart dtm-server over break-glass SSH), and start the RTO clock at the restart.

5. Verify the rewind, then watch the cluster heal it

This is a two-phase check, and the order is the point:

1. Immediately after restart, before convergence: query the restored node directly and confirm the canary is absent:

   dig @10.0.250.4 canary.drill.example.internal +short
   # (empty)   the node is answering from the snapshot: the rewind happened

   In this same pre-convergence window, zone and record counts on the node match the snapshot, if you want the fuller assertion.

2. Then keep querying. Within the convergence window the canary reappears on the restored node, served back to it by its peers, who hold the newer write. Stop your convergence stopwatch when it answers, and compare against the published replication convergence SLO of under 60 seconds.

Stop the RTO clock when curl -sk https://10.0.250.4:8443/readyz returns 200. After convergence, counts match the cluster, not the snapshot; that is the healing working, not the restore failing.

6. Clean up

• The swap leaves a rollback copy on the node. Either delete it now that the drill passed, or practice rolling back to it first; both are break-glass SSH tasks described on backup and restore.
• Delete the drill.example.internal. zone.
• Confirm the node’s disk usage is back to normal.

7. Assemble the evidence pack

Everything an auditor asks for came out of the drill:

curl -sk "https://10.0.250.4:8443/api/v1/audit?resource=backup" \
  -H "Authorization: Bearer $DTM_TOKEN"
# resource=backup: byte count and SHA256
curl -sk "https://10.0.250.4:8443/api/v1/audit?resource=restore" \
  -H "Authorization: Bearer $DTM_TOKEN"
# resource=restore: result=staged

File: measured RTO against the recovery objectives on disaster recovery; measured convergence against the SLO table; implied RPO as your backup cadence (the SLO table itself has no RTO/RPO rows; do not cite it for those). Plus the snapshot digest, the sidecar check, and the canary timestamps.

Know the drill’s boundary

Step 5 proved that a single-node restore in a live cluster cannot rewind cluster state: peers heal the node forward, every time, by design. Which is exactly the drill’s limit: when you genuinely need to rewind the whole cluster (bad bulk change replicated everywhere), that is a full-cluster restore, a different procedure with different stakes; read disaster recovery before you ever need it, and do not improvise it from this page.

Where next

• Run a quarterly maintenance pass: the recurring run that keeps this drill on the calendar.
• Drill a region loss and fail back: the infrastructure-loss sibling of this data drill.
• Disaster recovery: full-cluster restore and region-loss playbooks.
• Backup and restore: the scheduling and rollback mechanics this drill leans on.

Need a hand? Email [email protected].

Last validated: 2026-07-26