Disaster recovery

This page is a triage-first runbook: match your symptom to a scope, then follow the matching recovery. It assumes you take regular backups.

Recover confidence with a periodic DR drill that exercises the backup and restore path end to end. Run it at least quarterly so the procedure and your snapshots are proven before you need them.

Note

Routine operation never needs a shell on a node. The few recovery steps below that touch the OS are break-glass tasks: prefer the Azure control-plane alternative where one is shown, and use the deployment’s break-glass SSH (keys-only, restricted to the deploy-time admin username) for the rest. See security.

Triage

Symptom	Scope	Go to
One node not ready (/readyz = 503), peers fine	Single node, transient	Restart a node
One node crash-loops on start; logs mention a database error	Single node, corrupt data	Rebuild a node from peers
Whole cluster lost data, infrastructure intact	Cluster	Restore from backup
Nodes cannot re-form the cluster after an infra change	Cluster	Redeploy and restore

Single node, transient

The node is momentarily unhealthy but the cluster is otherwise fine. Restart it and let it rejoin and re-sync from its peers. Restarting the VM from the Azure control plane is enough; over break-glass SSH you can restart just the service instead:

az vm restart -g <resource-group> -n <vm-name>   # or: systemctl restart dtm-server
curl -sk https://<node>:8443/readyz              # wait for 200

No rebuild needed. Target: minutes.

Single node, corrupt database

The node crash-loops and its logs indicate a database problem. Hayami DTM checks the database for damage at startup before it opens it, so a corrupt or truncated file is never served as if it were good data.

On a healthy cluster this repairs itself automatically. When a node finds its database damaged at startup and can confirm at least one other node is healthy and ready, it:

1. Sets the damaged file aside (renames and keeps it, never deletes it) for post-mortem.
2. Starts with a fresh database, rejoins, and rebuilds a full copy of cluster state from a healthy peer.

The node stays out of service until it has fully caught up, so clients are never sent to a node that is missing your data. Usually no action is needed beyond confirming it recovered.

Safety limit. A node will never discard its only copy of the data without first confirming a healthy peer to rebuild from. A single-node deployment, or a cluster where every other node is also down, therefore keeps the damaged file and waits rather than risk data loss. In that case, restore from your last known-good backup; if the API on that node is unreachable, use the offline break-glass restore instead. If a node’s disk keeps damaging the database, self-heal stops retrying and waits for an operator, which is your signal to check the underlying disk.

If you ever need to do this by hand (for example self-heal has been turned off), it is a break-glass SSH task: stop the service, move the database file and any sidecar files aside (keeping the corrupt file), and start the service again; the node rebuilds from a peer exactly as above.

Verify with /readyz = 200 and by comparing zone and record counts (from GET /api/v1/stats, which returns both in one call) against the other nodes. Target: tens of minutes.

Whole cluster, restore from backup

If the cluster lost data but the infrastructure is intact, restore from your last known-good snapshot. Pick a snapshot from before the failure (not necessarily the newest). Follow the whole-cluster restore procedure. Target: around 30 minutes depending on data size.

Cluster cannot re-form

If nodes are up but cannot rejoin each other after a significant infrastructure change, the reliable path is to redeploy the cluster with a fresh, consistent set of deployment credentials applied to all nodes together, then restore your data from the latest snapshot. Perform cluster credential changes as an all-nodes-at-once operation rather than node by node.

Verify recovery

After any recovery, confirm:

• Cluster size: dtm_cluster_nodes (or GET /api/v1/nodes) equals your deployed node count, all alive.
• Data: zone and record counts match your pre-failure baseline.
• DNS: the same name resolves to the same answer from more than one node.
• The audit log is readable.

Recovery objectives

The target times above are goals, not guarantees; they depend on data size and your infrastructure. Set your backup cadence to match the data-loss window you can tolerate. See backup and restore for cadence guidance and SLA and SLO for availability targets.