Drill a region loss and fail back

Region loss is the scenario DTM is sold against, and the failover half genuinely runs itself. The half that deserves rehearsal is failback: rebuild, rejoin, repoint clients, deregister the corpse, in an order that spans four reference pages. This drill runs the whole arc in a scheduled window, so the real event is a re-run, not a first attempt.

1. Plan the drill

Pick the sacrificial region deliberately: the one WITHOUT the Key Vault (here, westeurope). The vault is the cluster’s secret store; a drill that deallocates the vault’s region tests Azure’s vault availability, not your runbook, and complicates the rebuild.
Confirm a current, verified snapshot exists in blob storage outside the sacrificial region.
Announce the window, and write the abort criteria down first (for example: abort and restore region B immediately if the survivor’s /readyz degrades or the error-rate alert fires).

2. Capture the baseline

Save what “healthy” looks like, in comparable numbers:

curl -sk https://10.0.250.4:8443/api/v1/nodes -H "Authorization: Bearer $DTM_TOKEN" > drill-nodes.json
curl -sk https://10.0.250.4:8443/api/v1/stats -H "Authorization: Bearer $DTM_TOKEN" > drill-stats.json
for n in 10.0.250.4 10.1.250.4; do dig SOA example.internal @$n +short; done
dig @10.1.250.4 app.example.internal +short   # westeurope client's local answer

Zone and record counts plus matching SOA serials are the comparison basis; there is no cluster-wide version number to quote.

3. Simulate the loss, reversibly

Stop and deallocate the westeurope VM. Do not delete anything yet: a deallocated VM is your abort lever. Then observe from a westeurope client: answers keep coming, now from the surviving region (your VNet DNS ordering decides how quickly clients move to the second server; note what you actually see, it is your real RTO input).

4. Triage as if it were real

Run the disaster-recovery triage exactly as the page prescribes, resisting drill-shortcuts: classify the scenario, check the survivor’s /readyz and dashboard, and confirm one node alone is inside SLO for the current query load. The point of the drill is to exercise the triage muscle, not to skip to the rebuild.

5. Decide the rebuild path

The triage outcome forks: an intact node restarts and rejoins (and a corrupt local store self-heals by reseeding from a healthy peer); destroyed infrastructure means redeploying into the region with the add-node plan. This drill takes the redeploy path on purpose, because that is the half nobody has practiced. (Your deallocated VM stays parked as the abort lever until the drill closes.)

6. Rebuild the region

Deploy the add-node template into westeurope against the cluster subnet, the shared (surviving) Key Vault, and the northeurope node as seed, per adding nodes. Two things the real event will also have:

The rebuilt node has a new IP. Nothing inherits the old one; every place the old IP lived must be updated explicitly (next step).
The cluster port glue: confirm the existing nodes’ NSG rule admits the new IP on 7946 (subnet-scoped rules already do; per-IP rules need the new address).

Gate on curl -sk https://<new-ip>:8443/readyz returning 200 before anything else touches it.

7. Fail back, in order

Update the westeurope VNet DNS list to the new node’s IP first (local node first, survivor second); reboot or lease-renew a test client and verify the local-first answer returned:
Terminal window
```
dig @<new-westeurope-ip> app.example.internal +short
```
Then deregister the dead original node from a surviving node (DELETE /api/v1/nodes/<id>); removal also revokes any tokens that node issued, so automation that authenticated against old 10.1.250.4 signs in again.
Only now delete the deallocated VM and its remnants (disks, NIC, orphaned role assignments), per removing nodes.

The ordering rule worth memorizing: clients local again before the corpse is buried; deregistering first costs nothing, but deleting Azure resources before clients are settled removes your abort lever.

8. Close the drill

Compare against step 2: node count and regions back to baseline, zone and record counts matching, SOA serials aligned across both nodes, local-first answers in both regions. Record time-to-recover for the runbook, plus the two numbers people always ask for afterward: how long clients took to move off the dead region (step 3) and how long the rebuild took end to end. Update the runbook with whatever surprised you; that sentence is the drill’s actual product. For a real event that deviates from the rehearsed script, the escalation path is a support bundle to [email protected].

Where next

Disaster recovery: the triage this drill exercises.
Run a backup-and-restore fire drill: the data-loss sibling of this infrastructure drill.
Scale out for an event, then back in: the voluntary version of the same mechanics.

Need a hand? Email [email protected].

Last validated: 2026-07-27

Privacy policyTerms of use