Scale out for an event, then back in

Adding a node and removing one are each documented; the round trip is where people get hurt, because the two halves have opposite ordering rules. Going out: a node goes into client DNS only after it proves ready. Coming back in: a node leaves client DNS before anything else happens to it. This tutorial runs the full arc for a seasonal traffic event, including the step both single pages omit: extending backups and monitoring so the event itself is observable.

Scenario

An admin at Bramwell Retail (fictional), two-node dual-region cluster (10.0.250.4 northeurope, 10.1.250.4 westeurope). Peak season needs a third node in northeurope for six weeks, deployed from the Marketplace add-node plan and removed afterward. Budget an hour each way, plus waits. API steps send an admin bearer token as $DTM_TOKEN (how to capture one).

1. Decide with metrics, not vibes

Read the dashboard against the SLO table: query rate trend, p99 latency headroom, and last year’s peak. A third node buys both capacity and the ability to lose one node during the event without losing failover. Pick the region the load lands in, and confirm the DTM subnet has free addresses.

2. Pre-flight the cluster

Topology changes start from a known-good state:

• Every node alive in GET /api/v1/nodes.
• Zone data converged: same SOA serial from every node (there is no version-compare API; the digs are the check):

for n in 10.0.250.4 10.1.250.4; do dig SOA example.internal @$n +short; done

• A fresh, verified backup in hand.

3. Deploy the add-node plan

Deploy the Marketplace add-node template (or the Terraform module) into the existing cluster’s subnet, pointing at the shared Key Vault and the existing node IPs as seeds; adding nodes is the full reference. Network note, split correctly: the template creates the new node’s own NSG for you; the one thing to confirm is that the existing nodes admit the new node’s IP on the cluster port 7946 (already covered when every node sits in the same DTM subnet’s cluster rule).

4. Gate on readiness, not liveness

The new node full-syncs cluster state before it will serve:

curl -sk https://10.0.250.6:8443/readyz   # 200 = ready; 503 = still syncing

Caution

Gate on /readyz, never /healthz: a syncing node answers liveness long before it can serve correct answers, and putting it in client DNS early is how a capacity add causes a resolution incident.

Then prove it answers correctly: dig @10.0.250.6 app.example.internal +short and compare with a settled node.

5. Put it in service

Add 10.0.250.6 to the northeurope VNet’s DNS server list (local nodes first, per the dual-region ordering); clients pick it up on restart or lease renewal. Verify from a client subnet through the new node.

6. Extend coverage to the new node

The step the event depends on and both single pages omit:

• Monitoring: add the node to the Prometheus scrape (or confirm the Azure Monitor streams are enabled in its config), and check the dashboard’s node selector shows all three (observability).
• Backups: if the scheduler is enabled in config, enable it on the new node too; snapshots are per node.
• Alert thresholds: any rule assuming a two-node cluster (node-count alerts) now expects three.

The event runs; the dashboard shows three nodes sharing it.

7. Scale back in, by exact mirror

Six weeks later, reverse in the strict order (removing nodes is the full runbook):

1. Drain client DNS first: remove 10.0.250.6 from the VNet DNS list, wait for query rate on the node to fall to zero on the dashboard.
2. Stop the node (systemctl stop dtm-server, or stop the VM).
3. Then deregister it, from a node you are keeping:

curl -sk -X DELETE https://10.0.250.4:8443/api/v1/nodes/<id> \
  -H "Authorization: Bearer $DTM_TOKEN" -H "X-DTM-CSRF: 1"

Removal also revokes tokens the departed node issued, so re-authenticate any automation that had logged in against it. Finally delete the Azure resources, including the orphaned role assignments.

8. Verify the end state

Symmetry check against step 2: remaining nodes alive and serving matching SOA serials, no alert referencing the removed node, scrape and backup configs pruned back to two nodes, and the subscription’s cost line back at baseline. The round trip is closed when the cluster looks exactly like step 2’s snapshot, plus one uneventful peak season in the metrics history.

Where next

• Adding nodes and removing nodes: the two halves in reference form.
• Drill a region loss: the involuntary version of this tutorial.
• SLA and SLO: the numbers the step-1 decision reads against.

Need a hand? Email [email protected].

Last validated: 2026-07-27