Removing nodes (scaling in)

Removing a node is the reverse of adding one: scale in after a migration peak, dispose of a failed VM, or retire capacity you no longer need. Done in the right order it is invisible to clients. Done in the wrong order it silently breaks resolution for every VNet that still lists the removed node’s IP as a DNS server, so the order below matters.

Caution

Do not scale a production cluster below two nodes. A single node has no failover and cannot take zero-downtime rolling upgrades. If you are replacing a node rather than removing one, use the stable-IP swap in the rolling upgrade procedure instead: it keeps the node’s IP, so client DNS settings never change. That includes a node that has already failed: the swap asks which cluster identity to retire and retires it for you, so there is nothing to remove by hand first (see replacing a node that has already failed).

Before you start

• A healthy cluster: every node you are keeping is listed and reports alive (see Verify the cluster).
• Know everywhere the node’s IP appears: VNet DNS server lists, resolver forwarding rules, NSG rules that list node addresses individually, and the seed lists in your Terraform configuration.
• Plan to call the API on a node you are keeping. A node refuses to remove itself, so the request must go to one of the remaining nodes.

Step 1: drain client DNS traffic

First, remove the node’s IP from every VNet DNS server list and any resolver forwarding rule that points at it (this is the reverse of the last step of adding a node). Then give clients time to move off it: watch the node’s queries per second fall on the Dashboard before continuing.

Skipping this step is the classic failure: the node disappears while VNets still list its IP, and clients that picked it keep timing out until their resolver fails over.

Step 2: stop the node

Before you stop it, note the node’s id: the id field for that node in GET /api/v1/nodes. Once the node stops it drops out of that list, so this is the easiest moment to read it.

Stop the VM from the Azure portal or CLI; no shell is needed:

az vm stop -g <resource-group> -n <vm-name>

The guest shutdown delivers SIGTERM and DTM exits gracefully (/readyz flips to draining first, so anything polling readiness sees it go). Stopping just the service (sudo systemctl stop dtm-server) works too, but that is break-glass SSH access; routine removal never needs it.

Either way, the remaining nodes notice within seconds and mark it failed. DNS keeps being answered by the nodes you kept; nothing about your zones or records changes.

Step 3: remove it from the cluster

Remove the stopped node over the API from a remaining node (admin role; capture the $DTM_TOKEN bearer token per the API overview). A stopped node has left the cluster membership, so it no longer appears on the Nodes page or in GET /api/v1/nodes: there is no row to remove in the UI, and a plain DELETE answers 404 (not a current cluster member) and removes nothing. Add ?force=true, which retires an id the cluster no longer lists. The removal is recorded in the audit log as a forced removal.

Caution

Removal is a cluster-side eviction: it revokes the node’s credentials and tells every peer to drop it, but it does not stop the node’s own service. A node removed while still running keeps answering DNS on port 53 from a datastore that no longer receives updates, with no error at any layer, a silent stale resolver for any client still pointed at its IP. That is why draining traffic (step 1) and stopping the node (step 2) come first.

# Confirm the id: every node this one holds a key for is listed as a "kid",
# including the stopped node (GET /api/v1/nodes no longer lists it)
curl -sk https://<remaining-node>:8443/api/v1/jwks \
  -H "Authorization: Bearer $DTM_TOKEN"

# Remove it
curl -sk -X DELETE "https://<remaining-node>:8443/api/v1/nodes/<id>?force=true" \
  -H "Authorization: Bearer $DTM_TOKEN" \
  -H "X-DTM-CSRF: 1"

Check the id before you send it. An id the node you call has no record of is refused with 404 (most likely a typo), but forcing a real but wrong id retires that other node. The ?force value must be exactly true; anything else is refused with 400.

Cleanup completes within about 30 seconds. Removal also revokes the departed node’s cluster credentials, so any API token that was issued by that node stops being accepted cluster-wide: if automation signed in against the removed node, sign it in again against a remaining one.

Step 4: delete the Azure resources

• With Terraform: remove the node’s module block (or lower the node count), then terraform plan and terraform apply. Update any seed list that referenced the removed node’s IP.
• In the portal: delete the VM, its OS and data disks, and its NIC. If an NSG cluster rule lists node addresses individually, remove the node’s IP from it. The VM’s system-assigned managed identity is deleted with the VM, but its role assignments (Key Vault Secrets User on the vault, Reader and Contributor on the resource group) are left behind as orphans; tidy them up under the resource group’s access control (IAM).

Verify

From a remaining node, confirm the removed node is gone and everything you kept is healthy:

# Removed node absent; every remaining node listed (count them) and alive
curl -sk https://<remaining-node>:8443/api/v1/nodes \
  -H "Authorization: Bearer $DTM_TOKEN"

# Each remaining node still answers
dig @<remaining-node-ip> <a-known-name> +short

Then spot-check that a client in each affected VNet resolves normally.

What changes when the cluster shrinks

• Answers follow membership. The apex NS records and glue reflect live cluster membership, so answers stop listing the removed node automatically; no zone edits are needed.
• Failure tolerance shrinks too. Going from three nodes to two keeps failover, but the cluster no longer tolerates a second concurrent failure.
• Retiring a whole region: drain that region’s node IPs from client DNS settings first, then either remove its nodes one at a time as above or tear down the region’s resource group as described in Deploy.

Next steps

• Adding nodes (scaling out): the reverse procedure.
• Rolling upgrades: replace a node without changing its IP.
• Observability: watch the remaining nodes take up the load.