Rolling upgrades

DTM ships as Azure Marketplace VM image versions. Upgrading a node means replacing its VM with one built from the new image version while keeping the node’s private IP, one node at a time. DNS stays available throughout because the other nodes keep answering, and each node drains gracefully before it stops. Nothing is downloaded onto a running node, no node needs outbound internet access, and routine upgrades never need a shell: everything runs from the Azure control plane and the DTM admin UI or API.

Before you start

Confirm the cluster is healthy: the UI Nodes page (or GET /api/v1/nodes) lists every node you deployed, each alive. Count the entries: a failed node drops out of the list rather than changing status.
Confirm DNS is answering, from any client machine: dig @<node-ip> <a-known-name> +short.
Confirm replication is current: zone version values match across nodes.
Note the version each node is running (the UI Lifecycle page lists them, with an update banner when a newer image version is available) so you can roll back.
Read the target version’s notes in release notes and the version-specific checks below.

Upgrading to v1.0.7

Zone transfer allow-list: if a node’s configuration file sets axfr.allowed_cidrs, check that every entry is a CIDR (10.0.0.5/32, not 10.0.0.5). From v1.0.7 an entry that is not one stops the node from starting. See configuration.
Audit log retention: a node that does not set observability.audit_retention keeps entries for 90 days from v1.0.7, so entries older than 90 days are removed within the hour of the upgrade. To keep them, export the log first (GET /api/v1/audit/export), or set audit_retention to a longer window before you upgrade. See audit log retention.
Credential headers: a health check that sends a credential header with tls_skip_verify and no ca_pem is refused at save from v1.0.7. Before you upgrade, give each such check a ca_pem pin or turn verification back on; a check saved that way stops probing until you do. See health probes.
Script templates: before you upgrade, re-save each template with "protocol": "script" with a script block, or change its protocol. See managed scripts.
Health probe policy: from v1.0.7 the policy (allow_insecure_authenticated_probes) applies to the whole cluster. After every node is upgraded, save it once, in Settings, through the API or with Terraform: until then each node keeps the value it had. With Terraform, keep a single dtm_health_probe_policy_config (remove any per-node copies) and run terraform apply -replace=<its address> once, because a plan with no change sends nothing. See health probes.

Choose your path

The mechanism depends on how the cluster was deployed. Both replace the VM and keep its IP; they differ in who owns the deployment’s desired state.

Deployed with	Upgrade with	How it works
ARM template, Bicep, or the Marketplace wizard	Stable-IP swap (Path A)	The Lifecycle page generates a pre-filled deployment plan for a replacement node on the new version, which you run under your own Azure credentials; DTM syncs it, moves the old node’s stable private IP to it, and deletes the old VM.
Terraform	Targeted apply per node (Path B)	Terraform replaces that node’s VM against the same network interface and the same data disk, so the IP and the data are retained by construction. You gate on readiness between nodes.

Whichever path you use, draining is automatic: stopping or replacing the VM signals the service, readiness flips to 503 so a load balancer polling /readyz evicts the node during the drain grace window, in-flight connections drain, the node leaves the cluster cleanly, and the database closes safely.

Path A: stable-IP swap (per node)

Open the Lifecycle page. It shows each node’s version and flags the nodes that have a newer image version available.
Start the swap with POST /api/v1/lifecycle/swap (admin), giving target_node_id, target_version, stable_ip and spare_ip (add "accept_outage": true only if you accept a single-node window). The Lifecycle page tracks the run and offers Retry and Abort; it does not start one. DTM first checks its gates: enough alive nodes for a no-outage swap (at least two, counting the node being replaced, overridable with accept_outage) and no other swap in progress; the replacement is admitted only once it joins the cluster and reports alive. If the node you are replacing has already failed, the first request is refused with a list of identities to choose from: see Replacing a node that has already failed.
Deploy the replacement VM using the plan the page generates: a runnable az deployment group create of the cluster-served add-node template, with the target image version, the seed IPs (the current alive nodes) and, from v1.0.5, the replaced node’s region, VM size, zone, disk and NSG prefixes already filled in. The remaining placeholders carry one-line guidance, and the free IP you choose for the replacement is the same value you give the swap as its spare. The deployment runs under your Azure credentials; DTM’s own identity is never used to deploy the replacement VM.
DTM syncs the replacement, drains the old node, then moves the node’s stable private IP to the replacement, so client DNS settings never change. Once the IP move succeeds, the swap’s final cleanup step deletes the old VM (and its network interface and OS disk) automatically. Nothing on the old node’s data disk is needed afterwards: the replacement replicated its data in before the swap. From v1.0.4 the templates delete the data disk with the VM; a node deployed from an earlier template keeps its data disk when the VM is deleted, so delete that disk yourself once you have verified the upgrade. It is named like the old node’s VM with -data in place of -vm (for example dtm-ha-n1-data beside dtm-ha-n1-vm), and it holds a copy of the cluster’s data and the retired node’s key material.
Verify, wait for convergence, and repeat for the next node.

The Lifecycle page’s cluster table, listing each node with its region, alive status, stable IP, running version, and whether it needs an upgrade

Moving the stable IP needs Microsoft.Network/virtualNetworks/subnets/join/action on the cluster subnet. Contributor on the DTM resource group includes it only when the VNet is in that group. With the VNet in another resource group, either deploy from v1.0.5 with grantSubnetJoin set to true (the Upgrades step of the Marketplace wizard, or the ARM or Bicep parameter), or have an administrator give every node’s managed identity Network Contributor on every cluster subnet, not only its own: a swap is driven from a node other than the one it replaces. The generated plan passes the grant on to the replacement when the role is assigned on the subnet itself. Since v1.0.6, when the old node’s role comes from above the subnet (its virtual network, resource group or subscription), the plan prints a notice: line naming each such role and its scope before the deploy, and passes nothing on: if that role is what let the old node move the stable IP, grant the new node the same before it drives a swap. The plan never creates a grant itself. A role assigned to an Entra group is not seen, so no notice does not prove nothing is inherited.

Replacing a node that has already failed

At its end, a swap retires the replaced node’s cluster identity (its key and its membership), so the old node cannot rejoin as itself. It finds that identity from the node still holding the stable IP. A node that has already failed holds nothing, so the swap cannot tell which identity it replaces.

In that case POST /api/v1/lifecycle/swap returns 400 before anything is deployed or changed, and the response lists retire_candidates: the cluster identities with no current node behind them, each with its node_id and, when the cluster still knows it, where it was last seen (last_seen_addr, status, last_seen). Send the same request again with one of:

"retire_node_id": "<node_id>", taken from retire_candidates: the swap retires that identity at the end. A node that is still listed as a cluster member is never a candidate, and naming one is refused, because retiring it would cut a working node out of the cluster (the replacement, or a mistyped id for another node). An identity you already retired by hand is accepted, and nothing more is retired.
"retire_skip": true: carry on and leave the identity behind. The choice is recorded in the audit log with your identity, and the swap history says the identity was left behind and how to retire it later.

A swap whose node is alive is unchanged: its identity is found automatically and neither field is needed.

The Lifecycle page’s swap history shows what each swap did with the replaced identity, in its Cluster identity column:

Shown	Meaning
retired `<id>`	The replaced node’s identity was retired.
already retired before the swap	The identity you named had already been retired by hand. Nothing was left behind.
left behind: skipped by `<user>`	You chose `retire_skip`.
left behind: not identified	The swap could not tell which identity it replaced.
left behind: was a current cluster member	The identity you named was listed as a cluster member again by the end of the swap, so it was not retired. Check it is the replaced node first.
left behind: retiring failed	The swap tried to retire the identity and could not.

A left-behind identity is retired with DELETE /api/v1/nodes/{id}?force=true, as in Removing nodes; the row shows the id where the swap knows it.

What a swapped node does not carry across

The replacement is a fresh node: an empty data disk and a new node identity. Zones, records, health checks, users and the default forwarder set arrive by replication before the IP moves, so DNS answers are unchanged.

Node-local settings are not replicated and come up as the deployment’s defaults on the replacement: rate limits, dynamic-DNS policy, cache size, forwarder hardening, OTLP export, the Azure Monitor shipper, scheduled backups, a per-node Entra SSO configuration and the region-discovery backend. Re-apply them for the new node after the swap, through the UI, the API (/api/v1/config/*), or the Terraform provider’s per-node resources. A cluster-wide Entra SSO configuration (v1.0.5) arrives by replication. AXFR policy (including NOTIFY) and DoT/DoH transport settings are cluster-wide: they replicate, and arrive with the zones and the default forwarder set.

Files on the node are not carried across either. The replacement boots from the new image with a fresh operating-system disk, so anything installed by hand on the old node is gone: a per-node UI certificate (the replacement mints a new self-signed pair), an API certificate placed under /etc/dtm/api-tls, or any other break-glass change. Re-install them on the replacement, following TLS certificates. An API certificate uploaded through the API is the exception: the replacement receives it from the cluster in its first sync and serves it.

Path B: targeted Terraform apply (per node)

Terraform-deployed clusters ship two nodes as separate module instances fed from a single image variable, so a plain terraform apply after bumping the image replaces both nodes at once. Target each node explicitly.

Set the image version variable to the new version. Pin the exact version rather than latest, so every node lands on the same one.

Apply to one node and wait for it to be ready before touching the next:

terraform apply -target=module.dtm_node1
# gate on readiness over the network, not on the VM being "running"
until [ "$(curl -sk -o /dev/null -w '%{http_code}' --max-time 3 https://<node1-ip>:8443/readyz)" = "200" ]; do sleep 5; done

Repeat for the next node, then run a final untargeted terraform apply. Expect no changes; if it wants to change a node, stop and read the plan.

The plan for an image change shows the VM being replaced, with the network interface, the data disk and its attachment untouched. The node keeps its IP, its data, its identity and its node-local settings, so nothing in DTM needs re-applying afterwards.

The operating-system disk is new, though. Configuration is re-rendered from your Terraform module, and the UI certificate is re-fetched from Key Vault, so those survive; anything placed on the node by hand outside the module (an API certificate under /etc/dtm/api-tls, a break-glass edit) does not. Re-install it on the replaced node, following TLS certificates. An API certificate uploaded through the API needs nothing: it lives on the retained data disk.

Verify after each node

All of this is done over the network; none of it needs a shell:

# 200 = ready to serve (startup can take a few seconds up to about a
# minute on nodes with many health checks; a replacement node's first
# boot can add several minutes while its Key Vault access propagates)
curl -sk https://<node-ip>:8443/readyz

The UI Nodes or Lifecycle page (or GET /api/v1/nodes) shows the node alive and reporting the new version.
dig @<node-ip> <a-known-name> +short answers, from any client machine.
Zone version values match the other nodes.

Then wait at least 30 seconds for the cluster to fully converge before starting the next node.

Mixed-version window

While old and new versions coexist mid-roll, hold off on configuring features introduced by the target version. An older node does not understand a new record field and can strip it as changes replicate; the record degrades to its safe fallback (for example, a weighted policy serves as round-robin), never to an outage, and heals once every node runs the new version. Finish the roll, then adopt the new capability.

OS patching

Operating-system security updates are separate from DTM image upgrades: the image installs them automatically and never reboots a node on its own (the deployments do not enrol the VMs in Azure Update Manager by default: its automatic guest patching does not support Marketplace images like DTM’s). When an update needs a reboot, schedule it yourself, one node at a time. The reboot is absorbed by the same drain-and-rejoin path, so a rebooted node leaves and rejoins the cluster the same way an upgraded one does.

Rollback

Roll back with the same path you upgraded with, using the previous image version, one node at a time. Released images keep the database format compatible, so your DNS data needs no restore.

Path A: a second swap per node with target_version set to the previous version. It needs a spare IP again and the previous version still listed in the offer.
Path B: set the image variable back and apply per node with the readiness gate, exactly as the forward roll. Never roll back with an untargeted apply.

Rolling back below v1.0.4 can sign out recently renewed sessions

From v1.0.4 nodes use a faster session-refresh format, adopted only once every member of the cluster runs v1.0.4 or later (a node that is down and not removed counts as older). An earlier version cannot read it, so:

Rolling a node back to an earlier version after every node reached v1.0.4 can sign out a user whose session renewed since then, when that node answers their next refresh. They sign in again; nothing else is affected.
The nodes still on v1.0.4 accept both forms, and go back to bcrypt as soon as they see the rolled-back node, so sessions renewed after that work on every node.
Sessions renewed before the last node reached v1.0.4 are not affected. Rolling forward never signs anyone out.

If something goes wrong

A node that does not come back ready: roll it back on the previous image version with the same path, or read its logs over break-glass OS access. See troubleshooting.
A node that rejoins but looks out of date: wait one convergence cycle; if it is still behind, make a no-op change (for example re-save a zone) on a healthy node to trigger a re-broadcast.
Multiple nodes down at once usually means the process moved too fast: bring a node back to restore availability, then slow down.
Never redeploy a node under its existing name with a fresh data disk. A node’s identity is tied to its name; a node that comes back with the same name but a new data disk is refused by its peers and cannot serve the API cluster-wide until its id is removed and readmitted, from a healthy node: DELETE /api/v1/nodes/<id> (add ?force=true if it is no longer listed), then POST /api/v1/nodes/<id>/readmit against the same node, then start or restart the replacement within ten minutes. If it misses the window, readmit again and restart. A removal without the readmit leaves the id refused indefinitely. Disaster recovery shows the commands. The two paths above avoid this by design: the swap deploys under a new name, and Terraform keeps the disk.

Privacy policyTerms of use