Troubleshooting

Find your symptom below. Each entry gives the likely cause and the fix. For health and cluster status, the web UI Health and Nodes pages and GET /api/v1/nodes are your first stops. Mid-incident and unsure where to start? The diagnose a resolution incident tutorial walks the localize-classify-inspect order that indexes into this page.

A zone is stuck in pending

A zone created with status pending (replication-aware activation) normally flips to active once peers acknowledge the broadcast; ordinary creates are active immediately. If a zone stays pending:

• Check every node is alive: GET /api/v1/nodes.
• Nudge a re-broadcast with a no-op update: PUT /api/v1/zones/<id> with the same settings.

DNS returns SERVFAIL

Usually a forwarding failure for a name DTM is not authoritative for:

• Check the default forwarder is reachable and correct: GET /api/v1/forwarders/default. On Azure the platform resolver is 168.63.129.16.
• Check the node’s logs for a forwarding error.

DNS returns REFUSED

DTM answers REFUSED when a query matches no zone or conditional forwarder and the default forwarder is disabled.

• Confirm the default forwarder is enabled if you expect recursion.
• A name on the blocklist returns NXDOMAIN, not REFUSED; if a name unexpectedly does not resolve, check GET /api/v1/blocklist.

A client receives BADVERS

A query that advertises an EDNS version greater than 0 receives BADVERS rather than being processed, per RFC 6891 section 6.1.3. This is standards-conformant behaviour: EDNS(0) is the only published version, so a client requesting a higher one is misconfigured or probing. Fix or update the client; ordinary resolvers are unaffected.

A health check disagrees between nodes

Health is evaluated per node and is not replicated, so different nodes can briefly hold different views (this is expected). If one node persistently disagrees:

• Test the backend’s reachability from that node.
• Give it time to cross the success threshold.
• Compare each node’s GET /api/v1/health/status.

Every script health check is unhealthy at once

Script probes run inside a separate sandboxed probe-runner service, and they fail closed: if the runner is unreachable, the check reports unhealthy rather than running the command inside the main service. So when every script check on a node goes unhealthy at the same time while tcp and http checks stay green, the cause is almost never a mass backend outage; it is the probe runner.

On the affected node (over SSH):

systemctl status dtm-probe-runner.socket    # should be active (listening)
journalctl -u dtm-server | grep -i "probe runner"

If the runner is healthy but one specific probe keeps failing, the script is probably depending on something the sandbox denies by design:

• Reading node state. The command runs as a dedicated unprivileged user that cannot read DTM’s configuration or data store. A probe that needs cluster state should ask the API over the network instead.
• Reaching the instance metadata service. 169.254.169.254 is blocked, so a script cannot fetch a managed-identity token. Use an https check with http_config.managed_identity instead; see authenticated health probes.

Both restrictions are deliberate; rewrite the probe rather than working around them.

Replication is not working

• Every node must share the same cluster encryption key (the dtm-gossip-key secret the deployment stores in Key Vault). A mismatch is rejected silently, so nodes never converge. Ensure all nodes were deployed with the same key.
• Nodes beyond the first need a reachable seed to join.
• The cluster uses TCP and UDP port 7946 between nodes. Allow it in the NSG / firewall for the DTM nodes.

A node is stuck joining or not becoming ready

Check the readiness detail:

curl -sk https://<node>:8443/readyz

A node that has joined but not yet received a full copy of cluster state refuses traffic until a peer syncs it, which it does automatically. If it stays that way, verify replication connectivity to its seeds (TCP and UDP 7946) and that the cluster encryption key matches.

If an established node dips to 503 during a bulk import, or on a cluster with a very large dataset, check which component the /readyz body reports as failing. The replication guards (replication_queue, replication_freshness) are a deliberate “drain me” signal while replication catches up, not a serving failure: DNS keeps answering, and the node returns to ready once the queue drains or the next full-state sync lands. The freshness window scales itself to your dataset’s sync cadence, and the queue threshold is tunable; see known limitations.

Clock drift causes odd update ordering

DTM assumes cluster nodes are time-synced. If you see edits appear to apply out of order, check clock skew and keep it under about a second across the cluster (Azure VMs are NTP-synced by default). Verify with your time daemon (for example chronyc tracking).

Port 53 is already in use

On Linux, systemd-resolved may hold 127.0.0.53:53. Bind DTM to the VM’s own private IP for DNS (its listen address), or disable systemd-resolved if you are not using it.

Cannot sign in after a password change

• Clear browser cookies and try again.
• Confirm the account is not disabled: GET /api/v1/users.
• If Entra SSO is configured, use Sign in with Microsoft; a local password does not apply to SSO users.
• If the admin password itself is lost, see the next entry.

Recover a lost admin password

If the local admin password is lost, the break-glass account has been disabled, or an Entra SSO misconfiguration has locked every admin out, reset the password offline. This is a break-glass SSH task: it runs against one node’s local data store while the service is stopped.

1. SSH to one node and stop the service:

   sudo systemctl stop dtm-server

2. Reset the admin password:

   sudo dtm-server --reset-admin-password --config /etc/dtm/dtm-server.yaml

   This prints a fresh random password to the terminal and exits. To choose the password yourself, use --admin-password-stdin instead: it reads the new password (16 characters minimum) from standard input, so it never appears in shell history.

   On a cluster with encrypted secrets, the reset needs the cluster encryption key, which it takes exactly as the service does: from the configuration file and environment. If the key is unavailable the command exits with an error and changes nothing, rather than leaving any stored secret unencrypted. If you hit that error, re-run it in the same environment the service unit uses.

3. Restart the service and sign in as admin with the new password:

   sudo systemctl start dtm-server

   The change replicates to the rest of the cluster.

After a reset, the admin account is re-enabled if it was disabled, and all of its existing sessions are revoked, so anything signed in as admin must sign in again. That also makes the reset a useful incident-response step after a suspected credential compromise.

Caution

Store the new password in your vault straight away. If you still keep the first-boot administrator password in Key Vault from the initial deployment, it is no longer current after a reset; update or retire that secret so nobody reaches for a stale credential in the next emergency.

Performance seems degraded

• Very many health checks can saturate the probe workers; reduce probe frequency or the number of checks per node.
• Check the cache hit rate (dtm_dns_cache_hits_total vs dtm_dns_cache_misses_total).
• If the database file has grown large, compact it to reclaim disk (watch dtm_storage_bbolt_free_pages).

The service will not start after a config edit

The server validates its YAML configuration strictly: an unknown or misspelled key anywhere in the file stops startup with an error naming the offending field (for example parsing config: field "require_tsigg" not found). This is deliberate, so a typo in a security-relevant key fails loudly instead of being silently ignored.

• Fix or remove the key named in the error; keys must match the names in the configuration reference exactly.
• An empty or comments-only file is fine; the server starts with defaults.
• If the error is about stored secrets or the encryption key rather than a config field: the node refuses to start when replication.encrypt_key no longer matches the key its stored secrets were written under, instead of silently discarding those secrets. Restore the original key value and the node starts again with nothing lost. Editing the key in configuration is not a rotation path; see rotating secrets.

Deployment (cloud-init) failed on an Azure VM

• Check cloud-init status --long and /var/log/cloud-init-output.log.
• Confirm the VM’s managed identity has Key Vault Secrets User on the Key Vault holding the cluster’s secrets. On the primary (first) node, also confirm Key Vault Secrets Officer: it generates and writes the cluster secrets at first boot, and a not-yet-propagated Officer grant makes the bootstrap retry until the role assignment lands.
• Cloud-init runs once, so reimage or re-run it after fixing the cause.

Capture debug output

For an intermittent problem, open a time-boxed debug window from Settings (or POST /api/v1/debug/enable, admin role) and reproduce the issue. The captured lines render in the Settings debug pane, with a text filter, pause/resume, and a row cap; over the API:

curl -sk https://<node>:8443/api/v1/debug/tail \
  -H "Authorization: Bearer $DTM_TOKEN" \
  -H "X-DTM-CSRF: 1"

Three things to know when reading a capture:

• It is per node. Debug capture is retained in memory on each node (the last 1000 lines, with a dropped counter when it overflows) and never replicated. An empty pane means this node saw nothing, not that the cluster did; repeat against the node that handled the traffic.
• It survives the window. The buffer is kept when debug switches off or its window expires (the pane marks it stale), so the capture you took is still there to read. DELETE /api/v1/debug/tail discards it.
• The journal is unchanged. Debug lines still land in the node’s journal as before; the pane is a convenience view, not a replacement for durable logs.

Still stuck?

Build a support bundle and email [email protected] with the symptom and the bundle attached (optionally encrypted). The bundle gathers the config, diagnostics, cluster state, and recent audit entries support will ask for, with secrets redacted; it can be built from the UI, the API, or offline on a node that will not start. If you cannot produce one, include the relevant node’s /readyz output and GET /api/v1/nodes instead. See also known limitations.