Troubleshooting

Find your symptom below. Each entry gives the likely cause and the fix. For health and cluster status, the web UI Health and Nodes pages and GET /api/v1/nodes are your first stops. Mid-incident and unsure where to start? The diagnose a resolution incident tutorial walks the localize-classify-inspect order that indexes into this page.

A Marketplace node refuses to start: certificate signed by unknown authority

At startup a node deployed from the Azure Marketplace verifies its Azure Marketplace entitlement. If the journal shows certificate signed by unknown authority and names a Microsoft TLS G2 RSA CA OCSP issuer that “is not bundled and could not be fetched”, Azure has started signing with a Microsoft issuing certificate this release does not carry. Azure moves between these certificates without notice.

From v1.0.5 you have two ways out:

1. Upgrade to a release that carries that certificate. This is the answer for a private network with no outbound internet access.
2. Allow outbound HTTP (port 80) from the node to www.microsoft.com, and restart it. The node fetches the certificate from Microsoft’s certificate repository, accepts it only if it chains to the Microsoft root built into DTM, keeps it for later restarts, and logs a warning that it did. It contacts no other address and follows no redirect.

Either way the check is as strict as before. A node that had to fetch a certificate counts it in dtm_entitlement_issuer_fallback_total: treat that as the cue to upgrade before you deploy a node with no outbound access.

A zone is stuck in pending

A zone created with status pending (replication-aware activation) normally flips to active once the node has finished sending it to its peers (normally within a few seconds, at most about 10); ordinary creates are active immediately. If a zone stays pending:

• Check that GET /api/v1/nodes lists every node you deployed, each alive. Count the entries: a failed node drops out of the list rather than changing status.
• Nudge a re-broadcast with a no-op update: PUT /api/v1/zones/<id> with the same settings.

DNS returns SERVFAIL

Usually a forwarding failure for a name DTM is not authoritative for:

• Check the default forwarder is reachable and correct: GET /api/v1/forwarders/default. On Azure the platform resolver is 168.63.129.16.
• Check the node’s logs for a forwarding error.

DNS returns REFUSED

DTM answers REFUSED when a query matches no zone or conditional forwarder and the default forwarder is disabled or declines to forward it.

• Confirm the default forwarder is enabled if you expect recursion.
• If it is enabled and you still get REFUSED, the default forwarder refused the query without asking any upstream. It always refuses ANY queries (test with a specific type such as A instead), and forwarder hardening refuses names outside its qname_allowlist and clients over their per_client_qps limit. Check GET /api/v1/forwarders/hardening.
• A name on the blocklist returns NXDOMAIN, not REFUSED; if a name unexpectedly does not resolve, check GET /api/v1/blocklist.

A client receives BADVERS

A query that advertises an EDNS version greater than 0 receives BADVERS rather than being processed, per RFC 6891 section 6.1.3. This is standards-conformant behaviour: EDNS(0) is the only published version, so a client requesting a higher one is misconfigured or probing. Fix or update the client; ordinary resolvers are unaffected.

A health check disagrees between nodes

Health is evaluated per node and is not replicated, so different nodes can briefly hold different views (this is expected). If one node persistently disagrees:

• Test the backend’s reachability from that node.
• Give it time to cross the success threshold.
• Compare each node’s GET /api/v1/health/status.

Every script health check is unhealthy at once

Script probes run inside a separate sandboxed probe-runner service, and they fail closed: if the runner is unreachable, the check reports unhealthy rather than running the command inside the main service. So when every script check on a node goes unhealthy at the same time while tcp and http checks stay green, the cause is almost never a mass backend outage; it is the probe runner.

On the affected node (over SSH):

systemctl status dtm-probe-runner.socket    # should be active (listening)
journalctl -u dtm-probe-runner -n 50        # the runner's own log ("probe-runner:" lines)

The main service does not log the runner failure itself. The reason is in the failing check’s error field in GET /api/v1/health/status, and starts with script probe runner: for example script probe runner unavailable at /run/dtm/probe.sock (the default socket path) when the runner is unreachable.

If the runner is healthy but one specific probe keeps failing, the script is probably depending on something the sandbox denies by design:

• Reading node state. The command runs as a dedicated unprivileged user that cannot read DTM’s configuration or data store. A probe that needs cluster state should ask the API over the network instead.
• Reaching the instance metadata service. 169.254.169.254 is blocked, so a script cannot fetch a managed-identity token. Use an https check with http_config.managed_identity instead; see authenticated health probes.

Both restrictions are deliberate; rewrite the probe rather than working around them.

Replication is not working

• Every node must share the same cluster encryption key (the dtm-gossip-key secret the deployment stores in Key Vault). A node with another key cannot read its peers’ replication traffic, so the nodes never converge. Each node logs the packets it rejects as memberlist: Decrypt packet failed: no installed keys could decrypt the message (search its journal for Decrypt packet failed). Ensure all nodes were deployed with the same key.
• Nodes beyond the first need a reachable seed to join.
• The cluster uses TCP and UDP port 7946 between nodes. Allow it in the NSG / firewall for the DTM nodes.

A node is stuck joining or not becoming ready

Check the readiness detail:

curl -sk https://<node>:8443/readyz

A node that has joined but not yet received a full copy of cluster state refuses traffic until a peer syncs it, which it does automatically. If it stays that way, verify replication connectivity to its seeds (TCP and UDP 7946) and that the cluster encryption key matches.

If an established node dips to 503, check which component the /readyz body reports as failing. replication_freshness is a deliberate “drain me” signal while the node catches up with its peers, not a serving failure: DNS keeps answering, and the node returns to ready once the next full-state sync lands. The freshness window scales itself to your dataset’s sync cadence. A deep replication queue (replication_queue, for example during a bulk import) is named in the body but no longer makes the node not-ready from v1.0.4; a sustained one raises DTMReplicationQueueBacklog, which usually points at an unhealthy peer. See known limitations.

Clock drift causes odd update ordering

DTM assumes cluster nodes are time-synced. If you see edits appear to apply out of order, check clock skew and keep it under about a second across the cluster (Azure VMs are NTP-synced by default). Verify with your time daemon (for example chronyc tracking).

Port 53 is already in use

On Linux, systemd-resolved may hold 127.0.0.53:53. Bind DTM to the VM’s own private IP for DNS (its listen address), or disable systemd-resolved if you are not using it.

Cannot sign in after a password change

• Clear browser cookies and try again.
• Confirm the account is not disabled: GET /api/v1/users.
• If Entra SSO is configured, use Sign in with Microsoft; a local password does not apply to SSO users.
• If the admin password itself is lost, see the next entry.

Recover a lost admin password

If the local admin password is lost, the break-glass account has been disabled, or an Entra SSO misconfiguration has locked every admin out, reset the password offline. This is a break-glass SSH task: it runs against one node’s local data store while the service is stopped.

1. SSH to one node and stop the service:

   sudo systemctl stop dtm-server

2. Reset the admin password. Run the command through systemd, exactly as shown:

   sudo systemd-run --quiet --pipe --wait --collect \
       -p User=dtm -p Group=dtm -p UMask=0027 \
       -p EnvironmentFile=/etc/dtm/dtm-server.env \
       -p ProtectSystem=strict -p ReadWritePaths=/var/lib/dtm \
       /usr/local/bin/dtm-server --config /etc/dtm/dtm-server.yaml --reset-admin-password

   This prints a fresh random password to the terminal and exits.

   To choose the password yourself (16 characters minimum), type it at a hidden prompt and hand it over on standard input:

   read -rs -p 'New admin password: ' pw; echo
   printf '%s\n' "$pw" | sudo systemd-run --quiet --pipe --wait --collect \
       -p User=dtm -p Group=dtm -p UMask=0027 \
       -p EnvironmentFile=/etc/dtm/dtm-server.env \
       -p ProtectSystem=strict -p ReadWritePaths=/var/lib/dtm \
       /usr/local/bin/dtm-server --config /etc/dtm/dtm-server.yaml --admin-password-stdin
   unset pw

   Why this shape:

   • The encryption key. The reset needs the cluster encryption key to unseal the cluster’s stored secrets. On a node deployed from the Marketplace, Bicep or Terraform templates, that key is not in dtm-server.yaml: it is DTM_REPLICATION_ENCRYPT_KEY in the root-only /etc/dtm/dtm-server.env, which systemd loads for the service. -p EnvironmentFile= has systemd read that file with the same parser it uses for the service, so the reset gets exactly the service’s key, and nothing in the file is executed. Run with plain sudo or sudo -u dtm instead, the command has no key, and DTM refuses to run it.
   • Never generate a new key for an existing node. Its secrets are sealed with the current one.
   • Do not load the env file with a shell (. /etc/dtm/dtm-server.env). A shell evaluates the file instead of reading it as data.
   • Ownership and sandbox. -p User=dtm -p Group=dtm -p UMask=0027 keep the data store owned by the service account, and -p ProtectSystem=strict -p ReadWritePaths=/var/lib/dtm let the command write the store and nothing else.
   • The password never goes on a command line. Do not pass it as an argument to --admin-password: a command line is visible to every local user while it runs, and both sudo and systemd log it to the journal. printf is a bash builtin, so the stdin form never puts the password in a process argument either.

3. Restart the service and sign in as admin with the new password:

   sudo systemctl start dtm-server

   The change replicates to the rest of the cluster.

After a reset, the admin account is re-enabled if it was disabled, and all of its existing sessions are revoked, so anything signed in as admin must sign in again. That also makes the reset a useful incident-response step after a suspected credential compromise.

Caution

Store the new password in your vault straight away. If you still keep the first-boot administrator password in Key Vault from the initial deployment, it is no longer current after a reset; update or retire that secret so nobody reaches for a stale credential in the next emergency.

Performance seems degraded

• Very many health checks can saturate the probe workers; reduce probe frequency or the number of checks per node.
• Check the cache hit rate (dtm_dns_cache_hits_total vs dtm_dns_cache_misses_total).
• If the database file has grown large, compact it to reclaim disk (watch dtm_storage_bbolt_free_pages).

The service will not start after a config edit

The server validates its YAML configuration strictly: an unknown or misspelled key anywhere in the file stops startup with an error naming the offending field and its line. In journalctl -u dtm-server it is a JSON log record whose msg reads, for example:

Failed to load config: parsing config /etc/dtm/dtm-server.yaml: yaml: unmarshal errors:\n  line 12: field require_tsigg not found in type config.DynamicDNSConfig

This is deliberate, so a typo in a security-relevant key fails loudly instead of being silently ignored.

• Find it with journalctl -u dtm-server -n 50 | grep "not found in type".
• Fix or remove the key named in the error; keys must match the names in the configuration reference exactly.
• An empty or comments-only file is fine; the server starts with defaults.
• If the error is about stored secrets or the encryption key rather than a config field: the node refuses to start when replication.encrypt_key no longer matches the key its stored secrets were written under, instead of silently discarding those secrets. Restore the original key value and the node starts again with nothing lost. Editing the key in configuration is not a rotation path; see rotating secrets.

Deployment (cloud-init) failed on an Azure VM

• Check cloud-init status --long and /var/log/cloud-init-output.log.
• Confirm the VM’s managed identity has Key Vault Secrets User on the Key Vault holding the cluster’s secrets. On the primary (first) node, also confirm Key Vault Secrets Officer: it generates and writes the cluster secrets at first boot, and a not-yet-propagated Officer grant makes the bootstrap retry until the role assignment lands.
• Cloud-init runs once, so reimage or re-run it after fixing the cause.

Capture debug output

For an intermittent problem, open a time-boxed debug window from Settings (or POST /api/v1/debug/enable, admin role) and reproduce the issue. The captured lines render in the Settings debug pane, with a text filter, pause/resume, and a row cap; over the API:

curl -sk https://<node>:8443/api/v1/debug/tail \
  -H "Authorization: Bearer $DTM_TOKEN" \
  -H "X-DTM-CSRF: 1"

Three things to know when reading a capture:

• It is per node. Debug capture is retained in memory on each node (the last 1000 lines, with a dropped counter when it overflows) and never replicated. An empty pane means this node saw nothing, not that the cluster did; repeat against the node that handled the traffic.
• It survives the window. The buffer is kept when debug switches off or its window expires (the pane marks it stale), so the capture you took is still there to read. DELETE /api/v1/debug/tail discards it.
• The journal is unchanged. Debug lines still land in the node’s journal as before; the pane is a convenience view, not a replacement for durable logs.

Still stuck?

Build a support bundle and email [email protected] with the symptom and the bundle attached (optionally encrypted). The bundle gathers the config, diagnostics, cluster state, and recent audit entries support will ask for, with secrets redacted; it can be built from the UI, the API, or offline on a node that will not start. If you cannot produce one, include the relevant node’s /readyz output and GET /api/v1/nodes instead. See also known limitations.