Diagnose a resolution incident

The expensive mistake in a DNS incident is acting before localizing: restarting nodes at random, then mailing support a vague symptom. This tutorial is the decision tree instead: a dig matrix to localize, the response code to classify, readiness and health to inspect, the audit log to catch a deploy-shaped cause, and, if it comes to it, an escalation package support can act on in one round trip.

Scenario

An on-call admin at Tarlow Systems (fictional), paged because an app team reports intermittent SERVFAIL for api.corp.example.internal from one spoke VNet. The three-node cluster looked healthy an hour ago; the admin has UI and API access. Budget: this is an incident, the tree is the budget. API steps send a bearer token as $DTM_TOKEN (how to capture one).

The anti-pattern, named up front: do not restart anything yet. A restart destroys the evidence and, if the fault is a change rather than a node, fixes nothing while adding a DNS gap of its own.

1. Localize with a dig matrix

Query the failing name against each node directly, from an affected and an unaffected subnet, bypassing VNet DNS ordering:

for n in 10.0.250.4 10.0.250.5 10.0.250.6; do
  echo "$n: $(dig @$n api.corp.example.internal +short +tries=1 +time=2)"
done

The matrix forks the whole investigation: one node failing points at that node (readiness, its probe view, its reachability); all nodes failing points at the record, the zone, or an upstream; only one subnet failing points at the network path (NSG, peering), not DNS.

2. Classify the response code

Match what the failing digs actually return against troubleshooting before touching anything: SERVFAIL is usually a forwarding failure for a non-authoritative name; REFUSED means no zone or forwarder matched with the default forwarder off; NXDOMAIN may be the blocklist; an empty NOERROR is NODATA (the name exists with no records of that type) or a record whose values are all disabled; health checks never empty an answer, because DTM fails open and returns all values when none are healthy. Intermittent SERVFAIL for a forwarded name, as here, smells like a flaky upstream or a not-ready node in the client’s rotation, which the next step separates.

3. Read readiness, not liveness

for n in 10.0.250.4 10.0.250.5 10.0.250.6; do
  echo "$n: $(curl -sk https://$n:8443/readyz)"
done

A node can be alive and not ready (still syncing, or deliberately draining during bulk writes), and clients round-robining across nodes then see exactly this incident’s shape: intermittent failures. A 503 body names the failing component; the troubleshooting entries decode it.

4. Run the built-in diagnostics on the suspect node

Two UI pages, two jobs: Tools is the ad-hoc query tester and reachability checker against the node the UI is talking to (its query result names the pipeline stage that decided the answer, which shortcuts the “which layer did this come from” question); Diagnose runs the read-only configuration and cluster-health checks (and can build a support bundle). Run Diagnose on the node the matrix implicated: it surfaces config, storage, and upstream findings without side effects, and re-running it after a fix is your regression check (UI tour).

5. Inspect the record’s dependencies

For a health-checked record, compare each node’s own probe view (health is per node, never replicated):

curl -sk https://<node>:8443/api/v1/health/status \
  -H "Authorization: Bearer $DTM_TOKEN"

For a forwarded name, check upstream health on the metrics surface: dtm_forwarder_upstream_healthy{upstream,scope} on /metrics is the per-upstream status the forwarding failover runs on.

6. Check the audit log for a smoking gun

Most incidents that start “nothing changed” end with a change. Filter the last few hours for the zone, its records, forwarders, and config:

curl -sk "https://10.0.250.4:8443/api/v1/audit?resource=forwarder&from=2026-07-27T06:00:00Z" \
  -H "Authorization: Bearer $DTM_TOKEN"

A deploy-shaped root cause is fixed by reverting the change, not by restarting nodes; the audit entry tells you who to talk to and exactly what to put back.

7. Fix, verify, and log the cost

Apply the fix the matched troubleshooting entry prescribes (re-enable the forwarder someone disabled, fix the upstream, wait out or drain the not-ready node). Verify by re-running the step-1 matrix until it is clean from every node and subnet, and watch the error-rate metric fall back under the SLO line. Write the incident window down against the error budget while the timestamps are fresh.

8. Escalate with a package, not a symptom

If the tree did not converge, escalate the way that gets a one-round-trip answer: build a support bundle (Diagnose page or API; if the node is down, the offline path builds it on the node with the service stopped), optionally passphrase-encrypted, and mail [email protected] with the bundle, your step-1 dig matrix, and the /readyz outputs. Those three artifacts are the difference between “we are looking into it” and an answer.

Where next

• Troubleshooting: the symptom-keyed reference this tree indexes into.
• Wire observability end to end: the alerts that page you before the app team does.
• Support bundles: the escalation artifact in full.

Need a hand? Email [email protected].

Last validated: 2026-07-27