Wire observability end to end

Your cluster is serving DNS, and you cannot see it. This tutorial gets you from black box to go-live observability: a dashboard showing every node, alerts mapped to the published SLOs, and, the part most teams skip, one test alert deliberately fired and cleared before the first real incident fires it for you.

Scenario

An SRE at a company whose DTM cluster (three nodes, 10.0.250.4 to .6) serves example.internal. The go-live gate requires a dashboard and paging alerts mapped to the SLO table (99.95 percent DNS answer success, p99 under 10 ms). Budget about 60 minutes on one path, plus up to 15 minutes of alert latency in the fire drill.

1. Pick your pager path

DTM feeds two stacks out of the box, and they can coexist; wire the one your pager actually listens to:

• Path A: Prometheus and Grafana, if you run your own metrics stack.
• Path B: Azure Monitor, if you are Azure-native (workbook, KQL log alerts, and per-node metric alerts).

Follow only your path’s sections, then rejoin at map alerts to SLOs. (There is also an OTLP push path for collector pipelines; it is out of scope here, see observability.)

2A. Scrape the nodes (Path A)

Every node serves Prometheus metrics at GET /metrics on port 8443, unauthenticated but NSG-gated; confirm your NSG admits the Prometheus subnet on 8443 and nothing else new. Add the scrape job from the observability page, with one deliberate deviation to bridge:

tls_config:
  insecure_skip_verify: true # temporary; see below

The observability page’s example shows insecure_skip_verify: false as the end state, and that is correct once the API listener has a CA-issued certificate. By default each node presents its own self-signed certificate, which no CA vouches for, so pinning a CA cannot work yet. Scrape insecurely now, and flip this to verified TLS when you complete the stable certificate step (the encrypted DNS tutorial does exactly that).

Verify: in Prometheus, Status, Targets shows all three nodes UP, and dtm_dns_queries_total returns series for each node.

3A. Import the shipped dashboard and alert rules (Path A)

DTM provides a pre-built Grafana dashboard (dtm-overview) and a Prometheus alert-rules file (dtm-alerts.yml); email support for the current versions for your release. Import the dashboard into Grafana, point it at your Prometheus data source, and load the rules file into your Prometheus rule configuration. Three notes:

• The dashboard’s node template variable filters every panel per node; use it to confirm all three nodes report.
• The DTMClusterUndersized rule in dtm-alerts.yml is marked EDIT ME and assumes a three-node cluster. Set its expected node count to yours, and review the other thresholds against your topology too, before trusting the pager.
• The rules page on the latency objective itself: DTMDNSLatencyAboveSLO (warning) when p99 latency stays above 10 ms for 15 minutes, and DTMDNSLatencyHigh (critical) when p95 exceeds 50 ms. DTMReplicationLagHigh applies only while a node has peers, and the Azure background-loop rules judge only a loop that has run, so a loop you never enabled raises nothing. Alerting explains how to route them.

Verify: the dashboard shows live query rates for all nodes, and Prometheus Status, Rules lists the DTM rules as healthy.

2B. Enable the two Azure Monitor streams (Path B)

The Azure Monitor assets ride two independent data streams, and the number-one trap is enabling one and expecting the other’s panels to fill:

• Query log to Log Analytics feeds the workbook’s traffic panels and all six alerts: set azure.monitor_enabled: true plus observability.query_log_enabled, observability.query_log_sample_rate, and a Logs Ingestion target (azure.logs_ingestion_endpoint, azure.logs_ingestion_dcr_immutable_id, and optionally azure.logs_ingestion_stream). Queries land in the DTMLogs_CL table through a data collection rule, and each node’s identity needs Monitoring Metrics Publisher on that rule. Create the table, the rule and the role first: Ship the query log to Log Analytics. (Changed in v1.0.6: earlier releases used a workspace ID and key, which are removed.)
• Custom metrics feed the workbook’s cluster-health section and the metric alerts: the same azure.monitor_enabled, with azure.monitor_metrics_enabled left at its default (true). Leave azure.monitor_resource_id and azure.monitor_region blank: each node then reads its own VM from the Azure Instance Metadata Service and publishes there, and every series carries a node dimension holding the node ID. Custom metrics land on, and are billed against, that VM, not the workspace, and each node’s identity needs the Monitoring Metrics Publisher role on its own VM. For logs only, set monitor_metrics_enabled: false.

Caution

azure.monitor_enabled is the master switch for both streams: with the Logs Ingestion settings in place but the switch off, the query log is collected and never shipped, and DTMLogs_CL stays silently empty. Set a query_log_sample_rate above 1 on busy nodes so you keep 1-in-N rather than every query.

Set the shipper on each node with PUT /api/v1/config/azure-monitor (admin role), the Azure Monitor card on the Settings page, or the dtm_azure_monitor_config Terraform resource; a saved setting overrides the azure.* keys in the server configuration (see the configuration reference). The exporter is built at startup, so then restart one node at a time, gating each on curl -sk https://<node>:8443/readyz returning 200. The query log itself is the exception: switch it on and set the sample rate at runtime with PUT /api/v1/config/query-log (admin role) or the dtm_query_log_config Terraform resource, which applies live on every node with no restart; the persisted runtime value then wins over the observability.query_log_* YAML keys at the next boot. Until the shipper is enabled, query-log entries stay local to each node. If a node cannot read its target from the Instance Metadata Service, it pushes no metrics rather than guess: it logs azure-monitor: target unresolved: <cause>; metrics push disabled until resolved and retries with backoff. GET /api/v1/config/azure-monitor reports the running exporter’s target (effective_resource_id, target_source); check it, and the node’s journal, if a stream stays empty.

Verify:

DTMLogs_CL | summarize count() by node_id_s | order by count_ desc

one row per node_id within a few minutes of restart. For the custom metrics, run this against each node’s VM:

az monitor metrics list-definitions --resource <node-vm-resource-id> \
  --namespace DTM --query "[?name.value=='dtm_heartbeat'].name.value" -o tsv

It prints dtm_heartbeat once that node has pushed at least once.

3B. Import the workbook and create the alerts (Path B)

Import the shipped workbook (portal: Monitor, Workbooks, Advanced Editor, paste the JSON; or Terraform: azurerm_application_insights_workbook with data_json = file(...)). Then create the six shipped KQL scheduled-query alerts, one az monitor scheduled-query create each: nxdomain-per-client, servfail-surge, refused-spike, error-response-rate, high-latency, and query-volume-collapse. The observability page lists what each covers.

Then create the metric alerts from the alerting sample, in Terraform, Bicep or ARM: 33 rules per node VM, mapped to the Prometheus rules, each scoped to one node’s VM and attached to an action group you already have. DTM’s deployment templates create no alert rules and take no alert email, so how you are notified is your choice. The sample also grants each node’s system-assigned identity Monitoring Metrics Publisher on its own VM, unless you turn that off. See Alerting.

Verify: the workbook’s traffic panels render from DTMLogs_CL, the cluster-health section renders from the DTM custom-metrics namespace (its metrics-resource parameter is one node’s VM, so it shows that node), az monitor scheduled-query list -o table shows all six log alerts enabled, and az monitor metrics alert list -o table shows 33 rules per node VM.

4. Map alerts to the SLOs

Tie each pager rule to the SLO line it guards, so an alert page answers “which promise is burning”:

Alert (A: rule / B: metric rule, KQL alert)	SLO line it guards
DTMQueryErrorRate / query-error-rate, error-response-rate	99.95% DNS answer success
DTMDNSLatencyAboveSLO / dns-latency-above-slo	p99 under 10 ms
DTMDNSLatencyHigh / dns-latency-high, high-latency	p99 under 10 ms (gross slowness)
servfail-surge, refused-spike (B, KQL)	answer success, leading signals
nxdomain-per-client (B, KQL)	abuse/misconfig ahead of success
query-volume-collapse (B, KQL)	serving availability

On Path A both latency rules count only the answers DTM gives itself (authoritative="true": from a zone it owns or a static override), which is what the objective covers, and need at least 100 such queries in 15 minutes. DTMDNSLatencyAboveSLO pages on the objective itself; DTMDNSLatencyHigh catches gross slowness (p95 above 50 ms). On Path B, the alerting sample’s dns-latency-above-slo and dns-latency-high rules judge the same answers with the same thresholds and the same 100-query gate: an Azure metric alert cannot take a percentile, so each node computes its 15-minute p99 and p95 and publishes them as metrics. The KQL high-latency alert (p95 above 50 ms over at least 100 queries in its 15-minute window) reads the query log, which does not separate DTM’s own answers from forwarded ones, so on a forward-heavy node it also reflects your upstream resolvers. Page on the metric rules for the objective, and keep high-latency as a second view.

The SLO page also sketches multi-window burn-rate alerting if your team runs error budgets; map, do not restate.

5. Fire one alert on purpose

An alert that has never fired is a hypothesis. Make SERVFAIL happen safely: create a conditional forwarder (or stub zone) for a fictional domain, say drill.invalid.internal, pointing at an unreachable upstream like 10.99.99.99:53, then sustain a dig loop against names under it:

while true; do dig @10.0.250.4 x$RANDOM.drill.invalid.internal +short; sleep 1; done

Forwarding to an unreachable upstream yields SERVFAIL (a name that simply matches nothing yields REFUSED when default forwarding is off, which is the wrong signal for this drill; see troubleshooting).

• Path A: DTMQueryErrorRate holds for 5 minutes before firing, so keep the loop running past that.
• Path B: servfail-surge evaluates every 5 minutes over a 10-minute window, and Log Analytics ingestion adds latency: expect 10 to 15 minutes before the alert fires. It is not broken at minute two.

When the page arrives, stop the loop, delete the drill forwarder, and watch the alert resolve.

Verify: one alert fired, one alert cleared, both visible in your pager’s history with timestamps you can quote.

6. The go-live checklist

Check	Path
Dashboard shows every node	A+B
Scrape targets all UP / both streams populated	A/B
Alert rules loaded, thresholds reviewed	A+B
Alerts mapped to SLO lines	A+B
One test alert fired and cleared	A+B

One housekeeping note for a cluster upgraded from 1.0.5: query-log shipping stops until each node has a Logs Ingestion target, and the workspace’s shared keys should be rotated afterwards. See upgrading from 1.0.5.

Where next

• Production readiness pass: the wider go-live gate this feeds into.
• Encrypted DNS: give the API a stable certificate and retire that insecure_skip_verify: true.
• SLA and SLO: burn-rate recipes for error-budget teams.

Need a hand? Email [email protected].

Last validated: 2026-07-26