Wire observability end to end

Your cluster is serving DNS, and you cannot see it. This tutorial gets you from black box to go-live observability: a dashboard showing every node, alerts mapped to the published SLOs, and, the part most teams skip, one test alert deliberately fired and cleared before the first real incident fires it for you.

Scenario

An SRE at a company whose DTM cluster (three nodes, 10.0.250.4 to .6) serves example.internal. The go-live gate requires a dashboard and paging alerts mapped to the SLO table (99.95 percent DNS answer success, p99 under 10 ms). Budget about 60 minutes on one path, plus up to 15 minutes of alert latency in the fire drill.

1. Pick your pager path

DTM feeds two stacks out of the box, and they can coexist; wire the one your pager actually listens to:

• Path A: Prometheus and Grafana, if you run your own metrics stack.
• Path B: Azure Monitor, if you are Azure-native (workbook plus KQL alerts).

Follow only your path’s sections, then rejoin at map alerts to SLOs. (There is also an OTLP push path for collector pipelines; it is out of scope here, see observability.)

2A. Scrape the nodes (Path A)

Every node serves Prometheus metrics at GET /metrics on port 8443, unauthenticated but NSG-gated; confirm your NSG admits the Prometheus subnet on 8443 and nothing else new. Add the scrape job from the observability page, with one deliberate deviation to bridge:

tls_config:
  insecure_skip_verify: true # temporary; see below

The observability page’s example shows insecure_skip_verify: false as the end state, and that is correct once the API listener has a stable certificate. By default it self-signs a fresh certificate at every service start, so pinning a CA cannot work yet. Scrape insecurely now, and flip this to verified TLS when you complete the stable certificate step (the encrypted DNS tutorial does exactly that).

Verify: in Prometheus, Status, Targets shows all three nodes UP, and dtm_dns_queries_total returns series for each node.

3A. Import the shipped dashboard and alert rules (Path A)

DTM provides a pre-built Grafana dashboard (dtm-overview) and a Prometheus alert-rules file (dtm-alerts.yml); email support for the current versions for your release. Import the dashboard into Grafana, point it at your Prometheus data source, and load the rules file into your Prometheus rule configuration. Two notes:

• The dashboard’s node template variable filters every panel per node; use it to confirm all three nodes report.
• The expected-node-count threshold is marked EDIT ME and assumes a three-node cluster; review the other thresholds against your topology too before trusting the pager.

Verify: the dashboard shows live query rates for all nodes, and Prometheus Status, Rules lists the DTM rules as healthy.

2B. Enable the two Azure Monitor streams (Path B)

The Azure Monitor assets ride two independent data streams, and the number-one trap is enabling one and expecting the other’s panels to fill:

• Query log to Log Analytics feeds the workbook’s traffic panels and all six alerts: set azure.monitor_enabled: true plus observability.query_log_enabled, observability.query_log_sample_rate, and the azure.log_analytics_workspace_id / _key pair. Queries land in the DTMLogs_CL table.
• Custom metrics feed the workbook’s cluster-health section: the same azure.monitor_enabled plus azure.monitor_resource_id and azure.monitor_region. monitor_resource_id is the DTM VM’s resource ID (custom-metrics billing lands on that resource, not the workspace).

Caution

azure.monitor_enabled is the master switch for both streams: with the workspace settings in place but the switch off, the query log is collected and never shipped, and DTMLogs_CL stays silently empty. Set a query_log_sample_rate above 1 on busy nodes so you keep 1-in-N rather than every query.

The azure.* shipper settings are file settings (see the configuration reference): there is no config reload, so edit /etc/dtm/dtm-server.yaml and restart one node at a time, gating each on curl -sk https://<node>:8443/readyz returning 200. The query log itself is the exception: switch it on and set the sample rate at runtime with PUT /api/v1/config/query-log (admin role) or the dtm_query_log_config Terraform resource, which applies live on every node with no restart; the persisted runtime value then wins over the observability.query_log_* YAML keys at the next boot. Until the shipper is enabled, query-log entries stay local to each node. If the exporter’s configuration is incomplete (a missing region, for example), it disables itself at startup and logs why, rather than pushing to a malformed endpoint; check the node’s journal if a stream stays empty.

Verify:

DTMLogs_CL | summarize count() by node_id_s | order by count_ desc

one row per node_id within a few minutes of restart.

3B. Import the workbook and the six alerts (Path B)

Import the shipped workbook (portal: Monitor, Workbooks, Advanced Editor, paste the JSON; or Terraform: azurerm_application_insights_workbook with data_json = file(...)). Then create the six shipped KQL scheduled-query alerts, one az monitor scheduled-query create each: nxdomain-per-client, servfail-surge, refused-spike, error-response-rate, high-latency, and query-volume-collapse. The observability page lists what each covers.

Verify: the workbook’s traffic panels render from DTMLogs_CL, the cluster-health section renders from the DTM custom-metrics namespace, and az monitor scheduled-query list -o table shows all six rules enabled.

4. Map alerts to the SLOs

Tie each pager rule to the SLO line it guards, so an alert page answers “which promise is burning”:

Alert (A: rule / B: KQL)	SLO line it guards
query error rate / error-response-rate	99.95% DNS answer success
high-latency	p99 latency under 10 ms
servfail-surge, refused-spike	answer success, leading signals
nxdomain-per-client	abuse/misconfig ahead of success
query-volume-collapse	serving availability

The SLO page also sketches multi-window burn-rate alerting if your team runs error budgets; map, do not restate.

5. Fire one alert on purpose

An alert that has never fired is a hypothesis. Make SERVFAIL happen safely: create a conditional forwarder (or stub zone) for a fictional domain, say drill.invalid.internal, pointing at an unreachable upstream like 10.99.99.99:53, then sustain a dig loop against names under it:

while true; do dig @10.0.250.4 x$RANDOM.drill.invalid.internal +short; sleep 1; done

Forwarding to an unreachable upstream yields SERVFAIL (a name that simply matches nothing yields REFUSED when default forwarding is off, which is the wrong signal for this drill; see troubleshooting).

• Path A: the query-error-rate rule holds for 5 minutes before firing, so keep the loop running past that.
• Path B: servfail-surge evaluates every 5 minutes over a 10-minute window, and Log Analytics ingestion adds latency: expect 10 to 15 minutes before the alert fires. It is not broken at minute two.

When the page arrives, stop the loop, delete the drill forwarder, and watch the alert resolve.

Verify: one alert fired, one alert cleared, both visible in your pager’s history with timestamps you can quote.

6. The go-live checklist

Check	Path
Dashboard shows every node	A+B
Scrape targets all UP / both streams populated	A/B
Alert rules loaded, thresholds reviewed	A+B
Alerts mapped to SLO lines	A+B
One test alert fired and cleared	A+B

One housekeeping note: both Azure ingestion paths ride APIs Microsoft is retiring in favour of the Azure Monitor Agent; the observability page tracks that migration and it preserves the same table and columns.

Where next

• Production readiness pass: the wider go-live gate this feeds into.
• Encrypted DNS: give the API a stable certificate and retire that insecure_skip_verify: true.
• SLA and SLO: burn-rate recipes for error-budget teams.

Need a hand? Email [email protected].

Last validated: 2026-07-26