Observability

DTM exposes Prometheus metrics out of the box and can optionally push them to an OTLP collector you manage. Both give you visibility into DNS traffic, health, replication, and cluster state. The paths coexist: enable any combination, and the local metrics endpoint always exposes everything regardless. For discrete operational events pushed to an incident channel (failover activated, backup failed) rather than scraped metrics, see notifications.

Prometheus metrics

Every node serves Prometheus-format metrics at:

GET /metrics

This endpoint is unauthenticated so your scraper can reach it; gate it at the network layer (NSG / firewall) to control who can. The example below verifies TLS, so it assumes the API on 8443 presents a certificate from your own CA: by default each node serves a self-signed certificate, which fails verification (unknown authority) until you give the API a stable certificate. Point Prometheus at each node:

scrape_configs:
  - job_name: dtm
    metrics_path: /metrics
    static_configs:
      - targets:
          - dtm-node-1:8443
          - dtm-node-2:8443
          - dtm-node-3:8443
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/dtm-ca.pem # the CA that issued the API certificate
      insecure_skip_verify: false

Key metric families

All DTM metrics are prefixed dtm_. The most useful:

Area	Metrics
DNS	dtm_dns_queries_total, dtm_dns_query_duration_seconds, dtm_dns_cache_hits_total, dtm_dns_cache_misses_total
Health	dtm_health_check_drops_total, dtm_health_warmup_duration_seconds, dtm_health_probe_refusals_total{guard} (checks a node declined to run, and why; from v1.0.5 including managed script reasons)
Cluster / replication	dtm_cluster_nodes, dtm_replication_last_sync_timestamp_seconds
API	dtm_api_requests_total
Storage	dtm_storage_zones, dtm_storage_records, dtm_storage_tombstones{entity_type}, dtm_storage_tombstone_oldest_timestamp_seconds{entity_type} (from v1.0.5; see below), dtm_storage_db_bytes, dtm_storage_bbolt_free_pages, dtm_storage_free_bytes, dtm_storage_artifact_files, dtm_storage_bbolt_page_writes_total, dtm_storage_bbolt_write_seconds_total, dtm_storage_bbolt_read_tx_total
Node / liveness	dtm_node_info (carries the version label), dtm_node_uptime_seconds, dtm_heartbeat, dtm_process_start_unixtime
Backups	dtm_backup_last_success_timestamp_seconds, dtm_backup_scheduled_failures_total{stage}; alerts on backup and restore
Notifications	dtm_notify_delivered_total{sink}, dtm_notify_failed_total{sink}, dtm_notify_dropped_total, dtm_notify_suppressed_total; see notifications
TLS	dtm_tls_certificate_expires_in_seconds{listener} (from 1.0.4): seconds left on the certificate the api or dns (DoT and DoH) listener presents, negative once expired; see TLS certificates
Marketplace	dtm_entitlement_issuer_fallback_total{outcome} (from v1.0.5): a Marketplace node had to fetch a Microsoft certificate it did not carry; see troubleshooting
Region discovery	dtm_region_discovery_mappings_pending_add, dtm_region_discovery_mappings_pending_remove (always 0 after an incomplete cycle), dtm_region_discovery_last_run_timestamp_seconds, dtm_region_discovery_last_cycle_complete (sustained 0 usually means a lost Reader grant), dtm_region_discovery_run_errors_total

The storage gauges added in v1.0.5 show what a node holds, not just how big its file is. dtm_storage_zones and dtm_storage_records count the live zones and records: a dtm_storage_db_bytes that grows while they stay flat is file growth, not data. dtm_storage_tombstones counts the delete markers held, by type. dtm_storage_tombstone_oldest_timestamp_seconds is the deletion time of the oldest marker due to be purged: if time() minus it passes 30 days, old delete markers are no longer being purged. They are sampled every 5 minutes.

dtm_dns_query_duration_seconds carries an authoritative label alongside type: "true" when DTM gave the answer itself (from a zone it owns or a static override), "false" otherwise (forwarded, a cached forwarded answer, blocklisted, or refused). A forwarded query’s duration is mostly the upstream resolver’s, so the latency objective and the shipped latency rules read only authoritative="true".

Grafana dashboard

A pre-built Grafana dashboard covering DNS traffic, latency, health, replication, and cluster state is available from support. Import it into your Grafana and point it at your Prometheus data source.

Useful things to alert on:

• Storage growth: dtm_storage_bbolt_free_pages above 50% of total pages sustained for 6 hours (the shipped DTMBboltFreelistBloat rule’s hold) suggests it is time to compact.
• Cluster health: dtm_cluster_nodes below your deployed node count (each node reports the number of cluster members it can currently see).
• Replication lag: a growing gap in dtm_replication_last_sync_timestamp_seconds.
• Unexpected write load: on a converged cluster with no configuration changes or dynamic updates in flight, the rate of dtm_storage_bbolt_page_writes_total should sit near zero; sustained page writes mean something is rewriting unchanged state. Rising dtm_storage_bbolt_write_seconds_total with a flat page-write rate points at a slow disk instead.

The shipped Prometheus rules (dtm-alerts.yml) cover the first three and page on the latency objective; Alerting explains how to route them and lists the Azure Monitor equivalents. See SLA and SLO for target thresholds and burn-rate alerting recipes.

OTLP export (optional)

Instead of (or as well as) being scraped, DTM can push its metrics to an OTLP/HTTP collector you run, which can route them onward to Azure Monitor and/or a Grafana/LGTM stack. Configure it over the API ($DTM_TOKEN is a bearer token; see authentication), or with Terraform:

curl -sk -X PUT https://dtm.internal:8443/api/v1/observability/otlp \
  -H "Authorization: Bearer $DTM_TOKEN" \
  -H "X-DTM-CSRF: 1" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "endpoint": "https://otel-collector.internal:4318",
    "headers": {"x-api-key": "<collector-token>"},
    "interval_seconds": 30
  }'

The change is applied to the live exporter without a restart. A few notes:

• These endpoints only turn DTM’s export on or off and set where to send it; they configure nothing on the receiving side.
• Header values are stored encrypted and returned redacted (***). Resubmitting *** or an empty value preserves the stored secret.
• An http:// endpoint sends metrics without transport security (the OTLP-standard meaning of “insecure”); use https:// with a trusted certificate for anything sensitive.
• The metrics exporter is configurable today; OTLP log export is not yet available.

For Azure deployments, DTM can also push metrics directly to Azure Monitor custom metrics (enabled in the server configuration or through the Azure Monitor settings API described below, authenticating with the VM’s managed identity). Each node pushes to its own VM: leave monitor_resource_id and monitor_region blank and the node reads both from the Azure Instance Metadata Service at startup; set them only to send the metrics to a different resource. Every pushed series carries a node dimension holding the node ID, and the node’s identity needs the Monitoring Metrics Publisher role on the VM it publishes to. Only DTM’s own dtm_-prefixed metrics are pushed, so the billed custom-metrics namespace is not filled with runtime series; the local /metrics endpoint is unaffected and still exposes everything.

If the node cannot read the metadata service, it pushes nothing rather than guess a target: it logs azure-monitor: target unresolved: <cause>; metrics push disabled until resolved and retries with backoff (5 seconds, doubling to 5 minutes). A read of GET /api/v1/config/azure-monitor reports the running exporter’s target (effective_resource_id, effective_region, target_source), and the dtm_azure_monitor_export_state{state} gauge is 1 for exactly one of disabled, target_unresolved, pushing and push_failing. Through the API, half a Logs Ingestion target (an endpoint without a DCR immutable ID, or an ID without an endpoint) with the exporter enabled is refused with a 400 when you save it.

How DTM publishes to Azure Monitor

Every node pushes its dtm_ metrics to its own VM once a minute. From DTM 1.0.2:

• Gauges are pushed as their current value.
• Counters are pushed as their increase since the node’s last successful push, which is how Azure custom metrics model a count. Chart or alert on them with Total (Sum): Total over a window is the number of events in it. A push that fails is carried into the next one, so the totals still add up after an outage, and a restart never produces a negative value.
• Histograms are pushed as the sum and count of that interval’s observations, with Minimum and Maximum set to the interval’s mean.

DTM 1.0.1 and earlier pushed each counter as its running total, which only Max read correctly. A chart or alert you built that way on a counter now sees per-interval values, so switch it to Total.

DTM also pushes metrics it computes for Azure only, so that an Azure metric alert, which compares one metric with a fixed value, can express what the Prometheus rules compute with a rate, ratio, percentile or age. They are not on /metrics, where Prometheus computes the same things from the raw series:

Metric	What it holds
dtm_process_starts	1 in the first push after the node’s process starts, 0 after that
dtm_<counter>_increase_<range>	the counter’s increase over that range, for the counters the Prometheus rules take a rate of (for example dtm_health_check_drops_increase_5m)
dtm_dns_authoritative_latency_p99_15m_seconds, dtm_dns_authoritative_latency_p95_15m_seconds	p99 and p95 of the time the node took to answer from its own zones over 15 minutes, computed the way histogram_quantile does; pushed only while there were such answers
dtm_dns_authoritative_queries_increase_15m	how many such answers that covers
dtm_dns_query_error_ratio_5m	the share of queries over 5 minutes answered with an error (not NOERROR or NXDOMAIN); 0 with no queries
dtm_replication_last_sync_age_seconds, dtm_region_discovery_last_run_age_seconds, dtm_tag_registration_last_run_age_seconds	seconds since the event; pushed only once it has happened
dtm_storage_free_ratio, dtm_storage_bbolt_free_page_ratio	free space on the data volume over its total, and free database pages over all allocated pages (free plus in use)

Azure Monitor workbook and KQL alerts

DTM ships an Azure-native counterpart to the Grafana dashboard: a pre-built Azure Monitor workbook plus six KQL alert queries (deploy/azure-monitor/), available from support. Two independent data streams feed it; enable the ones you need on each node, through the Azure Monitor settings API or the server configuration:

• Query log to Log Analytics (traffic panels and all six alerts): azure.monitor_enabled: true plus the query log and a Logs Ingestion target: azure.logs_ingestion_endpoint, azure.logs_ingestion_dcr_immutable_id and, optionally, azure.logs_ingestion_stream (default Custom-DTMLogs). Queries land in the DTMLogs_CL table through a data collection rule (DCR), authenticated by the node’s managed identity, which needs Monitoring Metrics Publisher on the DCR. Ship the query log to Log Analytics creates the table, the DCR and the role, and sets the values. The query log itself (enabled, sample_rate) is runtime-configurable: one PUT /api/v1/config/query-log (admin role) or the dtm_query_log_config Terraform resource applies live on every node with no restart, and the persisted runtime value wins over the observability.query_log_* YAML keys at the next boot. The shipper settings (all the azure.* keys here except azure.subscription_id, which stays a file setting) are per node: set them with PUT /api/v1/config/azure-monitor (admin role, for reads too), the Azure Monitor card on the Settings page, or the dtm_azure_monitor_config Terraform resource. Changes take effect at the node’s next dtm-server restart, and a saved setting overrides the azure.* keys in the server configuration. There is no secret among them: the node’s managed identity authenticates. monitor_enabled (enabled on the API) is the master switch for the Azure Monitor exporter: without it the query log is collected but never shipped, so the table stays empty even with the Logs Ingestion settings in place.
• Custom metrics (cluster-health charts and the metric alerts): the Azure Monitor push described above: azure.monitor_enabled, with monitor_metrics_enabled at its default (true) and monitor_resource_id and monitor_region blank so each node publishes to its own VM. For logs only, set monitor_metrics_enabled: false: no custom metric is pushed and no target is resolved, while the query log still ships. A hand-written PUT /api/v1/config/azure-monitor replaces the whole record, so on a logs-only node send "monitor_metrics_enabled": false every time; the Settings card and the Terraform resource always send it.

Import the workbook JSON via the portal (Monitor, Workbooks, Advanced Editor) or as a Microsoft.Insights/workbooks resource in your IaC. Its cluster-health section reads one metrics resource, by default a node’s own VM, so it shows that node. Its counter tiles chart Sum (Total), which from DTM 1.0.2 is the number of events in the chosen window. The alert queries cover NXDOMAIN floods per client, SERVFAIL surges, REFUSED spikes, error-response rate, high p95 latency, and query-volume collapse, each with a tunable threshold; create them with az monitor scheduled-query create or the equivalent Terraform resource. They work from the query log, which does not separate DTM’s own answers from forwarded ones.

Metric alerts on the custom metrics come from the alerting sample, not from the deployment templates, which create no alert rules and take no alert email. The sample creates 33 rules per node VM, mapped to the Prometheus rules and attached to an action group you already have. The rules that need a rate, a ratio, a percentile or an age read the metrics DTM computes for Azure (above), so the query error rate and DNS latency against the objective (judged on DTM’s own answers, as the Prometheus rules judge it) have metric alerts too. The one thing no metric alert can see is a whole-cluster outage. See Alerting and what Azure Monitor cannot alert on.

Changed in v1.0.6

The query log ships through the Azure Monitor Logs Ingestion API, to a data collection rule, authenticated by the node’s managed identity. Earlier releases used the HTTP Data Collector API with the workspace’s shared key; Microsoft ended support for that API, and it does not work behind Azure Monitor Private Link Scope (AMPLS). The Logs Ingestion path does: point logs_ingestion_endpoint at a data collection endpoint in your AMPLS. The DTMLogs_CL table keeps its columns (qname_s, rcode_s, latency_ms_d, and the rest), so the workbook, the KQL alerts and your own queries keep working. Two things read differently:

• TimeGenerated is now the query’s own time, not the time Azure ingested it.
• dtm_observability_push_total{exporter="azure_log"} counts batches (up to 500 rows each, one outcome per batch with its retries), where it counted one per row before.

The new dtm_observability_log_api{api} gauge says what the node ships through: 1 on logs_ingestion or on none, 0 on the other. Alert on {api="none"} == 1 where you expect the query log to ship. The custom metrics are not affected: they use a separate API and the VM’s managed identity.

Upgrading from 1.0.5

The azure.log_analytics_workspace_id and azure.log_analytics_workspace_key settings are removed in 1.0.6. Query-log shipping stops on the upgrade until you configure a Logs Ingestion target: a node that has only the old workspace settings ships no query log on 1.0.6. DNS is unaffected, and the node keeps starting and serving.

• This is the same whether the old settings are in dtm-server.yaml or were set through the API or the Settings page, which save them in the node’s Azure Monitor settings record. Either way they are logged at start-up as an ERROR: azure monitor: ... line that names the replacement settings, and are otherwise ignored. Nothing ships through them.
• Once a Logs Ingestion target is configured, the old values are still ignored and the query log ships through the new target.
• GET /api/v1/config/azure-monitor lists the leftover in warnings, and the next PUT /api/v1/config/azure-monitor rewrites the saved settings without them. A PUT that still sends either field is a 400 naming the replacement.

To move over, create the Logs Ingestion setup and set the three values on each node (Ship the query log to Log Analytics), and remove the two old keys from dtm-server.yaml. A workspace whose DTMLogs_CL table the old path created needs a one-time conversion first; that page has the command.

Rotate the workspace’s shared keys after upgrading

1.0.6 never uses the workspace’s shared key. But until that first PUT, a node’s saved settings still hold the key an older release stored, and a backup taken before then carries it too. Once every node has been saved again, regenerate the workspace’s shared keys (primary and secondary) so the old key is worthless wherever a copy remains.

Note

Health-aware forwarding also exposes dtm_forwarder_upstream_healthy{upstream,scope} on /metrics, the status surface for upstream failover.

Logs

DTM logs to the system journal by default. For deployments that ship logs centrally, forward the journal to your log pipeline (for example Azure Log Analytics) using your standard agent.

Two retention decisions are worth making deliberately:

• On the node, journald’s own limits decide how much local history survives a busy day or a reboot. Persistent storage with a sensible cap (Storage=persistent and a SystemMaxUse of a few hundred MB in journald.conf) keeps enough history for incident forensics without filling the OS disk.
• In Log Analytics, the workspace’s retention setting governs how long shipped logs stay queryable. The Terraform deployment’s monitoring module exposes it as retention_in_days (default 30). Use 90 or more for production, and 365 or more where a compliance regime such as SOC 2 or PCI applies; the module’s total_retention_in_days can keep older data in the low-cost archive tier so long retention stays affordable.

The audit log (who changed what in DTM) is separate from operational logs and has its own bounded retention; see audit log retention.