Alerting

DTM does not decide how you are notified. Whether an alert becomes an email, an SMS, a Teams message, a webhook, an incident ticket, or nothing at all is up to you. You set that up in your own alerting stack, and DTM supplies the signals and the rules. The deployment templates create no alerting.

There are two routes. Use the one your on-call already listens to; they can run side by side.

Route	What you run	Rules	Coverage
A. Prometheus and Alertmanager	Prometheus scraping every node, and Alertmanager or Grafana alerting	dtm-alerts.yml	Every rule, including DNS latency against the SLO and a whole-cluster outage
B. Azure Monitor	Azure Monitor metric alert rules, created by a Terraform, Bicep or ARM sample	The sample’s 33 rules per node	Every rule except three (a whole-cluster outage, a managed script check that is not running, and a private endpoint zone group the node cannot read), with a few others approximated; see what route B cannot alert on

Route A: Prometheus and Alertmanager

1. Scrape every node. Add the scrape job from the observability page.

2. Load the rules. DTM’s Prometheus alert-rules file is dtm-alerts.yml. Email support for the version that matches your release. Add it under rule_files: in your Prometheus configuration, and check it loads:

   promtool check rules dtm-alerts.yml

3. Set your cluster size. DTMClusterUndersized compares each node’s view of the cluster with a node count marked EDIT ME, which assumes three nodes. Set it to your node count.

4. Route the alerts. Each rule carries a severity label, critical or warning. Route on it in Alertmanager (or Grafana alerting) to whatever receivers you use.

Verify: in Prometheus, Status, Rules lists the DTM rule groups with no errors, and Alerts shows them inactive on a healthy cluster. To prove the whole path pages someone, follow the fire drill in wire observability end to end.

Two rules map directly to the published DNS latency SLO. Both judge DTM’s own answers only: queries for zones the cluster owns, and override records. Forwarded queries are left out, because their time is mostly the upstream resolver’s.

• DTMDNSLatencyAboveSLO (warning): p99 above 10 ms for 15 minutes.
• DTMDNSLatencyHigh (critical): p95 above 50 ms for 5 minutes.

Both need at least 100 such queries in 15 minutes, so a nearly idle node does not page on a handful of slow answers.

Route B: Azure Monitor

Each DTM node can publish its own metrics to Azure Monitor, as custom metrics on its own virtual machine. The alerting samples create metric alert rules on them, one set per node, and attach every rule to an action group you already have.

Before you deploy

1. Turn on the Azure Monitor exporter on every node, and leave the target blank so each node publishes to its own VM. Per node:

   curl -sk -X PUT https://<node>:8443/api/v1/config/azure-monitor \
     -H "Authorization: Bearer $DTM_TOKEN" \
     -H "X-DTM-CSRF: 1" \
     -H "Content-Type: application/json" \
     -d '{"enabled": true}'

   You can also use the Azure Monitor card on the Settings page, or the dtm_azure_monitor_config Terraform resource. The setting takes effect at the node’s next restart, so restart one node at a time and wait for curl -sk https://<node>:8443/readyz to return 200 before the next.

   • monitor_resource_id and monitor_region left out means “this VM”, read from the Azure Instance Metadata Service.
   • Every series then carries a node dimension holding the node ID.
   • Do not send "monitor_metrics_enabled": false: that is the logs-only setup, and it publishes no metrics.
   • If you also ship the query log to Log Analytics, include your logs_ingestion_* settings in the same request, because a PUT replaces the whole record (see Ship the query log to Log Analytics).

2. Let each node publish to its own VM. The node’s identity needs the Monitoring Metrics Publisher role on its own VM. The samples grant it to the VM’s system-assigned identity unless you turn that off. If your nodes run on a user-assigned identity you brought yourself, turn the grant off and give that identity the role on each VM instead.

3. Have an action group. Use an existing one, or create one with the receivers you want. For example, an email-only group:

   az monitor action-group create \
     --resource-group rg-monitoring \
     --name dtm-oncall \
     --short-name dtmoncall \
     --action email oncall [email protected]

Verify the metrics are arriving before you rely on any rule. GET /api/v1/config/azure-monitor on a node reports target_source: imds and that node’s VM as effective_resource_id. Then:

az monitor metrics list-definitions \
  --resource <node-vm-resource-id> --namespace DTM \
  --query "[?name.value=='dtm_heartbeat'].name.value" -o tsv

It prints dtm_heartbeat once the node has published at least once.

Deploy the sample

All three flavours create the same rules from the same inputs. Deploy into the resource group that holds your DTM node VMs.

Flavour	File
Terraform	main.tf
Bicep	main.bicep
ARM template (JSON)	azuredeploy.json

curl -O https://docs.hayami.io/samples/alerting/terraform/main.tf
terraform init
terraform apply \
  -var 'resource_group_name=rg-dtm' \
  -var 'action_group_id=<action-group-resource-id>' \
  -var 'node_vm_names=["dtm-node-1","dtm-node-2","dtm-node-3"]'

curl -O https://docs.hayami.io/samples/alerting/bicep/main.bicep
az deployment group create \
  --resource-group rg-dtm \
  --template-file main.bicep \
  --parameters actionGroupId=<action-group-resource-id> \
               nodeVmNames='["dtm-node-1","dtm-node-2","dtm-node-3"]'

The ARM template takes the same parameters as the Bicep file; deploy it the same way with --template-file azuredeploy.json.

Input (Terraform / Bicep and ARM)	Required	Meaning
resource_group_name / the deployment’s resource group	yes	The resource group holding the node VMs. The rules are created there too.
action_group_id / actionGroupId	yes	Your existing action group. Every rule notifies it.
node_vm_names / nodeVmNames	yes	The node VMs. Each rule is created once per VM, scoped to it.
expected_node_count / expectedNodeCount	no	Defaults to the number of node VMs.
expected_dns_listeners / expectedDnsListeners	no	Defaults to 2 (UDP and TCP). Add one if you enabled DNS over TLS. DNS over HTTPS is not counted: it has its own dtm_dns_doh_listener_up gauge. When in doubt, read dtm_dns_listeners_total from a healthy node.
dns_latency_p99_threshold_seconds / dnsLatencyP99ThresholdSeconds	no	Defaults to 0.010 (10 ms, the SLO objective). The p99 latency of the node’s own answers, in seconds, above which dns-latency-above-slo fires. Bicep and ARM take it as a decimal string ('0.010') that nothing checks, so write seconds, not milliseconds.
dns_latency_p95_threshold_seconds / dnsLatencyP95ThresholdSeconds	no	Defaults to 0.050 (50 ms). The p95 latency, in seconds, above which dns-latency-high fires. A decimal string in Bicep and ARM, like the p99 one.
grant_metrics_publisher / grantMetricsPublisher	no	Defaults to on: grants each node’s identity Monitoring Metrics Publisher on its own VM.
create_example_action_group / createExampleActionGroup	no	Example only. Creates an email-only action group to try the rules with. Replace it with your own for real use.

If your nodes are in more than one resource group, deploy once per group and set the expected node count to the whole cluster’s size.

Changed in v1.0.4

The sample gains tls-certificate-expiring. Nodes publish its metric from DTM 1.0.4, so against a node on an earlier version the rule never fires.

Changed in v1.0.2

The sample grew from 16 rules per node VM to 30, and the disk rules now read the node’s free-space ratio, so the data_volume_bytes / dataVolumeBytes input is gone: drop it from your variables or parameters when you redeploy an earlier copy. Many of the rules read metrics that nodes publish from DTM 1.0.2, which also pushes counters as per-interval increases (see how DTM publishes to Azure Monitor), so upgrade every node to DTM 1.0.2 before you deploy this sample. During a rolling upgrade the cluster runs mixed versions for a while: deploy the sample once the last node is upgraded. Against a node on an earlier version:

• the rules on the metrics that 1.0.2 adds never fire, because that node publishes none of them;
• expect bbolt-stale-lock-loop to fire continuously if the node recovered a stale lock at its last boot: an earlier node publishes that counter as a running total (1 every minute while the process runs), and the rule adds the values up with Total;
• memory-self-restart still fires once for a self-restart, as intended.

The rules

Thirty-three rules per node VM, so a three-node cluster gets 99 (changed in v1.0.6: swap-stable-ip-unrestored is new). Each is named <prefix>-<vm>-<rule>, for example dtm-dtm-node-1-heartbeat-missing. Thresholds are those of dtm-alerts.yml. Critical rules are Azure severity 0 and warnings severity 2.

An Azure metric alert compares one metric with a fixed value: it cannot take a rate, a ratio, a percentile, or the time since an event. So DTM computes those itself and publishes each result as a metric of its own, named for what it holds (for example dtm_health_check_drops_increase_5m, the increase over 5 minutes); see how DTM publishes to Azure Monitor. A Prometheus rule’s for: becomes the evaluation window, the smallest one Azure allows that covers it, and “holds for the whole window” is Minimum (for “above”) or Maximum (for “below”). So a rule does not fire earlier than its Prometheus counterpart, except the three that count restarts with Total and tls-certificate-expiring (see where route B differs).

Rule	Metric	Aggregation	Condition	Window	Sev	Prometheus rule
heartbeat-missing	dtm_heartbeat	Count	< 1	PT5M	0	DTMNodeDown (see the caution below)
cluster-undersized	dtm_cluster_nodes	Maximum	< expected node count	PT5M	0	DTMClusterUndersized
not-serving-dns	dtm_dns_listeners_alive	Maximum	< expected DNS listeners	PT5M	0	DTMNodeNotServingDNS (approximated)
stored-certificate-refused	dtm_tls_stored_pair_refused	Minimum	>= 1	PT15M	2	DTMStoredCertificateRefused
tls-certificate-expiring	dtm_tls_certificate_expires_in_seconds	Minimum	< 1814400 (21 days)	PT6H	2	DTMTLSCertificateExpiringSoon
member-unreachable-day	dtm_replication_longest_unreachable_member_seconds	Minimum	> 86400	PT15M	2	DTMMemberUnreachableLong
member-unreachable-week	dtm_replication_longest_unreachable_member_seconds	Minimum	> 604800	PT15M	0	DTMMemberUnreachableWeek
member-fenced	dtm_replication_fenced_members	Maximum	> 0	PT5M	0	DTMMemberFenced
stale-replica-blocked	dtm_replication_stale_replica_blocked	Maximum	> 0	PT5M	0	DTMStaleReplicaBlocked
swap-stable-ip-unrestored	dtm_lifecycle_swap_stable_ip_unrestored	Maximum	> 0	PT5M	0	DTMSwapStableIPUnrestored
security-advisory-critical	dtm_advisory_active, severity = critical	Maximum	>= 1	PT5M	0	DTMSecurityAdvisoryCritical
security-advisory	dtm_advisory_active, severity not critical	Maximum	>= 1	PT5M	2	DTMSecurityAdvisory
tag-registration-unclaimed	dtm_tag_registration_unclaimed_endpoints	Minimum	> 0	PT30M	2	DTMTagRegistrationUnclaimed
management-group-access-revoked	dtm_region_discovery_membership_lost_access	Minimum	> 0	PT1H	2	DTMManagementGroupAccessRevoked
region-discovery-incomplete	dtm_region_discovery_last_cycle_complete Maximum < 1 and dtm_region_discovery_last_run_timestamp_seconds Minimum > 0			PT6H	2	DTMBackgroundLoopIncomplete
tag-registration-incomplete	dtm_tag_registration_last_cycle_complete Maximum < 1 and dtm_tag_registration_last_run_timestamp_seconds Minimum > 0			PT6H	2	DTMBackgroundLoopIncomplete
data-disk-critical	dtm_storage_free_ratio	Maximum	< 0.05	PT5M	0	DTMStorageDiskCritical
data-disk-low	dtm_storage_free_ratio	Maximum	< 0.10	PT15M	2	DTMStorageDiskLow
restart-loop	dtm_process_starts	Total	> 2	PT15M	0	DTMRestartLoop (approximated)
bbolt-stale-lock-loop	dtm_storage_bbolt_stale_lock_recovered_total	Total	>= 2	PT30M	2	DTMBboltStaleLockLoop (approximated)
memory-self-restart	dtm_memory_self_restart_total	Total	> 0	PT1H	2	DTMMemorySelfRestart (approximated)
health-check-drops	dtm_health_check_drops_increase_5m	Minimum	> 0	PT15M	2	DTMHealthCheckDrops
query-error-rate	dtm_dns_query_error_ratio_5m	Minimum	> 0.05	PT5M	2	DTMQueryErrorRate
dns-latency-above-slo	dtm_dns_authoritative_latency_p99_15m_seconds Minimum > the p99 threshold and dtm_dns_authoritative_queries_increase_15m Minimum >= 100			PT15M	2	DTMDNSLatencyAboveSLO
dns-latency-high	dtm_dns_authoritative_latency_p95_15m_seconds Minimum > the p95 threshold and dtm_dns_authoritative_queries_increase_15m Minimum >= 100			PT5M	0	DTMDNSLatencyHigh
bbolt-freelist-bloat	dtm_storage_bbolt_free_page_ratio	Minimum	> 0.5	PT6H	2	DTMBboltFreelistBloat
replication-lag-high	dtm_replication_last_sync_age_seconds Minimum > 120 and dtm_cluster_nodes Minimum > 1			PT5M	2	DTMReplicationLagHigh
replication-queue-backlog	dtm_replication_broadcast_queue_depth Minimum >= 5000 and dtm_cluster_nodes Minimum > 1			PT15M	2	DTMReplicationQueueBacklog
region-discovery-stalled	dtm_region_discovery_last_run_age_seconds	Minimum	> 10800	PT15M	2	DTMBackgroundLoopStalled
tag-registration-stalled	dtm_tag_registration_last_run_age_seconds	Minimum	> 10800	PT15M	2	DTMBackgroundLoopStalled
region-discovery-failing	dtm_region_discovery_run_errors_increase_30m	Minimum	> 0	PT1H	2	DTMBackgroundLoopFailing
tag-registration-failing	dtm_tag_registration_run_errors_increase_30m	Minimum	> 0	PT1H	2	DTMBackgroundLoopFailing
azure-auth-denied	dtm_azure_auth_denials_increase_10m, split by backend	Minimum	> 0	PT15M	2	DTMAzureAuthDenied

Added in v1.0.6. swap-stable-ip-unrestored fires when a failed stable-IP swap could not leave a running VM holding the stable IP, so its clients may get no answer. Only the node that drove the swap reports it. It reports what the swap recorded, not a live check of the address: after fixing the address by hand, retry the swap to clear it.

Page on cluster-undersized, not a node’s own heartbeat rule

Azure Monitor records nothing, not zero, for a custom metric that stops arriving, so whether a stopped node trips its own heartbeat-missing rule depends on how Azure evaluates that gap. It may fire for a stopped node, but this is not guaranteed, so treat it as a second signal rather than the one to page on. The rule to rely on for a node going down is cluster-undersized: every surviving node sees the member count drop, so it fires on them. Neither can see the whole cluster go down at once, because then no node is publishing anything. For that, add an Azure VM availability alert or an external DNS probe.

Changed in v1.0.4

The sample gains replication-queue-backlog. Before v1.0.4 a deep replication queue turned /readyz not-ready instead; it is now reported in /readyz and alerted on here (see readiness during bulk writes). Earlier nodes publish the same metric, so the rule works against them too, but on those nodes /readyz still turns not-ready at the threshold.

Changed in v1.0.5

dtm_storage_bbolt_pages_in_use now counts the pages the database has allocated, less its free pages. Before v1.0.5 it counted the whole database file, which grows ahead of the data, so the free-page ratio read low. After an upgrade the ratio reads higher for the same database, so bbolt-freelist-bloat (and the Prometheus DTMBboltFreelistBloat rule) can fire on a node where it did not before. The rule itself is unchanged. When it fires, more than half of the database really is free pages: compact the node.

Where route B differs

• restart-loop, bbolt-stale-lock-loop and memory-self-restart are tied to a process restart, which ends any window a node keeps in memory, so they count events with Total over the window rather than holding a condition for the Prometheus for: duration. Each can therefore fire sooner than its Prometheus rule. A boot that dies before its first push (about a minute) is not counted by restart-loop; heartbeat-missing and cluster-undersized cover a node that never comes up. The stale lock rule wants two recoveries in 30 minutes, because one recovery after a single crash is not a loop.
• not-serving-dns compares with a number you supply, because an Azure metric alert cannot compare two metrics. If you enable DNS over TLS later, raise the expected DNS listeners by one. Do not raise it for DNS over HTTPS: a threshold the node can never reach fires this severity 0 rule permanently.
• Five rules are not split by node: region-discovery-incomplete, tag-registration-incomplete, dns-latency-above-slo, dns-latency-high and replication-lag-high have two criteria, and Azure allows a split on all values only in a single-criterion rule. They are correct while each node publishes to its own VM, which is what leaving the target blank does.
• tls-certificate-expiring is not split by listener: it takes the lowest time left across the node’s listeners (the API, and DoT and DoH, which share one certificate). It fires on the first value below 21 days, up to an hour before the Prometheus rule’s for: 1h elapses, and clears up to six hours after the certificate is renewed or replaced.
• cluster-undersized needs at least two nodes to mean anything. On a single-node deployment, leave the expected node count at 1 and the rule never fires.
• dns-latency-above-slo and dns-latency-high judge only the answers a node gives from its own zones (forwarded answers are excluded), over 15 minutes, and only once at least 100 of them were answered in that time, as the Prometheus rules do.
• region-discovery-stalled and tag-registration-stalled judge a loop only once it has run: a node publishes the age of its last run only after the first one. The two incomplete rules likewise wait for a first run, and tag registration is off until you enable it.
• replication-lag-high needs a first completed sync: a node publishes the age of its last sync only once there has been one. A node with peers that has never synced fires DTMReplicationLagHigh but not this rule; cluster-undersized and the member-unreachable rules cover a node that never joins.
• replication-queue-backlog fires when the queue has held 5000 or more for 15 minutes on a node that has peers. It has two criteria, so like the rules above it is not split by node. The queue is this node’s writes not yet passed on to enough peers, so a backlog usually points at an unhealthy peer rather than at the node that fires: check the member-unreachable rules and the Nodes page first.

What route B cannot alert on

Every rule in dtm-alerts.yml has a route B form except three:

• A whole-cluster outage (DTMAllNodesDown). Every route B rule reads metrics the nodes publish themselves, so when every node is down at once, nothing is publishing and no rule has anything to judge. Pair the sample with an Azure VM availability alert or an external DNS probe for that.
• A managed script check that is not running (DTMScriptCheckNotRunning, from v1.0.5). The sample has no rule for it. The metric it reads, dtm_health_probe_refusals_total, is published like any other, so add your own rule on it, for the guard values that start with script-, if you use managed scripts.
• A private endpoint zone group the node cannot read (DTMPrivateEndpointZoneGroupUnread). The sample has no rule for it. The metric it reads, dtm_private_endpoint_zone_group_unread, is published like any other, so add your own rule on it if you use Private Endpoint automation.

On Azure, the KQL log alerts add signals the metrics do not carry, such as NXDOMAIN floods per client. They need query-log shipping to Log Analytics turned on, and they do not separate DTM’s own answers from forwarded ones, so for the latency objective rely on the sample’s DNS latency rules, not the log alerts.