Health checks

Health checks are what make DTM’s answers health-aware: it probes each backend and returns only the ones that pass. A failing backend is dropped from DNS responses automatically, and traffic shifts to the remaining healthy values.

Health checking attaches to records through a health-check template on the record (or on an ALIAS or override record): DTM instantiates one check per value automatically and keeps them in sync as values change.

Separately, a monitor-only check can be created directly through the API or the dtm_healthcheck Terraform resource. It probes a target and reports status (the Health Checks page and the dtm_health_check_status metric) but has no DNS effect: it never gates any record’s answers. Use it as a lightweight synthetic monitor; use the template for anything that should influence answers.

Probe types

Protocol	What it does	Healthy when
tcp	Opens a TCP connection to the target port.	The connection succeeds.
http	Sends an HTTP request to a path.	The response status matches (default 200) and any expected body substring is present.
https	As http, over TLS.	Same, with certificate verification (or a pinned CA).

For auth-gated backends, HTTP/HTTPS probes can authenticate with a static token, OAuth2 client-credentials, mutual TLS, or Azure Managed Identity. See authenticated health probes.

HTTPS: the name the certificate is checked against

A probe from a health_check_template connects to each value’s IP address. Most real HTTPS backends, such as an App Service or Application Gateway private endpoint or an AKS ingress, present a certificate for their hostname, and a publicly trusted certificate cannot name a private IP. Set tls_server_name to the hostname the backend’s certificate names:

"health_check_template": {
  "protocol": "https", "port": 443, "path": "/healthz",
  "tls_server_name": "app.example.com"
}

• Only the name changes. The probe sends tls_server_name as SNI and verifies the certificate against it, but still connects to the value’s address (for a monitor-only check, its target). With tls_skip_verify and no ca_pem, the name is sent as SNI only and nothing is verified.
• Verification is not relaxed. With verification on (the default, or a ca_pem pin), the certificate must name tls_server_name.
• A hostname, on HTTPS only. No IP address, port, scheme or *, and only with "protocol": "https". Anything else is refused with 400.
• Every surface carries it. The web UI’s TLS server name field on an HTTPS check, the API’s health_check_template and http_config, and the Terraform health_check_template and dtm_healthcheck http_config blocks.
• Unset, a Host header can supply it. A Host header that is a hostname (a :port suffix is dropped) sets the name instead. With neither, the certificate is verified against the address, as before.

Changed in v1.0.5

Before v1.0.5 a Host header changed only the HTTP header. Now, when tls_server_name is not set, a Host header that is a hostname also sets the name the certificate is verified against. An HTTPS check that sends a Host header to a backend whose certificate names only its IP fails verification after the upgrade. Give the backend a certificate that names that hostname, or remove the Host header if the backend does not need it. Checks with no Host header behave as before.

Script probes (disabled by default)

A fourth probe type, script, runs a command on each node and treats exit code 0 as healthy. It sits behind two gates: it is off by default until you set allow_script_health_checks: true in the server configuration (the Terraform deployment exposes the same switch as a module variable), and even with the flag on, creating a script check, switching a check to the script protocol, or editing its script_config requires the admin role; editor-tier tokens, including editor service accounts, are refused with a 403 (a service account granted the admin role is treated as an admin). Changed in v1.0.6: every other change to an existing script check, its target, its protocol or only its interval, and deleting it, is admin-only too; a non-admin PUT or DELETE answers 403 and leaves the check as it was (before v1.0.6 those edits were editor-tier). Editors keep control of every other check, except that a Managed Identity probe also needs admin to create or change (see Health probes). While the flag is off, the API refuses script checks for everyone, and a node will not execute one under any circumstances; it reports unhealthy instead.

Managed scripts, from v1.0.5

Instead of a command that must already exist on every node, a script check can run a managed script: a script DTM stores and runs on every node, approved by a signing key you hold, and pinned to the exact version you approved. Both forms run in the same sandbox, behind the same setting and the admin role. From v1.0.7 a record’s or override’s health check template can run a managed script against each value; see In a record or override template.

Caution

Enabling script probes means an admin can run arbitrary operator-supplied commands on every node, so treat the flag as a security decision (editor-tier tokens cannot define script checks; an admin-role service account can). The commands execute inside a sandboxed probe-runner unit under a dedicated unprivileged user with no access to DTM’s configuration, data store, or the instance metadata service; if the runner is unavailable the probe fails closed and reports unhealthy. Before an HTTPS backend pushes you to scripts, check whether an authenticated probe covers it: those are fully supported in production.

A script check’s probe deadline is the check’s top-level timeout_seconds (default 5, and it may not exceed interval_seconds). If the script needs a different deadline, set script_config.timeout_seconds: when set it overrides the top-level value for the script run, under the same must-not-exceed-the-interval rule. It applies only to script checks, so a leftover script_config on a check whose protocol later changed does not affect that check’s timing. The runner also refuses any timeout above five minutes outright.

Because the sandbox fails closed, every script check on a node going unhealthy at once (while tcp and http checks stay green) points at the probe runner, not your backends; see troubleshooting.

Configuration

Setting	Meaning	Guidance
interval_seconds	How often each backend is probed.	Default 15. Minimum 5. Lower means faster detection but more probe traffic.
timeout_seconds	How long a single probe may take.	Keep it comfortably below the interval.
failure_threshold	Consecutive failures before a backend is marked unhealthy.	Higher resists flapping; lower fails over sooner. Settable on monitor-only checks; checks created from a template use 3.
success_threshold	Consecutive successes before an unhealthy backend returns.	Higher avoids flapping a recovering backend back in too soon. Settable on monitor-only checks; checks created from a template use 2.
expected_status	HTTP status codes treated as healthy.	Defaults to [200].
expected_body	Substring the HTTP body must contain.	Optional; use to catch “up but broken” backends.

Note

Every node in the cluster runs every check independently, and health state is not replicated between nodes. This keeps each node’s view of “is this backend reachable from here” accurate: a node’s own reachability stands in for its clients’, so it will not steer clients toward a region it cannot itself reach. The flip side: for cross-region failover, your peering and NSGs must let the DTM nodes probe backends in every region (Azure VNet peering is not transitive). It also means very short intervals across many backends multiply the probe load, so tune the interval to your fleet size.

How health drives routing

1. DTM probes each value on its own interval.
2. A value that crosses its failure threshold is marked unhealthy and removed from DNS answers.
3. The load-balancing policy is applied only to the remaining healthy values.
4. When the value recovers and crosses its success threshold, it is returned to the pool.

If every value is unhealthy, DTM fails open and returns all of them: an answer that can reach a recovering backend beats an empty one that strands every client.

Detection and failover time

Detection time is roughly interval_seconds x failure_threshold. Total end-to-end failover is detection time plus the record’s DNS TTL (the time clients keep caching the old answer). For a critical service, a short interval (for example 5s) with a low failure threshold and a low record TTL gives failover in the low tens of seconds. See load-balancing methods for the full picture.

Managing health checks

• Web UI: configure a health check inline when you add or edit a record. The Health Checks page lists every check worst-status first, with a summary bar and status filters for triage at scale.
• REST API: health_check_template on a record, or the /api/v1/healthchecks endpoints for monitor-only checks.
• Terraform: a health_check_template block, or the dtm_healthcheck resource (monitor-only).