Health checks

Health checks are what make DTM’s answers health-aware: it probes each backend and returns only the ones that pass. A failing backend is dropped from DNS responses automatically, and traffic shifts to the remaining healthy values.

Health checking attaches to records through a health-check template on the record (or on an ALIAS or override record): DTM instantiates one check per value automatically and keeps them in sync as values change.

Separately, a monitor-only check can be created directly through the API or the dtm_healthcheck Terraform resource. It probes a target and reports status (the Health Checks page and the dtm_health_check_status metric) but has no DNS effect: it never gates any record’s answers. Use it as a lightweight synthetic monitor; use the template for anything that should influence answers.

Probe types

Protocol	What it does	Healthy when
tcp	Opens a TCP connection to the target port.	The connection succeeds.
http	Sends an HTTP request to a path.	The response status matches (default 200) and any expected body substring is present.
https	As http, over TLS.	Same, with certificate verification (or a pinned CA).

For auth-gated backends, HTTP/HTTPS probes can authenticate with a static token, OAuth2 client-credentials, mutual TLS, or Azure Managed Identity. See authenticated health probes.

Script probes (disabled by default)

A fourth probe type, script, runs a command on each node and treats exit code 0 as healthy. It sits behind two gates: it is off by default until you set allow_script_health_checks: true in the server configuration (the Terraform deployment exposes the same switch as a module variable), and even with the flag on, creating a script check, switching a check to the script protocol, or editing its script_config requires the admin role; editor-tier tokens, including editor service accounts, are refused with a 403 (a service account granted the admin role is treated as an admin). Other edits to an existing script check, such as changing its interval or target, and deleting it, are editor-tier. While the flag is off, the API refuses script checks for everyone, and a node will not execute one under any circumstances; it reports unhealthy instead.

Caution

Enabling script probes means an admin can run arbitrary operator-supplied commands on every node, so treat the flag as a security decision (editor-tier tokens cannot define script checks; an admin-role service account can). The commands execute inside a sandboxed probe-runner unit under a dedicated unprivileged user with no access to DTM’s configuration, data store, or the instance metadata service; if the runner is unavailable the probe fails closed and reports unhealthy. Before an HTTPS backend pushes you to scripts, check whether an authenticated probe covers it: those are fully supported in production.

A script check’s probe deadline is the check’s top-level timeout_seconds (default 5, and it may not exceed interval_seconds). If the script needs a different deadline, set script_config.timeout_seconds: when set it overrides the top-level value for the script run, under the same must-not-exceed-the-interval rule. It applies only to script checks, so a leftover script_config on a check whose protocol later changed does not affect that check’s timing. The runner also refuses any timeout above five minutes outright.

Because the sandbox fails closed, every script check on a node going unhealthy at once (while tcp and http checks stay green) points at the probe runner, not your backends; see troubleshooting.

Configuration

Setting	Meaning	Guidance
interval_seconds	How often each backend is probed.	Default 15. Minimum 5. Lower means faster detection but more probe traffic.
timeout_seconds	How long a single probe may take.	Keep it comfortably below the interval.
failure_threshold	Consecutive failures before a backend is marked unhealthy.	Higher resists flapping; lower fails over sooner. Settable on monitor-only checks; checks created from a template use 3.
success_threshold	Consecutive successes before an unhealthy backend returns.	Higher avoids flapping a recovering backend back in too soon. Settable on monitor-only checks; checks created from a template use 2.
expected_status	HTTP status codes treated as healthy.	Defaults to [200].
expected_body	Substring the HTTP body must contain.	Optional; use to catch “up but broken” backends.

Note

Every node in the cluster runs every check independently, and health state is not replicated between nodes. This keeps each node’s view of “is this backend reachable from here” accurate: a node’s own reachability stands in for its clients’, so it will not steer clients toward a region it cannot itself reach. The flip side: for cross-region failover, your peering and NSGs must let the DTM nodes probe backends in every region (Azure VNet peering is not transitive). It also means very short intervals across many backends multiply the probe load, so tune the interval to your fleet size.

How health drives routing

1. DTM probes each value on its own interval.
2. A value that crosses its failure threshold is marked unhealthy and removed from DNS answers.
3. The load-balancing policy is applied only to the remaining healthy values.
4. When the value recovers and crosses its success threshold, it is returned to the pool.

If every value is unhealthy, DTM fails open and returns all of them: an answer that can reach a recovering backend beats an empty one that strands every client.

Detection and failover time

Detection time is roughly interval_seconds x failure_threshold. Total end-to-end failover is detection time plus the record’s DNS TTL (the time clients keep caching the old answer). For a critical service, a short interval (for example 5s) with a low failure threshold and a low record TTL gives failover in the low tens of seconds. See load-balancing methods for the full picture.

Managing health checks

• Web UI: configure a health check inline when you add or edit a record. The Health Checks page lists every check worst-status first, with a summary bar and status filters for triage at scale.
• REST API: health_check_template on a record, or the /api/v1/healthchecks endpoints for monitor-only checks.
• Terraform: a health_check_template block, or the dtm_healthcheck resource (monitor-only).