Authenticated health probes

Hayami DTM’s health probes are the engine behind its load balancing and failover: it only routes to endpoints that pass. Many real backends sit behind authentication, so DTM can probe auth-gated endpoints directly, with no public /healthz to expose, and (on the Managed-Identity path) no stored credentials.

Authentication is configured on the probe: on a record’s health_check_template (the same field you use on an ALIAS record or an override record), or on a monitor-only health check’s http_config. Every management surface carries the same fields: the REST API, Terraform, and the web UI. The health, load-balancing, and failover behaviour is identical; only how the probe authenticates changes.

DTM supports four auth methods:

a static Authorization header: a fixed token or API key;
OAuth2 client-credentials: DTM fetches and refreshes a bearer token;
mutual TLS: a client certificate, with optional CA pinning;
Azure Managed Identity: system- or user-assigned, with no stored credentials.

Static Authorization header

The simplest option: DTM sends the headers you configure on every probe. Use them for a static bearer token, an API key, or basic auth you pre-encode.

"health_check_template": {
  "protocol": "https", "port": 443, "path": "/healthz",
  "interval_seconds": 5,
  "headers": {
    "Authorization": "Bearer <static-token>"
  }
}

Header values are treated as secrets: masked in the audit log, never returned by the API or UI, and sent only to the probed endpoint.

Always set "protocol": "https" here. Unlike the three dynamic methods below, a static header is not blocked from running over plain HTTP.

OAuth2 client-credentials

DTM obtains a bearer token from your token endpoint using the client-credentials grant, caches it, and refreshes it before expiry, so the probe always carries a valid token without you rotating anything by hand.

"health_check_template": {
  "protocol": "https", "port": 443, "path": "/healthz",
  "interval_seconds": 5,
  "oauth2": {
    "token_url": "https://login.example.com/oauth2/token",
    "client_id": "<client-id>",
    "client_secret": "<client-secret>",
    "scope": "health.read"
  }
}

Mutual TLS (client cert + CA pin)

DTM presents a client certificate on the probe connection and, optionally, pins the endpoint’s CA so it only trusts a certificate chain you nominate.

"health_check_template": {
  "protocol": "https", "port": 443, "path": "/healthz",
  "interval_seconds": 5,
  "mtls": {
    "cert_pem": "<pem-client-certificate>",
    "key_pem": "<pem-private-key>"
  },
  "ca_pem": "<pem-ca-certificate>"
}

Leave ca_pem out to use the system trust store; set it to pin the probe to a specific CA. (ca_pem sits alongside mtls on the template and works with any auth method.)

Azure Managed Identity

The strongest Azure-native option: no stored credentials. DTM acquires a token from the Azure instance metadata endpoint using the VM’s identity and sends it as a bearer token, so there is no secret to store, rotate, or leak.

DTM supports both flavours:

System-assigned: the identity tied to the DTM VM itself. Just name the audience (resource) you want a token for:

"health_check_template": {
  "protocol": "https", "port": 443, "path": "/healthz",
  "interval_seconds": 5,
  "managed_identity": {
    "resource": "api://<app-id-uri>"
  }
}

User-assigned: a shared identity you attach to the VM; identify it by its client ID:

"health_check_template": {
  "protocol": "https", "port": 443, "path": "/healthz",
  "interval_seconds": 5,
  "managed_identity": {
    "client_id": "<user-assigned-identity-client-id>",
    "resource": "api://<app-id-uri>"
  }
}

Grant the identity access to the target (e.g. an app role or RBAC assignment) and DTM’s probes authenticate with nothing stored on the DTM side.

Every node probes on its own, so every node needs the identity and its access:

System-assigned: each node has its own identity, so grant each node’s identity access to the target. A node you add, and the replacement an upgrade creates, comes with a new identity that needs the same grant.
User-assigned: the Marketplace deployments give each node only its system-assigned identity. Attach your user-assigned identity to every node yourself, including a node you add and the replacement an upgrade creates. Until it is attached, that node’s probe fails and the node treats the backend as unhealthy. With the Terraform starter, both nodes can share an existing user-assigned identity (create_identity = false with existing_identity_id).

Admin-only, and which audiences are allowed

The token is minted from the node’s own Azure identity, and that identity can read the cluster’s Key Vault. So a Managed Identity probe is held to the same bar as a script probe:

Admin role required. Creating a Managed Identity probe, or changing anything about a check, record or override that uses one, needs the admin role. That includes a change to a record’s or override’s values alone, since every value is a probe target. An editor token, including an editor service account, gets 403. This applies to health checks, record and override templates, and records in a JSON zone import.
Audience policy. By default a probe may not ask for a token for Key Vault or Managed HSM, Azure Resource Manager, Microsoft Graph or Azure AD Graph, or Azure Storage, in any cloud, named by URL or by their well-known application IDs. Such a request gets 400. Your own application’s audience (api://..., or its App ID URI) is what this feature is for, and it is unaffected.
Allowlist mode. To accept only an exact list of audiences instead, set managed_identity_probe_audiences in the server YAML on every node (see configuration). Only the listed audiences are then accepted.
Checked again when the token is minted. A probe that reaches a node by replication or a restore is held to the same policy, and reports unhealthy with managed identity token refused by this node's audience policy rather than an IMDS error.

Good to know

Auth applies to the probe, not the answer. DTM authenticates when it health-checks a backend; the DNS answer it returns is unchanged.
Secrets are write-only. Tokens, client secrets, and keys are masked in the audit log and never returned by the API or UI. When you edit a probe in the web UI, secret fields load blank with a “leave blank to keep” hint: leave one blank to keep the stored value, or type a new one to rotate it. Over the API, sending the masked value *** back, or leaving the block out, keeps the stored secret the same way. Terraform likewise keeps the values you applied in state, so a refresh shows no spurious drift.
Moving a secret means re-entering it. Keeping a stored secret only works while its destination is unchanged. When an editor changes where it is sent, every secret the probe keeps must be sent again in clear in the same request, or the request gets 400 naming what moved and which secrets to re-enter. The destination is a health check’s protocol, target and oauth2.token_url, or a template’s protocol, port and oauth2.token_url together with the record’s or override’s values: adding a value counts, removing one does not, and so does turning a record’s auto_populate on. Clearing the template needs no re-entry, and an admin is not bound by the rule. A dynamic DNS update cannot add a new value to a record whose template carries a secret or uses Managed Identity; add it through the API. (Changed in v1.0.4.) From v1.0.7 the destination also covers where the probe trusts the server: changing tls_skip_verify, ca_pem, tls_server_name or the Host header needs the kept secrets re-entered too.
A credential header needs verified TLS. Added in v1.0.7. A check or template that sends a credential header (Authorization, Cookie, or a header whose name contains auth, key, token, secret, passw, pwd, session, credential, signature, bearer, jwt or assertion) with tls_skip_verify and no ca_pem is refused at save, as an oauth2, mtls or Managed Identity probe already was. A check saved that way before v1.0.7 stops probing until you give it a ca_pem pin or turn verification back on. The cluster’s health-probe TLS policy (allow_insecure_authenticated_probes) still relaxes this.
Guard rails are built in. A probe carries at most one bearer-token method (oauth2 or managed_identity); a client certificate (mtls) can be combined with either. Dynamically fetched credentials require HTTPS: a probe configuring oauth2, mtls, or managed_identity is rejected with "protocol": "http" at create and at update (an existing one cannot be switched to plain HTTP), and by default the backend’s TLS certificate is verified, against the system trust store or your ca_pem pin, before any credential is sent. An admin can relax the certificate-verification requirement (for self-signed backends you cannot pin) via PUT /api/v1/config/health-probe-policy. Changed in v1.0.7: the policy applies to the whole cluster, so one PUT, or one dtm_health_probe_policy_config, reaches every node (it was per node before). After upgrading every node to v1.0.7, save it once: until then each node keeps the value it had. A refused probe resets the check to unknown, which counts as healthy: that node keeps serving the backend without checking it, and a backend it had already marked unhealthy goes back into its answers until a real probe runs. The change is audit-logged, and plain HTTP stays rejected for those three methods regardless. A static Authorization header is not covered by that guard: DTM will accept it with "protocol": "http" and send the token in cleartext, so set "protocol": "https" yourself.
Verified against the right name. Credentials go only to a backend whose certificate verifies, so a backend whose certificate names its hostname needs tls_server_name set to that hostname, or a Host header naming it. (Added in v1.0.5.) See the name the certificate is checked against.
One engine, everywhere. The same health_check_template (and every auth method above) works on plain records, ALIAS records, and override records, and the identical auth fields apply to monitor-only health checks (/api/v1/healthchecks or the dtm_healthcheck resource; the UI’s Health Checks page edits them).

Need a hand? Email [email protected].

Privacy policyTerms of use