Authenticated health probes

Hayami DTM’s health probes are the engine behind its load balancing and failover: it only routes to endpoints that pass. Many real backends sit behind authentication, so DTM can probe auth-gated endpoints directly, with no public /healthz to expose, and (on the Managed-Identity path) no stored credentials.

Authentication is configured on the probe: on a record’s health_check_template (the same field you use on an ALIAS record or an override record), or on a monitor-only health check’s http_config. Every management surface carries the same fields: the REST API, Terraform, and the web UI. The health, load-balancing, and failover behaviour is identical; only how the probe authenticates changes.

DTM supports four auth methods:

• a static Authorization header: a fixed token or API key;
• OAuth2 client-credentials: DTM fetches and refreshes a bearer token;
• mutual TLS: a client certificate, with optional CA pinning;
• Azure Managed Identity: system- or user-assigned, with no stored credentials.

Static Authorization header

The simplest option: DTM sends the headers you configure on every probe. Use them for a static bearer token, an API key, or basic auth you pre-encode.

"health_check_template": {
  "protocol": "https", "port": 443, "path": "/healthz",
  "interval_seconds": 5,
  "headers": {
    "Authorization": "Bearer <static-token>"
  }
}

Header values are treated as secrets: masked in the audit log, never returned by the API or UI, and sent only to the probed endpoint.

Always set "protocol": "https" here. Unlike the three dynamic methods below, a static header is not blocked from running over plain HTTP.

OAuth2 client-credentials

DTM obtains a bearer token from your token endpoint using the client-credentials grant, caches it, and refreshes it before expiry, so the probe always carries a valid token without you rotating anything by hand.

"health_check_template": {
  "protocol": "https", "port": 443, "path": "/healthz",
  "interval_seconds": 5,
  "oauth2": {
    "token_url": "https://login.example.com/oauth2/token",
    "client_id": "<client-id>",
    "client_secret": "<client-secret>",
    "scope": "health.read"
  }
}

Mutual TLS (client cert + CA pin)

DTM presents a client certificate on the probe connection and, optionally, pins the endpoint’s CA so it only trusts a certificate chain you nominate.

"health_check_template": {
  "protocol": "https", "port": 443, "path": "/healthz",
  "interval_seconds": 5,
  "mtls": {
    "cert_pem": "<pem-client-certificate>",
    "key_pem": "<pem-private-key>"
  },
  "ca_pem": "<pem-ca-certificate>"
}

Leave ca_pem out to use the system trust store; set it to pin the probe to a specific CA. (ca_pem sits alongside mtls on the template and works with any auth method.)

Azure Managed Identity

The strongest Azure-native option: no stored credentials. DTM acquires a token from the Azure instance metadata endpoint using the VM’s identity and sends it as a bearer token, so there is no secret to store, rotate, or leak.

DTM supports both flavours:

• System-assigned: the identity tied to the DTM VM itself. Just name the audience (resource) you want a token for:

  "health_check_template": {
    "protocol": "https", "port": 443, "path": "/healthz",
    "interval_seconds": 5,
    "managed_identity": {
      "resource": "api://<app-id-uri>"
    }
  }

• User-assigned: a shared identity you attach to the VM; identify it by its client ID:

  "health_check_template": {
    "protocol": "https", "port": 443, "path": "/healthz",
    "interval_seconds": 5,
    "managed_identity": {
      "client_id": "<user-assigned-identity-client-id>",
      "resource": "api://<app-id-uri>"
    }
  }

Grant the identity access to the target (e.g. an app role or RBAC assignment) and DTM’s probes authenticate with nothing stored on the DTM side.

Good to know

• Auth applies to the probe, not the answer. DTM authenticates when it health-checks a backend; the DNS answer it returns is unchanged.
• Secrets are write-only. Tokens, client secrets, and keys are masked in the audit log and never returned by the API or UI. When you edit a probe in the web UI, secret fields load blank with a “leave blank to keep” hint: leave one blank to keep the stored value, or type a new one to rotate it. Terraform likewise keeps the values you applied in state, so a refresh shows no spurious drift.
• Guard rails are built in. A probe carries at most one bearer-token method (oauth2 or managed_identity); a client certificate (mtls) can be combined with either. Dynamically fetched credentials require HTTPS: a probe configuring oauth2, mtls, or managed_identity is rejected with "protocol": "http" at create and at update (an existing one cannot be switched to plain HTTP), and by default the backend’s TLS certificate is verified, against the system trust store or your ca_pem pin, before any credential is sent. An admin can relax the certificate-verification requirement cluster-wide (for self-signed backends you cannot pin) via PUT /api/v1/config/health-probe-policy; the change is audit-logged, and plain HTTP stays rejected for those three methods regardless. A static Authorization header is not covered by that guard: DTM will accept it with "protocol": "http" and send the token in cleartext, so set "protocol": "https" yourself.
• One engine, everywhere. The same health_check_template (and every auth method above) works on plain records, ALIAS records, and override records, and the identical auth fields apply to monitor-only health checks (/api/v1/healthchecks or the dtm_healthcheck resource; the UI’s Health Checks page edits them).

Need a hand? Email [email protected].