Managed script health checks

Added in v1.0.5

A managed script is a health-check script that Hayami DTM stores, replicates and runs on every node, instead of a file you place on each node yourself. You approve its code with a signing key that DTM never holds. Before every run, each node checks that a key it trusts signed exactly that script, and that the script has not been revoked.

The steps are: create a key, write the script, then draft, test, sign and activate it, and attach it to a check.

What runs where

• Only where script checks are allowed. A script runs only on nodes whose allow_script_health_checks is true, in the same sandboxed probe runner as any script probe: an unprivileged user with no access to DTM’s configuration, data store or the instance metadata service. A node with the setting off holds the check but never runs it, and DTMScriptCheckNotRunning fires there. Turn the setting on for every node that should enforce the check.
• Interpreter and size. The interpreter is sh or python3. The script reaches it on standard input, so a managed script cannot read standard input itself. A script is at most 64 KiB of UTF-8 text.
• Inputs and result. The check’s target is in the DTM_CHECK_TARGET environment variable, and its arguments are $1, $2 and so on (sys.argv[1:] in Python). Exit status 0 is healthy; any other status is unhealthy.
• Admin only. Every managed-script API call, reads included, needs the admin role. On a node with allow_script_health_checks off, every call is refused with 403.

1. Create a signing key

Keep the private key on a workstation, or in the pipeline that holds your other secrets. Never put it on a cluster node. A node that can sign can approve code for every node.

Two key types are accepted: Ed25519, and ES256 (ECDSA P-256). Create the key either way.

In the web UI. The Scripts page’s Create a signing key card creates the key pair in your browser and downloads the private key as a PKCS#8 PEM file named after the key ID. It shows the public key, with the dtm-server.yaml entry and the Terraform vm module form ready to copy. No copy of the private key is kept anywhere: store the file safely, because a lost key means signing with a new one.

With openssl (OpenSSL 3), where the key will live. The second command prints the public key:

On your workstation: an Ed25519 key

openssl genpkey -algorithm ed25519 -out ops-2026.key
openssl pkey -in ops-2026.key -pubout -outform DER | base64 | tr -d '\n'; echo

Or an ES256 key

openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out ops-2026.key
openssl pkey -in ops-2026.key -pubout -outform DER | base64 | tr -d '\n'; echo

Add the public key to dtm-server.yaml on every node:

dtm-server.yaml, on every node

script_signing_keys:
  - id: ops-2026
    public_key: MCowBQYDK2VwAyEA...

Restart each node after you change it: the keys are read once, at startup. On a cluster deployed with the Terraform vm module, set the module’s script_signing_keys variable (a list of {id, public_key}) on every node’s module block instead.

Keep script_signing_keys and script_revoked_hashes the same on every node. Each node trusts only its own list, so nodes with different lists disagree about which scripts they will run.

To keep the private key in Azure Key Vault, use ES256: Key Vault offers ECDSA P-256 but not Ed25519. Create the key in Key Vault, put its public key in script_signing_keys, and sign digest.bin from the Ed25519 commands in step 3 (the SHA-256 the nodes verify) with Key Vault’s ES256 sign operation. DTM accepts an ES256 signature in ASN.1 DER or raw r||s form.

2. Write the script, and treat its inputs as untrusted

Your key approves a script’s code, but not the target and arguments a check gives it: those are set by whoever saves the check. Write every script so that hostile input does nothing worse than fail the check.

Rely only on the interpreter and what ships with it. Python’s standard library covers most probes.

Shell: quote every expansion, and never eval an input or pass one to sh -c.

Good: inputs are compared and printed, never run

role="${1:-primary}"
[ "$role" = "primary" ] || [ "$role" = "replica" ] || { printf 'unknown role %s\n' "$role"; exit 2; }

Python: treat inputs as data, pass argument lists to anything you run, and never use shell=True.

A TCP connect probe

import os, socket, sys
host, _, port = os.environ["DTM_CHECK_TARGET"].rpartition(":")
try:
    socket.create_connection((host, int(port)), timeout=3).close()
except (OSError, ValueError):
    sys.exit(1)

Do not put credentials in a script: scripts are replicated, backed up, and readable by admins through the API.

3. Draft, test, sign, activate

Save the script as a draft. A draft has no signature. It stays on the node you saved it on, never replicates, never runs on a schedule, and cannot be attached to a check.

curl -sk -X POST https://<node>:8443/api/v1/scripts \
  -H "Authorization: Bearer $TOKEN" -H "X-DTM-CSRF: 1" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --rawfile body check.sh '{name:"orders-db", interpreter:"sh", body:$body}')"

Test it on that node, through the real probe runner, before you sign it. A test runs once, on the node you call, at most once every 10 seconds per caller, and is audited.

curl -sk -X POST https://<node>:8443/api/v1/scripts/<id>/test \
  -H "Authorization: Bearer $TOKEN" -H "X-DTM-CSRF: 1" \
  -H "Content-Type: application/json" \
  -d '{"target":"10.0.0.5:5432","args":["5432"]}'

Signing does not limit admins

Testing is the one place an unsigned script runs. With allow_script_health_checks on, an admin token can save an unsigned draft and run it once on the node it calls. Signing stops an unsigned script from being published, replicated or scheduled; it does not stop an admin running one on demand. Treat an admin token as able to run code on any node with the setting on, and leave the setting off on clusters that do not use script checks.

Sign it in the web UI (below), or with openssl where the key lives. A key signs the SHA-256 of dtm-script-sig-v1, a zero byte, the interpreter, a zero byte, and the draft’s content hash. The commands work from the script file, check.sh: the hash they compute must equal the content hash the draft shows. Set interp to the draft’s interpreter.

On your workstation: sign with an Ed25519 key

interp=sh
hash="sha256:$(openssl dgst -sha256 -r check.sh | cut -d' ' -f1)"
{ printf 'dtm-script-sig-v1\0%s\0' "$interp"; printf '%s' "$hash"; } | openssl dgst -sha256 -binary > digest.bin
openssl pkeyutl -sign -inkey ops-2026.key -rawin -in digest.bin | base64 | tr -d '\n'; echo

Or sign with an ES256 key

interp=sh
hash="sha256:$(openssl dgst -sha256 -r check.sh | cut -d' ' -f1)"
{ printf 'dtm-script-sig-v1\0%s\0' "$interp"; printf '%s' "$hash"; } | openssl dgst -sha256 -sign ops-2026.key | base64 | tr -d '\n'; echo

Each prints the signature. Activate the draft with it:

curl -sk -X POST https://<node>:8443/api/v1/scripts/<id>/draft/activate \
  -H "Authorization: Bearer $TOKEN" -H "X-DTM-CSRF: 1" \
  -H "Content-Type: application/json" \
  -d '{"key_id":"ops-2026","signature":"<base64>"}'

The node checks the signature against its own keys, publishes the script as its next version, and replicates it to every node. An automated pipeline can skip the draft: send key_id and signature with the body in the first POST. A signature that does not verify is refused with 400, naming why: an unknown key ID, a signature over a different script, or a revoked hash.

In the web UI

The Scripts page (admins only, on a node with script checks allowed) does the same steps:

1. New script saves a draft, marked “Draft: not signed, will not run”.
2. Test draft runs it once on this node.
3. Sign and activate here shows the draft’s body, which is what you approve, so read it. Pick your private key file (PKCS#8 PEM, Ed25519 or ES256), enter its key ID, and choose Sign and activate. The page hashes the body it shows and refuses to sign if that does not match the draft’s content hash. It signs with the key type the file holds and says which. A file it cannot use (encrypted, RSA, another curve, an EC PRIVATE KEY file, or a public key) is refused by name.
4. Sign with openssl instead shows the commands above, with a form to paste a key ID and a signature made elsewhere.
5. New version saves a new draft for the same script. Checks keep their version until you move them.

How the page handles your key. The file is read in the browser tab, used for one signature and dropped, and every buffer that held it is wiped. It is never sent, stored or logged: the page can send requests only to the node that served it, and it sends only the signature. A second signature needs the file picked again. The browser’s own copies, the file it read and the download a new key came from, are released when you close the tab.

Where to sign

The Scripts page is served by the node you open it on. Someone with root on that node could change the page and capture the key of the next person who signs there, then approve code for every node. Sign with openssl for any key that approves scripts on more than one cluster, or if your threat model includes a compromised node. Sign in a browser profile without extensions.

The page needs HTTPS, and Ed25519 keys need a current Chrome, Edge, Firefox or Safari. Where the browser cannot sign, the page says so and shows the openssl commands instead.

On the Health Checks page, a script check’s edit dialog picks the script and version it uses, and a node that refuses to run a check shows the reason on the check’s badge.

4. Attach it to a check

Create a health check with "protocol": "script" whose script_config names the script. With no script_version (or 0), the check pins the latest version at the moment you save it, and the response shows the version and hash it pinned.

{
  "name": "orders-db",
  "protocol": "script",
  "target": "10.0.0.5:5432",
  "interval_seconds": 30,
  "timeout_seconds": 5,
  "script_config": { "script_id": "<id>", "args": ["5432"] }
}

The pin never moves on its own. Publishing a newer version does not change what an existing check runs. To move a check to the new version, update it with a PUT, giving the new script_version.

With Terraform

The dtm_script resource takes the script and a signature made outside Terraform; Terraform never holds the key. A dtm_healthcheck with protocol = "script" takes a script_config block naming the script, and cannot send a free-form command.

resource "dtm_script" "orders_db" {
  name        = "orders-db"
  interpreter = "sh" # sh or python3
  body        = file("${path.module}/scripts/orders-db.sh")
  key_id      = "ops-2026"
  signature   = var.orders_db_signature # from the openssl signing commands
}

resource "dtm_healthcheck" "orders_db" {
  name             = "orders-db"
  protocol         = "script"
  target           = "10.0.0.5:5432"
  interval_seconds = 30
  timeout_seconds  = 5

  script_config {
    script_id = dtm_script.orders_db.id
    args      = ["5432"]
    # To move the check along when the body changes, also set:
    # script_version = dtm_script.orders_db.version
    # script_hash    = dtm_script.orders_db.content_hash
  }
}

A changed body publishes the script’s next version, and the plan shows the content_hash your pipeline should sign. See Health checks and load balancing for the details.

In a record or override template

Added in v1.0.7. A record’s or override’s health_check_template can run a managed script against each of its values. Give it "protocol": "script" and a script block; DTM creates one script check per value, with the value as its target:

"health_check_template": {
  "protocol": "script",
  "port": 5432,
  "interval_seconds": 30,
  "timeout_seconds": 5,
  "script": {"script_id": "<id>", "script_version": 0, "args": ["--max-lag", "30"]}
}

• Pinned like a check. script_version 0 pins the latest version when you save, and the response shows the version and hash it pinned. The args are masked as *** in responses and the audit log; sending *** back keeps the stored argument while the template keeps the same script. With a different script, a masked argument is refused with 400: send it in clear.
• Admin only, on a ready cluster. Attaching or changing one needs the admin role (403 otherwise), and allow_script_health_checks on the node you save on (403). It is refused with 409 until every node in the cluster runs v1.0.7 or later, and with 400 on a record with auto_populate, on an ALIAS record and on a linked override, whose values you do not choose.
• The record stays guarded. Once a record or override carries one, changing its values, template, auto_populate, type or name is admin only, and a dynamic DNS update cannot add a value to it (removing one is allowed). Deleting the record is not admin only.
• A value whose check is not running is left out of answers. Where the check has not run yet, or cannot run on a node (see When a check does not run), that node leaves the value out of its answers rather than serving it unchecked. Right after a save this can last up to one replication interval on some nodes; the save response says so. GET /api/v1/diagnose lists each held value and why. The one exception is the all-down rule: a node where no value passes answers with the values whose checks have not failed (all of them if every check failed), rather than nothing.
• protocol: script needs the script block. Without one, the template is refused with 400. The JSON zone import refuses a script template; add the health check after the import.

Upgrading to v1.0.7, and rolling back

Before upgrading, find each template with "protocol": "script", and re-save it with a script block, or change its protocol. A template that has no script block has its checks held after the upgrade, so its values leave the answers.

Before rolling back below v1.0.7, remove the script templates (or the records and overrides that carry them) first.

5. Stop a script: revoke it, then restart

Deleting a script does not revoke it. A signed script stays signed for as long as its key is trusted, so a deleted version can be sent to the cluster again and installed again. To stop a script from ever running again:

1. Add its content_hash (shown on every version, and in every delete response) to script_revoked_hashes on every node, or in the Terraform vm module’s script_revoked_hashes variable.
2. Restart each node. The list is read at startup. The next scheduled run after a node restarts is refused, and the check reports script-revoked.

To retire a key, for example one you think is compromised, remove it from script_signing_keys and restart each node: every script that key signed then stops running (script-orphaned). Sign the scripts you still want with a new key first.

Deleting a script or version that a check uses is refused with 409, listing the checks. ?force=true deletes it anyway; those checks then refuse to run (script-pending) until you point them at another version.

When a check does not run

A managed check that cannot run is refused, not failed: the node keeps serving the backend, so the check protects nothing on that node until it runs again. dtm_health_probe_refusals_total{guard} counts why, and the DTMScriptCheckNotRunning alert fires after 15 minutes of refusals. GET /api/v1/healthchecks/{id}/status on that node gives the exact reason in its error field.

Reason (guard)	What it means	What to do
script-pending	The pinned version is not on this node: it has not arrived yet, or it was deleted	If it persists, check the node is in the cluster, or point the check at a version that exists
script-pin-mismatch	The version here is not the script the check pinned	Point the check at the version you meant
script-corrupt	The stored script no longer matches its hash	Publish the script again
script-revoked	Its hash is in this node’s script_revoked_hashes	Expected after a revocation: point the check at an approved version
script-orphaned	No key in this node’s script_signing_keys signed it	Add the key on this node and restart, or re-sign with a trusted key
script-signature	The signature does not verify	Re-sign the script and publish it again
script-gate-off	allow_script_health_checks is off on this node	Turn it on for every node that should run the check
script-invalid	The check’s script reference is incomplete	Fix the check’s script_config
script-unreadable	The stored version could not be read	Publish the script again

Limits

Across the cluster: 64 scripts, 16 versions per script, 1,024 versions, and 16 MiB of script text. A create over a limit is refused with 409, naming the limit.