Respond to a suspected credential exposure

A credential exposure is the day-2 event with the tightest clock and the easiest mistake: reaching for a password reset and believing access is dead. It is not; the instant kill is something else. This tutorial walks one incident in the order that works, ending at the decision every response plan should pre-make: when rotation suffices, and when the event becomes a rebuild.

Scenario

An admin at Nordvale Media (fictional). A contractor’s laptop is missing. On it: a logged-in browser session for the contractor’s admin-role DTM account, the CI service-account token in a local env file, and a repo checkout containing a TSIG key. The cluster itself is believed intact. Budget about an hour of focused work. API steps send an admin bearer token as $DTM_TOKEN (how to capture one), issued to your account, not the exposed one.

1. Capture evidence before anything rotates

First, the audit trail, while it still tells the story cleanly. Pull what the exposed identities did, and export the window to immutable storage:

curl -sk "https://10.0.250.4:8443/api/v1/audit?user_name=contractor.a&from=2026-07-25" \
  -H "Authorization: Bearer $DTM_TOKEN"
curl -sk "https://10.0.250.4:8443/api/v1/audit/export?from=2026-07-25" \
  -H "Authorization: Bearer $DTM_TOKEN" -H "X-DTM-CSRF: 1" -o incident-audit.ndjson

Anything the laptop’s credentials did after it went missing is scoped here; anything they do after this point should be nothing, which step 8 verifies.

2. Contain at the network edge

While you work through credentials, confirm the NSG story: 8443 and 8080 restricted to the admin prefixes, tightened further for the incident if warranted (see operator responsibilities). Network containment buys calm for everything below.

3. Kill the user’s access, in the order that actually works

Danger

A password reset does not end live access: it revokes the account’s refresh tokens, but an outstanding access token stays valid for the rest of its 24-hour life. The instant kill is disabling the account: enabled status is re-checked on every request, so a disabled user’s token dies with its next call.

So, for the contractor’s account, in this order:

# 1. Disable: live sessions and tokens stop working on their next request
curl -sk -X PUT https://10.0.250.4:8443/api/v1/users/<contractor-id> \
  -H "Authorization: Bearer $DTM_TOKEN" -H "X-DTM-CSRF: 1" \
  -H "Content-Type: application/json" -d '{"enabled": false}'

# 2. Then reset the password (revokes refresh tokens), before any re-enable

If the account should keep existing, re-enable it only after the password change, with the new password delivered out of band. Repeat for any other account the laptop could have held a session for. (If the exposed account were your only admin, the offline break-glass reset is the recovery path; check now, calmly, that your team knows it.)

4. Rotate the CI service-account token

The version-gated rotate mints a fresh secret and invalidates the old token immediately; it is idempotent under retry, so a nervous double-call cannot double-rotate:

curl -sk -X POST https://10.0.250.4:8443/api/v1/service-accounts/<id>/rotate \
  -H "Authorization: Bearer $DTM_TOKEN" -H "X-DTM-CSRF: 1" \
  -H "Content-Type: application/json" -d '{"version": 2}'

Write the new token to its Key Vault sink and confirm the pipeline picks it up. If the account is Terraform-managed, do the rotation through Terraform’s version bump instead, never this endpoint directly (the manage-as-code tutorial explains the desync it avoids). If the account should not survive the incident, revoke it instead (POST /api/v1/service-accounts/<id>/revoke).

5. Rotate the TSIG key, as a staged cutover

Dynamic-update clients hold the key too, so this is a cutover, not a flip: create a new key bound to the same zones, move the DHCP/ExternalDNS clients onto it, verify updates still land (nsupdate plus a dig, per dynamic DNS), then delete the exposed key. Deleting first takes registration down estate-wide; order is the whole game here.

6. Rotate the Entra client secret, if it was in scope

If the repo or laptop could have held SSO app credentials, rotate the client secret at the app registration and update DTM’s config, per Entra ID SSO; verify a sign-in afterward.

7. Face the cluster-key question

The cluster encryption key is not rotatable in place, so make this decision by rule, not under adrenaline:

• Tokens, passwords, TSIG, or the Entra secret leaked (this scenario): the rotations above suffice. The key never leaves Key Vault and the nodes; a laptop holding sessions and tokens never had it.
• The cluster key itself may be exposed (Key Vault access compromised, a node’s filesystem read), and snapshots may be too: treat snapshots as decryptable offline, keep 7946 locked node-to-node, and plan the rebuild: a fresh cluster born with its own key, data moved by zone bundle export/import, with support in the loop. The rotating secrets page carries the full exposure checklist.

8. Close out with evidence

Re-run the step-1 audit query and confirm zero activity from the old credentials after your rotations; that line, plus the exported NDJSON and your timeline, is the incident record. Two follow-ups while it is fresh: check the audit for anything the exposed identities changed that needs reverting, and calendar a drill of this runbook so the next reader has done it before they need it.

Where next

• Rotating secrets: the credential-by-credential rotation index this incident walked.
• Run a quarterly maintenance pass: where the access review that catches stale accounts lives.
• Users and access: the role model and account baseline.

Need a hand? Email [email protected].

Last validated: 2026-07-27