Run a quarterly maintenance pass

A DTM cluster runs quietly for months, which is exactly how preventive work gets skipped: certificates age toward their two-year cliff with no built-in warning, reclaimable dead space accumulates in each node’s local data store, backups go unverified, and leavers keep accounts. This is the quarterly run that catches all of it, with pass criteria per check and a deliberate order.

Scenario

An admin at Meridex Logistics (fictional) operating a three-node, twelve-zone cluster (10.0.250.4 to .6). Comfortable with curl and az, holds an admin bearer token as $DTM_TOKEN (how to capture one), has never done structured maintenance on DTM. Budget about 90 minutes; the only service impact is the optional compaction restarts at the end, one node at a time.

Two ground rules for the whole run: anything that restarts a node happens one node at a time, gated on /readyz; and every finding is either stop the run (the cluster is not healthy enough to maintain; fix first) or spawn a follow-up (record it, keep going). Each check below says which.

1. Cluster health review

curl -sk https://10.0.250.4:8443/api/v1/nodes \
  -H "Authorization: Bearer $DTM_TOKEN"

Pass: every node alive, and every node reporting the same software version. The version field is the DTM software version, and a mismatch is a real finding: a half-finished upgrade. Then confirm zone data agrees everywhere:

for n in 10.0.250.4 10.0.250.5 10.0.250.6; do
  dig SOA example.internal @$n +short
done
# identical serials on every node

Finally, replication lag from Prometheus: time() - dtm_replication_last_sync_timestamp_seconds under 120.

Any disagreement here is stop the run: troubleshooting first, maintenance after.

2. Alert spot check

You cannot test-fire alerts from DTM, so verify the rules are still loaded and enabled instead.

On the Prometheus path, confirm the storage alert (DTMBboltFreelistBloat, which holds for 6 hours) and the lag alert (DTMReplicationLagHigh) are loaded: GET /api/v1/rules, or promtool check rules against the shipped dtm-alerts.yml.

On the Azure Monitor path those two rule names do not exist. Confirm instead that the six KQL alerts from the observability tutorial are enabled (az monitor scheduled-query list -o table), and, if you deployed the shipped metric alerts (they are created only when you set an alert email), that they are enabled too (az monitor metrics alert list -o table).

Either way, confirm that the SLO targets your pager guards are still the published ones: 99.95 percent answer success, p99 under 10 ms, convergence under 60 seconds (SLA and SLO).

Pass: the rules for your path are present and enabled. Anything missing: follow-up via the observability tutorial.

3. Certificate expiry audit

On each node (the certificate directory is group-restricted, so the local read needs sudo), or over the wire from anywhere that reaches the UI:

sudo openssl x509 -in /etc/dtm/tls/cert.pem -noout -enddate

# or, over the wire:
openssl s_client -connect <node-ip>:8080 </dev/null 2>/dev/null \
  | openssl x509 -noout -enddate

Caution

Two surfaces, two clocks: this file is the web UI certificate (730-day first-boot validity, no auto-renewal, no built-in expiry warning). The API, DoT, and DoH listener self-signs at every service start unless listen.tls points at real files, and a renewed file is only picked up by a restart, one node at a time. The built-in Diagnose warning covers only a configured API certificate, at 30 days, and never the UI certificate, so this manual check is the UI cert’s only guard.

Pass: more than 90 days remaining, everywhere. Under 90 days is this page’s editorial trigger: schedule the full rotation campaign now, using the procedure matching your deployment dialect (Terraform: one cluster-wide Key Vault bundle; Marketplace: per node). Record which dialect the cluster uses in your run notes.

4. Backup verification

Confirm backups are landing on cadence (hourly for a single node, 6-hourly for a cluster, 15 minutes under compliance regimes), whichever trigger you run: the built-in scheduler or an external cron flow.

az storage blob list --account-name <account> --container-name dtm-backups --auth-mode login -o table
# built-in scheduler: also check the pipeline's own pulse
# time() - dtm_backup_last_success_timestamp_seconds  < 2x your interval

Details that matter: with the built-in scheduler, alert on dtm_backup_last_success_timestamp_seconds going stale and on rate(dtm_backup_scheduled_failures_total{stage="upload"}[30m]) > 0 (local snapshots can keep landing while the off-node copy silently breaks). On the external flow, verify the newest snapshot against its own checksum sidecar, never snapshot-to-snapshot (identical state still hashes differently). And confirm blob versioning and soft delete are still enabled on the container.

Pass: fresh snapshot, matching sidecar, container protections on.

Tip

Verification proves the pipeline; only a restore proves the backup. The full rehearsal is its own session: run a restore drill.

5. Access review, then audit export

People first: list users and service accounts, disable leavers (PUT /api/v1/users/{id} with {"enabled": false}; disable, not delete, so history keeps its subject), revoke unused service accounts (revocation is soft; they stay listable for audit), and confirm the break-glass local admin still signs in while SSO is primary (users, Entra SSO).

Then export the quarter’s audit log to immutable storage now: retention pruning runs continuously, and the export is the durable record:

curl -sk "https://10.0.250.4:8443/api/v1/audit/export" \
  -H "Authorization: Bearer $DTM_TOKEN" -H "X-DTM-CSRF: 1" -o audit-2026Q3.ndjson

(Audit retention is bounded; the export is the durable record.)

Pass: no unexplained accounts, break-glass verified, export archived.

6. Storage health and compaction

Compute the free-space ratio from Prometheus:

dtm_storage_bbolt_free_pages
  / (dtm_storage_bbolt_free_pages + dtm_storage_bbolt_pages_in_use)

Pass: ratio at or below 0.5. Above it, the node’s local data store is carrying reclaimable dead space: stage a compaction with POST /api/v1/admin/compact (admin role, X-DTM-CSRF header), restart to apply, one node at a time gated on /readyz, then confirm free pages dropped to near zero and delete the rollback copy. The full procedure is on backup and restore.

Danger

Never stage a compaction and a restore in the same window: they share one staging path, and whichever file was staged last before the restart wins.

7. Close out the run

Record the findings (one line per check: pass, or follow-up filed), calendar the next run, and keep the escalation line short: symptoms you could not explain during the run go to [email protected] with a support bundle attached.

Check	Pass criteria	On failure
Cluster health	all alive, one version, serials match	stop the run
Alert rules	the rules for your path loaded and enabled	follow-up
Certificates	over 90 days remaining, both surfaces known	schedule rotation
Backups	fresh snapshot verifies against its sidecar	follow-up now
Access and audit	no strays, break-glass works, quarter exported	follow-up
Storage	free-space ratio at or below 0.5	compact

Where next

• Run a restore drill: the rehearsal this run keeps promising.
• Zero-downtime upgrades: if step 1 found a version mismatch, finish the upgrade properly.
• Rotating secrets: the credential half of periodic hygiene.

Need a hand? Email [email protected].

Last validated: 2026-07-26