Backup and restore

DTM keeps all of its persistent state (zones, records, users, audit log, TSIG keys, forwarders, blocklist, and more) in a single database file. Two admin endpoints let you snapshot and restore it. The examples send an admin bearer token as $DTM_TOKEN; the API overview shows how to capture one.

Scheduled backups (built-in, recommended)

DTM can schedule its own snapshots: no cron, no token file, and, crucially, an alertable failure signal (a quietly dead cron job is invisible until the day you need a restore). Enable it in the server configuration on each node:

backup:
  schedule_enabled: true
  interval_minutes: 360 # 0 = default 1440 (daily); compliance setups often use 15
  retention_count: 7 # local snapshots kept, oldest pruned first
  dir: /var/lib/dtm/backups # default: a backups directory beside the data store
  # Optional off-node copy, uploaded with the VM's managed identity:
  blob_container_url: https://<account>.blob.core.windows.net/dtm-backups

How it behaves:

• Same protection as the API path: scheduled snapshots carry the same signed (HMAC-authenticated) envelope, keyed to the cluster.
• Local-first: a snapshot is durably on local disk before any upload is attempted, so a broken upload degrades to local-only plus a failure metric, never to no backup at all.
• Off-node copy via managed identity only: grant the node’s identity Storage Blob Data Contributor on the container. No SAS tokens or account keys; a container URL carrying a query string is rejected at startup. A role granted after boot heals without a restart.
• Every node schedules independently (a snapshot is node-local state). Point all nodes at one container: blob names are prefixed with the node ID, so they never collide.
• Retention: retention_count prunes files matching the scheduler’s naming pattern (dtm-backup-*.dtmb) in dir, oldest first. It keeps no record of which of them it wrote, so a snapshot you download and save into the same directory under its default name is a prune candidate too: keep copies you care about outside dir, or rename them so they no longer match the pattern. Remote retention is deliberately DTM’s job not to do: put an Azure Storage lifecycle-management rule on the container.

Alert on it (the reason the built-in scheduler exists):

# Backups have stopped: no success for more than 2 intervals. The gauge
# starts at 0, so "never succeeded since boot" fires too.
time() - dtm_backup_last_success_timestamp_seconds > 2 * 6 * 3600

# Off-node copy broken while local snapshots still land:
rate(dtm_backup_scheduled_failures_total{stage="upload"}[30m]) > 0

The full key list, defaults, and DTM_BACKUP_* environment overrides are in the configuration reference.

Take a backup on demand (external trigger)

GET /api/v1/backup streams a consistent point-in-time snapshot of the database; it is the same snapshot the scheduler takes, and it remains the right tool for pre-change snapshots and for teams that prefer their own trigger (cron, a systemd timer, an Azure Function). Requires an admin token and the CSRF header.

curl -sk https://dtm.internal:8443/api/v1/backup \
  -H "Authorization: Bearer $DTM_TOKEN" \
  -H "X-DTM-CSRF: 1" \
  -o "dtm-$(date +%Y%m%d-%H%M%S).dtmb"

Snapshots use the .dtmb extension (the name DTM’s own UI download and scheduler use); restore accepts older files whatever they are called.

Recommended practice, whichever trigger you use:

• Cadence: hourly for a single node; every few hours for a multi-node cluster (peers cover the gap, so the backup is really for total loss); more often where compliance requires it.
• Store off the node: ship snapshots to an Azure Storage account (with blob versioning and soft-delete enabled) rather than leaving them on the node’s disk.
• Protect them: a snapshot contains sensitive data, so store it encrypted at rest with restricted, logged access, or set a passphrase so the file itself is encrypted.
• Save a checksum alongside each snapshot (external-trigger flows; the endpoint also returns the digest in an X-DTM-Backup-SHA256 response trailer).

What a snapshot contains

Included: zones, records, SOA and RRsets, users and password hashes, the audit log, TSIG keys, the blocklist, forwarders, and discovery scopes.

Not included (re-provisioned per environment): TLS certificates and keys on disk, the Key Vault-held first-boot secrets (the cluster encryption key and initial admin password), OS and image state, and network configuration. The node’s JWT signing key is stored in the database and travels with the snapshot, protected by encryption at rest keyed off the cluster encryption key. A node restored without its cluster credentials boots but cannot participate in the cluster until those are re-issued.

Because a snapshot carries those credentials, any copy you have downloaded outlives the cluster it came from. Deleting the DTM resources does nothing to a file sitting on a workstation, a file share, or a storage account. Decommissioning a cluster covers tracking those copies down when you retire a deployment.

Encrypt backups with a passphrase (optional)

By default a snapshot is signed (HMAC-authenticated, keyed to the cluster) but not encrypted, which is why the guidance above says to treat the file as sensitive wherever it is stored. Setting a passphrase encrypts the snapshot payload itself with AES-256-GCM (the key is derived from the passphrase with Argon2id), so the file is unreadable without the passphrase no matter where it ends up.

• On-demand backups: send the passphrase in the X-DTM-Backup-Passphrase request header on the download (never in the URL); the backup card on the Settings page has a matching passphrase field:

  curl -sk https://dtm.internal:8443/api/v1/backup \
    -H "Authorization: Bearer $DTM_TOKEN" \
    -H "X-DTM-CSRF: 1" \
    -H "X-DTM-Backup-Passphrase: <passphrase>" \
    -o "dtm-$(date +%Y%m%d-%H%M%S).dtmb"

• Scheduled backups: set backup.passphrase in the server configuration, or the DTM_BACKUP_PASSPHRASE environment variable (preferred, so the value stays out of the YAML), and every scheduled snapshot is encrypted before it reaches local disk or blob storage.

• Restoring: supply the same passphrase when staging the snapshot (step 1 below). An encrypted snapshot staged without its passphrase, or with the wrong one, is rejected with a clear error before the live database is touched.

Danger

There is no passphrase recovery. The passphrase is not stored or escrowed anywhere: lose it and the encrypted snapshot is permanently unreadable. Keep it in your secret manager alongside (not inside) your backups.

Restore a single node

Restore is staged and applied atomically on the next start, so the live database is never touched mid-flight.

Note

Backups and restore staging are pure API operations. The restart that promotes a staged snapshot only needs an Azure VM restart; the file-level steps (rolling back, clearing a peer’s database) are break-glass SSH tasks using the deployment’s keys-only admin login. Routine operation never needs a shell; see security.

1. Upload the snapshot (the service stays running; this only stages the file, the live database is not changed yet). The Settings page’s restore form drives this same staged flow, including a passphrase field for encrypted snapshots; over the API:

   curl -sk -X POST https://dtm.internal:8443/api/v1/restore \
     -H "Authorization: Bearer $DTM_TOKEN" \
     -H "X-DTM-CSRF: 1" \
     --data-binary @dtm-20260101-000000.dtmb

   If the snapshot is passphrase-encrypted, add -H "X-DTM-Backup-Passphrase: <passphrase>": a wrong or missing passphrase rejects the upload and nothing is staged.

2. Restart the node: restart the VM from the Azure portal or CLI, or restart just the service (systemctl restart dtm-server) over break-glass SSH. On boot DTM sets the current database aside (as a rollback copy) and promotes the snapshot.

3. Verify: GET /api/v1/zones returns the expected data.

4. Once satisfied, remove the rollback copy (a break-glass SSH task). To roll back instead, stop the service, move the rollback copy back, and start.

Restore offline (break-glass)

The staged API restore needs a running, reachable server. When the node is dead or every admin is locked out, restore over the deployment’s break-glass SSH instead, with the service stopped:

sudo systemctl stop dtm-server
sudo -u dtm /usr/local/bin/dtm-server restore --local \
  --config /etc/dtm/dtm-server.yaml \
  /var/backups/dtm/dtm-backup-20260501T100000Z.dtmb
sudo systemctl start dtm-server

Because the service is stopped, the subcommand stages and applies in one shot: it validates the snapshot envelope against the cluster key from the configuration, swaps the payload in atomically, keeps the previous database as dtm.db.pre-restore for rollback, and prints the restored zone and record counts. For a passphrase-encrypted snapshot, add --passphrase-file <path>: reading the passphrase from a file keeps it out of the process list.

Do not mix the two paths: a stopped server cannot accept the API upload, and a running one should be restored through the staged API flow above, never the offline CLI.

Caution

The subcommand’s bypass flags (--accept-cross-cluster for a deliberate cross-cluster migration, --key to override the configured cluster key, --insecure-backup for unkeyed clusters) skip only the signature check, never the envelope validation. A routine restore never needs them; reach for them only when you understand exactly why the signature will not verify.

Restore a whole cluster

1. Stop every node except the one you will restore on (pick the most recently backed up). Stopping the VMs from the Azure portal or CLI is enough; the service drains gracefully.
2. Restore on that node using the single-node procedure above (stage the snapshot, then restart), and wait for /readyz to return 200.
3. On every other node, remove the database file so it re-syncs cleanly (a break-glass SSH task).
4. Start the remaining nodes one at a time; each joins and syncs a full copy of state from the restored node before becoming ready.
5. Verify: GET /api/v1/nodes shows every node alive.

Caution

Prefer a recent snapshot for a cluster-wide restore. Restoring a very old snapshot across the whole cluster can bring back entities that were deleted after the snapshot was taken.

Reclaim disk with compaction

The database file grows over time; space freed by deletes and log pruning is reused internally but not returned to the filesystem, so the file can grow several times larger than the live data. Watch dtm_storage_bbolt_free_pages (alert when free pages exceed 50% of total, sustained for 6 hours: the shipped DTMBboltFreelistBloat rule’s hold; a longer hold is fine if you tune your own rule).

Compact per node, one at a time:

curl -sk -X POST https://dtm.internal:8443/api/v1/admin/compact \
  -H "Authorization: Bearer $DTM_TOKEN" \
  -H "X-DTM-CSRF: 1"
az vm restart -g <resource-group> -n <vm-name>   # or: systemctl restart dtm-server

The compact stages a smaller copy (no downtime during staging); the restart swaps it in. The file typically shrinks 30-60%. Across a cluster, do this one node at a time, waiting for /readyz = 200 between nodes. Do not run compaction and restore at the same time. A quarterly cadence (or alert-driven) suits most deployments.

Related

• Disaster recovery for recovering from node or cluster failure.
• Rolling upgrades for zero-downtime image upgrades.
• Decommissioning a cluster for retiring a deployment, including what deleting the VMs does not remove.