Roll out encrypted DNS

Enabling DoT or DoH is one API call; doing it before the listener has a CA-issued certificate hands your resolvers each node’s self-signed certificate, which they cannot verify. This tutorial does the rollout in the order that works: stable CA-issued certificate first, encrypted transports second, clients third, and then a sweep to retire every -sk and insecure_skip_verify the self-signed era forced on you.

Scenario

An admin at a company with a working three-node cluster (10.0.250.4 to .6, names dtm1 to dtm3.example.internal). Compliance requires DoT for the Linux estate’s resolvers and DoH for a VDI fleet, both validating against the corporate CA. You can issue certificates from that CA (directly or via Key Vault). Budget about 60 minutes, including one rolling restart. API steps send an admin bearer token as $DTM_TOKEN (how to capture one).

1. Know your two TLS surfaces

DTM terminates TLS in two places with different lifecycles: the web UI (8080) serves a managed two-year certificate, while the API listener (8443), DoT (853), and the dedicated DoH listener (443) share listen.tls material: on a deployed node, a self-signed certificate each node writes at first boot (kept across restarts, valid for two years, and renewed under a new key in its last 30 days) until you install your own. This tutorial touches only the second surface; the full inventory lives in TLS certificates, and which clients refuse a self-signed certificate (and which accept one without verifying it) is in client trust.

This tutorial installs the certificate as files on each node. You can instead upload it once for the whole cluster, with no restart, as TLS certificates describes; from step 5 on, the tutorial is the same either way.

2. Issue and place the certificate

Issue one server certificate from the corporate CA with subject alternative names covering every node name and stable IP (dtm1.example.internal, 10.0.250.4, and so on). If the CA publishes through Key Vault, download it yourself onto each node; this is a customer-managed step, DTM has no built-in Key Vault fetch for the API listener certificate (the Key Vault bundle flow you may have read about applies to the web UI certificate only):

az keyvault secret download --vault-name kv-corp-ca \
  --name dtm-api-tls --file ~/bundle.pem
# split ~/bundle.pem into ~/cert.pem and ~/key.pem, then install them
# for the dtm service user:
sudo install -d -o root -g dtm -m 0750 /etc/dtm/api-tls
sudo install -o dtm -g dtm -m 0644 ~/cert.pem /etc/dtm/api-tls/cert.pem
sudo install -o dtm -g dtm -m 0600 ~/key.pem /etc/dtm/api-tls/key.pem
shred -u ~/bundle.pem ~/key.pem && rm ~/cert.pem

dtm-server runs as the unprivileged dtm user, so the key must be owned by dtm and readable by its owner only (0600). The server refuses to start on a group- or world-readable key, and on a missing or unreadable file it stops with a clear error rather than silently self-signing: a key left owned by root means the node does not come back from the restart in step 3.

Caution

Files placed by hand live on the node’s OS disk, and both rolling upgrade paths replace the VM’s OS disk. After upgrading a node, place the files and the listen.tls setting from step 3 on it again.

3. Configure and roll it out

On each node, point /etc/dtm/dtm-server.yaml at the files (both keys set together; see the configuration reference):

listen:
  tls:
    cert_file: /etc/dtm/api-tls/cert.pem
    key_file: /etc/dtm/api-tls/key.pem

The deployed file already has a tls block with managed: true and paths under /var/lib/dtm. Replace the two paths and remove managed: true: managed mode hands those paths to the cluster, which tries to write any certificate uploaded through the API over your files. Under /etc/dtm that write fails, so the node keeps serving your files while reporting a failed apply. If you would rather not place files on every node, upload the certificate once instead, as TLS certificates describes.

Caution

There is no configuration reload: the file is read at startup, so this is a restart, and a restarting node stops answering DNS. Roll one node at a time, gating each restart on curl -sk https://<node>:8443/readyz returning 200 before moving on, the same discipline as a rolling upgrade.

4. Verify the certificate is yours, and stable

Compare the served certificate’s serial against the on-disk file, on every node, without any extra restarts (each one costs you a node’s DNS):

openssl s_client -connect 10.0.250.4:8443 </dev/null 2>/dev/null \
  | openssl x509 -noout -serial
sudo openssl x509 -in /etc/dtm/api-tls/cert.pem -noout -serial
# serials match on every node

The rolling restarts you just performed are themselves the stability proof: the serial survived them. If you want a belt-and-braces demonstration, restart one node again and confirm its serial is unchanged.

5. Enable DoT and DoH cluster-wide

Now the payoff of step 2: talk to the API with --cacert, not -sk, and flip both transports with the live runtime toggle:

curl --cacert corp-ca.pem -X PUT https://dtm1.example.internal:8443/api/v1/transport-config \
  -H "Authorization: Bearer $DTM_TOKEN" \
  -H "X-DTM-CSRF: 1" \
  -H "Content-Type: application/json" \
  -d '{"dot_enabled": true, "doh_enabled": true}'

The change applies live and replicates cluster-wide, no restarts; a misspelled field is a 400, never a silent no-op. Toggle mechanics, including why the runtime value wins over the YAML at the next boot, live on DNS over TLS and HTTPS.

Caution

/dns-query is unauthenticated by design on its dedicated listener (443), and DoT opens 853. Before clients arrive, scope both ports in the NSG to exactly the resolver and VDI subnets that should reach them; the dot-doh page spells out the decision.

6. Verify both transports against the corporate CA

# DoT, validating the chain and the SAN
kdig @10.0.250.4 app.example.internal +tls \
  +tls-ca=corp-ca.pem +tls-hostname=dtm1.example.internal

# DoH
kdig @10.0.250.4 app.example.internal +https=/dns-query \
  +tls-ca=corp-ca.pem +tls-hostname=dtm1.example.internal

Both should match plain dig answers exactly. If 853 does not answer on one node, check that node’s logs: a node that cannot load its TLS material leaves DoT down and logs why, rather than serving a broken listener.

7. Point real clients at it

The client half exists on no other page. For a Linux host or estate resolver running systemd-resolved, in /etc/systemd/resolved.conf:

[Resolve]
DNS=10.0.250.4#dtm1.example.internal 10.0.250.5#dtm2.example.internal
DNSOverTLS=yes

The #hostname suffix is what systemd-resolved validates against the certificate’s SAN, so it must match a name you issued in step 2, and the corporate CA must be in the client’s trust store. Restart systemd-resolved and verify end to end:

resolvectl query app.example.internal   # answered, with DNSOverTLS=yes active

Point the VDI fleet’s browser or OS DoH settings at https://dtm1.example.internal/dns-query (the dedicated DoH listener on 443, so no port suffix is needed) per your platform’s policy tooling.

8. De-insecure the estate

The self-signed era left -sk and skip verify scattered through your tooling; retire them now, while the change is fresh:

• Scripts and automation: replace curl -sk with curl --cacert corp-ca.pem.
• Prometheus: set the CA in the scrape job’s tls_config and flip insecure_skip_verify to false (the end state the observability page shows).
• Terraform: the provider can now verify TLS too; drop insecure_skip_verify from the provider block if you had set it.

Verify: scrapes stay green and DoT clients keep resolving after the sweep. Anything that breaks was silently trusting the old certificate.

9. Plan the renewal before you forget

• Calendar the CA certificate’s expiry with margin. Renewal is replacing the files at the same paths (same ownership and mode) and repeating step 4’s serial check on each node: every listener serves the new pair from its next full handshake, so no restart is needed. If you uploaded the certificate, upload the renewed one instead. After a key compromise, still do a rolling restart, because clients resuming an earlier TLS session keep the old certificate.
• The web UI certificate is a separate surface with its own two-year clock and rotation procedure; do not assume this rollout renewed it.
• Update any monitor that pins the API certificate so renewal does not page you as an incident.

Where next

• Wire observability end to end: if your scrape still says insecure_skip_verify: true, close that loop now.
• Production readiness pass: the wider hardening sequence this slots into.
• TLS certificates: the full two-surface reference, including bringing the web UI onto your CA.

Need a hand? Email [email protected].

Last validated: 2026-07-26