Rotate every certificate before the two-year mark

Twenty months in, the certificate clock is the quiet risk: the web UI’s first-boot certificate expires at two years with no auto-renewal, the API listener has been re-self-signing on every restart since day one, and by now a small crowd of scripts, scrapers, and resolvers has opinions about which certificates they trust. This tutorial is the campaign: inventory both surfaces and their consumers, rotate each surface once, and leave alerts behind so the next expiry is a calendar entry, not an outage.

Scenario

An admin at Halvern Health (fictional), three-node cluster, 20 months old. The 730-day web UI certificate expires in under four months, security wants the API listener on corporate-CA certificates, and monitoring keeps breaking on self-signed churn. Budget a half day, including one rolling restart. API steps send an admin bearer token as $DTM_TOKEN (how to capture one).

If all you need is the API listener on a corporate CA ahead of a DoT/DoH rollout, the encrypted DNS tutorial is that slice on its own; this page is the both-surfaces campaign.

1. Inventory what you actually have

Two surfaces, two lifecycles (the full table). Measure both on every node:

# Web UI (8080): the first-boot 730-day certificate
openssl s_client -connect 10.0.250.4:8080 </dev/null 2>/dev/null \
  | openssl x509 -noout -enddate -issuer

# API listener (8443; DoT 853 and DoH 443 serve the same certificate):
# self-signed per start unless listen.tls is configured
openssl s_client -connect 10.0.250.4:8443 </dev/null 2>/dev/null \
  | openssl x509 -noout -enddate -issuer

Write the table down: surface, per node, issuer, expiry. On a 20-month-old default deployment you will find a UI certificate a few months from death and an API issuer that changes on every restart; both are what this campaign fixes.

2. Enumerate the consumers

Every consumer that pins or validates these certificates needs the new CA chain, and finding them now is what stops the rotation from becoming its own incident:

• Prometheus scrape tls_config (or its insecure_skip_verify: true standing in for one; see observability).
• curl -sk and --cacert flags in backup jobs, automation, and runbooks.
• DoT and DoH clients, if enabled: estate resolvers and the VDI fleet validate the API surface’s chain.
• Browsers and anything else hitting the UI on 8080.
• The Terraform provider’s TLS settings.

3. Issue and stage the certificates

From the corporate CA, issue for the names and IPs consumers actually use (SANs per surface: node names and stable IPs for the API/DoT surface; the UI’s reachable names for 8080), and stage through Key Vault per your CA workflow. Two file rules that bite later, stated now: the API listener refuses to start on a group- or world-readable private key, and a missing file stops startup with a clear error rather than silently self-signing.

4. Rotate the API/DoT/DoH surface first

This surface is first because its consumers are machines you can fix immediately, and because the rolling restart it needs doubles as the stability proof. The procedure is the stable certificate step: place the files, set listen.tls.cert_file / key_file, restart one node at a time gated on /readyz returning 200 (no reload exists; a restarting node stops answering DNS, so treat it as a rolling-upgrade step).

Verify per node as you go, using the restarts the procedure already performs (no extra ones): after each node’s restart, compare the served serial with the on-disk file, and confirm the serial is the same one the previous node served:

openssl s_client -connect <node>:8443 </dev/null 2>/dev/null \
  | openssl x509 -noout -serial
kdig @<node> app.example.internal +tls +tls-ca=corp-ca.pem   # if DoT is on

Caution

Do not set listen.tls.ca_file while you are in there. It is not a chain-trust option: it turns on mutual TLS and every client of 8443, including each node’s own UI service, must then present a certificate. Leave it unset unless mTLS is a deliberate project.

5. Rotate the web UI certificate

Per your deployment dialect, using the rotation procedure: Terraform-managed clusters rotate one cluster-wide Key Vault bundle and re-run the bootstrap per node; Marketplace clusters rotate per node. Then verify the new expiry everywhere:

for n in 10.0.250.4 10.0.250.5 10.0.250.6; do
  openssl s_client -connect $n:8080 </dev/null 2>/dev/null \
    | openssl x509 -noout -enddate
done

Only the UI service restarts here; DNS never blinks.

6. De-insecure the consumer list

Walk the step-2 inventory and retire every workaround the self-signed era forced: --cacert instead of -sk, the CA in Prometheus tls_config with insecure_skip_verify: false, provider TLS verification on. The encrypted DNS tutorial walks this sweep; the verification is that every consumer stays green because it now validates, not despite it.

7. Make year four boring

The product will not remind you about the UI certificate: the built-in Diagnose warning covers only a configured API certificate, at 30 days, and never the UI surface; and 30 days is too late to start corporate-CA issuance anyway. So the guard is yours:

• Record both new expiry dates where your team plans work.
• Alert at 90 days out (a calendar entry, or a probe-based cert-expiry alert in your monitoring stack against both ports).
• The quarterly maintenance pass carries the recurring expiry audit; with this campaign done, its certificate check becomes a ten-second confirmation.

Where next

• Roll out encrypted DNS: put the now-stable API surface to work for DoT and DoH clients.
• TLS certificates: the two-surface reference this campaign leans on.
• Run a quarterly maintenance pass: where the expiry check lives from now on.

Need a hand? Email [email protected].

Last validated: 2026-07-27