Notifications

DTM can push structured JSON events when something operationally significant happens: a health check flips state, cross-region failover activates or recovers, a node joins or leaves the cluster, or a scheduled backup fails. That means “tell my incident channel when DTM fails over” works without standing up Prometheus or a Log Analytics pipeline first.

Notifications are off by default: DTM makes no unsolicited outbound calls unless you configure a sink.

Configuration

Two sinks, usable together; set at least one (a server with enabled: true and no sink refuses to start, because an operator who enabled notifications believes an incident channel is covered):

notifications:
  enabled: true
  # Sink 1: a generic HTTPS webhook (Teams/Slack relay, incident bot).
  webhook_url: https://hooks.example.internal/dtm
  webhook_secret: '<hmac key>' # prefer the environment override
  # Sink 2: an Azure Event Grid custom topic, published with the node's
  # managed identity. No access keys; a URL with a query string is rejected.
  eventgrid_topic_endpoint: https://dtm-events.westeurope-1.eventgrid.azure.net/api/events
  # Optional shaping:
  event_types: [] # empty = all types
  min_interval_seconds: 60 # flap-dedupe window

The key list and DTM_NOTIFICATIONS_* environment overrides are in the configuration reference; deliver webhook_secret via the environment rather than the YAML. Every node emits independently, and each event carries its node ID, so a receiver can tell N sightings of one cluster-wide condition from N distinct events; deduplicate on the receiver if you want one line per cluster event.

Event types

Type	Fired when
health.check_transition	A health check’s aggregated status flips (per transition, not per probe).
failover.activated	Every local-region backend for a record went unhealthy and cross-region failover started.
failover.recovered	A local backend recovered and cross-region failover stopped.
cluster.node_joined	A peer joined the cluster.
cluster.node_left	A peer left the cluster or was declared unreachable (a clean leave and a failed node both fire this).
backup.failed	A scheduled backup run failed.

Webhook contract

Each event is one JSON POST:

{
  "type": "failover.activated",
  "time": "2026-07-26T14:00:00Z",
  "node": "dtm-we-01",
  "data": { "record_id": "rec-9", "subject": "rec-9" }
}

with headers Content-Type: application/json, X-DTM-Event: <type> (route without parsing the body), and X-DTM-Signature: sha256=<hex>: an HMAC-SHA256 of the exact body bytes with your webhook_secret.

data.subject is present on every event (the record ID, check ID, node ID, or backup) and is what the Event Grid sink uses to build dtm/<node>/<key>.

Caution

Verify the signature before trusting a payload. Anything that can reach your webhook URL can POST to it; the signature is what makes the event DTM’s. It is sent only when a secret is configured; an empty secret sends unsigned, with a warning at startup.

Receiver-side verification in shell:

expected="sha256=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -r | cut -d' ' -f1)"
[ "$expected" = "$SIGNATURE_HEADER" ] || exit 1

Delivery: 10 second timeout, two retries on network errors and 5xx (1s, then 5s), and no retry on 4xx (the receiver actively rejected the payload; replaying identical bytes cannot fix a signature mismatch). https:// is required; plain http:// is allowed only for loopback collectors during development.

Event Grid contract

Events publish in the standard Event Grid schema, with eventType: Hayami.DTM.<type> and subject: dtm/<node>/<key>:

[
  {
    "id": "dtm-we-01-failover.activated-1753538400000000000",
    "eventType": "Hayami.DTM.failover.activated",
    "subject": "dtm/dtm-we-01/rec-9",
    "eventTime": "2026-07-26T14:00:00Z",
    "data": { "record_id": "rec-9", "subject": "rec-9" },
    "dataVersion": "1"
  }
]

Authentication is the node’s managed identity: grant it EventGrid Data Sender on the topic (no aeg-sas-key; a role granted after boot heals without a restart). Route onward with Event Grid subscriptions and subject filters, for example subject beginsWith dtm/dtm-we- for one region’s nodes, into Teams or email via Logic Apps, queues, or Functions.

Flap protection and delivery semantics

• Repeats of the same event about the same subject inside min_interval_seconds are suppressed (and counted), so a flapping backend cannot flood an incident channel. During a flap you may see the “unhealthy” edge and not the “healthy” edge that follows inside the window: treat events as triggers to look, not a state ledger; GET /api/v1/health/status is the state of record.
• Event delivery never blocks DNS or health checking: events queue on a bounded buffer and are dropped (and counted) if a slow sink backs it up.
• Delivery is at-least-once per sink in the happy path, best-effort overall.

Watch the notifier itself

The notifier is observable in Prometheus:

Metric	Alert on
dtm_notify_delivered_total{sink}	(baseline)
dtm_notify_failed_total{sink}	any sustained rate: a sink is down or rejecting
dtm_notify_dropped_total	any increase: queue overflow, the sink is too slow
dtm_notify_suppressed_total	a high rate means a flapping source, not a notifier fault

Notifications complement metrics rather than replacing them: discrete events with context for humans and incident tooling, metrics for thresholds and trends. Keep your alerting as the quantitative layer.