Ship the query log to Log Analytics

Added in v1.0.6

From DTM 1.0.6 the query log ships through the Azure Monitor Logs Ingestion API, to a data collection rule (DCR). The node’s managed identity authenticates, so there is no key or secret to store or rotate. The settings earlier releases used (azure.log_analytics_workspace_id and _key) are removed; see upgrading from 1.0.5.

The shipped Azure Monitor workbook and KQL alerts read the DTMLogs_CL table (see Observability), so set this up before you import them.

DTM node --(HTTPS, managed identity token)--> data collection endpoint (DCE)
                                                      |
                                       data collection rule (DCR)
                                       stream Custom-DTMLogs + transform
                                                      |
                                    Log Analytics workspace, table DTMLogs_CL

Which path is yours

You deployed with	What creates the setup
The Terraform modules/monitoring module	logs_ingestion_enabled = true, see Deploy with Terraform
The module-based Bicep tree	enableLogsIngestion = true, see Deploy with Bicep
The Azure Marketplace wizard, or anything else	You create it, as this page describes. The Marketplace templates create no query-log target.

Whichever creates it, the cluster side is the same: set the three values and turn the query log on.

What to create

Four resources and one role assignment, all in your workspace’s region (Azure requires the DCR, the DCE and the workspace in one region):

Resource	Setting	Value
Table	name	DTMLogs_CL, a DCR-based custom table
Table	columns	TimeGenerated dateTime; client_ip_s, qname_s, qtype_s, rcode_s string; answers_d, latency_ms_d real; backend_s, node_id_s string; timestamp_t dateTime; timestamp_s string
DCE	region	the workspace’s region
DCR	region, endpoint	the workspace’s region; the DCE above
DCR	stream declaration Custom-DTMLogs	timestamp datetime; client_ip, qname, qtype, rcode string; answers, latency_ms long; backend, node_id string
DCR	destination	the workspace
DCR	data flow	stream Custom-DTMLogs, output stream Custom-DTMLogs_CL, the transform below
Role	role, scope, members	Monitoring Metrics Publisher, on the DCR only, for each node’s managed identity

The DCR transform maps the stream onto the table’s column names, the ones the workbook and the KQL alerts read:

DCR transform

source
| project TimeGenerated = timestamp,
          client_ip_s = client_ip, qname_s = qname, qtype_s = qtype, rcode_s = rcode,
          answers_d = todouble(answers), latency_ms_d = todouble(latency_ms),
          backend_s = backend, node_id_s = node_id

Read each node’s managed identity object ID with:

az vm show -g <rg> -n <vm-name> --query identity.principalId -o tsv

List every node. A node without the role on the DCR ships nothing. Scope the role to the DCR and nothing wider: a node only sends rows to this one rule, so do not grant it on the resource group, the subscription or the workspace.

A workspace that already has DTMLogs_CL

A DTMLogs_CL table created by a release before 1.0.6 is a Data Collector table. Convert it once, before you create the DCR. It keeps its data and columns:

az monitor log-analytics workspace table migrate \
  --resource-group <rg> --workspace-name <ws> --name DTMLogs_CL

Behind Azure Monitor Private Link

If the workspace is reachable only through an Azure Monitor Private Link Scope (AMPLS), add the DCE to your AMPLS, disable its public network access, and point logs_ingestion_endpoint at that DCE. The Terraform module’s dce_public_network_access_enabled = false and the Bicep tree’s logsIngestionDcePublicNetworkAccess = 'Disabled' are for this case.

Read the three values

Read them from the resources rather than typing them:

DTM setting	What it is	Looks like
logs_ingestion_endpoint	the DCE’s logs ingestion URI	https://<dce>.<region>-1.ingest.monitor.azure.com
logs_ingestion_dcr_immutable_id	the DCR’s immutable ID (not its name or resource ID)	dcr- and 32 lower-case hex digits
logs_ingestion_stream	the stream the DCR declares	Custom-DTMLogs, the default when left empty

az monitor data-collection rule show -g <rg> -n <dcr-name> --query immutableId -o tsv

DTM refuses an endpoint that is not an https:// origin on a host under .monitor.azure.com (no path, no IP address, no port other than 443), because the node sends its managed-identity token there. A host outside .ingest.monitor.azure.com is accepted with a warning: it is probably not a logs ingestion endpoint.

Set the values on every node

The Azure Monitor settings are per node: they are not replicated, and a change takes effect at the node’s next restart. On each node, use the Azure Monitor card on the Settings page, the dtm_azure_monitor_config Terraform resource, or the API (admin role):

Logs only, on one node

curl -sk -X PUT https://<node>:8443/api/v1/config/azure-monitor \
  -H "Authorization: Bearer $DTM_TOKEN" \
  -H "X-DTM-CSRF: 1" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "monitor_metrics_enabled": false,
    "logs_ingestion_endpoint": "https://<dce>.<region>-1.ingest.monitor.azure.com",
    "logs_ingestion_dcr_immutable_id": "dcr-<32 hex digits>",
    "logs_ingestion_stream": "Custom-DTMLogs"
  }'

• The PUT replaces the whole record, so send every field you want kept. Leave out monitor_metrics_enabled: false if the node also pushes custom metrics.
• With enabled: true, DTM checks the values when you save them: an endpoint without a DCR immutable ID (or the reverse), an ID that is not dcr-<32 hex>, or an endpoint it will not send a token to is a 400 that names the field. (A record saved with enabled: false is not checked.)
• The response carries pending_restart: true. Restart the node, one node at a time, and wait for curl -sk https://<node>:8443/readyz to return 200 before the next.

In dtm-server.yaml the same settings are azure.logs_ingestion_endpoint, azure.logs_ingestion_dcr_immutable_id and azure.logs_ingestion_stream (with azure.monitor_enabled: true); a value saved through the API, the Settings page or Terraform overrides them. Start-up checks values from the file the same way and logs an ERROR for a target it refuses; DNS still serves.

GET /api/v1/config/azure-monitor shows log_api (logs_ingestion once a target is stored, otherwise none) and a warnings list for anything DTM will not ship to.

Turn the query log on

The query-log switch is cluster-wide and applies live, with no restart. The PUT replaces the whole record, so send both fields:

curl -sk -X PUT https://<node>:8443/api/v1/config/query-log \
  -H "Authorization: Bearer $DTM_TOKEN" \
  -H "X-DTM-CSRF: 1" \
  -H "Content-Type: application/json" \
  -d '{"enabled": true, "sample_rate": 1}'

sample_rate keeps 1 in N queries: 1 logs every query. Use a higher number on a busy cluster. The dtm_query_log_config Terraform resource does the same.

Check that rows arrive

The first rows through a new DCR can take several minutes. Then:

DTMLogs_CL
| where TimeGenerated > ago(1h)
| summarize count() by node_id_s, rcode_s

You should see every node, with NOERROR and, for names that do not exist, NXDOMAIN. TimeGenerated is the query’s own time, so filter and bin on it.

How the node ships, and when it cannot

DNS never waits on shipping. Each query’s row is queued, and the node sends the queue in batches of up to 500 rows or 1 MB, about every 5 seconds. A batch that cannot be delivered is dropped, not kept and resent later, and its rows are counted in dtm_observability_querylog_dropped_total. The node logs a line starting querylog: logs ingestion of N row(s) that says why, without the row data.

What the node sees	What DTM does
400 or 413 (the batch’s content was rejected)	drops that batch only and carries on with the next
429 (throttled) or 503 (unavailable)	retries the batch, up to 3 attempts within 60 seconds, honouring Retry-After; if it still fails, drops the batch and the rest of that send
401 or 403	drops its cached token and retries once with a fresh one, so a role you just granted takes effect; a second rejection drops the batch and the rest of that send, and the log line says the identity needs Monitoring Metrics Publisher on the DCR
404 (unknown DCR or stream), another error status, or a connection or token error	drops the batch and the rest of that send, since every later batch would fail the same way

Each send takes every row queued since the last one, so a failure that ends the send loses all of them; nothing is held back for a later try. The queue cap of 20,000 rows only bounds what builds up between sends and during retry waits: a row that arrives when it is full is dropped and counted. An outage longer than the retry budget therefore loses its query-log rows (DNS is not affected).

Azure re-times a row whose TimeGenerated is more than two days behind, or more than a day ahead of, the time it received it. dtm_observability_querylog_retimed_rows_total counts such rows; a non-zero value means the node’s clock is wrong, so check NTP.

Metric	What it tells you
dtm_observability_log_api{api}	1 on logs_ingestion while a shipper is running, otherwise 1 on none; the other series is 0. Alert on {api="none"} == 1 where you expect shipping.
dtm_observability_push_total{exporter="azure_log",result}	one outcome per batch (retries included), success or error
dtm_observability_querylog_dropped_total	rows that never reached Azure: queue full, or their batch rejected or not sent
dtm_observability_querylog_retimed_rows_total	rows Azure re-timed because of the node’s clock

When no rows appear, check that:

• the role is on the DCR, for each node’s identity, and has had a few minutes to propagate (a 403 right after granting it is normal);
• the endpoint and DCR immutable ID belong to this DCE and DCR (a 404 means the ID or the stream is wrong);
• the workspace, the DCE and the DCR share a region;
• the node was restarted after the PUT, and GET /api/v1/config/azure-monitor shows no warnings;
• dtm_observability_log_api{api="logs_ingestion"} is 1 on the node.

Last validated: 2026-10-23