Fail over between Key Vault Private Endpoints in two regions

A Key Vault with a Private Endpoint in each of two regions gives clients two network paths to the same vault. Azure Private DNS can hold only one address for the vault’s name, so clients use one path and lose access if it breaks. This tutorial puts both endpoints behind one DTM override record, with a health check that proves the vault answers through each endpoint. Each region’s clients get their nearby endpoint, and move to the other region’s endpoint when their own path fails. (Added in v1.0.5: tls_server_name pins the name the check verifies the certificate against. Without it, the check uses the Host header when that is a valid hostname.)

The pattern works for any Azure service with Private Endpoints in two regions. Key Vault is the worked example.

What this protects, and what it does not

Protects: clients against a failure of one region’s path to the vault: that region’s endpoint, its subnet or NSG, the VNet, or the peering or ExpressRoute leg.
Does not protect against an outage of the vault’s own region. Both endpoints reach the same vault. Recovery from a regional Key Vault outage is Microsoft’s failover to the paired region.
Detects only what the DTM nodes can see. The health checks run on the DTM nodes. A path that breaks for clients but still works from the DTM nodes is not detected, so this protects the paths the nodes share with their clients.
When every endpoint is down, DTM still answers rather than returning nothing. Clients then get the vault’s errors.

Clients keep using the vault’s own name, so TLS, the vault’s host routing and the Azure SDKs’ challenge checks are untouched. Two separate vaults behind one name cannot work this way: each vault answers only to its own name.

1. Create an endpoint in each region

Create a subnet for the endpoints in each VNet, with Private Endpoint network policies enabled so an NSG applies to them, and an NSG on each. The NSG is what step 5 uses to test failover.

for r in ne we; do
  az network nsg create -g rg-net -n nsg-kvpe-$r \
    -l $([ $r = ne ] && echo northeurope || echo westeurope)
  az network vnet subnet create -g rg-net --vnet-name vnet-$r -n snet-kvpe \
    --address-prefixes $([ $r = ne ] && echo 10.10.20.0/27 || echo 10.20.20.0/27) \
    --private-endpoint-network-policies Enabled \
    --network-security-group nsg-kvpe-$r
done

Create the two endpoints without a private DNS zone group. DTM answers the vault’s name, and a zone group would write one of the two addresses into a privatelink.vaultcore.azure.net zone:

KV_ID=$(az keyvault show -n kv-contoso --query id -o tsv)
for r in ne we; do
  az network private-endpoint create -g rg-kv -n pe-kv-$r \
    -l $([ $r = ne ] && echo northeurope || echo westeurope) \
    --subnet "$(az network vnet subnet show -g rg-net --vnet-name vnet-$r \
      -n snet-kvpe --query id -o tsv)" \
    --private-connection-resource-id "$KV_ID" --group-id vault \
    --connection-name kv-$r
done
az network private-endpoint show -g rg-kv -n pe-kv-ne \
  --query 'customDnsConfigs[0].ipAddresses[0]' -o tsv   # 10.10.20.4
az network private-endpoint show -g rg-kv -n pe-kv-we \
  --query 'customDnsConfigs[0].ipAddresses[0]' -o tsv   # 10.20.20.4

Make sure both VNets are covered by subnet mappings, so DTM tags 10.10.20.4 as northeurope and 10.20.20.4 as westeurope.

2. Create the override and its health check

Create the override before any client queries the vault’s name through DTM. A name DTM has already forwarded is answered from its cache until that answer’s TTL runs out.

curl -sk -X POST https://10.10.0.4:8443/api/v1/overrides \
  -H "Authorization: Bearer $DTM_TOKEN" -H "X-DTM-CSRF: 1" \
  -H "Content-Type: application/json" -d '{
  "fqdn": "kv-contoso.vault.azure.net",
  "type": "A",
  "target_mode": "static",
  "ttl": 10,
  "values": [{"value": "10.10.20.4"}, {"value": "10.20.20.4"}],
  "lb_policy": {"method": "failover", "failover_order": ["10.10.20.4", "10.20.20.4"]},
  "health_check_template": {
    "protocol": "https",
    "port": 443,
    "path": "/secrets?api-version=7.5",
    "expected_status": [401],
    "expected_body": "AKV10000",
    "interval_seconds": 5,
    "timeout_seconds": 3,
    "tls_server_name": "kv-contoso.vault.azure.net",
    "headers": {"Host": "kv-contoso.vault.azure.net"}
  }
}'

What each part does:

The check proves the vault answered through that endpoint. It connects to each endpoint’s address, sends the vault’s name for TLS and in the Host header, and verifies the vault’s certificate against that name. An unauthenticated request gets Key Vault’s own 401 with its own error code, which only the vault sends. An open port alone does not pass.
Each region’s clients get their own endpoint. DTM narrows the values to the client’s region first, then applies the failover order. A client moves to the other region’s endpoint only when its own is unhealthy. Behind a single forwarder, every client gets the forwarder’s region; see known limitations.
Detection takes three failed probes at the 5-second interval, then the 10-second TTL for clients to ask again.

3. Check what the health check proves

Before relying on it, show that the check fails when it should. Create throwaway checks with POST /api/v1/healthchecks against 10.10.20.4:443, each with one thing wrong, and confirm each reads unhealthy:

Changed	Why it must fail
`tls_server_name` and `Host` set to another name	The vault’s certificate does not match it
The target is a different vault’s Private Endpoint, if you have one	That endpoint does not serve `kv-contoso`, so Key Vault answers `403`
`expected_status` set to `[200]`	Key Vault answers `401` without a token

Delete them afterwards. If any of them reads healthy, the check proves less than this page says, so stop and check the template.

4. Check the answers from each region

From a client in each region:

dig +short kv-contoso.vault.azure.net
# northeurope client: 10.10.20.4
# westeurope client:  10.20.20.4

Then read a secret as each client normally would, for example with the Azure CLI or an SDK. The name, and therefore TLS, is unchanged.

5. Test failover

Block the northeurope endpoint with a deny rule on its NSG:

az network nsg rule create -g rg-net --nsg-name nsg-kvpe-ne -n drill-deny-443 \
  --priority 100 --direction Inbound --access Deny --protocol Tcp \
  --destination-address-prefixes 10.10.20.0/27 --destination-port-ranges 443

Watch the northeurope client’s answer move to 10.20.20.4, and its secret reads continue through the westeurope endpoint. The westeurope client does not change.

What to expect. These are computed from the settings in this tutorial, not measured: a 5-second probe interval, 3 failed probes to mark an endpoint unhealthy, 2 successful probes to bring it back, and a 10-second TTL. Your numbers will differ with your settings.

Step	Expected time
Deny rule applied to the check reporting unhealthy	About 15 seconds (3 failed probes, 5 seconds apart), give or take one interval depending on where in the probe cycle the rule lands
Deny rule applied to the first `10.20.20.4` answer	The time above plus up to one TTL (10 seconds): roughly 15 to 25 seconds
Failed secret reads on the northeurope client	From the rule landing until the client asks again: the same window as the row above, longer if the client caches DNS answers past the TTL
Rule removed to the first `10.10.20.4` answer again	5 to 10 seconds (2 successful probes, 5 seconds apart) plus up to one TTL: roughly 5 to 20 seconds

Remove the rule, and the answer returns to 10.10.20.4 after two consecutive successful probes:

az network nsg rule delete -g rg-net --nsg-name nsg-kvpe-ne -n drill-deny-443

Next steps

Override records: the rest of what an override can do.
Health checks: intervals, thresholds and TLS.
The DTM nodes probe both endpoints, so any firewall between them and the endpoint subnets must allow 443 from the DTM node subnets.

Need a hand? Email [email protected].

Privacy policyTerms of use