Skip to content
Hayami Docs

Adding nodes (scaling out)

Adding a node is how you scale Hayami DTM. It adds both capacity (more query and update throughput) and resilience (more nodes that can fail without taking DNS down). A new node joins the existing cluster, replicates the full DNS state, and starts answering queries; nothing about your zones or records changes.

You add nodes the same two ways you deploy: from the Azure Marketplace (the add-node template) or with Terraform. Either way, the new node receives the same managed-identity grants and reads the existing cluster’s secrets from Key Vault, so it comes up as a full member of the cluster you already have.

Before you start

Section titled “Before you start”
  • A healthy cluster. Confirm every existing node is listed and reports alive before you add another (see Verify the cluster). Count the entries: a failed node drops out of the list rather than changing status.
  • The existing cluster’s subnet, its Key Vault, and one or more existing node IPs to use as seeds. The new node contacts a seed to join.
  • The same deployment permissions as the first deploy: rights to create resources and to assign roles in the target resource group.

How a new node joins

Section titled “How a new node joins”

However you add it, the new node:

  • gets its own system-assigned managed identity, granted Key Vault Secrets User on the existing vault plus Reader and Contributor on the resource group, the same rights every other node already has;
  • reads the existing cluster secrets from Key Vault (administrator password and cluster encryption key; in the Terraform customer-starter environment the web UI TLS bundle is also shared via the dtm-tls secret, or via the CA-signed secret you named, while Marketplace and sample-deployed nodes generate their own self-signed certificate at first boot). It does not get Key Vault Secrets Officer and does not regenerate secrets: only the original primary node ever generated them (see how the cluster’s secrets are created);
  • joins via the seed IPs you point it at, then replicates all zones and records from the cluster.

How a new node joins the cluster, in six steps. Deployed from the Marketplace add-node template or Terraform with its own system-assigned managed identity, the identity is granted Key Vault Secrets User plus Reader and Contributor on the resource group (never Secrets Officer), and the node reads the existing secrets from the existing Key Vault: the admin password and cluster encryption key, with Terraform deployments also sharing the TLS bundle. It then contacts a seed node via the seed IPs and replicates the full DNS state (every zone and record) from the in-sync cluster. An after-the-join panel shows node-c in service as a full member: ready and fully caught up, in sync with node-a and node-b, with DNS clients in your VNets and peered VNets querying it on port 53 and getting answers such as A 10.0.1.10, once you add node-c's IP to your VNet DNS settings or resolver forwarding rules. Zones, records, and existing client DNS settings are untouched throughout: the cluster simply gains another complete, current copy. How a new node joins the cluster, in six steps. Deployed from the Marketplace add-node template or Terraform with its own system-assigned managed identity, the identity is granted Key Vault Secrets User plus Reader and Contributor on the resource group (never Secrets Officer), and the node reads the existing secrets from the existing Key Vault: the admin password and cluster encryption key, with Terraform deployments also sharing the TLS bundle. It then contacts a seed node via the seed IPs and replicates the full DNS state (every zone and record) from the in-sync cluster. An after-the-join panel shows node-c in service as a full member: ready and fully caught up, in sync with node-a and node-b, with DNS clients in your VNets and peered VNets querying it on port 53 and getting answers such as A 10.0.1.10, once you add node-c's IP to your VNet DNS settings or resolver forwarding rules. Zones, records, and existing client DNS settings are untouched throughout: the cluster simply gains another complete, current copy.

From the Azure Marketplace (add-node)

Section titled “From the Azure Marketplace (add-node)”

Deploy the add-node template and point it at the cluster you already have. The running cluster serves its own copy of the template from the Lifecycle page (/lifecycle/add-node-template.json when signed in to the UI), so the template you deploy always matches the image version you are running. Point it at:

  • the existing cluster’s subnet,
  • its Key Vault, and
  • one or more existing node IPs as seeds.

The template creates a new VM, data disk, NIC, managed identity, and NSG, and grants the new node’s identity read access to the existing Key Vault plus Reader and Contributor on the resource group. On first boot the node reads the shared cluster secrets and joins through the seed IPs.

One step to confirm: the existing nodes must accept the new node on the cluster port (7946). A cluster deployed from v1.0.4 allows that port from its own subnet (on a dual-region deployment, both regions’ subnets) and denies it from the rest of the VNet, so:

  • A new node in an existing node subnet needs no NSG change.
  • A new node in a different subnet or region needs that subnet added to each existing node’s port 7946 (TCP and UDP) allow rules, or the deny refuses it. In the add-node wizard, enter the existing cluster’s other subnets in Other cluster subnets so the new node’s own NSG accepts them.

A cluster deployed before v1.0.4 has no such deny, but its multi-node NSGs list the existing nodes’ individual IPs: add the new node’s IP to each existing node’s port 7946 rules so the allowlist stays meaningful. If the new node stays not-ready (/readyz 503), cluster-port reachability between it and the existing nodes is the first thing to check.

The same applies to the nodes’ API calls to each other on 8443 when adminAddressPrefix is narrowed. A node added from the v1.0.5 template allows them on its own NSG; an existing node deployed before v1.0.5 does not, so add the rules to its NSG as described in known limitations.

The new node must also reach the Key Vault at its first boot, which depends on how the cluster’s vault is exposed (see Key Vault network access):

  • Private (the default): a node in the cluster’s VNet resolves the existing private endpoint as it is. A node in a different, peered VNet needs that VNet linked to the cluster’s privatelink.vaultcore.azure.net private DNS zone first; neither the template nor the samples create that link.
  • ServiceEndpoint: the new node’s subnet needs the Microsoft.KeyVault service endpoint and a network rule on the vault.

Until it can read the vault, the node does not start DTM. Its journalctl -u dtm-bootstrap names the secret it could not fetch.

With Terraform

Section titled “With Terraform”

Add another node to your DTM configuration and apply:

  • add one more node module block, or
  • raise the node count if your configuration is written that way.
Terminal window
terraform plan
terraform apply

Terraform provisions the new node with the same identity grants as the existing ones. On first boot it reads the existing administrator password and cluster encryption key from Key Vault (only the primary ever generates them) and joins via the existing nodes’ IPs. Because every node shares one Key Vault and one cluster encryption key, the new node comes up as a full cluster member with no further configuration.

Verify the new node

Section titled “Verify the new node”

Check the new node is ready:

Terminal window
# 200 = ready to serve; 503 = still starting or joining
curl -sk https://<new-node-ip>:8443/readyz

Then confirm the API lists every node, including the new one, each alive (see Verify the cluster for how to capture a token), and that the new node answers DNS:

Terminal window
dig @<new-node-ip> <a-known-name> +short

Only once the new node reports ready and is answering correctly should you add its IP to your VNet DNS settings or resolver forwarding rules.

Next steps

Section titled “Next steps”
Privacy policyTerms of use