3.4 High availability

Part 3 · Architecture & operationsArchitect⏱ ~25 min✓ Knowledge check

High availability in SpatiumDDI is three separate problems: redundant DNS, redundant DHCP, and a redundant control plane. Each uses different mechanics and fails in a different way. If you mix them up you can end up with a three-node cluster that still drops DNS when one box reboots. This module takes each layer in turn, then puts them together in a failure table you can use in design reviews.

What you’ll learn

• Build redundant DNS with several servers per group, explain why no zone transfer is involved, and know the DNSSEC exception
• Configure a Kea HA pair, and separate what SpatiumDDI renders from what Kea does itself
• Choose a control-plane HA path: the appliance or Helm, since Compose HA isn’t supported in 1.0
• Predict what keeps serving for each common failure, and why HA doesn’t replace backups

Three layers, scaled independently

Layer	How you make it redundant	Protects
DNS	More servers in the same DNS server group	Name resolution
DHCP	Two Kea servers in one DHCP server group (DHCPv4)	Leases and renewals
Control plane	Several api/worker/frontend replicas, plus HA PostgreSQL and Redis	The UI, the API and your ability to make changes

The layers don’t cover for each other. The appliance docs say it bluntly: control-plane HA is not serving HA. A cluster whose DNS and DHCP roles sit on one node still loses DNS and DHCP when that node goes down.

DNS: more servers in the group

Several DNS servers in one group, and a Kea HA pair · Adapted from the SpatiumDDI documentation (Apache-2.0)

Add a second (or third) agent-managed server to the same group and you have redundant authoritative DNS. Unless the group’s zones are DNSSEC-signed, nothing else needs configuring.

• Every enabled server renders each of the group’s zones as an independent authoritative copy, and gets its own queued op for every record change.
• is_primary doesn’t make one server “the writer” in an agent-managed group. It only orders the fan-out and picks whose result the audit trail reports.
• If one server is unreachable, its ops wait as pending and drain when it returns, while the others apply straight away. The zone’s per-server serial status shows who has converged.
• Hand clients more than one of the servers, for example in DHCP option 6, so they can fail over.

Signed zones are the exception. Every agent engine signs with keys held on that server, so two servers in a group would serve two DNSKEY sets against one DS. SpatiumDDI therefore signs only on a group with one agent server, and refuses to add or move a server into a group that holds signed zones; disabled servers count (#1762). For signed zones, keep one signing server and add redundancy with secondary zones in another group (module 3.2 shows the pattern).

A zone already signed on several servers shows a red DNSSEC: multiple signers badge and raises the critical dnssec_multi_signer alert. An agent that registers itself into such a group through AGENT_GROUP isn’t refused, because it would retry forever, so check for that alert after you add agents.

Why DNS HA needs no zone transfers

Older DNS designs have a primary that takes writes and secondaries that copy it by NOTIFY and AXFR. SpatiumDDI’s control plane is the source of truth, so it skips that step: it fans every change out to every server directly. There’s no chain of servers where one lagging secondary holds up the rest. A zone only uses transfers if you declare it secondary with a masters list on purpose, usually to back a server SpatiumDDI doesn’t manage, or to feed copies of a signed zone into another group.

DHCP: a Kea HA pair on the group

Put two Kea servers in one DHCP server group, pick an HA Mode on the group, and fill in the HA Peer URL on each server. Each server’s URL is its own HA listener, typically http://<that server's address>:8000/ on the shipped image; SpatiumDDI renders every member’s URL into each peer’s list, which is how the partner finds it. On appliances, empty URLs are filled in for you when the second Kea member joins a host-networked HA group (#2963). When both URLs are set, SpatiumDDI renders Kea’s HA hook into both configs.

• Hot Standby (one active, one passive): the primary serves every client, and the standby takes over when Kea decides the partner is down.
• Load Balancing (both active): Kea splits clients between the peers by a hash of the client identifier.

Scopes, pools, reservations and classes live on the group, so both peers always render the same config.

Caution

Kea HA covers DHCPv4 only (#1238). A DHCPv6 scope on a group with two Kea members is served by each member on its own, so both can hand out the same address. The UI tags such a scope v6: no HA, and every HA pill reads HA v4: <state>. Until DHCPv6 HA ships (#1258), serve DHCPv6 from a group with one Kea member.

SpatiumDDI does	Kea does
Renders the HA hook with an identical peer list in both configs, plus your heartbeat and delay tuning	Replicates leases and runs the HA state machine, peer to peer over TCP 8000
Generates a per-group CA and peer certificate, so only the group’s own peers can use the HA listener; a superadmin rotates it with POST /api/v1/dhcp/server-groups/{id}/ha-peer-credential/rotate	Requires that certificate from its partner, over HTTPS, once every member’s agent supports it
Resolves peer hostnames to IP addresses (Kea only accepts IP literals) and re-checks them every 30 s	Decides when to fail over, including after the control plane is gone
Polls status-get about every 15 s to drive the HA v4 pill in the UI	Keeps answering from its own lease store if replication breaks

A third Kea member now has defined behavior. With its own HA Peer URL it joins as a backup peer that receives lease updates but takes no part in the heartbeat. Without one it’s sidelined: no HA hook and none of the group’s scopes, and the group page says “More than two Kea members” (#3151). Two things still aren’t built: driving HA state transitions (maintenance, continue, force-sync) from the UI, and gating DDNS writes on HA state, so in some windows both peers could try to write the same record.

From the field

Adding a third Kea server to an HA group once produced a config Kea refused to load, because the third server’s name wasn’t in the two-entry peer list. Today it renders as a backup peer. But roles follow server order, so a third server’s URL can take the primary role from the live pair, which is why appliances never fill it in for you. Keep HA groups at two Kea servers plus, at most, a backup whose URL you set on purpose in a maintenance window.

Source: GitHub issue #332

Caution

Kea 3.0 can’t be rolled into an HA pair. Its HA wire protocol rejects peers older than 2.7.0, so a 2.6 to 3.0 upgrade needs both members upgraded in the same window. On appliances, the rolling-upgrade preflight warns about this (kea_ha_version_skew). DHCP keeps serving throughout, because each Kea answers from its own lease store; only replication between the peers breaks until both are on 3.0.

Windows DHCP uses its own failover relationships. Since 2026.10.02-1 SpatiumDDI reads them on every lease sync, writes each scope only to the members that hold it (both partners when a relationship covers it), and can create, edit and replicate relationships from the group’s Windows DHCP failover panel over the CredSSP transport (#1110). Kea and Windows DHCP can’t share a group; module 3.2 has the details.

The control plane

Topology 7: a 3-node appliance control plane behind one VIP · Adapted from the SpatiumDDI documentation (Apache-2.0)

In every shape the rules are the same: api, worker and frontend are stateless and scale out, PostgreSQL and Redis need their own HA, and beat stays a singleton.

OS appliance

• Install every node after the first as an Additional node, then promote from /appliance → Fleet → Manage control plane cluster…. Sizes are odd only (1, 3, 5, 7); the API refuses an even total. A second First node would be a separate SpatiumDDI, not a member.
• Within about 60 seconds of promoting two nodes you have etcd quorum across three, CloudNativePG at three instances, a 3-node Redis Sentinel set, and three api, worker and frontend replicas.
• Size every member at 4 vCPU, 8 GiB and a 48 GiB SSD: on 40 GiB disks a slot upgrade followed by a rolling upgrade reached the disk-usage warning.
• CloudNativePG replicates asynchronously, so a failover can lose the last few commits.
• Per-role node labels keep control-plane workloads on control-plane nodes, so a DNS-only appliance never gets PostgreSQL.
• Set a MetalLB pool and control-plane VIP under Appliance → Network & Host. The self-signed Web UI certificate grows to cover the VIP.
• You can’t demote the etcd seed, and you can’t revoke a node while it’s a live cluster member; demote it first.
• k3s takes an etcd snapshot every 6 hours and keeps 8. Restoring one resets etcd to a single node, so it’s for disaster recovery only.

Helm chart

• postgresql.kind: cnpg renders a CloudNativePG cluster (instances: 3, or 5 or 7 for larger installs). Install the CloudNativePG operator first. The apps connect through the -rw Service, which always points at the current primary.
• Replication is asynchronous: a failover can lose the last few commits. If you need zero data loss, configure synchronous replication on the cluster yourself and accept the write latency.
• The chart sets 20-second tolerations on the database pods, so a hard node loss fails over in about a minute instead of about five.
• redis.kind: sentinel with replicas: 3 gives a quorum of 2; the apps find the master through a sentinel:// URL.
• The api’s replica count comes from its autoscaler (api.autoscaling.minReplicas and maxReplicas, 2 to 10 by default, on CPU); api.replicas applies only with autoscaling off. Scale worker and frontend with their replica counts. Beat stays a single replica with the Recreate strategy.
• Promote/demote and the VIP are appliance features; on a plain cluster you front the frontend with your own Ingress or load balancer.

Hand-built (Compose or VMs)

• Compose HA isn’t supported in 1.0 (#1236). The repo’s Patroni overlay (k8s/ha/postgres-docker-compose.yaml) doesn’t work: Patroni never starts, and on an existing install it comes up on empty volumes.
• The release docker-compose.yml hard-codes DATABASE_URL, REDIS_URL, CELERY_BROKER_URL and CELERY_RESULT_BACKEND, so pointing .env at HAProxy or Sentinel changes nothing. A hand-built HA control plane means maintaining your own compose files or VM services. Making Compose HA real is #137.
• If you build one anyway: two or more api hosts behind a load balancer. The API holds no session state, so stickiness only matters for streaming (SSE) connections.
• One beat. Leader election for beat isn’t shipped, so if its host dies, recreate beat elsewhere.
• For a supported HA control plane, use the appliance or Kubernetes with CloudNativePG.

Point agents at the VIP

An agent follows one control-plane URL. Point DNS and DHCP agents (and operators’ browsers) at the VIP or load balancer, never at one node’s IP: an agent pinned to a node loses its control plane whenever that node is down, even while the cluster is healthy. It keeps serving from cache, but it gets no changes until the node returns. When you install an Additional node, give the VIP as its control plane URL. Cluster members themselves are safe: their supervisors reach the api through the in-cluster Service.

What survives which failure

Failure	Impact	What keeps serving
One DNS server in a group	Its ops wait as pending; marked unreachable after 90 s	The group’s other servers answer and take changes; the stale one catches up on return
One Kea in an HA pair	The HA v4 pill shows the change of state	The surviving peer, under Kea’s own HA state machine
Whole control plane unreachable	No UI, API, changes or scheduled jobs; agents spool leases, query logs and metrics and replay them on reconnect	Every agent from its last-known-good cache; Kea HA peer to peer
Redis down	Celery loses its broker, so DNS bundles stop being rendered. The long-poll falls back to polling: DHCP changes land within about 2 s, and DNS changes wait for the api, which builds the bundle itself after 120 s	Config delivery through the long-poll, and all DNS and DHCP
Worker not consuming bundles, or no worker at all	No DNS bundle renders. If the rest of the worker runs, agent_bundle_render_failed fires after 10 minutes; with no worker, alerts and other background jobs stop too	DNS changes through the api’s fallback after 120 s (it can time out on very large groups); DHCP config delivery
PostgreSQL primary lost (HA)	The api can’t reach the database until a replica is promoted (about a minute on Helm); the last few commits can be lost	Agents from cache; the api resumes on the new primary
One of three appliance control-plane nodes	etcd keeps quorum (2 of 3), the database fails over (possibly losing the last few commits), the VIP moves	The UI on the same address, and DNS/DHCP on every node that carries the roles
Beat stopped	Scheduled jobs stop; the dashboard’s Platform Health card flags Beat (celery-beat)	Everything interactive, and all DNS and DHCP
A bad config bundle	The agent reverts, quarantines it and raises agent_config_rejected	That server, on its previous good config

Tip

Rehearse the first row before production. Stop one DNS agent, change a record, and start the agent again: you should see its op wait as pending, then drain, while the other server answers the whole time.

HA is not a backup

Replication copies mistakes as fast as it copies good changes. A deleted zone disappears from every DNS server and every database replica within seconds. Your undo buttons are the Trash (30 days) and your backups, so keep both working:

• Schedule backups to a remote destination and run Restore Drills against them. A drill replays the newest archive into a scratch database and checks it without touching the live one. The backup_failed and backup_stale alerts are on by default for every scheduled destination (#1262).
• On a hand-built Patroni cluster behind HAProxy, point restores at the read/write port, not the read-only one.
• DNSSEC keys live on each agent’s own volume, whatever the engine, not in the backup.
• An etcd snapshot restore is disaster recovery for the cluster, not a way to undo a bad change.

From the field

It’s tempting to treat the appliance’s A/B slot rollback as the undo for a bad upgrade. This issue spells out why it isn’t safe across a database migration: /var and PostgreSQL are shared by both slots, migrations only go forward, and readiness checks the schema version. The canonical recovery is to reinstall from the ISO and restore a backup, which only works if you took one first.

Since 2026.10.02-1 the appliance refuses a slot rollback, a Fleet boot change or an older-image upgrade onto a release that can’t run on the database (409 schema_rollback_unsafe) unless you acknowledge it; the automatic trial-boot revert isn’t checked. Back up the spatium-control-app-keys Secret too: it holds the appliance’s SECRET_KEY (#1448).

Source: GitHub issue #782

Knowledge check

Knowledge check · 6 questions

High availability

1. Ridgeline adds a second BIND9 server to the default group. What else must you set up so that it serves corp.example.edu?

Zone transfers (allow-transfer) from dns-bind9 to the new serverNothing: every enabled server in the group renders the group's zones and gets its own op for each changeMark the new server as is_primaryA catalog zone for the group

Every agent-managed server in a group is an independent authoritative copy. The control plane fans each record change out as one op per server, so no primary-to-secondary transfer is involved. The one exception is a group whose zones are DNSSEC-signed: each server would sign with its own keys, so SpatiumDDI refuses a second server there.

2. A DHCP group has two Kea servers, run from the standalone agent compose file, in Hot Standby mode, but only one server has its HA Peer URL filled in. What happens?

SpatiumDDI fills in the missing URL from the server's hostNeither peer loads the HA hook, so the pair runs without HAOnly the server with a URL runs HAThe API refuses to save the group

HA renders only when both Kea members have a non-empty peer URL. Otherwise the bundle drops the failover block and neither peer loads the hook, so both serve the group's scopes uncoordinated. It isn't silent: the group page says "HA is not running" and the API lists the member in ha_peer_urls_missing. Only appliance-hosted members get an empty URL filled in for them.

3. Which of these is Kea's own job, not SpatiumDDI's?

Rendering an identical peer list into both servers' configsReplicating leases between the peers and deciding when to fail overResolving peer hostnames to IP addressesPolling status-get to drive the HA pill

SpatiumDDI renders the HA config, resolves peer hostnames (Kea only accepts IP literals) and reports state. Kea's HA hook does the lease replication and failover itself, peer to peer, whether or not the control plane is reachable.

4. You upgrade a Kea HA pair from 2.6 to 3.0 one node at a time. What should you expect?

A rolling upgrade with no interruption to HAThe pair falls out of sync until both are on 3.0, while each Kea keeps serving from its own lease storeDHCP stops on both nodes until the upgrade finishesSpatiumDDI's preflight refuses to start the upgrade

Kea 3.0's HA protocol rejects peers older than 2.7.0, so there's no rolling path for a pair. The preflight warns but never blocks, because finishing the upgrade is what brings the pair back together. DHCP itself keeps serving.

5. Why should off-cluster DNS and DHCP agents use the control-plane VIP rather than a node's IP?

The VIP gives faster DNS answersAn agent pinned to one node loses its control plane whenever that node is down, even while the cluster is healthyAgents can't register against a node IPThe VIP carries client DNS queries to the agents

The VIP floats to a surviving node. An agent pinned to one node still serves from cache when that node is down, but it can't receive changes until the node returns.

6. Which statements about control-plane HA are true?

Select all that apply.

An appliance control-plane cluster must have an odd number of membersBeat runs as a single replica in every HA shapeA 3-node appliance cluster automatically keeps DNS serving on all three nodesDocker Compose ships a supported Patroni overlay for PostgreSQL HA

Clusters grow 1, 3, 5, 7 and the API refuses an even total. Beat is always a singleton. DNS and DHCP only run on nodes that carry those roles: control-plane HA isn't serving HA. And Compose HA isn't supported in 1.0: the repo's Patroni overlay doesn't work, so use the appliance or Kubernetes with CloudNativePG.

Check answers

Go deeper

• System architecture: high availability: the three HA tiers
• Deployment topologies: topologies 3, 4 and 7 and how they fail
• Sizing: the three-node cluster figures and why its disks are 48 GiB
• Install the OS appliance: Additional nodes and growing to a high-availability control plane
• OS appliance: promote/demote, the VIP, etcd snapshots and rolling back
• Kubernetes deployment: CloudNativePG, Sentinel and autoscaling settings
• DHCP features and DHCP drivers: Kea HA, peer authentication, Windows failover and the Kea 3.0 caveat
• System administration: backup and restore: archives, restore, restore drills and backup alerts

Finished 3.4 High availability?

Mark it complete to track your progress (stored only in this browser).

Mark complete