3.1 How SpatiumDDI works

Part 3 · Architecture & operationsArchitect⏱ ~30 min✓ Knowledge check

When a record doesn’t resolve, or a server shows a rejected config, you need to know where the change is and who is supposed to act on it next. This module gives you that model: one control plane that holds the truth, DNS and DHCP servers that are projections of it, and the few mechanisms in between. Every later decision in Part 3 (engines, topologies, HA, hardening, migration) builds on it.

What you’ll learn

• Name the control-plane components and say what each one does
• Tell the data-plane shapes apart, including the one exception to “read-only”
• Walk an agent through its lifecycle, from bootstrap key to heartbeat
• Trace one record change from the API to a dig answer
• Explain what keeps serving when the control plane is unreachable

One source of truth, many projections

Three rules shape the whole design, and the project treats them as non-negotiable:

• API-first. Every UI action is a REST call under /api/v1. The browser, your scripts and the Copilot’s MCP endpoint share one surface, and the API checks permissions itself.
• The control plane is the source of truth. PostgreSQL holds the desired state. SpatiumDDI renders config for the DNS and DHCP daemons and runs them; the servers hold no authoritative state of their own.
• Agents survive a control-plane outage. Every DNS and DHCP agent caches its last-known-good config on disk and keeps serving if the control plane is unreachable.

Clients, control plane and data plane · Adapted from the SpatiumDDI documentation (Apache-2.0)

The control plane

In Docker Compose the control plane is seven services. Kubernetes and the OS appliance run the same set from the Helm chart.

Component	What it does	Scaling
api	FastAPI app: the REST API and its OpenAPI docs (/api/docs), sign-in and RBAC, the agent register, heartbeat and long-poll endpoints, and the Copilot’s MCP endpoint	Stateless; run several
worker	Celery workers for anything that mustn’t block a request: rendering each DNS server’s config bundle, IPAM↔DNS sync, health checks, lease sweeps, integration syncs, alert evaluation. Tasks are idempotent and safe to retry	Several, by queue (ipam, dns, dhcp, default, bundles); a custom queue list must keep bundles
beat	The Celery scheduler. It only enqueues periodic tasks	Exactly one; two would double-enqueue
frontend	nginx serving the React UI and proxying /api	Stateless
migrate	A one-shot alembic upgrade head. On Compose the api and worker wait for it to finish	Once per upgrade
PostgreSQL 16	The source of truth: IPAM, DNS, DHCP, auth, audit, settings	HA with CloudNativePG (Helm chart, OS appliance); no supported Compose HA in 1.0
Redis	Celery broker, cache and sessions, and the agent-wake bus	HA with Sentinel

Treat PostgreSQL and Redis as internal only: never publish them to the host or the network. The api’s readiness probe, /health/ready, compares the schema version its image expects with the database, so a new api pod stays out of service until the schema reaches that version. During a rolling upgrade, a pod that was already serving keeps serving a newer schema only while every table and column it uses still exists (#2777).

From the field

An operator’s API container started on new code before alembic upgrade head had run (for example make up before make migrate) and logged about 2,440 UndefinedColumnError 500s. The api’s /health/ready already compared schema heads; since 2026.07.04-1 the worker and beat check them too, and since 2026.10.02-1 the Compose api and worker don’t start until migrate has completed.

If an upgrade leaves the UI throwing 500s, confirm that migrate finished first. In the Academy lab on 1.0.0-beta.3, a migrate run while the previous release was still up stopped once with deadlock detected; running it again resumed it, because each revision commits on its own.

Source: GitHub issue #565

The data plane: four shapes and one exception

You can mix these freely in one install. Behind all of them, the services talk only to a neutral driver interface (DNSDriver, DHCPDriver), never to BIND9, Kea or PowerShell directly.

Shape	Members	How SpatiumDDI reaches them
Agented, on-prem	BIND9, PowerDNS, Technitium and Kea, shipped as the dns-bind9, dns-powerdns, dns-technitium and dhcp-kea images with the agent baked in	The agent calls out to the api; nothing calls in
Agentless, on-prem	Windows DNS, Windows DHCP, and an existing Technitium server (technitium_api, experimental in 1.0)	The api dials out: RFC 2136 on port 53, WinRM on 5985/5986, or Technitium’s HTTP API
Agentless, cloud DNS	Cloudflare, Route 53, Azure DNS, Google Cloud DNS, DigitalOcean, Hetzner, Linode, Vultr	The api calls the provider’s API
Read-only integration mirrors	Kubernetes, Docker, Proxmox VE, AWS/Azure/GCP, Tailscale, NetBird, UniFi, OPNsense, Palo Alto, Fortinet, Meraki	Scheduled pulls into IPAM, never written back

FortiGate DHCP is also agentless: the api pushes whole scopes over the FortiOS REST API.

The exception to “never written back” is active block sync: an opt-in, off-by-default path that pushes a SpatiumDDI-owned block set to an OPNsense alias, a UniFi client, a Palo Alto dynamic address group or a Meraki client policy, or serves it as a feed the firewall polls. It needs the manage_block_sync permission, and the Palo Alto and Meraki targets also need manage_firewall_enforcement, which no built-in role carries.

On the OS appliance, a host-side spatium-supervisor looks after the machine itself: pairing, role assignment, A/B upgrades, firewall and host settings. The DNS and DHCP containers only do service work.

The agent lifecycle

DNS and DHCP agents follow the same lifecycle. Here it is for a DNS agent such as Ridgeline’s dns-bind9.

Bootstrap: a pre-shared key is exchanged for a rotating token · Adapted from the SpatiumDDI documentation (Apache-2.0)

1. Bootstrap with a pre-shared key. The agent starts with CONTROL_PLANE_URL and DNS_AGENT_KEY (the DHCP agent uses SPATIUM_API_URL and SPATIUM_AGENT_KEY, the key checked against the control plane’s DHCP_AGENT_KEY). It keeps a stable agent ID on disk and calls POST /api/v1/dns/agents/register. On the OS appliance you never paste a key: the supervisor pairs with an 8-digit code and hands the per-role keys to the containers.
2. Receive a JWT. The control plane creates or updates the server row and returns an agent token: a JWT signed with SECRET_KEY, valid for 24 hours, rotated through the heartbeat and scoped to that one server. On a 401 or 404 the agent drops the token and bootstraps again, so a server row deleted by mistake reappears within a heartbeat.
3. Long-poll for config. The agent holds GET /dns/agents/config open with its last ETag. The control plane answers at once when the ETag differs or ops are waiting, and with 304 when the poll times out. For DNS the bundle isn’t built in that request: the worker renders each server’s bundle once per change and stores it, and the poll streams the stored bundle with the server’s pending ops (#1111). DHCP bundles are still built per request. A Redis wake, published when a render lands or a DHCP change commits, makes changes land in well under a second.
4. Keep a last-known-good copy. The cache in /var/lib/spatium-dns-agent/config/ holds the bundle last fetched (current.json) and the last one that applied cleanly (previous.json). On boot the agent applies its cache before it contacts the control plane. The one exception is a bundle the daemon died on: then the agent asks the control plane first, and serves the cache only if the control plane is unreachable or sends no bundle within 10 minutes (AGENT_BOOT_HOLD_MAX_WAIT_S).
5. Revert and quarantine a bad bundle. Applying is phased: render, validate, then swap and reload. BIND9 validates with named-checkconf and named-checkzone in a staging directory, then confirms each changed zone’s new serial after the reload. Kea’s config is checked with kea-dhcp4 -t in a separate process, so a rejected config never reaches the running daemon. If an apply fails, the agent reverts to previous.json, quarantines the failed ETag, retries on a 60 s → 5 min → 15 min ladder, and reports it: a chip on the server row, a banner on the server, and the agent_config_rejected alert.
6. Apply record changes per server, over loopback. Every enabled agent-based server in a group gets its own queued op. BIND9 applies it with TSIG-signed nsupdate against its own named; PowerDNS and Technitium use their REST APIs on 127.0.0.1. In a group without views, record changes never reload the daemon; in a group with views, a record change re-renders the zone instead. Structural changes (zones, views, ACLs, options) take the render, validate and reload path.
7. Heartbeat. Every 30 seconds (±3 s) the agent reports its daemon’s status and acknowledges applied ops. A beat job runs every 60 seconds and marks a server unreachable after three missed heartbeats (90 s).

What happens to config delivery without Redis

The wake is only advisory. Without Redis the long-poll falls back to a 2-second loop. With Redis up, a 12-second safety tick catches any wake that goes missing.

DNS also needs the worker, and Redis is the Celery broker that carries its render tasks, so with Redis down no new DNS bundle gets rendered. The api then builds the DNS bundle itself once the stored one has waited 120 seconds for the worker (dns_agent_bundle_inline_fallback_after_seconds): DNS changes still arrive, only later. The same fallback covers a worker that is down or doesn’t consume the bundles queue. While the rest of the worker runs, the agent_bundle_render_failed alert fires once changes have waited 10 minutes with no render.

From the field

On a clean 2026.08.12-1 install, DHCP option 6 never reached Kea’s config: the agent looked options up under different names than the control plane sent. The fix shipped in a new dhcp-kea agent image. Agents render their daemon’s config themselves, so a control-plane upgrade alone changes none of the rendered files. Recent release notes say it plainly: pull the new agent images too.

Source: GitHub issue #856

Trace a record change end to end

A Ridgeline operator allocates print-01 at 10.20.21.10 in Staff VLAN 21, with the zone corp.example.edu. Here’s the path that change takes.

One record change, from API call to applied op · Adapted from the SpatiumDDI documentation (Apache-2.0)

1. The api commits. It checks the operator’s permissions, writes the address and its two records (print-01.corp.example.edu A 10.20.21.10, and the PTR 10 in 21.20.10.in-addr.arpa, the reverse zone created when the subnet was saved), and writes the audit row in the same transaction. The response returns once the write is durable in PostgreSQL.
2. Ops are queued. One DNSRecordOp is written per enabled agent-based server in the default group (at Ridgeline that’s only dns-bind9), and the same transaction marks that server’s bundle dirty. The op carries the zone serial the control plane has bumped (YYYYMMDDNN).
3. The worker renders. After the commit, a render of dns-bind9’s bundle is queued on the worker’s bundles queue. The worker renders and stores it, then publishes a wake on Redis.
4. The agent wakes. dns-bind9’s waiting long-poll returns with the stored bundle and the pending ops that render covers. An op committed after the render read the database rides with the next render.
5. No reload. A record change doesn’t move the bundle’s structural ETag, so the agent skips render-and-reload and only runs the ops.
6. nsupdate over loopback. The agent sends a TSIG-signed RFC 2136 update to its own named on 127.0.0.1. The TSIG key never leaves the container.
7. Acknowledge. The next heartbeat acknowledges the op and it becomes applied. A failed apply is retried with a growing backoff, from 30 seconds up to 15 minutes. After 8 attempts, about 45 minutes, the op becomes failed and the default-on dns_record_op_failed alert (“DNS record change not applied”) fires. The server’s Sync tab lists each op with its state and attempts.

Check it from the lab host:

dig @127.0.0.1 -p 1053 print-01.corp.example.edu +short
dig @127.0.0.1 -p 1053 -x 10.20.21.10 +short

If a second server joined default, it would get its own op and apply it independently. If that server were offline, its op would wait as pending and drain when it came back. Agentless servers (Windows DNS, technitium_api and the cloud drivers) have no agent: the op row is written pending in the same transaction, the control plane calls the server right after the commit, and a refused write is retried with backoff (#1539).

Tip

When a record “didn’t make it”, check three places in order: the Audit Log (did the api accept the change?), the zone’s per-server serial status and the server’s Sync tab (did every server apply its op?), and the wire (dig, or the zone’s Drift tab). The first place that disagrees is where to look.

What keeps working when the control plane is down

Why DNS and DHCP outlive the control plane

No query path depends on the control plane. The daemons answer from their own data: BIND9’s zone files, PowerDNS’s LMDB store, Kea’s config and lease file. A container that restarts during an outage checks its quarantine, applies its cached bundle and starts serving without waiting for the control plane. On Kubernetes and multi-node appliances the cache lives on the node itself, so each node keeps serving on its own. Kea HA pairs keep working peer to peer, and even maintenance mode leaves the agent endpoints open.

Keeps working	Stops until the control plane is back
DNS answers from every agent-managed server	The UI and the API
DHCP leases, including failover inside a Kea HA pair	Every change: records, zones, scopes, reservations
Windows and cloud DNS, which never needed SpatiumDDI to answer	Scheduled jobs: syncs, health checks, alerts, integration pulls
Agent restarts, which come up warm from cache	Fresh lease, query and metric data in IPAM, logs and dashboards

That last item arrives late rather than never. DNS and DHCP agents spool query logs, metrics, DHCP activity and Kea lease events to disk, and replay them in order, with their original timestamps, when the control plane answers again (#1077). The spool holds up to 256 MiB per agent, and query-log batches older than 24 hours are dropped. The server list shows Replaying … backlog while a backlog drains and Spool trimmed if the cap forced a drop.

Every change is audited in the same transaction

Each mutation writes an append-only audit row in the same database transaction as the change, before the response returns. An audit gap and a rolled-back write are therefore the same event: you can’t get one without the other. Three more guarantees sit on top:

• The log is hash-chained: each row stores the previous row’s hash. A nightly job verifies the rows added since its last run and re-walks the whole chain about once a week, so an edit to an old row can take about eight days to surface. A break raises the critical audit-chain-broken alert, and the Audit Log page shows Chain verified or Tampering detected.
• Database triggers refuse DELETE and UPDATE on the table, even for superadmins.
• Forwarding to a SIEM happens after the commit and is best-effort, so a dead collector never rolls back a change.

The data model at a glance

DNS is group-centric: configuration lives on the group · Adapted from the SpatiumDDI documentation (Apache-2.0)

• IPAM is a strict tree: IP space, then blocks (which can nest), then subnets, then addresses. Deleting a space or block moves everything under it (blocks, subnets and their DHCP scopes) to Trash as one batch, and one restore brings it all back.
• DNS and DHCP are group-centric. Zones, views and options live on the DNS server group; scopes, pools, reservations and client classes live on the DHCP server group. Servers are members that render the group’s config, which is why a DNS group runs a single engine. DHCP leases are per server, because each Kea keeps its own lease file.
• Settings inherit. Unless you override them, a subnet takes its DNS and DHCP group bindings and DDNS settings from the block and space above it. At Ridgeline, the default groups and corp.example.edu are pinned on 10.20.0.0/16 and flow down to Staff VLAN 21, which is also why saving the subnet created its reverse zone, 21.20.10.in-addr.arpa, on its own.
• Deletes are soft. Spaces, blocks, subnets, zones, records and scopes go to Trash for 30 days, and a cascade comes back in one restore. A single deleted IP address is the exception: it becomes an orphan row instead.

Knowledge check

Knowledge check · 6 questions

How SpatiumDDI works

1. A group has two BIND9 agents and one of them is offline when an operator changes a record. What happens to the offline server's copy of the change?

The change is refused for the whole group until every server is backIts op waits as pending and is applied when the server returns; the online server applies it straight awayThe online server transfers the zone to it by AXFR when it returnsIt misses the change until someone presses Sync with Servers

Every enabled agent-based server in the group gets its own DNSRecordOp, and op state is per server. There is no primary-to-secondary push between agents: the offline server drains its own queue when it comes back.

2. Redis goes down. What happens to config delivery to the agents?

Agents stop receiving changes until Redis returnsChanges still arrive, more slowly: the long-poll falls back to plain polling, and DNS bundles wait for the api to render them itselfAgents stop polling and serve from cache onlyThe api refuses every write until Redis returns

The Redis wake is advisory, never the only delivery path. Without it the long-poll degrades to a 2-second poll, so DHCP changes still land within about 2 seconds. DNS bundles are rendered by the worker, whose Celery broker is Redis, so with Redis down the api builds the DNS bundle itself once the stored one has waited 120 seconds for the worker. Slower convergence, not lost changes.

3. An agent receives a bundle that renders a config named rejects. What does the agent do?

Retries the same bundle on every poll until an operator fixes itReverts to the last bundle that applied cleanly, quarantines the failed ETag and reports the rejectionStops named so clients fail over to another serverDeletes its cache and re-registers with the bootstrap key

previous.json holds the last bundle that applied cleanly. The failed ETag is quarantined and retried on a 60 s → 5 min → 15 min ladder, and the verdict rides the heartbeat to a chip, a banner and the agent_config_rejected alert. A reverted agent keeps serving, which is why the report matters.

4. Why does a SpatiumDDI install run exactly one beat?

It holds the database connection poolIt only enqueues periodic tasks, and two beats would enqueue every task twiceIt serves the agents' config long-pollIt runs the Redis Sentinel election

Beat schedules and workers execute. Two beats would double-fire every scheduled task. The project chose not to add leader election because every task is idempotent and the broker is already HA.

5. Which statement about the audit log is true?

A background job writes audit rows a few seconds after each changeA change and its audit row commit in the same transaction, so a change with no audit row can't existSuperadmins can delete audit rows from the Audit Log pageIf the SIEM collector is down, the change is rolled back

The audit row joins the mutation's own transaction. The table is append-only (database triggers refuse DELETE and UPDATE) and hash-chained. Forwarding runs after the commit and is best-effort, so a dead collector never undoes a change.

6. The control plane is unreachable for an hour. Which of these keep working?

Select all that apply.

DNS answers from agent-managed BIND9 serversDHCP leases from Kea, including failover inside an HA pairCreating a new DNS record in the UIAn agent container that restarts and serves again from its cache

Agents cache their last-known-good bundle, and with the control plane unreachable a restarting agent applies that cache and serves; a Kea HA pair runs peer to peer. Any change needs the api, so the new record has to wait.

Check answers

Go deeper

• System architecture: the control plane, config delivery and the HA design
• DNS agent and container architecture: bootstrap, stored bundles, the config cache, revert and quarantine, the outage spool, per-server fan-out
• DNS driver specification: how each engine applies changes
• Data model: the schema, grouped by domain
• Observability: the audit log, forwarding and health endpoints

Finished 3.1 How it works?

Mark it complete to track your progress (stored only in this browser).

Mark complete