Troubleshooting playbook

When something breaks, resist the urge to click Sync or restart a container straight away. First work out which layer is failing; the fix is usually obvious once you know. This page gives you a method, two interactive diagnostics for the most common complaints, a playbook of entries that all follow the same shape (Symptom → Likely causes → Check → Fix → Prevent), and a checklist for upgrading from 2026.09.04-1.

Commands assume Docker Compose. In the Academy lab, run docker compose commands from inside the lab/ folder, and remember that the lab’s BIND9 answers on 127.0.0.1 port 1053 and the web UI and API answer on http://localhost:8077. API examples use $SPATIUM for that base URL and $TOKEN for an API token, as module 2.3 does.

How to troubleshoot SpatiumDDI

Split the problem into four layers

SpatiumDDI itself never answers a DNS query or hands out a lease. The control plane decides what the configuration should be, an agent next to each daemon applies it, the daemon serves it, and clients consume it. Most confusing problems come from looking for the fault in the wrong layer.

Layer	What it is	Typical symptoms	Look here first
Control plane	api, worker, beat, frontend, Postgres, Redis	Errors in the UI, 5xx responses, scheduled jobs not running	/health/ready, /health/platform, docker compose logs api worker
Agent	The sidecar in dns-bind9 or dhcp-kea that registers, long-polls and renders config	Server shows unreachable or pending, config-rejected chip, changes never arrive	Server status and chips, the server detail, docker compose logs dns-bind9
Daemon	BIND9, PowerDNS, Technitium or Kea	Wrong or missing answers while the UI looks correct	dig against the server, the rendered config, the Logs page
Client and network	Resolvers, DHCP clients, relays, firewalls, VLANs	One client or one segment fails while others work	dig from the client, tcpdump on the server

A good order is outside-in: reproduce the problem at the client, ask the daemon directly, then look at the agent, and only then at the control plane.

Why DNS and DHCP keep working when the control plane doesn't

Each agent caches the last configuration that applied successfully and keeps serving from it, whether the control plane is down or has sent a config the daemon rejects. It also spools the logs, metrics and lease events it couldn’t deliver and replays them on reconnect (the server row shows Replaying … backlog). The upside is resilience. The downside is that a stale or rejected config looks healthy from the outside: the agent keeps heartbeating, so check the server’s config-apply chip and the Sync tab, not only its status dot.

Where to look

• Status chips and dots. Server rows in DNS and DHCP groups show the server’s status, a config-rejected chip when the agent had to revert, and the HA state for Kea pairs. In IPAM, the address table’s DNS column shows whether each IP’s records are in sync, and the Seen dot shows when the address was last observed.
• The server detail. For a DNS server: Overview (status, last heartbeat, config ETag, pause), Zones (serial per zone), Sync (pending, applied and failed record operations) and Events. BIND9 servers add Logs, Stats and Config, which shows the rendered named.conf and zone files to superadmins.
• The Logs page. DNS Queries (once query logging is on), DHCP Activity from Kea, and the Event Log and DHCP Audit tabs for Windows servers. The server picker reads one server, a whole group or All servers. Rows are kept for 24 hours by default (Settings → Data Retention).
• Platform Insights and Diagnostics. Platform Insights shows Postgres, Redis and containers. Diagnostics lists captured errors from the api, worker and beat, and builds a scrubbed support bundle.
• Container logs. docker compose logs -f api worker beat for the control plane; docker compose logs -f dns-bind9 or dhcp-kea for an agent and its daemon.
• Health endpoints. curl -s http://localhost:8077/health/ready answers “is the API ready to serve”, including whether the database schema is at head. /health/platform gives the per-component rollup to a signed-in caller or an API token (-H "Authorization: Bearer $TOKEN"); anonymous callers get only the overall status.
• The audit log. Administration → Audit Log shows who changed what, and when, which is often the fastest answer to “it worked yesterday”. Refused requests are there too, as denied rows.

From the field

Don’t wipe an install because an error looks like corruption. A Discord user saw an IntegrityError in Diagnostics after re-creating a DHCP scope and nearly wiped the install. Nothing was corrupt: a soft-deleted scope still held its unique slot (fixed in 2026.07.03-1). A UniqueViolationError in #564 also “read like a corrupted DB” and was a race between writers. Take a backup, and ask on GitHub or Discord before you reset anything.

Source: GitHub issue #474

Interactive diagnostics

A DNS record doesn’t resolve

Interactive · Why doesn't my DNS record resolve?

Answer a few questions about the record you expected to see. Each result names the likely cause and a concrete fix. The questions build on the official Troubleshooting the first IP checklist.

A DHCP client doesn’t get the right lease

Interactive · Why doesn't this client get the right lease?

Walk through the checks from the official Kea doesn't respond to DISCOVER recipe and the reservation checklist. You need a shell on the DHCP host for some of them. In the Academy lab, Kea runs in a bridged container, so it can't hear real LAN clients; use the lab to check configuration.

Playbook

The DNS record never appeared

Symptom. You allocated an IP with a hostname, but no A or PTR record shows up in the zone, and dig @127.0.0.1 -p 1053 print-01.corp.example.edu returns NXDOMAIN.

Likely causes. The official checklist lists six, in this order: no primary DNS zone on the subnet, a zone that isn’t on a server group, a group with no enabled server, an unhealthy server, and, for Windows, a zone that only accepts secure dynamic updates (Path A) or a server with no WinRM credentials (Path B).

Check. Work down the table and stop at the first row that fails.

#	Check	Fix
1	Open the subnet: is a primary DNS zone set or inherited?	Set it on the subnet, or on the block and let the subnet inherit
2	Open the zone: which server group is it on?	Put it on the group whose servers should serve it
3	Open the group’s server list: is at least one server enabled?	Enable a server, or add one
4	Click Sync with Servers and read the per-server status	Fix what it reports: network path, service down, credentials
5	Windows Path A: is the zone’s dynamic-update setting “Secure only”?	Set it to “Nonsecure and secure” in DNS Manager
6	Windows Path B: are WinRM credentials stored on the server?	Store them, and apply the fix from row 5 as well

Fix. Once the cause is fixed, open the subnet’s Sync ▾ → DNS report. It lists exactly what is missing and lets you apply it. Also check the IP itself: it needs both a hostname and a DNS zone.

Prevent. Pin the DNS group and zone on the block, watch the DNS column in the address table, and once things are stable turn on IPAM → DNS Reconciliation in Settings (off by default, additive only).

Sources: Getting Started, troubleshooting the first IP · DNS features §14

The zone isn’t served

Symptom. The zone exists in SpatiumDDI, but dig @127.0.0.1 -p 1053 corp.example.edu SOA returns no SOA (NXDOMAIN, REFUSED or no answer).

Likely causes.

• The zone’s group has no enabled, healthy server (see the entry above).
• The agent rejected the new configuration and went back to its last good config. The server keeps serving and heartbeating, so only the config-rejected chip gives it away.
• The group uses views and your query matches a view that doesn’t carry the zone, or no view at all. Once any view exists, a client that matches no view gets no answer.
• On Windows without WinRM credentials, SpatiumDDI can’t create zones; the zone exists only in SpatiumDDI.
• The agent image is older than the control plane, so a fix from the release notes hasn’t reached the daemon.

Check.

1. Query the SOA directly: dig @127.0.0.1 -p 1053 corp.example.edu SOA +short.
2. In the group’s server list, look for an unreachable status or a config-rejected chip.
3. Open the server detail: the Zones tab shows each zone’s serial on that server, and superadmins can read the rendered zone list under Config.
4. Read the agent’s log: docker compose logs --tail=100 dns-bind9.

Fix. Correct whatever the rejected-config chip or the agent log names; the agent retries a rejected config on a 60-second, 5-minute, 15-minute ladder, and at once when the control plane serves a new config. Add a catch-all view matching any if you use views. For Windows, create the zone in DNS Manager, then click Sync with Servers. After an upgrade, pull and recreate the agent containers.

Prevent. Keep a catch-all view in every group that has views, enable the agent_config_rejected alert rule, and upgrade agents together with the control plane.

Sources: DNS agent §3, last-known-good revert · DNS features §8 and §13

The PTR record is missing

Symptom. The forward A record resolves, but dig @127.0.0.1 -p 1053 -x 10.20.21.10 returns NXDOMAIN.

Likely causes. A subnet whose DNS group or zone is set or inherited normally gets its reverse zone (one per /24 it spans) the moment you save it, and its addresses’ PTRs, the gateway’s included, go there. When a PTR is missing, it’s usually one of these:

• The subnet was created through the API with skip_reverse_zone, which is remembered on the subnet.
• The DNS group runs a cloud driver without Auto-create reverse zones at the provider. The subnet’s DNS sync report then says “No reverse zone is created for this subnet…”.
• The group already holds a secondary, stub or forward zone with the reverse zone’s name. SpatiumDDI never writes PTRs into a zone another server owns.
• Two zones cover the address, such as a hand-made 20.10.in-addr.arpa and the auto-created 21.20.10.in-addr.arpa. The more specific zone answers, so the PTR must be in it; SpatiumDDI judges each address against the most specific reverse zone in the group.
• Overlapping IP spaces share one DNS group. A second space’s subnet can’t attach to a reverse zone another space created, so its IPs get no PTR (logged as reverse_zone_cross_space_conflict).

Check.

1. Look in the DNS group’s Zones tab for a zone that covers the address. At Ridgeline that’s 21.20.10.in-addr.arpa, created with Staff VLAN 21, where the PTR for 10.20.21.10 is the record 10.
2. Open the subnet’s Sync ▾ → DNS report and see which PTR records it lists as missing.
3. Query the reverse zone’s SOA: dig @127.0.0.1 -p 1053 21.20.10.in-addr.arpa SOA +short.

Fix. Open Sync ▾ → DNS on the subnet. Its first step creates the subnet’s missing reverse zones, and the report then lists the missing PTRs for you to apply. Where SpatiumDDI won’t create the zone, decide where it belongs: turn on Auto-create reverse zones at the provider on the cloud group, or create it yourself in the group’s Zones tab with Add Zone (kind Reverse lookup). For overlapping spaces, give each space its own DNS server group.

Prevent. Pin DNS on the block so every subnet inherits it, use skip_reverse_zone only on purpose, and run Sync ▾ → DNS on subnets you created on an older release.

Sources: DNS features §4 (#844, #1149, #1840) · Getting Started §4 and §8 · Verified in the Academy lab, 1.0.0-beta.3

Kea doesn’t answer DHCPDISCOVER

Symptom. A client on the same layer-2 segment as Kea gets no lease. A capture shows the DHCPDISCOVER going out and no DHCPOFFER coming back, although the Kea server has a green “healthy” chip.

Likely causes. The healthy chip only means the Kea process is up. There are exactly two ways a DISCOVER ends in silence:

• Class A: the DISCOVER never reaches Kea’s socket, because the group is Relay-only, the host firewall drops it, or the container isn’t on the host’s network.
• Class B: Kea hears it but drops it, because no active IPv4 scope attached to this server’s group contains the IP of the interface the packet arrived on, or the matching scope has no dynamic pool. For a first setup, class B is by far the most common.

Check. Split the two classes in about two minutes, on the DHCP host:

# 1. Does the broadcast reach the host NIC? (then renew the client's lease)
sudo tcpdump -ni any -v 'port 67 or port 68'

# 2. Watch Kea react
docker compose logs -f dhcp-kea

# 3. What is Kea actually running?
docker compose exec dhcp-kea \
    cat /var/lib/spatium-dhcp-agent/rendered/kea-dhcp4.json | grep -A4 subnet4
docker compose exec dhcp-kea \
    cat /var/lib/spatium-dhcp-agent/rendered/kea-dhcp4.json | grep socket-type

tcpdump shows the DISCOVER	Kea log	Meaning
Yes	DISCOVER logged, no OFFER (often DHCP4_SUBNET_SELECTION_FAILED)	Class B: scope or subnet mismatch
Yes	Nothing	Class A: the packet reaches the host but not Kea
No	—	The broadcast never reaches the host (VLAN or port group)

Fix.

• Class B: run ip -4 addr and note the host’s address on the client-facing interface. Make sure a scope exists whose subnet contains that address, that it is active and IPv4, that it has a dynamic pool, and that it is attached to the same DHCP server group as this Kea. An empty "subnet4": [] proves no scope is reaching Kea.
• Class A: if the rendered dhcp-socket-type is udp on a directly attached LAN, set the group’s Client reachability to Directly attached/mixed. On Docker, run Kea with host networking. On an appliance, sudo nft list chain inet filter input | grep -E 'dport (67|68|547)' should show UDP 67 and 68 accepted, and 547 for DHCPv6; if not, re-save the DHCP role assignment in Fleet.
• Relayed DHCPv6 gets no answer: a relay sends to the server’s global IPv6 address, so kea-dhcp6 needs a unicast socket there. ss -ulpn6 | grep 547 should show the global address, not only fe80::… and ff02::1:2. The agent adds one for every stable global address on its next sync loop (about 30 seconds) and logs dhcp6_unicast_addresses_changed; a temporary, deprecated or tentative address gets none.

Prevent. Create the scope before you point clients at the server, and set Client reachability when you create the group.

In the Academy lab

The lab’s Kea container is on a bridged Docker network with UDP 67 published on host port 6767, so it can’t hear DHCP broadcasts from real clients. Use the lab to check scopes, reservations and the rendered config; serving a real LAN needs host networking.

Sources: Troubleshooting, Kea doesn’t respond to DISCOVER (#1139, #1140) · DHCP features §1 · Docker Compose deployment §10

A reservation isn’t applied

Symptom. A client with a reservation gets a dynamic address instead, or the reservation you expected isn’t in the Static Assignments tab at all.

Likely causes.

• The reservation was never created. A blank MAC is rejected with a 422, so nothing is saved. A MAC with no scope on the subnet, or with several matching scopes, saves only the IPAM row, and so does a user without write on dhcp_static; the response carries a dhcp_static_warning saying which.
• The MAC was entered Cisco-style (aabb.ccdd.eeff) or without separators, which older releases couldn’t match.
• The scope is inactive, so it and all its reservations are left out of Kea’s config.
• A reserved IP outside the scope’s subnet (from an old or imported row) makes Kea reject the whole config; the agent keeps serving the previous one, so no reservation on that server updates.
• The scope’s server group has no Kea member, or the scope is a SLAAC-only IPv6 scope, which never has reservations.

Check.

1. DHCP → the server group → Static Assignments: is the reservation there, with the right MAC and IP?
2. The same list through the API: GET /api/v1/dhcp/scopes/<scope-id>/statics.
3. The rendered config: reservations sit inside each subnet4 entry, not in a top-level list. Read kea-dhcp4.json and find the reservations array in the right subnet.
4. A config-rejected chip on the Kea server row.

Fix. Create the reservation from Static Assignments → New static assignment, or allocate the IP with status static_dhcp and the MAC; SpatiumDDI puts it on the subnet’s scope. If the IPAM row exists without its reservation, fix what the warning named and save the row again. Enter MACs as aa:bb:cc:dd:ee:ff, activate the scope, and keep every reserved IP inside the subnet’s CIDR. A MAC can be reserved only once per server group.

Prevent. Create exactly one scope per subnet before you reserve anything, standardize on colon-form MACs, and give reservation authors the DHCP Editor role (or write on dhcp_static).

Sources: DHCP features §3 and §16 · GitHub issue #472 · GitHub issue #476 · GitHub issue #1628

Clients get a lease but no default gateway

Symptom. Clients get an address but no default route. In the lab, ./lab/check.sh 4 reports routers=—.

Likely causes.

• The scope’s Routers (option 3) was cleared, or never set because the scope came from the API or an importer without it. Create Scope pre-fills it from the subnet’s gateway.
• The subnet has no gateway set, so there was nothing to pre-fill.
• The subnet is IPv6. Option 3 is DHCPv4-only; IPv6 hosts learn their default router from router advertisements.
• The scope is right but the agent hasn’t applied it: look for a config-rejected chip, or an agent image older than the control plane.

Check. The rendered Kea config shows whether a router reached Kea:

docker compose exec dhcp-kea \
    cat /var/lib/spatium-dhcp-agent/rendered/kea-dhcp4.json | grep -B2 -A2 routers

No match means no scope on that server has a router. With several scopes, check that a match sits in the right subnet’s entry.

Fix. Edit the scope, set Routers (option 3) (10.20.21.1 for Staff VLAN 21), and save. The Kea agent picks up the change through its config long-poll within seconds; nothing needs pushing by hand.

Prevent. Check Routers (option 3) when you create a scope, and confirm it in the rendered config after you save.

Sources: Getting Started §9 · GitHub issue #1154 · Verified in the Academy lab, 1.0.0-beta.3

An agent shows unreachable or pending

Symptom. A DNS or DHCP server row turns red (“unreachable”), a newly registered server sits waiting for approval, or a server looks healthy but never picks up changes.

Likely causes.

• Unreachable: the container has stopped or is crash-looping; the agent can’t reach its control-plane URL (wrong URL, name resolution, firewall); TLS verification fails against a control plane with a private or self-signed certificate (remote agents verify by default); or the bootstrap key was rotated, so re-registration is refused. A server is marked unreachable after three missed 30-second heartbeats.
• Pending: the control plane runs with DNS_REQUIRE_AGENT_APPROVAL=true (or DHCP_REQUIRE_AGENT_APPROVAL=true), so new agents wait for approval. An agent that re-registers with a changed identity fingerprint also goes back to pending, as an anti-hijack measure.
• Healthy but stuck: two agent processes share one identity, such as a cloned VM or two containers on one state directory. The control plane serves only the newest process and answers the other’s config polls with 409, so it keeps serving a configuration that never changes. The default-on dns_agent_duplicate_instance and dhcp_agent_duplicate_instance alerts fire after two minutes, and a DHCP server shows a Duplicate agent chip.

Check.

docker compose ps
docker compose logs --tail=100 dns-bind9      # or dhcp-kea
curl -s <control-plane-url>/health/live       # from the agent's host

Then open the server detail’s Overview for the last heartbeat and the approval state (Pending approval on a DNS server), and compare the agent’s key with Settings → Security → Agent bootstrap keys.

Fix. Approve a pending DHCP agent with Approve on its server page (superadmin). A DNS server shows Pending approval but has no button yet: approve it with POST /api/v1/dns/groups/<group-id>/servers/<server-id>/approve (superadmin). Correct the control-plane URL or DNS. For a private CA, mount the CA bundle into the agent and set TLS_CA_PATH; SPATIUM_INSECURE_SKIP_TLS_VERIFY=1 is for labs only. After a key rotation, update DNS_AGENT_KEY (or SPATIUM_AGENT_KEY for DHCP) on the agent and restart it. For a duplicate, stop the extra process; a clone meant to be a server of its own needs a fresh state directory.

Prevent. Keep the bootstrap keys in your secret store, give agents a stable control-plane address, turn on agent approval in production, and never clone an agent VM with its state.

Sources: DNS agent §2 and §4 (#1121, #2044) · Docker Compose deployment §10 (#1220) · Observability §9.1 (#3143)

An appliance never shows up in Fleet

Symptom. An appliance you paired with a code never appears in Fleet, or its console’s Pairing row reads “Pair failed — regenerate code on control plane”.

Likely causes.

• On a Helm or other generic Kubernetes control plane, appliance registration is off by default, so supervisor registration returns 404 and the supervisor idles.
• The code expired. A single-use code lasts 15 minutes by default, a persistent code 30 days.
• Registration is throttled: ten wrong codes from one address, or a hundred across the install, in 15 minutes refuse further attempts with 429 until the window passes.

Check. On the control plane, look at the Appliance registration switch under Appliance → Fleet → Pairing codes, and at the code’s expiry. Revealing the keys under Settings → Security → Agent bootstrap keys (after re-entering your password, or a TOTP code for SSO users) shows whether each key is configured; a DNS or DHCP role assigned while its key is missing is held back and reported.

Fix. Enable appliance registration before the first pairing, or set supervisor_registration_enabled: true with PUT /api/v1/settings. Mint a fresh code, or use the bootstrap key for re-installs and air-gapped sites.

Prevent. Set both agent keys when you install the control plane, and enable registration on Helm installs before you hand out codes.

Sources: GitHub issue #407 · OS appliance §9 (#1062, #1356)

You deleted a DNS or DHCP server by mistake

Symptom. You deleted a server row that turned out to be a real, running BIND9 or Kea container.

Likely causes. Deleting a server only removes its database row; the container keeps running with its cached token and agent ID.

Check. Wait one heartbeat (about 30 seconds) and look at the group’s server list. The agent treats the resulting 401 or 404 as “my registration is gone” and registers again with its bootstrap key.

Fix. Usually nothing: the row comes back by itself. If the bootstrap key was rotated, update it on the agent and restart. If recovery is stuck, remove only the two identity files; the config cache stays, so the daemon keeps serving:

docker compose exec dns-bind9 sh -c \
    'rm -f /var/lib/spatium-dns-agent/agent_token.jwt /var/lib/spatium-dns-agent/agent-id'
docker compose restart dns-bind9

For DHCP, use dhcp-kea and /var/lib/spatium-dhcp-agent/. Settings you had edited only in the UI (notes, Path B credentials, per-server overrides, rotated TSIG keys) don’t come back and must be applied again. Zones, records, scopes and pools are unaffected, because they live on the group and the subnet.

Prevent. To take a server out of service, pause it (per-server maintenance mode) instead of deleting it.

Sources: Troubleshooting, recovering a deleted server · GitHub issue #182

You’re locked out of the admin account

Symptom. Nobody knows the admin password, the account is locked after failed sign-ins, or the only superadmin can’t get past the second factor. (If sign-in answers 429, see the next entry.)

Likely causes. A forgotten password; account lockout (only if a threshold was set, and only for local users); a lost authenticator; or an SSO outage with no working local account.

Check. A locked account is refused before its password is even checked. Ask another superadmin to look at Administration → Users, where a locked account shows a locked badge.

Fix.

• Another superadmin can use the unlock (open padlock) or reset-password (key) action on the user’s row in Administration → Users. Resetting a superadmin’s password asks the superadmin doing it for their own password (or authenticator code) first.
• For a lost authenticator, sign in with a recovery code (Use a recovery code instead on the two-factor screen).
• With no superadmin left, reset the password from the CLI. In the Academy lab, ./lab/reset-admin.sh puts admin back to admin with a forced change. On any Compose install:

docker compose exec -T api python - <<'EOF'
import asyncio
from sqlalchemy import update
from app.core.security import hash_password
from app.db import AsyncSessionLocal
from app.models.auth import User
async def reset():
    async with AsyncSessionLocal() as db:
        await db.execute(update(User).where(User.username == "admin")
            .values(hashed_password=hash_password("<new-password>"), force_password_change=True,
                    failed_login_count=0, failed_login_locked_until=None))
        await db.commit()
asyncio.run(reset())
EOF

Prevent. Keep at least two superadmins, store MFA recovery codes somewhere safe, and keep a working local admin for when the IdP is down.

Sources: Troubleshooting, resetting the admin password · Auth & permissions (#1355) · GitHub issue #71

Sign-in answers 429 for everyone

Symptom. Signing in fails with 429 “Too many login attempts. Try again shortly.”, for one username or for everyone, a superadmin included.

Likely causes.

• A sign-in throttle tripped. /auth/login allows 30 attempts a minute per source address, 20 failed passwords per username in 15 minutes (LOGIN_FAIL_MAX_PER_USERNAME) and 500 failed passwords across the whole install in 15 minutes (LOGIN_FAIL_MAX_GLOBAL). A refusal lasts until its window ends.
• You run your own reverse proxy in front of SpatiumDDI and it isn’t listed in FRONTEND_TRUSTED_PROXY_CIDRS, so every user arrives from the proxy’s address and shares one per-source budget. If you narrowed the api’s TRUSTED_PROXY_CIDRS, it may no longer cover the frontend.
• Someone is guessing passwords. The per-username and install-wide budgets are a bounded denial of service by design: they stop guessing, at the price of refusing the right password too until the window ends.

Check. In Administration → Audit Log, each refusal is a login row with result failure and reason login_throttled. Look at the source IPs of recent login rows: they must be your real clients, not the proxy. The 429 carries Retry-After with the seconds left in the window.

Fix. Wait out the window (15 minutes at most), or end it early by deleting the counter in Redis:

# Install-wide budget
docker compose exec redis redis-cli DEL login_fail:global
# One username's budget: the key ends in the first 32 hex characters of
# the SHA-256 of the lower-cased name (on macOS use: shasum -a 256)
docker compose exec redis redis-cli DEL \
    "login_fail:user:$(printf '%s' 'admin' | sha256sum | cut -c1-32)"

Behind your own proxy, set FRONTEND_TRUSTED_PROXY_CIDRS to the address the proxy connects from. Existing sessions, API tokens and OIDC or SAML sign-in keep working throughout.

Prevent. Set FRONTEND_TRUSTED_PROXY_CIDRS when you put a proxy in front, keep an API token for scripts rather than scripted password sign-ins, and raise LOGIN_FAIL_MAX_GLOBAL, or set it to 0, only if you’d rather allow more password guessing than risk an install-wide refusal.

Sources: Auth & permissions, sign-in throttles · Running behind a reverse proxy, troubleshooting · Docker Compose deployment §3

A subnet delete is refused

Symptom. DELETE /api/v1/ipam/subnets/<id>?permanent=true returns 409 Conflict with “Subnet is not empty: N allocated IP addresses, M DHCP scopes…”.

Likely causes. This is deliberate. A permanent delete of a non-empty subnet used to cascade silently and wipe IP rows and DHCP scopes out from under running servers. The normal delete (without permanent) is a soft delete that never refuses.

Check. Count the subnet’s own IPs and scopes. The network and broadcast placeholder rows, the untouched “Gateway” row and lease-mirrored rows don’t count as blockers. A permanent delete is superadmin-only.

Fix. Either empty the subnet first (release non-system IPs, detach DHCP scopes) and retry, or add ?force=true to cascade. The same cleanup still runs: agentless Windows DHCP gets a WinRM remove-scope and Kea gets a config update, so no lease is left orphaned. If you only wanted it gone for now, use the normal delete and restore it from Administration → Trash within 30 days.

Prevent. Use the soft delete by default; keep permanent and force for scripted cleanups you have reviewed.

Sources: Troubleshooting, subnet delete is refused · IPAM features §16

Names stop resolving after you trash a subnet

Symptom. You moved a subnet to Trash, and dig now returns NXDOMAIN for the hosts that were in it, forward and reverse.

Likely causes. This is expected. Moving a subnet, or its block or space, to Trash withdraws the A, AAAA and PTR records SpatiumDDI published for its addresses, the gateway’s PTR included, and trashes them in the same batch. A reverse zone created for the subnet goes to Trash with it unless another subnet still uses it. Records you created by hand stay where they are.

Check. Administration → Trash lists the subnet’s batch. In the lab, trashing Scratch with scratch-01 in it listed five rows: the subnet, 99.20.10.in-addr.arpa., the A and PTR of scratch-01, and the gateway’s PTR. Both names answered NXDOMAIN within two seconds.

Fix. If the subnet should come back, Restore it: the confirmation says how many rows return, and the names resolve again within seconds. Then click Refresh IP spaces in IPAM, because the tree doesn’t show a restored subnet until you refresh or reload. A restore that answers 409 names a record someone created by hand while the subnet sat in Trash, or a parent block that is still in Trash; deal with that row first and restore again.

Prevent. Treat Trash as “out of service, recoverable for 30 days”: names go dark at once. Before you trash a subnet, make sure nothing still depends on its hosts’ names.

Sources: Verified in the Academy lab, 1.0.0-beta.3 · IPAM features §15.16 (#1906) · System administration, Trash

An IP address is owned by two integrations

Symptom. An address that two integrations both see (a UniFi client that is also an OPNsense DHCP lease, a VM seen by Proxmox and UniFi) stopped updating: its hostname or MAC is stale, or it stayed in IPAM after the host went away.

Likely causes. Before 2026.10.02-1, an integration mirror could claim an address another integration already owned, and the claim marked the row as edited by hand. From then on neither integration updates or removes it, and deleting either integration’s target deletes the row. New claims are now refused (“owned by another integration; not claiming”), but rows claimed before the upgrade stay as they are, because a claim can’t be told apart from a real operator edit.

Check. In a psql session on the SpatiumDDI database (on Compose, docker compose exec -T postgres psql -U spatiumddi spatiumddi), list the addresses with two or more ownership columns set:

SELECT id, address, hostname, user_modified_at
FROM ip_address
WHERE num_nonnulls(kubernetes_cluster_id, docker_host_id, proxmox_node_id,
        tailscale_tenant_id, unifi_controller_id, cloud_endpoint_id,
        opnsense_router_id, netbird_instance_id, panos_firewall_id,
        fortinet_firewall_id, meraki_org_id) >= 2
ORDER BY address;

Fix. Decide which integration should own each address (usually the one that hands it out, such as the DHCP server for a LAN lease) and clear the other’s column. If nobody has edited the row by hand, clear user_modified_at too, so the remaining integration keeps it current:

-- e.g. keep OPNsense, drop the UniFi claim
UPDATE ip_address
SET unifi_controller_id = NULL, user_modified_at = NULL
WHERE id = '<id from the query>';

Leave user_modified_at alone if someone did edit the row, or their hostname, description and MAC are overwritten on the next sync. Edits made in SQL bypass the audit log, so note what you changed.

Prevent. Upgrade, so new double claims are refused, and give each integration a clear area of ownership.

Sources: Troubleshooting, an IP address is owned by two integrations (#1135)

An integration took IP addresses from another IP space

Symptom. Hand-allocated addresses disappeared from one IP space after an integration (UniFi, Docker, Kubernetes, Proxmox, Tailscale, NetBird or Cloud) was pointed at another space covering the same range. Allocating a missing address again says it is already allocated, the address shows up in the integration’s space, and Sync ▾ → DNS on the original subnet may offer to delete many records.

Likely causes. Before 1.0.0-beta.2, those integrations looked addresses up across every IP space, adopted rows operators had allocated in another space, and moved them, with their DNS records, into their own. Upgrading stops it, and an integration releases what it owns outside its own space on its next sync, but nothing already moved is put back automatically.

Check. As a superadmin, run the read-only report (or ask the Operator Copilot to run find_cross_space_integration_adoptions):

curl -s -H "Authorization: Bearer $TOKEN" \
    "$SPATIUM/api/v1/ipam/integration-space-audit" | jq .

Rows marked likely_cross_space are the ones to repair. needs_review rows need their owner’s confirmation first. In home_candidates, the subnet with address_free: true in the space the address went missing from is almost always where it came from.

Fix. Don’t commit DNS deletions from Sync ▾ → DNS on an affected subnet until the rows are back: those records are what you’re restoring. For each row, confirm the home subnet with whoever owns the address, then move it back and release the claim, using the column for the integration the report names, and keep user_modified_at:

BEGIN;
UPDATE ip_address
SET subnet_id = '<home_candidates[].subnet_id>',
    unifi_controller_id = NULL
WHERE id = '<ip_address_id>';
COMMIT;

Then run Sync ▾ → DNS on the home subnet and confirm the address’s records are in sync. Rows the integration already deleted are gone; the upstream recipe has an audit-log query that lists them so you can re-create them.

Prevent. Upgrade first, and read an integration’s sync warnings after you change its IP space.

Sources: Troubleshooting, an integration took IP addresses from another IP space (#2577)

/health/platform says celery-beat is unhealthy

Symptom. The Platform Health card on the dashboard, or /health/platform called with a credential (curl -s -H "Authorization: Bearer $TOKEN" "$SPATIUM/health/platform"), shows celery-beat as error or warn.

Likely causes. Beat only schedules the heartbeat task; a worker runs it and writes the key the check reads. So “no tick landed” means either beat has stopped, or no worker is running tasks. celery-workers can stay green even when every worker slot is stuck.

Check.

# 1. Is beat scheduling? Expect a "Sending due task" line every 30 s.
docker compose logs --tail=20 beat | grep 'Sending due task'

# 2. Is a worker running it? Look for the matching beat_tick lines.
docker compose logs --tail=50 worker | grep beat_tick

# 3. Can the pool run anything? If "active" equals the concurrency, it can't.
docker compose exec worker celery -A app.celery_app inspect active --timeout 5

What you see	Where the fault is
No “Sending due task”	Beat has really stopped; check its container (on Kubernetes, its wait-for-migrate init container)
Sending, but no beat_tick succeeds	The worker isn’t consuming; check the pool (step 3) and Redis
Active tasks equal the concurrency	The pool is wedged, and every scheduled job has stopped, not only the heartbeat

Fix. Restart the stopped or wedged component. A warn that says “last tick … in the future (clock skew)” means two nodes disagree about the time: fix NTP on both rather than touching Celery.

Prevent. Keep NTP healthy on every node, and treat a red celery-beat as “scheduled jobs aren’t running”, because that is what it means.

Sources: Troubleshooting, celery-beat is unhealthy (#925) · Observability §7 (#2935)

After an upgrade, nothing changed on the servers

Symptom. You upgraded, but a fix from the release notes has no effect, and the rendered configs look exactly as before.

Likely causes. The agent containers weren’t pulled and recreated. Release notes since 2026.08.12-1 say operators must pull new agent images, because “a control-plane upgrade alone changes none of the rendered files”. Agents on separate hosts, deployed with the standalone agent compose files, are often forgotten. So are containers started with --profile on the command line: pull and up -d act only on active profiles.

Check. docker compose images shows which tag each container is running. Compare every agent host, not only the control plane.

Fix. Pull and recreate with the agent profiles active (the lab sets them in COMPOSE_PROFILES; otherwise pass the same --profile flags to both commands), then repeat on every agent host:

docker compose pull
docker compose up -d

On 1.0.0-beta.3, the first config apply after the agent upgrade reloads BIND9 and restarts PowerDNS once; that’s expected.

Prevent. Pin SPATIUMDDI_VERSION so every component moves to the same tag, set COMPOSE_PROFILES in .env, and make “pull agent images” a line on your upgrade checklist.

Sources: Release notes 1.0.0-beta.3 · GitHub issue #856 · Docker Compose deployment §8

After an upgrade, the API errors or never becomes ready

Symptom. The api container exits at start, the API returns 500s mentioning UndefinedColumnError, /health/ready returns 503, or on Kubernetes the api, worker and beat pods sit in their wait-for-migrate init containers.

Likely causes.

• SECRET_KEY is the .env.example placeholder, under 32 characters, or reads like a placeholder: the api refuses to boot on it (since 2026.10.02-1). A CREDENTIAL_ENCRYPTION_KEY that isn’t a valid Fernet key stops it too.
• The new code started before the migration ran, so it expects columns the database doesn’t have yet.
• The migration itself failed. Run while the old release still serves, it can stop with deadlock detected.
• The database is ahead of the code: an older release redeployed after an upgrade (its migrate step fails with Can't locate revision, and its api, worker and beat wait forever), a restore with allow_newer_schema=true, or an appliance slot rollback.

Check.

curl -s http://127.0.0.1:8000/health/ready     # on the Docker host: the API port is local-only
docker compose logs migrate
docker compose logs --tail=200 api | grep -i -E 'SECRET_KEY|UndefinedColumn'

On an appliance, kubectl -n spatium logs deploy/spatium-control-spatiumddi-api -c wait-for-migrate prints “The database was migrated by a NEWER SpatiumDDI release” when the database is ahead.

Fix.

• Placeholder key: follow Rotating SECRET_KEY in the Docker docs. It saves the old key, sets a new one, and re-encrypts every stored credential with python -m app.core.rotate_secret_key; then start the stack.
• Migration: run docker compose run --rm migrate followed by docker compose up -d. Re-running migrate is always safe, and each revision commits on its own, so after deadlock detected it resumes where it stopped (docker compose up -d also runs it first, and the api and worker wait for it). On Kubernetes, each Helm revision renders its own migrate Job; read its logs.
• Database ahead of the code: go forward, not back. Re-apply the newer release: on Compose or Helm, deploy the newer tag again; on an appliance, Schedule OS upgrade in Fleet with the same image, or POST /api/v1/appliance/slot-upgrade/apply. Restoring the pre-upgrade backup alongside the older release is the only other way.

Prevent. Always upgrade in the order pull, migrate, up; take a backup first; and pin SPATIUMDDI_VERSION so nothing redeploys an older tag.

From the field

A/B slot rollback is not a way back across a migration. /var and Postgres are shared by both appliance slots, Alembic only migrates forward, and readiness compares schema heads, so a rolled-back slot whose database is ahead never becomes ready. SpatiumDDI refuses such a rollback (409 schema_rollback_unsafe) unless you resend it with acknowledge_schema_rollback: true, and you should only do that if you’ll restore a pre-upgrade copy of the database. The trial-boot auto-revert isn’t checked.

Source: GitHub issue #782

Sources: GitHub issue #565 · Docker Compose deployment §3 and §8 (#1222) · Kubernetes deployment §4 · OS appliance, rolling back (#1227) · Verified in the Academy lab, 1.0.0-beta.3

An appliance upgrade won’t download or start

Symptom. A slot upgrade fails with Name or service not known, an image upload fails with 413, the other nodes of a cluster can’t find an image you uploaded, or the upgrade is refused before it starts.

Likely causes.

• The host fetches the image with its own resolver, from a URL built from the address your browser used to reach the UI. A name only your workstation can resolve fails on the appliance.
• SpatiumDDI refused the upgrade in its pre-flight: an image for the wrong architecture, not enough free space on /var, or an older release than the database can run on (409 schema_rollback_unsafe). The refusal says which.
• On 2026.07.21-1 or earlier, uploading a slot image through the Web UI returns 413 before the API sees it.
• An image uploaded while the appliance was still single-node sits on that node’s local disk, not the shared mirror.
• A URL pins a release tag but uses the unversioned asset name, which is pruned once a newer release ships (404).

Check. On the appliance host:

grep fetching /var/log/spatiumddi/slot-upgrade.log | tail -5   # redact the ?t= token before sharing
resolvectl query <hostname-from-that-url>
spatium-upgrade-slot status

Fix. Open the UI by IP address and schedule the upgrade again. Free space on /var, or pick the image for the node’s architecture. For the 413, use the URL source or the CLI over SSH:

sudo spatium-upgrade-slot apply <versioned-image-url> --checksum <versioned-sha256-url>
sudo spatium-upgrade-slot set-next-boot
sudo reboot

Re-upload images after a cluster forms, and always use the versioned asset names.

Prevent. Reach the UI by a name the appliance itself can resolve, and read the release notes’ upgrade-path section before you start.

Sources: GitHub issue #864 · OS appliance, update mechanism (#787, #789, #1026, #1227, #2400)

Pods stuck in ErrImageNeverPull after a slot rollback

Symptom. After rolling an appliance back to its previous slot, the spatium-supervisor pod and the DNS and DHCP agent pods on that node (and the control plane, on a control-plane node) sit in ErrImageNeverPull at the old tag.

Likely causes. k3s imports image tarballs from one directory on the shared /var, and its import cache already lists the newer slot’s tarballs there. After the rollback, the old slot’s k3s imports nothing, and every SpatiumDDI pod uses imagePullPolicy: Never, so it only starts if containerd still holds the old images.

Check. kubectl get pods -A shows the stuck pods; kubectl describe pod names an image that k3s crictl images doesn’t list.

Fix. As root on the affected node, on the slot you rolled back to:

mkdir -p /run/slotroot && mount --bind / /run/slotroot
ls /run/slotroot/var/lib/rancher/k3s/agent/images/*.tar.zst   # must list files
cp /run/slotroot/var/lib/rancher/k3s/agent/images/*.tar.zst \
   /var/lib/rancher/k3s/agent/images/
echo '{}' > /var/lib/rancher/k3s/agent/images/.cache.json
umount /run/slotroot
systemctl restart k3s

Wait 2–3 minutes while k3s re-imports, then check k3s crictl images | grep <old tag> and kubectl get pods -A. If the ls lists nothing (a slot the installer laid down), don’t copy the other slot’s images, which are the wrong version: apply that release’s slot image to the inactive slot with spatium-upgrade-slot, boot it, and switch back.

Prevent. Roll back only when you must, and remember that after a database migration a rollback is refused anyway.

Sources: Troubleshooting, pods stuck in ErrImageNeverPull (#3169)

Local backup archives disappear

Symptom. On Docker Compose, archives of a Local volume backup target vanish after the containers are recreated, or scheduled runs succeed but their archives never show in the target’s list.

Likely causes.

• You upgraded from 2026.09.04-1, whose docker-compose.yml left the spatium_backups volume commented out. Archives written before the upgrade lived in the api and worker containers’ writable layers, and the upgrade recreated the containers.
• The target’s path isn’t on a volume mounted on both the api and the worker. The worker writes scheduled runs and the api lists them, so each keeps its own copy, and every copy goes with the next recreate.

Check. Test connection on the target warns when its path isn’t on a mounted volume. Current compose files mount spatium_backups on both services at /var/lib/spatiumddi/backups, the path a new target suggests.

Fix. Point the target at /var/lib/spatiumddi/backups, or mount a volume at your own path on both services, then Run now and run a restore drill. Archives already lost with a container can’t be recovered; take a fresh backup.

Prevent. Before upgrading from 2026.09.04-1, copy the archives out with docker compose cp api:/var/lib/spatiumddi/backups ./backups-api (and the same for worker). Keep a copy off the host, or use a remote destination.

Sources: Docker Compose deployment §9 (#1160) · System administration §2.9

SSO sign-in fails

Symptom. After signing in at the identity provider, you land back on the SpatiumDDI login page with an error such as saml_state_missing, saml_requires_https or “OIDC login rejected: your account could not be provisioned…”, or an LDAP sign-in is refused with 401.

Likely causes.

• The External URL is empty, wrong, or http://. SAML with a hosted IdP needs HTTPS, because the IdP posts the assertion back from another site. Behind your own proxy, FRONTEND_TRUSTED_PROXY_CIDRS must list it, or callbacks are built as http:// (the api logs sso_external_url_not_set).
• The user’s IdP groups match no group mapping. There is no fallback group, so the login is rejected.
• Auto-create users on first login is off, so a first-time user is refused.
• username_collision: the username already belongs to another account, local or linked to another provider. SpatiumDDI never adopts an account by username.
• account_link_required: an account from before provider linking that the upgrade couldn’t attribute to one provider, or whose identifier changed at the IdP (an LDAP DN after an OU move). It shows an unlinked chip on the Users page.
• The SAML IdP sends a transient NameID, which changes on every sign-in and is refused.
• The callback URL registered at the IdP doesn’t match SpatiumDDI’s.

Check.

1. Settings → Branding & URL → External URL: behind a proxy that terminates TLS, it must be the public https:// address.
2. The provider’s test button (Probe discovery URL for OIDC, Probe metadata / settings for SAML).
3. The provider’s group mappings, against the groups the test user really has.
4. The audit log: every login attempt writes a login row, and a failed one records the reason, such as no_group_mapping_match, username_collision or account_link_required.

Fix. Set the External URL. Register https://<external-url>/api/v1/auth/<provider-id>/callback at an OIDC provider, or give a SAML IdP the SP metadata at /api/v1/auth/<provider-id>/metadata. Add the missing group mapping, turn on auto-create, or pre-create the account (New User with Signs in through set to the provider). For account_link_required, open Users → Edit → Sign-in provider, pick the provider and Link: the next sign-in through it claims the account, and its sessions end. For username_collision, decide whose name it is; a local account can’t be linked to a provider, so rename one of the two, or have the IdP send another username. Configure the IdP to release a persistent NameID.

Prevent. Set the External URL before you configure SSO, test with a dedicated test user, and keep a local admin that doesn’t depend on the IdP.

From the field

Every SAML login against a hosted IdP failed with saml_state_missing. The flow cookie was SameSite=Lax, so browsers didn’t send it with the IdP’s cross-site POST. Since 2026.08.22-1, an https:// External URL gets a cookie that survives the POST; an http:// deployment reports saml_requires_https instead.

Source: GitHub issue #873

Sources: Auth & permissions, provider linking, SAML and rules (#1235, #1291) · Getting Started §1 and §2 · Running behind a reverse proxy

The UI loads but the API returns 502 on Kubernetes

Symptom. On a Helm install, the web UI loads, but signing in and every API call return 502. The frontend pod’s nginx logs api could not be resolved.

Likely causes.

• A chart older than 2026.5.17-1 (release 2026.05.17-1): the frontend’s nginx was hard-wired to the Compose service name api and Docker’s DNS resolver.
• A non-default topology, such as the api in another namespace or under a custom Service name, without the matching override.
• The api pods aren’t ready yet, for example because they are still waiting for the migrate Job.

Check.

kubectl -n <namespace> get pods
kubectl -n <namespace> logs <frontend-pod> | grep -i resolve

Fix. Upgrade the chart. Since 2026.05.17-1, the frontend proxies to the release’s api Service and detects the cluster resolver at start-up. Only for a non-default topology, set frontend.apiUpstream.host, frontend.apiUpstream.port and frontend.nginxLocalResolvers. The workaround the original reporter used was a second Ingress that sends /api to the api Service.

Prevent. Keep the default topology where you can, and pin chart versions.

Sources: GitHub issue #176 · Kubernetes deployment §2

A Postgres standby is lost and never rejoins (CNPG)

Symptom. On a multi-node appliance or another CloudNativePG install, the cluster-health database row reads degraded (fewer ready instances than configured) and one standby never catches up; the default-on postgres_instance_unhealthy alert may fire.

Likely causes. With no WAL archive, WAL lives only in each instance’s pg_wal, and a replication slot may hold back at most 3 GB. A standby that falls further behind loses its slot (wal_status = lost) and can never catch up by streaming; a former primary whose pg_rewind needs WAL that’s gone loops the same way. CNPG retries both forever.

Check. Find the stuck instance with kubectl -n spatium get pods, and find the primary by its cnpg.io/instanceRole=primary label. The stuck instance must not be the primary.

Fix. Rebuild that one instance from the primary:

# <n> is the stuck instance's number, never the primary's
kubectl -n spatium cnpg destroy <cluster> <n>
# or, without the plugin: delete its PVC, then its pod
kubectl -n spatium delete pvc <cluster>-<n>
kubectl -n spatium delete pod <cluster>-<n>

CNPG joins a fresh copy from the primary. Do one instance at a time and wait for it to be Ready.

Prevent. Act on the postgres_instance_unhealthy alert early, and bring a node that was down for a long time back promptly.

Sources: Troubleshooting, a Postgres standby is lost (#2317) · Kubernetes deployment §6 · Observability §9.1 (#2856)

Imported or API-created records don’t resolve

Symptom. Records show in the UI and are even written to the zone file on the agent, but dig returns NXDOMAIN until you open each record and save it again.

Likely causes. An agent image older than the control plane. Older agents didn’t apply imported records, or API changes on groups with views, to the running zone (#707), and any agent that lags the control plane misses the fixes its release notes describe.

Check.

dig @127.0.0.1 -p 1053 <name>.corp.example.edu +short
docker compose exec dns-bind9 ls /var/lib/spatium-dns-agent/rendered/zones/
docker compose logs --tail=200 dns-bind9 | grep record_op_applied

Then compare the agent image tag with the control plane’s (docker compose images), and look at the server detail’s Sync tab for pending or failed operations.

Fix. Pull and recreate the agent containers so they run the control plane’s tag. On an old agent you can’t upgrade yet, re-saving a record queues an operation for it, which was the reporter’s workaround.

Prevent. Upgrade agents with every control-plane upgrade, and after an import, spot-check a few names with dig.

Sources: GitHub issue #707 · Release notes 2026.08.12-1

A page or API call returns 404

Symptom. An API call returns 404 for something that should exist, or a page or menu entry the docs describe isn’t there.

Likely causes. The feature module is switched off. A disabled module’s API returns 404, not 403, and its menu entries are hidden. A fresh install turns on only 16 of the 57 modules (IPAM, DNS, DHCP, sites, VLANs, VRFs and a few tools); Conformity, Windows cutover, nmap, packet capture, NAT, approvals and every integration ship off. Missing permissions can also hide menu entries and search results.

Check. Read the response body: a disabled module answers {"detail": "Feature '<module>' is disabled."}. Then look the module up under Features & Integrations. GET /api/v1/auth/me/permissions shows your effective permissions.

Fix. Ask a superadmin to enable the module, or to grant the permission. Turning a module off also stops its scheduled jobs, so turning it on starts them again.

Prevent. Keep a note of which modules your install has enabled, and check it before you debug a URL.

Sources: REST API §4 and §5 (#1069, #2279) · Verified in the Academy lab, 1.0.0-beta.3

The API docs ask you to sign in

Symptom. /api/docs (Swagger) or /api/redoc shows only “Sign in to SpatiumDDI to view the API documentation.” with a Sign in link, or /api/openapi.json returns 401.

Likely causes. This is by design: the schema names every route and the exact running version, so only signed-in users get it. The pages read it with the console’s sign-in cookie, so they fail when this browser has no live console session on the same host name (signed out, idle past the session timeout, or the console opened under another name), or when the account still has a forced password change pending.

Check. Open the console in the same browser, at the same address as the docs (http://localhost:8077 in the lab), and sign in.

Fix. Sign in, change the forced password if asked, then reload the docs page. From a script, fetch the schema with an API token: curl -s -H "Authorization: Bearer $TOKEN" "$SPATIUM/api/openapi.json". For code generation, use the openapi.json attached to the release, which needs no credential.

Prevent. Keep the release’s pinned openapi.json with your automation.

Sources: Verified in the Academy lab, 1.0.0-beta.3 · REST API §2 (#1157, #2935)

Alerts never arrive

Symptom. Alert events appear on the Alerts page, but nothing reaches your syslog server, webhook or mailbox.

Likely causes.

• The rule has no delivery channel ticked, or there is no enabled Audit Event Forwarding target of that kind.
• A Teams target uses the retired Office 365 “Incoming Webhook” connector (…webhook.office.com/webhookb2/…), which no longer delivers. A Workflows webhook accepts the post (202) before its flow runs, so Test succeeds even when the flow fails.
• The target’s Minimum Severity is above the alert’s severity. Critical alerts always pass.
• The rule is disabled, its condition hasn’t matched yet, or the event was resolved by hand: a manual Resolve stays in force while the subject keeps matching.

Check. On Administration → Alerts, confirm open events exist, then open the rule and check its Delivery channels. In Settings → Security → Audit Event Forwarding, check each target’s kind, status, minimum severity and resource-type filter. Evaluate now (superadmin) forces an evaluation pass. For Teams, read the flow’s run history in Power Automate.

Fix. Tick the channel, enable a target, or lower its threshold. For Teams, create the webhook with the Workflows app (template Send webhook alerts to a channel). Then confirm delivery with a real alert event, because Test sends an audit-shaped payload.

Prevent. After any change to rules or targets, trigger a real alert and watch it arrive.

Sources: Observability §5.1 and §9.1 (#1031, #1504, #2498)

Upgrading from 2026.09.04-1

An install still on 2026.09.04-1, as the first edition of this course was, crosses two big changes on its way to 1.0.0-beta.3: the version scheme moves from CalVer to SemVer, and several defaults tighten. Plan it as two upgrades, each with a backup, and read the “Before you upgrade” notes of 2026.10.02-1, 1.0.0-beta.1, 1.0.0-beta.2 and 1.0.0-beta.3 first.

Go through the bridge, and never go back

Upgrade to 2026.10.02-1, the bridge release, before anything newer. It’s the last CalVer release, and the version you upgrade from decides what’s newer: only the bridge and later know that 1.0.0 comes after 2026.x, so an older install is never offered 1.0.0 and its preflight refuses it. Once upgraded, don’t redeploy an older release: its migrate step fails with Can't locate revision, and its api, worker and beat wait forever. The way back is the backup you took before, restored alongside the older release.

On Docker Compose

1. Back up on 2026.09.04-1. Use Administration → Backup → Manual → Build + download and store the archive off the host. Then copy any Local volume archives out of the containers, because that release’s compose file kept them inside: docker compose cp api:/var/lib/spatiumddi/backups ./backups-api, and the same for worker.

2. Check SECRET_KEY in .env. If it’s the .env.example placeholder, under 32 characters, or reads like a placeholder, the bridge refuses to run on it, migrate included, and its error says what to do. Follow Rotating SECRET_KEY in the Docker docs at that point: set a new key, re-encrypt the stored credentials with python -m app.core.rotate_secret_key (it reads the old key from OLD_SECRET_KEY), then carry on with the hop. The Academy lab generated a random 64-character key, so it isn’t affected.

3. Refresh the compose file. Run git pull in your checkout, or take docker-compose.yml and .env.example from the release you’re installing. A compose file from 2026.09.04-1 or older pulls from ghcr.io/spatiumddi/, which now answers denied; the current one pulls from ghcr.io/spatiumnorth/. If your .env sets API_PORT=127.0.0.1:8000 to keep the API local, set it back to 8000: the API is published on 127.0.0.1 by default now (API_BIND).

4. Hop to the bridge. Set SPATIUMDDI_VERSION=2026.10.02-1 in .env, then:

   docker compose pull
   docker compose run --rm migrate
   docker compose up -d
   docker compose images          # every container on 2026.10.02-1

   Sign in and check that DNS and DHCP still serve before you go on. Take another backup.

5. Hop to 1.0.0-beta.3. Set SPATIUMDDI_VERSION=1.0.0-beta.3 and run the same commands. If migrate stops with deadlock detected (the old release’s traffic held a table it needed), run it again, or run docker compose up -d, which runs migrate first and makes the api and worker wait for it: each revision commits on its own, so it resumes where it stopped. Pass the same --profile flags to pull and up -d if you don’t use COMPOSE_PROFILES.

6. Finish the job. Rotate each DNS server group’s TSIG key once (Edit Group → Group TSIG key → Rotate…): it’s encrypted at rest since 1.0.0-beta.1, but every database copy and backup from before holds it in clear, and the key allows zone transfers and dynamic updates on every zone the group serves. Then work through the table below.

On an appliance

• Back up the Secret that holds SECRET_KEY before you leave 2026.09.04-1, and keep the file off the appliance: kubectl -n spatium get secret spatium-control-spatiumddi-app -o yaml > spatium-control-app-secret.yaml. That release’s chart doesn’t mark the Secret to be kept, so a failed first install on the new slot can mint a new key and leave every credential encrypted at rest unreadable. From the release after the bridge, the key lives in spatium-control-app-keys; back that one up from then on.
• Upgrade to the bridge first, then to 1.0.0-beta.3, with a backup before each.
• On a multi-node control plane, read #1445 before the bridge hop: the bridge’s built-in rolling upgrade doesn’t complete on a multi-node cluster.
• Expect each appliance’s SSH host key to change once, at 1.0.0-beta.1: every appliance now gets its own keys.
• A slot rollback onto a release that can’t run on the database is refused (409 schema_rollback_unsafe); if a node is stuck anyway, re-apply the newer release.

On Kubernetes (Helm)

• Hop to the bridge chart first, then to 1.0.0-beta.3. CalVer charts drop the leading zeros (the bridge is chart 2026.10.2-1) and always need an explicit --version.
• Upgrade once with --reset-then-reuse-values: --reuse-values keeps the old chart’s api and worker sizing.
• The DNS and DHCP agent Services now default to ClusterIP. Before you upgrade, set service: { type: LoadBalancer } (with the same address) on every server clients reach, or it releases its external IP. On k3s, disable servicelb, which would take the node’s port 53.
• Changing dnsAgents.storageClass or dhcpAgents.storageClass on an existing install needs kubectl delete sts <name> --cascade=orphan first; helm upgrade stops and names the command.

What changes under you

After the upgrade	Why	What to do
The api won’t start	Placeholder or weak SECRET_KEY refused (#1222)	Rotate it (step 2)
Prometheus gets 401 from /metrics	/metrics needs a bearer token (#1159)	Scrape with PROMETHEUS_METRICS_TOKEN or an API token
A remote agent stopped connecting	Agents verify TLS by default (#1220)	Mount the CA and set TLS_CA_PATH
Another host can’t reach port 8000	The API binds to 127.0.0.1 (#1221)	Use the web port, or set API_BIND and narrow TRUSTED_PROXY_CIDRS
Some recursive clients get REFUSED	Recursion limited to local and private networks (#1817)	Add their networks to allow-recursion
A custom role sees less of IPAM or DNS	Each route checks its own type (#3058, #3073)	Grant read on each IPAM and DNS type
Token-minting scripts get 403	Minting needs a step-up (#1355)	Send stepup_password or stepup_totp_code
Tooling reads "version": "restricted" or gets 401 from the schema	Version, schema and health details need a credential (#2935)	Send an API token
A script finds no refresh_token	The refresh token is the HttpOnly spatium_refresh cookie	Use an API token for scripts
An SSO user is refused	Accounts are tied to their provider (#1235)	Link it: Users → Edit → Sign-in provider
A new /24 reverse zone appears beside one you made by hand	Inheriting subnets get their own reverse zones (#1149)	Run Sync ▾ → DNS on each subnet and apply what it lists
Alerts you never set up start firing	New default-on rules; Minimum Severity now filters properly (#1031, #1262)	Review them under Administration → Alerts

Sources: README, upgrading · Release notes 2026.10.02-1 · Release notes 1.0.0-beta.1 · Release notes 1.0.0-beta.3 · Docker Compose deployment §8 and §9 · OS appliance, rolling back · Kubernetes deployment §3 and §8 · Verified in the Academy lab, 1.0.0-beta.3 (both hops, on the first edition’s lab data)

Go deeper

• SpatiumDDI Troubleshooting: the upstream recovery recipes this playbook builds on
• Getting Started: troubleshooting the first IP
• DHCP features §3: troubleshooting reservations
• DNS agent: registration, heartbeats and the last-known-good cache
• Observability: health endpoints, logs and alerting
• Running behind a reverse proxy: its own symptom-and-fix table for proxied installs
• What’s new in this edition: what changed in the course since the 2026.9 edition (SpatiumDDI 2026.09.04-1)
• Tips and tricks and Quick reference