--- type: decision tags: [parking, deployment, fleet, komodo, netbird, offline-first, threat-model] sources: [] updated: 2026-09-03 status: settled --- # Fleet deployment — Komodo Periphery over a NetBird mesh How the parking appliance is deployed and managed **at fleet scale**, superseding the single-box, SSH-and-`booth.sh` model. The image build/tag/registry pipeline ([[container-deployment]]) is unchanged — this decides only the *control plane* that drives those same compose files onto many booths. Settled 2026-06-27. ## The problem `booth.sh` couldn't solve [[container-deployment|`scripts/booth.sh`]] is a thin wrapper over `docker compose -f base -f prod --env-file .env`. It works for **one** appliance you can get a shell on, but as the fleet grows (the stated direction is **many/growing** sites) it gives us none of: - **Remote, no-SSH operation** — an update means someone gets a root shell on the booth. - **A fleet view** — which booth runs which `dev-`, which is healthy/offline. - **A deploy audit trail** — who deployed what, when. - **One-click rollback** to a previous immutable `dev-`. These are exactly the gaps a deployment controller fills. We already run every prerequisite (a **Komodo Core**, a **NetBird** zero-trust mesh, the **Gitea registry**), so the marginal cost is low. ## Decision Adopt **Komodo Periphery** on each appliance, driven by the existing **Komodo Core** over the **NetBird** mesh. Keep the compose files and the [[container-deployment|image pipeline]] verbatim — Komodo consumes them as a *Stack*; it does not replace them. `booth.sh` is demoted to a **break-glass local fallback** for when the mesh/Core is unreachable. ``` Gitea push ─▶ build-images.yml ─▶ registry (parking-server:dev-, parking-vision:dev-) │ Komodo Core (off-site) ──── NetBird mesh ─┼─▶ Periphery @ booth-A ─▶ docker compose up (pinned sha) • fleet table / history / rollback ├─▶ Periphery @ booth-B • per-booth secret injection └─▶ Periphery @ booth-C … • NO deploy webhook (manual + pinned) ``` ### The three load-bearing choices (settled with the user 2026-06-27) 1. **Fleet size: many/growing.** Komodo is treated as load-bearing infrastructure, not a convenience. This is what tips the decision away from "SSH-over-NetBird + a playbook". 2. **Deploy trigger: always manual + pinned.** **No deploy webhook on a booth Stack.** A human deploys a specific immutable `TAG=-` from Core. This preserves the determinism we chose when pinning the booth tag (a moving tag auto-redeploying a booth is the surprise we explicitly rejected). This holds for **every** booth — including the **staging** booth: it tracks the `stage` *branch* but still runs a **pinned `stage-`** (an earlier sketch floated a moving `:dev` + webhook for staging; rejected in favour of pinned-everywhere). See "Promotion tiers". 3. **Secrets: Komodo-managed (per-booth, unique).** Core's secret store injects `JWT_SECRET`, `EVENT_SIGNING_KEY` and `BACKUP_KEY` into the Stack at deploy. This scales (no SSH-to-N-booths to rotate a key) — but see the threat-model tension below; the keys MUST be **distinct per booth**. ## Promotion tiers — dev → stage → main (settled 2026-06-29) Three git branches map to three image tags and three booth roles: | Branch | Image tag | Role | Deploy | | --- | --- | --- | --- | | `dev` | `:dev` / `:dev-` | working branch | no booth runs it | | `stage` | `:stage` / `:stage-` | **staging booth (`park-buzi`)** — real-world test | **manual + pinned** `stage-` | | `main` | `:main` / `:main-` | **production** booths | manual + pinned `main-` | - **Promotion is a merge, not a build trigger.** When `dev` is confident-ready, **merge `dev → stage`**; CI (`build-images.yml`, which now triggers on `dev`/`stage`/`main`) builds `:stage` + `:stage-`; then **bump `TAG=stage-`** in `komodo/resources.toml` and deploy from Core. `main` only ever receives what survived staging. - **`stage` was branched from `dev`** (2026-06-29) so the first real-world test carries the full current app, not a `main` that predates this work. `park-buzi`'s Stack `branch` + `TAG` both point at `stage`. - **DB migrations run at boot** (`migrate-runtime.mjs`), so a promotion auto-migrates the staging booth's SQLite ledger — staging is exactly where a bad migration is caught before production. - **Per-booth secrets must pre-exist** in Core for `park-buzi`: `[[park_buzi_jwt_secret]]`, `[[park_buzi_event_signing_key]]`, `[[park_buzi_backup_key]]` — distinct, never shared. Once the booth signs real entries under its `EVENT_SIGNING_KEY`, that key is load-bearing for its ledger forever (escrow it; see [[backup-recovery]]). ## Why this is safe (against the project's two forces) ### Offline-first ([[offline-first]]) — Core is orchestration, never a runtime dependency The booth must run **fully when the mesh is down**. Komodo's agent model satisfies this: Periphery + the local containers keep operating if Core is unreachable; we lose *remote management* until the mesh returns, **not operation**. There must be **no runtime path** from booth operation to Core — Core only deploys. (Periphery's own liveness is irrelevant to entry/ exit; the Fastify server and SQLite ledger run independently of it.) ### Threat model — the adversary is the booth operator ([[threat-model]]) This is the sharp edge, and the reason this page is explicit rather than a footnote. - **Periphery is a root-capable remote-exec agent on the appliance.** If the operator compromises the box, the agent is a lever. Mitigations: bind Periphery **only to the NetBird interface** (never `0.0.0.0`), enforce its **passkey + TLS**, and fold the agent into the [[disk-os-hardening]] surface. It is part of the trusted computing base now. - **`EVENT_SIGNING_KEY` is the anti-fraud root.** It signs the [[append-only-event-chain| append-only ledger]] — the control between us and a booth operator forging entry/exit events. Holding it in Core means **a Core compromise can forge any booth's ledger that shares a key**. Two mitigations make central management acceptable: - **Per-booth, unique keys.** Never reuse a signing key across sites, so a single leak taints one booth, not the fleet. - **The [[atecc608|ATECC608]] is the real long-term signer.** The `EVENT_SIGNING_KEY` HMAC is the *interim* mechanism; once the secure element signs the chain, the key in Core stops being the fraud root. Tracked in [[open-questions]]. - **Core becomes a Tier-0 asset.** It now holds login + ledger keys for the whole fleet, so it must be hardened to the booths' bar: Komodo API bound to the NetBird mesh only, never a public interface; access-controlled; backed up. ### Licensing — Komodo is GPL-3.0, and that's fine here The hard MIT/Apache/BSD constraint ([[technology-stack]]) is about **shipped app dependencies** (code we distribute/link). Komodo is **external ops tooling we self-host and don't distribute**, so its GPL-3.0 does not taint the product — exactly like the [[vision-service|AGPL ANPR exception]] reasoning (a separate process / external boundary, not a linked dependency). Noted here so it isn't re-litigated. ## What lives where | Concern | Where | Notes | | --- | --- | --- | | Image build + tags | Gitea CI ([[container-deployment]]) | unchanged: `:dev` moving + `:dev-` immutable | | Compose files | the repo + on the booth | unchanged base + `docker-compose.prod.yml` | | Stack / deploy definition | **Komodo Core** | git-synced from `komodo/` (infra-as-code) | | Which sha is deployed | **Komodo Core**, manual | `TAG=dev-`, pinned, no webhook | | `JWT_SECRET`, `EVENT_SIGNING_KEY` | **Komodo Core** secret store | **per-booth, unique** | | `COOKIE_SECURE=0`, `TAG`, `REGISTRY` | Komodo Stack env | per-environment | | Registry pull creds | **Komodo Core** | so Periphery can pull from Gitea | | Local break-glass | `booth.sh` + a local `.env` | mesh-down fallback only | ## Setup outline **On each appliance** (after [[appliance-provisioning]]): 1. Install **Komodo Periphery** (binary or container), bound **only** to the NetBird interface; set its passkey/TLS. 2. Point its compose/stack dir at `/opt/parking_systems/` (the existing files). 3. Keep `booth.sh` + a minimal local `.env` (no real secrets) as break-glass. **In Komodo Core:** 1. Add the booth as a **Server**, address = its **NetBird IP** (mesh, not LAN/WAN). 2. Define the **Stack** = base + `docker-compose.prod.yml`, env from Core's secret store, secrets **per booth**. 3. **No deploy webhook** on the booth Stack — deploys are manual; set `TAG=dev-` explicitly. 4. Add Gitea registry creds so Periphery can pull. 5. Sync the Stack/Server definitions from the repo's `komodo/` directory (infra-as-code: `komodo/resources.toml` + README) so the control plane is itself reviewable + version-controlled. ## A non-booth stack (2026-09-06) The Stack model turned out to fit a service that is *not* a booth: the Car Wash review collector ([[vision-review-outbox]]) runs on the reviewer's GPU host as its own `[[stack]]` (`wash-collector`, `server = "art-docker-station"`, `file_paths = ["docker-compose.collector.yml"]`). Same repo, branch and pinned `TAG` promotion, its own secret references, and — because a stack names its compose files — nothing booth-side lands on that host and nothing of it on a booth. The same stack carries the phase-B **trainer** as a compose *profile* (`train`, [[bodytype-classifier-training]]): a deploy never starts it; the owner runs it by hand on the host with `docker compose … --profile train run --rm trainer …`. Its two env lines (`TRAINER_OUT`, the `TRAINER_PUBLISH_TOKEN` secret reference) stay commented in `resources.toml` until the first publish. ## Open / not yet done - **Per-booth secret generation + rotation flow** — how a new site's unique `EVENT_SIGNING_KEY` is generated and registered in Core (vs. on-site `openssl rand`). Tie-in: [[open-questions]] JWT-key item. - **ATECC608 as the signer** supersedes `EVENT_SIGNING_KEY`-in-Core as the fraud root — until then central secrets carry the blast-radius noted above. - **Periphery hardening checklist** folded into [[disk-os-hardening]] (interface binding, passkey, TLS, agent as TCB). - **Staging vs production booth split** — ✅ MODELLED 2026-06-29 (see "Promotion tiers" below). `park-buzi` is the first **staging** booth, tracking the `stage` branch / `:stage` image, deployed **manual + pinned** (`TAG=stage-`, no webhook — we hold the no-moving-tag-on-a-booth line even on staging, *not* the webhook-on-staging option the earlier sketch floated). `komodo/resources.toml` + `komodo/README.md` updated. - **Core backup / DR** — Core is now Tier-0; its loss = no fleet management (operation unaffected, per offline-first). Backup story TBD. ## Supersedes / relates - **Supersedes** the "SSH + `booth.sh` is the deploy mechanism" assumption in [[container-deployment]] (that page's *build/tag/registry* content stands; its `booth.sh`-as- primary-deploy framing is now the fallback). Cross-linked there. - Companion: the `komodo/` infra-as-code sketch (in the repo, not the wiki), [[appliance-provisioning]] (what runs *before* Periphery), [[disk-os-hardening]] (the appliance's hardening surface). ## park-lab — the lab bench joins the fleet (2026-07-07) Second `[[stack]]` in `komodo/resources.toml`: **`park-lab`** (server = the lab box's Periphery `connect_as`), the first non-booth member and the proof of the tier model in practice: | Stack | compose branch | image tag | secrets | | --- | --- | --- | --- | | park-lab | `dev` | **moving `dev`** (a lab may float) | `park_lab_*` | | park-buzi | `stage` | pinned `stage-` | `park_buzi_*` | The three knobs are independent per stack — the ResourceSync's own branch only governs where the FILE is read from, each stack's `branch` picks its compose files, `TAG` picks the image. Per-box secrets even in the lab (blast radius). The lab box earned its keep immediately: it caught the USB close-cancel truncation, the printer/controller wizard gate, and the Periphery v2.2.0 root_directory default before any of them reached a real booth ([[appliance-provisioning]] §7a). ## ResourceSync branch drift — the exact gotcha this page already warned about (2026-09-03) This page's own §"park-lab" note (2026-07-07) already spelled it out: *"the ResourceSync's own branch only governs where the FILE is read from"* — independent of any `[[stack]]`'s own `branch` field. It bit anyway. `resource-sync-park-systems` in Komodo Core was pointed at **`dev`**, while `park-buzi` and `park-2` are `stage`-tier Stacks (`branch = "stage"`, pinned `TAG=stage-`, per the promotion-tiers model above). `resources.toml` had been byte-identical on `dev` and `stage` since park-buzi's Stack was first written, so this had **zero observable effect for months** — until a desktop-app debugging session (see [[desktop-shell-tauri]]) landed 9 real commits on `dev` (including a `WS_ALLOWED_ORIGINS` fix) that were never merged to `stage`, creating the first genuine divergence between the two branches. **Symptom:** merged `dev` → `stage`, pushed, bumped `TAG` in `resources.toml` on `stage`, committed, pushed — then destroyed + recreated the `park-2` Stack in Komodo Core and it STILL came back running the old image. Every sync was silently re-reading `resources.toml` from `dev` (which still had the stale `TAG`), overwriting the correct value just committed on `stage`. No error, no warning — the sync just quietly did what it was configured to do, from the wrong branch. **Fix:** pointed `resource-sync-park-systems` at `stage` in Komodo Core's UI (Sync config → branch field), then re-synced + redeployed `park-2` — confirmed via `/api/version` (previously 404, proving a stale image; correctly 401-auth-gated after the fix, proving the new image + route exist). **Standing lesson, now written twice:** a `[[stack]]`'s promotion tier (which branch its own `branch`/`TAG` fields track) and the ResourceSync resource's own git branch are **two independently configured settings in Komodo Core — nothing enforces they agree**, and a mismatch is invisible until the two branches' `resources.toml` actually diverge. **Check this FIRST** whenever a redeploy doesn't pick up an expected `resources.toml` change, before assuming the change itself, the CI build, or the deploy step is broken.