239 lines
15 KiB
Markdown
239 lines
15 KiB
Markdown
---
|
|
type: decision
|
|
tags: [parking, deployment, fleet, komodo, netbird, offline-first, threat-model]
|
|
sources: []
|
|
updated: 2026-09-03
|
|
status: settled
|
|
---
|
|
|
|
# Fleet deployment — Komodo Periphery over a NetBird mesh
|
|
|
|
How the parking appliance is deployed and managed **at fleet scale**, superseding the
|
|
single-box, SSH-and-`booth.sh` model. The image build/tag/registry pipeline
|
|
([[container-deployment]]) is unchanged — this decides only the *control plane* that drives
|
|
those same compose files onto many booths. Settled 2026-06-27.
|
|
|
|
## The problem `booth.sh` couldn't solve
|
|
|
|
[[container-deployment|`scripts/booth.sh`]] is a thin wrapper over `docker compose -f base -f
|
|
prod --env-file .env`. It works for **one** appliance you can get a shell on, but as the fleet
|
|
grows (the stated direction is **many/growing** sites) it gives us none of:
|
|
|
|
- **Remote, no-SSH operation** — an update means someone gets a root shell on the booth.
|
|
- **A fleet view** — which booth runs which `dev-<sha>`, which is healthy/offline.
|
|
- **A deploy audit trail** — who deployed what, when.
|
|
- **One-click rollback** to a previous immutable `dev-<sha>`.
|
|
|
|
These are exactly the gaps a deployment controller fills. We already run every prerequisite
|
|
(a **Komodo Core**, a **NetBird** zero-trust mesh, the **Gitea registry**), so the marginal
|
|
cost is low.
|
|
|
|
## Decision
|
|
|
|
Adopt **Komodo Periphery** on each appliance, driven by the existing **Komodo Core** over the
|
|
**NetBird** mesh. Keep the compose files and the [[container-deployment|image pipeline]]
|
|
verbatim — Komodo consumes them as a *Stack*; it does not replace them. `booth.sh` is demoted
|
|
to a **break-glass local fallback** for when the mesh/Core is unreachable.
|
|
|
|
```
|
|
Gitea push ─▶ build-images.yml ─▶ registry (parking-server:dev-<sha>, parking-vision:dev-<sha>)
|
|
│
|
|
Komodo Core (off-site) ──── NetBird mesh ─┼─▶ Periphery @ booth-A ─▶ docker compose up (pinned sha)
|
|
• fleet table / history / rollback ├─▶ Periphery @ booth-B
|
|
• per-booth secret injection └─▶ Periphery @ booth-C …
|
|
• NO deploy webhook (manual + pinned)
|
|
```
|
|
|
|
### The three load-bearing choices (settled with the user 2026-06-27)
|
|
|
|
1. **Fleet size: many/growing.** Komodo is treated as load-bearing infrastructure, not a
|
|
convenience. This is what tips the decision away from "SSH-over-NetBird + a playbook".
|
|
2. **Deploy trigger: always manual + pinned.** **No deploy webhook on a booth Stack.** A human
|
|
deploys a specific immutable `TAG=<branch>-<sha>` from Core. This preserves the determinism we
|
|
chose when pinning the booth tag (a moving tag auto-redeploying a booth is the surprise we
|
|
explicitly rejected). This holds for **every** booth — including the **staging** booth: it tracks
|
|
the `stage` *branch* but still runs a **pinned `stage-<sha>`** (an earlier sketch floated a moving
|
|
`:dev` + webhook for staging; rejected in favour of pinned-everywhere). See "Promotion tiers".
|
|
3. **Secrets: Komodo-managed (per-booth, unique).** Core's secret store injects `JWT_SECRET`,
|
|
`EVENT_SIGNING_KEY` and `BACKUP_KEY` into the Stack at deploy. This scales (no SSH-to-N-booths to
|
|
rotate a key) — but see the threat-model tension below; the keys MUST be **distinct per booth**.
|
|
|
|
## Promotion tiers — dev → stage → main (settled 2026-06-29)
|
|
|
|
Three git branches map to three image tags and three booth roles:
|
|
|
|
| Branch | Image tag | Role | Deploy |
|
|
| --- | --- | --- | --- |
|
|
| `dev` | `:dev` / `:dev-<sha>` | working branch | no booth runs it |
|
|
| `stage` | `:stage` / `:stage-<sha>` | **staging booth (`park-buzi`)** — real-world test | **manual + pinned** `stage-<sha>` |
|
|
| `main` | `:main` / `:main-<sha>` | **production** booths | manual + pinned `main-<sha>` |
|
|
|
|
- **Promotion is a merge, not a build trigger.** When `dev` is confident-ready, **merge `dev → stage`**;
|
|
CI (`build-images.yml`, which now triggers on `dev`/`stage`/`main`) builds `:stage` + `:stage-<sha>`;
|
|
then **bump `TAG=stage-<sha>`** in `komodo/resources.toml` and deploy from Core. `main` only ever
|
|
receives what survived staging.
|
|
- **`stage` was branched from `dev`** (2026-06-29) so the first real-world test carries the full current
|
|
app, not a `main` that predates this work. `park-buzi`'s Stack `branch` + `TAG` both point at `stage`.
|
|
- **DB migrations run at boot** (`migrate-runtime.mjs`), so a promotion auto-migrates the staging booth's
|
|
SQLite ledger — staging is exactly where a bad migration is caught before production.
|
|
- **Per-booth secrets must pre-exist** in Core for `park-buzi`: `[[park_buzi_jwt_secret]]`,
|
|
`[[park_buzi_event_signing_key]]`, `[[park_buzi_backup_key]]` — distinct, never shared. Once the booth
|
|
signs real entries under its `EVENT_SIGNING_KEY`, that key is load-bearing for its ledger forever
|
|
(escrow it; see [[backup-recovery]]).
|
|
|
|
## Why this is safe (against the project's two forces)
|
|
|
|
### Offline-first ([[offline-first]]) — Core is orchestration, never a runtime dependency
|
|
|
|
The booth must run **fully when the mesh is down**. Komodo's agent model satisfies this:
|
|
Periphery + the local containers keep operating if Core is unreachable; we lose *remote
|
|
management* until the mesh returns, **not operation**. There must be **no runtime path** from
|
|
booth operation to Core — Core only deploys. (Periphery's own liveness is irrelevant to entry/
|
|
exit; the Fastify server and SQLite ledger run independently of it.)
|
|
|
|
### Threat model — the adversary is the booth operator ([[threat-model]])
|
|
|
|
This is the sharp edge, and the reason this page is explicit rather than a footnote.
|
|
|
|
- **Periphery is a root-capable remote-exec agent on the appliance.** If the operator
|
|
compromises the box, the agent is a lever. Mitigations: bind Periphery **only to the NetBird
|
|
interface** (never `0.0.0.0`), enforce its **passkey + TLS**, and fold the agent into the
|
|
[[disk-os-hardening]] surface. It is part of the trusted computing base now.
|
|
- **`EVENT_SIGNING_KEY` is the anti-fraud root.** It signs the [[append-only-event-chain|
|
|
append-only ledger]] — the control between us and a booth operator forging entry/exit events.
|
|
Holding it in Core means **a Core compromise can forge any booth's ledger that shares a key**.
|
|
Two mitigations make central management acceptable:
|
|
- **Per-booth, unique keys.** Never reuse a signing key across sites, so a single leak taints
|
|
one booth, not the fleet.
|
|
- **The [[atecc608|ATECC608]] is the real long-term signer.** The
|
|
`EVENT_SIGNING_KEY` HMAC is the *interim* mechanism; once the secure element signs the
|
|
chain, the key in Core stops being the fraud root. Tracked in [[open-questions]].
|
|
- **Core becomes a Tier-0 asset.** It now holds login + ledger keys for the whole fleet, so it
|
|
must be hardened to the booths' bar: Komodo API bound to the NetBird mesh only, never a public
|
|
interface; access-controlled; backed up.
|
|
|
|
### Licensing — Komodo is GPL-3.0, and that's fine here
|
|
|
|
The hard MIT/Apache/BSD constraint ([[technology-stack]]) is about **shipped app dependencies**
|
|
(code we distribute/link). Komodo is **external ops tooling we self-host and don't distribute**,
|
|
so its GPL-3.0 does not taint the product — exactly like the [[vision-service|AGPL ANPR
|
|
exception]] reasoning (a separate process / external boundary, not a linked dependency). Noted
|
|
here so it isn't re-litigated.
|
|
|
|
## What lives where
|
|
|
|
| Concern | Where | Notes |
|
|
| --- | --- | --- |
|
|
| Image build + tags | Gitea CI ([[container-deployment]]) | unchanged: `:dev` moving + `:dev-<sha>` immutable |
|
|
| Compose files | the repo + on the booth | unchanged base + `docker-compose.prod.yml` |
|
|
| Stack / deploy definition | **Komodo Core** | git-synced from `komodo/` (infra-as-code) |
|
|
| Which sha is deployed | **Komodo Core**, manual | `TAG=dev-<sha>`, pinned, no webhook |
|
|
| `JWT_SECRET`, `EVENT_SIGNING_KEY` | **Komodo Core** secret store | **per-booth, unique** |
|
|
| `COOKIE_SECURE=0`, `TAG`, `REGISTRY` | Komodo Stack env | per-environment |
|
|
| Registry pull creds | **Komodo Core** | so Periphery can pull from Gitea |
|
|
| Local break-glass | `booth.sh` + a local `.env` | mesh-down fallback only |
|
|
|
|
## Setup outline
|
|
|
|
**On each appliance** (after [[appliance-provisioning]]):
|
|
1. Install **Komodo Periphery** (binary or container), bound **only** to the NetBird interface;
|
|
set its passkey/TLS.
|
|
2. Point its compose/stack dir at `/opt/parking_systems/` (the existing files).
|
|
3. Keep `booth.sh` + a minimal local `.env` (no real secrets) as break-glass.
|
|
|
|
**In Komodo Core:**
|
|
1. Add the booth as a **Server**, address = its **NetBird IP** (mesh, not LAN/WAN).
|
|
2. Define the **Stack** = base + `docker-compose.prod.yml`, env from Core's secret store, secrets
|
|
**per booth**.
|
|
3. **No deploy webhook** on the booth Stack — deploys are manual; set `TAG=dev-<sha>` explicitly.
|
|
4. Add Gitea registry creds so Periphery can pull.
|
|
5. Sync the Stack/Server definitions from the repo's `komodo/` directory (infra-as-code:
|
|
`komodo/resources.toml` + README) so the control plane is itself reviewable +
|
|
version-controlled.
|
|
|
|
## A non-booth stack (2026-09-06)
|
|
|
|
The Stack model turned out to fit a service that is *not* a booth: the Car Wash review
|
|
collector ([[vision-review-outbox]]) runs on the reviewer's GPU host as its own `[[stack]]`
|
|
(`wash-collector`, `server = "art-docker-station"`, `file_paths = ["docker-compose.collector.yml"]`).
|
|
Same repo, branch and pinned `TAG` promotion, its own secret references, and — because a stack
|
|
names its compose files — nothing booth-side lands on that host and nothing of it on a booth.
|
|
The same stack carries the phase-B **trainer** as a compose *profile* (`train`,
|
|
[[bodytype-classifier-training]]): a deploy never starts it; the owner runs it by hand on the host
|
|
with `docker compose … --profile train run --rm trainer …`. So after a deploy of that stack
|
|
`docker ps` shows one container — expected; and a deploy pulls nothing for the profile, the
|
|
first `run` does (host Docker must be logged in to the registry). Its two env lines (`TRAINER_OUT`, the
|
|
`TRAINER_PUBLISH_TOKEN` secret reference) stay commented in `resources.toml` until the first
|
|
publish.
|
|
|
|
## Open / not yet done
|
|
|
|
- **Per-booth secret generation + rotation flow** — how a new site's unique `EVENT_SIGNING_KEY`
|
|
is generated and registered in Core (vs. on-site `openssl rand`). Tie-in: [[open-questions]]
|
|
JWT-key item.
|
|
- **ATECC608 as the signer** supersedes `EVENT_SIGNING_KEY`-in-Core as the fraud root — until
|
|
then central secrets carry the blast-radius noted above.
|
|
- **Periphery hardening checklist** folded into [[disk-os-hardening]] (interface binding, passkey,
|
|
TLS, agent as TCB).
|
|
- **Staging vs production booth split** — ✅ MODELLED 2026-06-29 (see "Promotion tiers" below).
|
|
`park-buzi` is the first **staging** booth, tracking the `stage` branch / `:stage` image, deployed
|
|
**manual + pinned** (`TAG=stage-<sha>`, no webhook — we hold the no-moving-tag-on-a-booth line even
|
|
on staging, *not* the webhook-on-staging option the earlier sketch floated). `komodo/resources.toml`
|
|
+ `komodo/README.md` updated.
|
|
- **Core backup / DR** — Core is now Tier-0; its loss = no fleet management (operation
|
|
unaffected, per offline-first). Backup story TBD.
|
|
|
|
## Supersedes / relates
|
|
|
|
- **Supersedes** the "SSH + `booth.sh` is the deploy mechanism" assumption in
|
|
[[container-deployment]] (that page's *build/tag/registry* content stands; its `booth.sh`-as-
|
|
primary-deploy framing is now the fallback). Cross-linked there.
|
|
- Companion: the `komodo/` infra-as-code sketch (in the repo, not the wiki),
|
|
[[appliance-provisioning]] (what runs *before* Periphery), [[disk-os-hardening]] (the
|
|
appliance's hardening surface).
|
|
|
|
## park-lab — the lab bench joins the fleet (2026-07-07)
|
|
|
|
Second `[[stack]]` in `komodo/resources.toml`: **`park-lab`** (server = the lab box's Periphery
|
|
`connect_as`), the first non-booth member and the proof of the tier model in practice:
|
|
|
|
| Stack | compose branch | image tag | secrets |
|
|
| --- | --- | --- | --- |
|
|
| park-lab | `dev` | **moving `dev`** (a lab may float) | `park_lab_*` |
|
|
| park-buzi | `stage` | pinned `stage-<sha>` | `park_buzi_*` |
|
|
|
|
The three knobs are independent per stack — the ResourceSync's own branch only governs where the
|
|
FILE is read from, each stack's `branch` picks its compose files, `TAG` picks the image. Per-box
|
|
secrets even in the lab (blast radius). The lab box earned its keep immediately: it caught the
|
|
USB close-cancel truncation, the printer/controller wizard gate, and the Periphery v2.2.0
|
|
root_directory default before any of them reached a real booth ([[appliance-provisioning]] §7a).
|
|
|
|
## ResourceSync branch drift — the exact gotcha this page already warned about (2026-09-03)
|
|
|
|
This page's own §"park-lab" note (2026-07-07) already spelled it out: *"the ResourceSync's own
|
|
branch only governs where the FILE is read from"* — independent of any `[[stack]]`'s own `branch`
|
|
field. It bit anyway. `resource-sync-park-systems` in Komodo Core was pointed at **`dev`**, while
|
|
`park-buzi` and `park-2` are `stage`-tier Stacks (`branch = "stage"`, pinned `TAG=stage-<sha>`, per
|
|
the promotion-tiers model above). `resources.toml` had been byte-identical on `dev` and `stage`
|
|
since park-buzi's Stack was first written, so this had **zero observable effect for months** — until
|
|
a desktop-app debugging session (see [[desktop-shell-tauri]]) landed 9 real commits on `dev`
|
|
(including a `WS_ALLOWED_ORIGINS` fix) that were never merged to `stage`, creating the first genuine
|
|
divergence between the two branches.
|
|
|
|
**Symptom:** merged `dev` → `stage`, pushed, bumped `TAG` in `resources.toml` on `stage`, committed,
|
|
pushed — then destroyed + recreated the `park-2` Stack in Komodo Core and it STILL came back running
|
|
the old image. Every sync was silently re-reading `resources.toml` from `dev` (which still had the
|
|
stale `TAG`), overwriting the correct value just committed on `stage`. No error, no warning — the
|
|
sync just quietly did what it was configured to do, from the wrong branch.
|
|
|
|
**Fix:** pointed `resource-sync-park-systems` at `stage` in Komodo Core's UI (Sync config → branch
|
|
field), then re-synced + redeployed `park-2` — confirmed via `/api/version` (previously 404,
|
|
proving a stale image; correctly 401-auth-gated after the fix, proving the new image + route exist).
|
|
|
|
**Standing lesson, now written twice:** a `[[stack]]`'s promotion tier (which branch its own
|
|
`branch`/`TAG` fields track) and the ResourceSync resource's own git branch are **two independently
|
|
configured settings in Komodo Core — nothing enforces they agree**, and a mismatch is invisible
|
|
until the two branches' `resources.toml` actually diverge. **Check this FIRST** whenever a
|
|
redeploy doesn't pick up an expected `resources.toml` change, before assuming the change itself,
|
|
the CI build, or the deploy step is broken.
|