Files
parking_solution/wiki/concepts/vision-review-outbox.md
T
julian f4b806a538 fix(collector,trainer): migrate an existing collector DB on open; trainer handlers answer 500 JSON
The reviewer host's collector.sqlite was created by an earlier build, before the
`kind` column. CREATE TABLE IF NOT EXISTS shapes only a new database, so every query
naming the column failed: the collector's /health (container unhealthy), every
booth ingest, and the trainer's readiness — whose stdlib server printed the
traceback and dropped the socket, which the collector could only render as
"trainer not reachable: fetch failed". Nine days like that.

- CollectorDb.#migrate(): PRAGMA table_info against the list of columns added
  since the first deploy; ALTER TABLE ADD COLUMN for each missing one (all
  nullable or defaulted). Append to that list whenever a column joins the CREATE.
  Test replays the original schema: health, ingest, stats, a legacy row reads
  back with the defaults.
- Trainer Handler._guarded(): any unexpected exception → 500 JSON naming it,
  never a dropped connection; /health keeps answering. Test drives readiness
  against an old-schema DB.
- The collector's training status proxy includes the trainer's error text.

Wiki: the incident and the schema rule (vision-review-outbox), what the message
means (bodytype-classifier-training), log. Deploy: the new collector migrates on
start; nothing manual.

Claude-Session: https://claude.ai/code/session_01FWncR69HgGPuei1dLrW3cU
2026-09-16 10:33:47 +02:00

180 lines
13 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Vision review outbox — harvesting the operator's category choice for a trusted reviewer
type: concept
status: booth side built 2026-09-06; collector pending
related: [venue-modules, opencv-anpr-service, threat-model, append-only-event-chain, network-isolation]
---
# Vision review outbox
**The idea (user, 2026-09-06).** The Car Wash desk asks the operator for the vehicle's category,
and the entry camera now proposes one ([[venue-modules]] §Vehicle category from vision). The
operator's choice is what we would love to train the body-type classifier on — but the
operator **cannot be fully trusted** (mistake or intent; the [[threat-model]]). So the booth
hands each decision to a **trusted party** who reviews the picture and the label remotely,
and *that* verdict is the training label — and, per operator, the honest-mistake / fraud rate.
The booths sit on a private zero-trust overlay (**Netbird**), so the hand-off can go to a very
locked-down collector without exposing anything to the open internet.
## Rules (all enforced in `apps/server/src/modules/carwash/review-outbox.ts`)
1. **Offline-first, never on the intake path.** Creating a wash order *queues* a package (fire
and forget — a failure is a log line); a background loop drains the queue when the overlay
is up. The wash never waits on the network.
2. **One-way.** The booth POSTs; nothing ever comes back into the booth's decisions. The signed
ledger ([[append-only-event-chain]]) stays the only record of what happened at the wash.
Reviewer verdicts stay central and reach the owner as a report per site.
3. **Nothing that names the site leaves the booth.**
- Only the vehicle **crop** (the detector's box + 8 % margin, ≤ 640 px) — no walls, no camera
OSD (date / camera name burned into the frame), no bystanders.
- The **plate is blurred inside the crop** on the booth, from the plate detector's own box.
- The booth is a **pseudonymous id** set at deploy (`CARWASH_REVIEW_BOOTH_ID`); the operator
is a **keyed hash** (`sha256(boothId:username)[:16]`). The mapping back to places and
people is the reviewer's, held off the collector. The dataset export drops even those.
- Boxes are stored as **fractions of the frame** on the vision read, so the crop is cut from
the stored (downscaled) snapshot copy.
4. **The network is not the auth.** A per-booth bearer token on top of the overlay; the booth
can do nothing at the collector but this one POST. Payloads are small (a crop ≈ 50–80 kB).
5. **Data minimisation.** Queued only when there is a vehicle box (no box = no sample); the
image is dropped from the row once delivered; a voided order is abandoned unsent; anything
older than 14 days is abandoned ("expired") rather than resurfacing a fortnight in a burst.
## The entry stream — the real accelerator (built 2026-09-07)
The wash stream is small; the **entry camera photographs every car**, in exactly the view the
classifier is trained on, with zero domain shift. So the booth can also queue **one in N entry
vehicle reads** as pure training material: the crop and the camera's class, *no* order, *no*
operator, *no* category — same crop-and-blur pipeline, same one-way path, same privacy
properties. `CARWASH_REVIEW_ENTRY_SAMPLE=N` = one in N entries; **`1` = every entry, and that is
the setting park-2 runs** (user, 2026-09-07: 4 TB on the collector host, bandwidth not an issue —
the only limit was ever the reviewer's time; the reviewer labels what they have time for, the
rest waits and stays useful once a first model exists, as the unlabelled pile it is measured on).
0/unset = off; needs the three upload settings. A washed car arrives twice, as an entry sample
and as the wash decision — intended, the `kind` keeps them apart.
Seam: the core announces every vehicle read (`deviceEvents.emitVehicleRead`, snapshot.ts, entry
and exit) and the Car Wash module decides — it samples entry reads in-process (`sampleEntry()`,
exactly one in N) and calls `enqueueEntry()`; the core never imports the module. Packages carry
`kind: "wash" | "entry"`; the collector stores the kind, the review screen shows an entry sample
as "entry stream — label the vehicle", the export carries a `kind` column, and **operator
agreement is computed from wash items only** (an entry sample has no operator decision).
An internet feed was considered the same day and kept OUT of the collector's ingest: licensed
sets only, in a separate folder with provenance, used as warm-up and weighted down, and never
the judge of accuracy — the evaluation set is gate crops only.
## The package
`multipart/form-data`: `meta` (JSON) + `image` (JPEG). Meta = `{ v, booth, item, order, at,
operator (hash), operatorCategory {id,name}, service, vision {class, confidence, categoryId},
downgraded, image {width, height, plateBlurred} }`. Headers: `Authorization: Bearer <token>`,
`X-Booth-Id`.
## Draining
Every `CARWASH_REVIEW_INTERVAL_SEC` (60): due items oldest-first, 20 per pass. `2xx` → sent
(image cleared). `400/404/413/415/422` → abandoned (the collector refused the package itself).
Anything else (auth not yet fixed, 429, 5xx, timeout, no route) → retry with backoff
`1 min · 2^attempts`, capped at 6 h. `GET /api/carwash/review/status` (site:read) and a line in
Setup → Car wash show queued / delivered / abandoned + the last error.
## Config
`CARWASH_REVIEW_URL`, `CARWASH_REVIEW_TOKEN`, `CARWASH_REVIEW_BOOTH_ID` — all three or the outbox
is off and **nothing is queued** (an unbounded queue nobody drains is worse than none). Set per
booth in the Komodo stack env; compose forwards them.
- **URL by Netbird DNS name** (`http://docker-station.nb.infra:8090/ingest`): the server container
runs on the host network in prod, so it uses the booth's resolver and Netbird's DNS answers
`*.nb.infra`; a collector that moves address costs no booth change. A failed lookup behaves like
a collector outage (defer, backoff). The collector's own `COLLECTOR_BIND` must be the raw overlay
**IP** — Docker port bindings take no hostname.
- **Secrets: one per booth, two consumers.** `wash_review_token_booth_2` is referenced by the
booth's stack as its `CARWASH_REVIEW_TOKEN` *and* by the collector's stack inside
`COLLECTOR_BOOTH_TOKENS=booth-2:[[wash_review_token_booth_2]],booth-3:[[…]]` — one value, nothing
to keep in sync, rotating a booth touches one secret. (A first cut had one combined secret for
the whole list; replaced the same day — rotation was all-or-nothing and the value lived twice.)
Token format: opaque, `openssl rand -hex 32`; the collector only demands ≥ 16 chars and the list
splits on commas/whitespace, which hex never contains. Never share a token between booths — it
is what names the booth. Total Komodo secrets for one booth + the collector: two (the booth's
token, the reviewer's password).
- **The operator hash needs no variable**: `sha256(boothId + ":" + username)[:16]`, computed on
the booth from values already set; the owner recomputes it from the booth's usernames to map a
hash back, the collector never can.
## The collector — skeleton built 2026-09-06 (`apps/collector`)
A deliberately small Fastify + SQLite service **in this monorepo** (so it imports the payload
contract and the class vocabulary from `@parking/shared` — the two ends cannot drift), delivered
to the reviewer's host by **its own Komodo stack** (`wash-collector` in `komodo/resources.toml`
→ `docker-compose.collector.yml` only; the booth stacks never see it and it never sees booth
services). Image `parking-collector:<branch>-<sha>` from the same workflow as the others.
Three surfaces, nothing else — it must not grow into a fleet console:
- **`POST /ingest`** — bearer token **per booth** (`COLLECTOR_BOOTH_TOKENS`, `boothId:token`
pairs; constant-time compare), `X-Booth-Id` must match the token's booth, multipart `meta` +
`image` (JPEG magic checked, 2 MB cap), `meta` validated field by field against the contract
above (unknown vision class, non-id item, wrong booth → 422), **idempotent on the item id**
(a retry after a lost 2xx → 200 `duplicate`). Stored: `crops/<booth>/<item>.jpg` on the
volume + one `items` row. The booth now also sends `operatorCategory.classes` (the classes
the chosen category covers at that site) so a reviewer's CLASS can be judged against the
operator's CATEGORY without the site's setup.
- **`/review`** (+ `/api/items`, `/api/items/:id/image`, `/api/items/:id/review`, `/api/stats`)
— the reviewer's screen, served by the process itself (no build, no framework): one pending
crop at a time, the operator's pick and the camera's pick beside it, one button (and one
key) per vocabulary class + *unusable* + *skip*. HTTP Basic, one login
(`COLLECTOR_REVIEWER_USER/PASS`), over the overlay. Stats: per booth received / pending /
reviewed; per operator (booth + hash) **agree / disagree / unusable** — disagree = the
reviewer's class is outside the operator's chosen category. That column is the honest-mistake
/ fraud rate.
- **`GET /export/labels.csv`** — reviewed, usable rows: item, booth, crop path, the reviewer's
label, the operator's category + classes, the camera's class + confidence, downgraded, at.
Crops are not packaged: the phase-B trainer runs **on the same host** and reads the SQLite
+ crops straight off the volume, read-only ([[bodytype-classifier-training]]: CPU-only, the
Xeon is enough) — the `trainer` service beside the collector in
`docker-compose.collector.yml` (the CSV export stays for a human with a spreadsheet).
- **Training section on `/review`** (+ `/api/training/status|jobs|jobs/:id|versions/:v/report`)
— a thin proxy, behind the same reviewer login, to the trainer's job API on the compose
network (`COLLECTOR_TRAINER_URL`, unset = hidden): labels per class vs the minimum, Train
(mode / backbone / floor), the running job's log, the versions with Report / Evaluate /
Publish. The collector forwards only a fixed set of paths and knobs; the trainer validates
values and answers 409 while a job runs.
**Where the data lives.** The collector writes to `/data` in its container: `collector.sqlite`
and one JPEG per item at `crops/<booth-id>/<item-id>.jpg`. `/data` is the named Docker volume
`collector-data` (compose), on the host under Docker's volume directory — normally
`/var/lib/docker/volumes/wash-collector_collector-data/_data/` (`docker volume inspect
wash-collector_collector-data` confirms). The trainer mounts the same volume read-only at its
own `/data`; nothing is copied or exported for training.
**Deploy notes.** Bind the published port to the host's **Netbird address** (`COLLECTOR_BIND`),
never `0.0.0.0` on a host with a public interface; Netbird policy: booths → this host:8090 and
nothing else. The host must be onboarded as a Komodo server like the booths. `TAG` is pinned
and promoted with the booths (one sha for all stacks) — fine while the collector stays small;
its own repo the day it needs its own cadence. Deploy the collector BEFORE a booth that sends a
package kind it does not know (a 422 is abandoned, not retried). The export neutralises cells
that start like a spreadsheet formula (category/service names are booth-supplied text).
> **Incident 2026-09-16 — the collector's DB predated the `kind` column; nothing worked for 9
> days and nothing said so.** The reviewer opened /review: *Training — trainer not reachable:
> fetch failed*. On the host: collector `stage-2d9bb15` **unhealthy** (`/health` → 500 *no such
> column: kind*), trainer healthy but every `/readiness` a Python traceback; the volume's
> `collector.sqlite` (created 2026-09-07 by the previous build, **0 items**) had the original
> column set. `CREATE TABLE IF NOT EXISTS` shapes only a NEW database — an existing volume keeps
> its old columns, so every query naming `kind` failed: the collector's health, **every ingest**
> (booths would have got 500s and kept retrying — the log shows none ever arrived, a separate
> question), and the trainer's readiness. The trainer's stdlib server printed the traceback and
> dropped the socket, which the collector could only render as "fetch failed".
>
> Fixes (same day): `CollectorDb` now **migrates on open** — `PRAGMA table_info` vs a list of the
> columns added since the first deploy, `ALTER TABLE … ADD COLUMN` for each missing one (all
> nullable or defaulted; **append to that list whenever a column joins the CREATE**); the trainer's
> handlers are guarded — an unexpected exception is a **500 JSON** naming the error, never a
> dropped connection; the collector's status proxy surfaces the trainer's error text. Rule going
> forward: the collector owns the schema; the trainer only reads; a deploy that changes the table
> must be accompanied by a migration entry, and the Training section is the first place a
> schema/DB mismatch shows — read its error text before suspecting the network.
**Status (2026-09-07).** Live: the collector runs on `art-docker-station` and park-2 is wired to
it (`stage-dbbb051` on both stacks, every entry sampled). The review screen at
`http://docker-station.nb.infra:8090/review` is filling; no labels reviewed yet.