From f9cb973fe9a4d972843e91aac26ae2f32b3374e1 Mon Sep 17 00:00:00 2001 From: Julian Cuni Date: Mon, 7 Sep 2026 10:01:39 +0200 Subject: [PATCH] docs(wiki): collector live on park-2; secrets shape, DNS vs bind, token format, every-entry sampling, CI extra rule Claude-Session: https://claude.ai/code/session_01FWncR69HgGPuei1dLrW3cU --- wiki/concepts/vision-review-outbox.md | 33 ++++++++++++++++++++-- wiki/decisions/vision-service-packaging.md | 5 ++++ wiki/log.md | 10 +++++++ 3 files changed, 46 insertions(+), 2 deletions(-) diff --git a/wiki/concepts/vision-review-outbox.md b/wiki/concepts/vision-review-outbox.md index f6f8fdc..a2fbf38 100644 --- a/wiki/concepts/vision-review-outbox.md +++ b/wiki/concepts/vision-review-outbox.md @@ -45,7 +45,12 @@ The wash stream is small; the **entry camera photographs every car**, in exactly classifier is trained on, with zero domain shift. So the booth can also queue **one in N entry vehicle reads** as pure training material: the crop and the camera's class, *no* order, *no* operator, *no* category — same crop-and-blur pipeline, same one-way path, same privacy -properties. `CARWASH_REVIEW_ENTRY_SAMPLE=N` (0/unset = off; needs the three upload settings). +properties. `CARWASH_REVIEW_ENTRY_SAMPLE=N` = one in N entries; **`1` = every entry, and that is +the setting park-2 runs** (user, 2026-09-07: 4 TB on the collector host, bandwidth not an issue — +the only limit was ever the reviewer's time; the reviewer labels what they have time for, the +rest waits and stays useful once a first model exists, as the unlabelled pile it is measured on). +0/unset = off; needs the three upload settings. A washed car arrives twice, as an entry sample +and as the wash decision — intended, the `kind` keeps them apart. Seam: the core announces every vehicle read (`deviceEvents.emitVehicleRead`, snapshot.ts, entry and exit) and the Car Wash module decides — it samples entry reads in-process (`sampleEntry()`, exactly one in N) and calls `enqueueEntry()`; the core never imports the module. Packages carry @@ -78,6 +83,24 @@ Setup → Car wash show queued / delivered / abandoned + the last error. is off and **nothing is queued** (an unbounded queue nobody drains is worse than none). Set per booth in the Komodo stack env; compose forwards them. +- **URL by Netbird DNS name** (`http://docker-station.nb.infra:8090/ingest`): the server container + runs on the host network in prod, so it uses the booth's resolver and Netbird's DNS answers + `*.nb.infra`; a collector that moves address costs no booth change. A failed lookup behaves like + a collector outage (defer, backoff). The collector's own `COLLECTOR_BIND` must be the raw overlay + **IP** — Docker port bindings take no hostname. +- **Secrets: one per booth, two consumers.** `wash_review_token_booth_2` is referenced by the + booth's stack as its `CARWASH_REVIEW_TOKEN` *and* by the collector's stack inside + `COLLECTOR_BOOTH_TOKENS=booth-2:[[wash_review_token_booth_2]],booth-3:[[…]]` — one value, nothing + to keep in sync, rotating a booth touches one secret. (A first cut had one combined secret for + the whole list; replaced the same day — rotation was all-or-nothing and the value lived twice.) + Token format: opaque, `openssl rand -hex 32`; the collector only demands ≥ 16 chars and the list + splits on commas/whitespace, which hex never contains. Never share a token between booths — it + is what names the booth. Total Komodo secrets for one booth + the collector: two (the booth's + token, the reviewer's password). +- **The operator hash needs no variable**: `sha256(boothId + ":" + username)[:16]`, computed on + the booth from values already set; the owner recomputes it from the booth's usernames to map a + hash back, the collector never can. + ## The collector — skeleton built 2026-09-06 (`apps/collector`) A deliberately small Fastify + SQLite service **in this monorepo** (so it imports the payload @@ -114,4 +137,10 @@ Three surfaces, nothing else — it must not grow into a fleet console: never `0.0.0.0` on a host with a public interface; Netbird policy: booths → this host:8090 and nothing else. The host must be onboarded as a Komodo server like the booths. `TAG` is pinned and promoted with the booths (one sha for all stacks) — fine while the collector stays small; -its own repo the day it needs its own cadence. +its own repo the day it needs its own cadence. Deploy the collector BEFORE a booth that sends a +package kind it does not know (a 422 is abandoned, not retried). The export neutralises cells +that start like a spreadsheet formula (category/service names are booth-supplied text). + +**Status (2026-09-07).** Live: the collector runs on `art-docker-station` and park-2 is wired to +it (`stage-dbbb051` on both stacks, every entry sampled). The review screen at +`http://docker-station.nb.infra:8090/review` is filling; no labels reviewed yet. diff --git a/wiki/decisions/vision-service-packaging.md b/wiki/decisions/vision-service-packaging.md index 7221ddd..d40e6ae 100644 --- a/wiki/decisions/vision-service-packaging.md +++ b/wiki/decisions/vision-service-packaging.md @@ -87,6 +87,11 @@ The skeleton is **built and wired** (no recognizer models yet): (env `VISION_*`), `schemas.py` (the `/analyze` contract incl. a not-yet-populated `vehicle` field for Job 2), `recognizer.py` (a `Recognizer` **Protocol** + `StubRecognizer` and `FastAlprRecognizer` — the [[device-adapter-pattern]] applied to the model). +- **CI runs WITHOUT the extra** (`uv sync --frozen` in ci.yml and build-images.yml): a test that + imports numpy/cv2 at module level breaks collection there even though it passes in a local venv + that has `alpr`. Rule (2026-09-07, after three red runs): pure post-processing tests get numpy + from the **dev group**; anything needing OpenCV uses `pytest.importorskip("cv2")`; the service + itself imports both lazily inside functions. - **Light-core, heavy-optional:** core deps boot in **stub mode** (no model download) so `uv sync` + tests work offline; the real stack is the `alpr` extra (`uv sync --extra alpr` → fast-alpr + onnxruntime). `VISION_RECOGNIZER=fast_alpr` switches it on. diff --git a/wiki/log.md b/wiki/log.md index 08875ba..a77bbab 100644 --- a/wiki/log.md +++ b/wiki/log.md @@ -3143,3 +3143,13 @@ run; the Quadro FX 3800 is unusable (cc 1.3), the HD P530 irrelevant, the Xeon E (feature-extraction head in minutes, full fine-tune ~1 h); trainer image = CPU-only torch, the compose seam drops the GPU reservation; cloud GPU rejected (crops stay on premises). Linked from [[opencv-anpr-service]], [[vision-review-outbox]], index. User: "No build just yet." + +## [2026-09-07] ingest | Collector live on park-2; secrets shape, DNS vs bind, token format, CI rule +Deployed: collector on art-docker-station + park-2 at stage-dbbb051, every entry sampled; review +screen filling. Recorded on [[vision-review-outbox]]: one secret per booth referenced by both stacks +(the combined-list secret was replaced the same day), URL by Netbird DNS name vs bind by IP, token = +`openssl rand -hex 32` (opaque, ≥16, no separators, never shared), the operator hash needs no +variable, deploy the collector before a booth that sends a new package kind, the export's formula +neutralisation (security review finding), and why every entry is sent. On +[[vision-service-packaging]]: CI syncs without the alpr extra — numpy in the dev group, cv2 tests +importorskip (three red runs on 2026-09-07).