feat(trainer): training from the collector UI — the trainer becomes a job service, the review page gains a Training section

Trainer: `parking-trainer serve` — a stdlib HTTP job API on the compose network (never
published): /health, /readiness, /versions, /versions/<v>/report, /jobs. One job at a
time; each job runs the CLI as a subprocess with its output captured, state + log
persisted under /out/jobs/ so a restart keeps history. `publish` takes its URL from
TRAINER_PUBLISH_URL. Dockerfile: CMD serve, EXPOSE 8091, healthcheck.

Collector: COLLECTOR_TRAINER_URL + /api/training/{status,jobs,jobs/:id,versions/:v/report}
— a reviewer-gated proxy that forwards a fixed set of paths and whitelisted knobs and
passes the trainer's status codes through (409 while a job runs; 503 unconfigured, 502
unreachable). /review gains the Training section: labels per class vs the minimum with
Train disabled until two classes clear it, mode / backbone / floor, the running job's
live log, the versions with Report / Evaluate / Publish (publish confirms), and the
reminder that pinning stays a git commit. Fixed on the way: an apostrophe in the page's
inline script broke the whole page — a test now parses the script.

Compose: `trainer` is a service (restart: unless-stopped, read-only data volume, its own
trainer-out volume), the `train` profile and TRAINER_OUT are gone; the Docker-socket
route was rejected (root on the host for a service booths upload to). Verified with both
images running together: a Train started through the proxy finished, version and report
came back, the page rendered.

Wiki: bodytype-classifier-training (loop, running it, operating notes superseded),
vision-review-outbox, fleet-deployment-komodo, log.

Claude-Session: https://claude.ai/code/session_01FWncR69HgGPuei1dLrW3cU
This commit is contained in:
2026-09-07 14:34:13 +02:00
parent 3e77a4ad7c
commit 4ff31557a8
17 changed files with 915 additions and 75 deletions
+8 -3
View File
@@ -130,9 +130,14 @@ Three surfaces, nothing else — it must not grow into a fleet console:
label, the operator's category + classes, the camera's class + confidence, downgraded, at.
Crops are not packaged: the phase-B trainer runs **on the same host** and reads the SQLite
+ crops straight off the volume, read-only ([[bodytype-classifier-training]]: CPU-only, the
Xeon is enough) — `docker-compose.collector.yml` carries it as the `trainer` service under
`profiles: ["train"]`, a one-off job never started by a deploy (built 2026-09-07; the CSV
export stays for a human with a spreadsheet).
Xeon is enough) — the `trainer` service beside the collector in
`docker-compose.collector.yml` (the CSV export stays for a human with a spreadsheet).
- **Training section on `/review`** (+ `/api/training/status|jobs|jobs/:id|versions/:v/report`)
— a thin proxy, behind the same reviewer login, to the trainer's job API on the compose
network (`COLLECTOR_TRAINER_URL`, unset = hidden): labels per class vs the minimum, Train
(mode / backbone / floor), the running job's log, the versions with Report / Evaluate /
Publish. The collector forwards only a fixed set of paths and knobs; the trainer validates
values and answers 409 while a job runs.
**Where the data lives.** The collector writes to `/data` in its container: `collector.sqlite`
and one JPEG per item at `crops/<booth-id>/<item-id>.jpg`. `/data` is the named Docker volume
+35 -33
View File
@@ -18,8 +18,13 @@ Xeon". This page is the loop as built; what is still outstanding is at the end.
Each step is a place where a person decides. Nothing here runs on its own.
1. **Train** — `apps/trainer` (`parking-trainer`, Python/uv like the vision service; its own
image `parking-trainer`, a one-off job on the collector's host — never a booth service).
`train` reads the collector's `collector.sqlite` and `crops/` **straight off the volume**
image `parking-trainer`, the `trainer` service beside the collector on the reviewer's host —
never a booth service). **Started from the collector's UI:** the Training section of
`/review` (readiness, a Train button with mode / backbone / floor, the live log, the
versions with Report / Evaluate / Publish) drives a small job API the trainer serves on the
compose network (`serve`; stdlib HTTP, one job at a time, each job the CLI as a subprocess
with its log persisted under `/out/jobs/`). The collector proxies it behind the reviewer's
login; the trainer is never published. `train` reads the collector's `collector.sqlite` and `crops/` **straight off the volume**
(read-only), takes only reviewed, usable rows (the operator's pick and the camera's class are
never labels), **splits by TIME** (validation = the newest 20 % by *time seen*, so the number
reflects tomorrow's traffic), drops classes with fewer than `--min-per-class` (20) labels from
@@ -129,47 +134,44 @@ What the owner has: an **NVIDIA Quadro FX 3800** (in hand, not installed), and i
host, so nothing moves.
- **Consequences for the build (done):** the trainer image is **CPU-only PyTorch** (torch
2.14+cpu, ~200 MB of wheels, not the ~5 GB CUDA build); the `trainer` service in
`docker-compose.collector.yml` is real now — `profiles: ["train"]`, no device reservation
(one block to add if a modern card ever lands; the trainer would pick up CUDA), the collector
volume mounted read-only, output to `TRAINER_OUT` on the host (default `./models` beside the
compose file).
`docker-compose.collector.yml` is real — always on, serving the job API, no device
reservation (one block to add if a modern card ever lands; the trainer would pick up CUDA),
the collector volume mounted read-only, models/reports/logs in its own `trainer-out` volume.
- **If faster is ever wanted:** a used mid-range card of the last few generations (~€200) turns
the hour into a minute, given a slot and a PSU. **Renting a cloud GPU is rejected**: the crops
would leave the premises, and even scrubbed of plates and site that runs against the whole
privacy design of the outbox.
## Running it (on the collector host)
## Running it
```
docker compose -f docker-compose.collector.yml --profile train run --rm trainer inspect
docker compose -f docker-compose.collector.yml --profile train run --rm trainer train --min-accuracy 0.85
docker compose -f docker-compose.collector.yml --profile train run --rm trainer evaluate --model /out/<version>/bodytype.onnx
docker compose -f docker-compose.collector.yml --profile train run --rm trainer publish /out/<version> --url https://git.infra.msai.al/api/packages/mca/generic/parking-bodytype
```
From the collector's `/review` page, Training section: **Train** (mode, backbone, floor) when
readiness says enough labels; watch the log; read the report under Versions; **Evaluate** a
written version against labels reviewed since; **Publish** it (needs `TRAINER_PUBLISH_TOKEN`
in the `wash-collector` stack — commented until the first publish). Then, in git: write the
version into `apps/vision/models/bodytype.version`, commit, let the build produce the image,
bump the booth's `TAG`. The pin stays a commit on purpose — it is the deploy control.
Then: write the version into `apps/vision/models/bodytype.version`, commit, let the build produce
the image, bump the booth's `TAG`. The trainer is never started by a deploy (a profile), and the
`wash-collector` stack's `TRAINER_OUT` / `TRAINER_PUBLISH_TOKEN` lines stay commented until the
first publish.
The CLI is still there for debugging, inside the running container:
`docker compose -f docker-compose.collector.yml exec trainer parking-trainer inspect`.
## Operating notes (first deploy, 2026-09-07)
## Operating notes (2026-09-07)
- **Only the collector shows as running — that is correct.** The trainer is not a service; it is
behind the `train` compose *profile*, so a deploy never starts it and `docker ps` on
`art-docker-station` lists one container. The trainer runs when invoked by hand, does its job,
exits, and leaves nothing behind (`--rm`).
- **A deploy does not pull profile services either.** The first `run` pulls the image itself, so
the host's Docker must be logged in to the registry (`docker compose -f
docker-compose.collector.yml --profile train pull trainer` is the check; if refused, `docker
login git.infra.msai.al` on the host first).
- **Why the wash-collector TAG bump mattered** although the collector code did not change: the
trainer service uses the stack's TAG, so a bump makes `run` resolve to an image that exists
(`stage-f7a262a` is the first tag that carries `parking-trainer`).
- **park-2 does not need the bump** until a model is pinned: the new vision image ships with an
- **First deploy (`stage-f7a262a`) shipped the trainer as a compose *profile*** — a one-off
job the owner had to start by hand with `docker compose … --profile train run …` from
wherever Komodo's periphery had cloned the repo (`/etc/komodo/stacks/wash-collector/`).
The user rightly called that "not so smart": the host runs a periphery, and the reviewer is
already in the collector's UI. **Superseded the same day:** the trainer is now a
**service** (`restart: unless-stopped`, the `serve` command) and the collector's
`/review` page carries the Training section. A deploy starts both containers; `docker ps`
shows two.
- **Why not a Docker socket in the collector** (the other way to a button): it would hand
root on the host to a service that accepts uploads from booths — the party the
[[threat-model]] distrusts. The job API keeps the trainer a normal container with a
read-only data mount and its own `trainer-out` volume.
- **park-2 does not need a bump** until a model is pinned: the vision image ships with an
empty `bodytype.version`, phase B off, nothing for a booth to gain.
- **What to run, from the stack's directory on the host:** `inspect` first (label counts per
class, `ready: false` until enough are reviewed), then `train`, read `report.md` under
`TRAINER_OUT/<version>/`, then `publish`, pin, push, bump the booth. Commands under §Running it.
- **Reviewing is the bottleneck**: the Training section shows labels per class against the
minimum and keeps Train disabled until two classes clear it.
## Packaging rule (same as the vision service)
+4 -5
View File
@@ -158,11 +158,10 @@ collector ([[vision-review-outbox]]) runs on the reviewer's GPU host as its own
(`wash-collector`, `server = "art-docker-station"`, `file_paths = ["docker-compose.collector.yml"]`).
Same repo, branch and pinned `TAG` promotion, its own secret references, and — because a stack
names its compose files — nothing booth-side lands on that host and nothing of it on a booth.
The same stack carries the phase-B **trainer** as a compose *profile* (`train`,
[[bodytype-classifier-training]]): a deploy never starts it; the owner runs it by hand on the host
with `docker compose … --profile train run --rm trainer …`. So after a deploy of that stack
`docker ps` shows one container — expected; and a deploy pulls nothing for the profile, the
first `run` does (host Docker must be logged in to the registry). Its two env lines (`TRAINER_OUT`, the
The same stack carries the phase-B **trainer** as a second service ([[bodytype-classifier-training]]):
a deploy starts both, `docker ps` shows two containers, and the trainer is driven from the
collector's UI, never from the host's shell (a first cut as a compose *profile* run by hand was
replaced the same day — the host runs a periphery, nobody should be typing compose there). Its two env lines (`TRAINER_OUT`, the
`TRAINER_PUBLISH_TOKEN` secret reference) stay commented in `resources.toml` until the first
publish.
+14
View File
@@ -3144,6 +3144,20 @@ run; the Quadro FX 3800 is unusable (cc 1.3), the HD P530 irrelevant, the Xeon E
compose seam drops the GPU reservation; cloud GPU rejected (crops stay on premises). Linked from
[[opencv-anpr-service]], [[vision-review-outbox]], index. User: "No build just yet."
## [2026-09-07] build | Training from the collector UI — the trainer becomes a job service
User: the compose-profile trainer is "not so smart" (where is the compose file on a periphery
host? why not a button on the collector UI?). Built: `parking-trainer serve` — a stdlib job API
(`/health`, `/readiness`, `/versions`, `/versions/<v>/report`, `/jobs`), one job at a time, each
job the CLI as a subprocess with state + log persisted under `/out/jobs/`; the collector gained
`COLLECTOR_TRAINER_URL` + `/api/training/*` (reviewer-gated proxy, fixed paths, whitelisted
knobs, trainer status codes passed through, 503 unconfigured / 502 unreachable) and a Training
section on `/review` (readiness table, Train with mode/backbone/floor, live log, versions with
Report / Evaluate / Publish, the pin reminder). Compose: `trainer` is a service now
(`restart: unless-stopped`, `serve`, read-only data, own `trainer-out` volume, not published);
the Docker socket route was rejected (root on the host for a service booths upload to). Tests:
trainer 14, collector 7. Pages: [[bodytype-classifier-training]] (loop, running it, operating
notes superseded), [[vision-review-outbox]], [[fleet-deployment-komodo]].
## [2026-09-07] ingest | Trainer deployed as a profile; operating notes
User pushed `stage-f7a262a`, bumped the `wash-collector` TAG, redeployed, and asked why only one
service runs on art-docker-station. Expected: the trainer is a compose profile, never started or