From 20a3cb3e807a8074ba0c0f5ce5588924b463eabd Mon Sep 17 00:00:00 2001 From: Julian Cuni Date: Sun, 6 Sep 2026 19:53:11 +0200 Subject: [PATCH] =?UTF-8?q?feat(vision):=20vehicle=20stage,=20phase=20A=20?= =?UTF-8?q?=E2=80=94=20YOLOX-S=20(Apache-2.0=20ONNX)=20beside=20the=20plat?= =?UTF-8?q?e=20recognizer?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fills /analyze vehicle.body_type + confidence (car / motorcycle / bus / truck from COCO, mapped to the shared vocabulary) for the Car Wash desk's category suggestion (venue-modules.md §Vehicle category from vision). Advisory: the operator decides, a confident downgrade is flagged, nothing is gated on it. - vision_service/vehicle.py: pure numpy/cv2 letterbox (pad 114, raw BGR), stride-grid decode, class-agnostic NMS, one vehicle per frame (the box holding the plate's centre, else the largest); YoloxVehicleDetector on onnxruntime CPU, 2 intra-op threads. - recognizer.py: WithVehicle composes the stage over any plate recognizer (stub included); a failing stage yields vehicle=null + a "vehicle: …" note in /health.detail — never costs the plate read. model_version reads "+yolox:yolox_s.onnx@640". - settings: VISION_VEHICLE_MODEL_PATH (unset = off), _INPUT_SIZE (640), _MIN_CONFIDENCE (0.4, the detector's floor; the flag threshold is site config). - Dockerfile bakes yolox_s.onnx (best-effort curl at build; no network → stage off) and sets the path; compose forwards it (empty = off); .env.example documents it. - Measured on four real dev entry frames (DS-2CD1047G3H, 2560×1440): car at 0.83–0.88 in ~240–330 ms; empty lane with a person → none. - tests/test_vehicle.py: decode/NMS/pick/letterbox on synthetic tensors, the composition, and a missing-model /health. Wiki: opencv-anpr-service, venue-modules, log. Claude-Session: https://claude.ai/code/session_01FWncR69HgGPuei1dLrW3cU --- apps/vision/.env.example | 9 + apps/vision/Dockerfile | 12 +- apps/vision/tests/test_vehicle.py | 143 +++++++++++++ apps/vision/vision_service/recognizer.py | 56 ++++- apps/vision/vision_service/settings.py | 10 + apps/vision/vision_service/vehicle.py | 247 +++++++++++++++++++++++ docker-compose.yml | 3 + wiki/decisions/venue-modules.md | 12 +- wiki/entities/opencv-anpr-service.md | 35 +++- wiki/log.md | 9 + 10 files changed, 521 insertions(+), 15 deletions(-) create mode 100644 apps/vision/tests/test_vehicle.py create mode 100644 apps/vision/vision_service/vehicle.py diff --git a/apps/vision/.env.example b/apps/vision/.env.example index f738ca8..3b2ab55 100644 --- a/apps/vision/.env.example +++ b/apps/vision/.env.example @@ -20,3 +20,12 @@ VISION_OCR_MODEL=cct-xs-v2-global-model # Confidence floor — a best plate below this is flagged low_confidence so the Node side # treats it as advisory and falls back to the ticket path. Keep in sync with the server. VISION_MIN_CONFIDENCE=0.5 + +# Vehicle stage (phase A): a YOLOX ONNX graph (Apache-2.0) run on the same frame after the +# plate read; fills /analyze `vehicle.body_type` (car/truck/bus/motorcycle) + confidence for +# the Car Wash desk's category suggestion. Unset = off. The Docker image bakes the weights +# at /app/models/yolox_s.onnx; locally: curl the release file into apps/vision/models/. +# https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_s.onnx +# VISION_VEHICLE_MODEL_PATH=models/yolox_s.onnx +# VISION_VEHICLE_INPUT_SIZE=640 +# VISION_VEHICLE_MIN_CONFIDENCE=0.4 diff --git a/apps/vision/Dockerfile b/apps/vision/Dockerfile index 754efc5..63bec35 100644 --- a/apps/vision/Dockerfile +++ b/apps/vision/Dockerfile @@ -14,7 +14,7 @@ ENV UV_LINK_MODE=copy \ # System libs the recognizer stack needs (opencv/onnxruntime): GL + glib. Kept minimal. RUN apt-get update \ - && apt-get install -y --no-install-recommends libgl1 libglib2.0-0 \ + && apt-get install -y --no-install-recommends libgl1 libglib2.0-0 curl \ && rm -rf /var/lib/apt/lists/* # ---- deps: resolve + install the venv from the lockfile (cache-friendly) ---- @@ -26,6 +26,13 @@ RUN --mount=type=cache,target=/root/.cache/uv \ # ---- project source ---- COPY vision_service/ ./vision_service/ COPY README.md ./ +# Vehicle stage weights (phase A): YOLOX-S, Apache-2.0, ~36 MB, baked into the image so the +# air-gapped appliance never fetches at runtime and no operator-writable path holds a model +# (vision-service-hardening.md). Best-effort at build: without network the stage stays off. +ARG YOLOX_URL=https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_s.onnx +RUN mkdir -p /app/models \ + && (curl -fsSL -o /app/models/yolox_s.onnx "$YOLOX_URL" \ + || (echo "[build] yolox weights not fetched (no network) — vehicle stage off" && rm -f /app/models/yolox_s.onnx)) RUN --mount=type=cache,target=/root/.cache/uv \ uv sync --frozen --extra alpr @@ -49,7 +56,8 @@ RUN uv run python -c "from fast_alpr import ALPR; ALPR()" \ # Default to the stub recognizer (offline, no model load); override to fast_alpr in prod. ENV VISION_RECOGNIZER=stub \ VISION_HOST=0.0.0.0 \ - VISION_PORT=8089 + VISION_PORT=8089 \ + VISION_VEHICLE_MODEL_PATH=/app/models/yolox_s.onnx EXPOSE 8089 HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \ CMD python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:8089/health').status==200 else 1)" || exit 1 diff --git a/apps/vision/tests/test_vehicle.py b/apps/vision/tests/test_vehicle.py new file mode 100644 index 0000000..042f995 --- /dev/null +++ b/apps/vision/tests/test_vehicle.py @@ -0,0 +1,143 @@ +"""Vehicle stage (phase A) — pure post-processing on synthetic tensors, and the +recognizer composition over the stub with a fake detector. No weights needed.""" + +from __future__ import annotations + +import os + +import numpy as np +from fastapi.testclient import TestClient + +from vision_service.schemas import BBox, VehicleResult +from vision_service.vehicle import ( + COCO_VEHICLE_CLASSES, + Detection, + decode, + letterbox, + nms, + pick_vehicle, + vehicles_from_output, +) + +SIZE = 64 # tiny "model" input: grids 8x8 + 4x4 + 2x2 = 84 rows +ROWS = (SIZE // 8) ** 2 + (SIZE // 16) ** 2 + (SIZE // 32) ** 2 + + +def raw_output(hits: list[tuple[int, int, int, float, float, float, float]]) -> np.ndarray: + """Build a YOLOX-style raw tensor [ROWS, 85] with the given (row, coco_class, _, obj, + cls_score, log_w, log_h) hits; everything else is background.""" + raw = np.zeros((ROWS, 85), dtype=np.float32) + raw[:, 2:4] = -10.0 # exp → ~0 size for background rows + for row, cls, _, obj, score, lw, lh in hits: + raw[row, 0:2] = 0.5 # centre of its grid cell + raw[row, 2] = lw + raw[row, 3] = lh + raw[row, 4] = obj + raw[row, 5 + cls] = score + return raw + + +def test_decode_maps_grid_offsets_and_log_sizes_to_pixels() -> None: + raw = raw_output([(0, 2, 0, 1.0, 1.0, np.log(2.0), np.log(3.0))]) + dec = decode(raw, SIZE) + # Row 0 = stride-8 grid cell (0,0): centre (0.5+0)*8 = 4, size exp(log 2)*8 = 16 / 24. + assert dec[0, :4].tolist() == [4.0, 4.0, 16.0, 24.0] + # Last row = stride-32 cell (1,1): centre (0.5+1)*32 = 48. + raw2 = raw_output([(ROWS - 1, 7, 0, 1.0, 1.0, 0.0, 0.0)]) + dec2 = decode(raw2, SIZE) + assert dec2[ROWS - 1, :4].tolist() == [48.0, 48.0, 32.0, 32.0] + + +def test_vehicles_only_above_floor_mapped_to_vocabulary_and_scaled_back() -> None: + raw = raw_output( + [ + (0, 2, 0, 0.9, 0.9, np.log(2.0), np.log(2.0)), # car, score .81 + (1, 0, 0, 0.99, 0.99, np.log(2.0), np.log(2.0)), # person → ignored + (2, 7, 0, 0.5, 0.5, np.log(2.0), np.log(2.0)), # truck, score .25 → below floor + ] + ) + found = vehicles_from_output(raw, SIZE, scale=0.5, min_confidence=0.4) + assert [d.body_type for d in found] == ["car"] + assert round(found[0].confidence, 2) == 0.81 + # Box 16px wide in the letterboxed input → 32px in the original (scale 0.5). + assert round(found[0].x2 - found[0].x1) == 32 + assert set(COCO_VEHICLE_CLASSES.values()) == {"car", "motorcycle", "bus", "truck"} + + +def test_nms_keeps_the_best_of_overlapping_boxes() -> None: + boxes = np.array([[0, 0, 10, 10], [1, 1, 11, 11], [50, 50, 60, 60]], dtype=np.float32) + scores = np.array([0.5, 0.9, 0.7], dtype=np.float32) + assert sorted(nms(boxes, scores, 0.45)) == [1, 2] + + +def test_pick_prefers_the_box_holding_the_plate_else_the_largest() -> None: + near = Detection("car", 0.9, 0, 0, 100, 100) + far = Detection("truck", 0.8, 200, 200, 400, 400) # larger + inside = Detection("car", 0.7, 10, 10, 60, 60) # tighter box also holding the plate + assert pick_vehicle([near, far], None) is far + assert pick_vehicle([near, far], BBox(x1=20, y1=20, x2=30, y2=30)) is near + assert pick_vehicle([near, far, inside], BBox(x1=20, y1=20, x2=30, y2=30)) is inside + assert pick_vehicle([near, far], BBox(x1=900, y1=900, x2=910, y2=910)) is far # plate outside every box + assert pick_vehicle([], None) is None + + +def test_letterbox_keeps_aspect_and_pads_with_114() -> None: + frame = np.zeros((30, 60, 3), dtype=np.uint8) + tensor, scale = letterbox(frame, 64) + assert tensor.shape == (1, 3, 64, 64) and tensor.dtype == np.float32 + assert abs(scale - 64 / 60) < 1e-9 + assert tensor[0, 0, 63, 63] == 114.0 # padding + assert tensor[0, 0, 0, 0] == 0.0 # image + + +class FakeDetector: + model_version = "fake-vehicle" + + def __init__(self, result: VehicleResult | None) -> None: + self.result = result + self.calls: list[BBox | None] = [] + + def detect(self, image_bytes: bytes, plate: BBox | None) -> VehicleResult | None: + self.calls.append(plate) + return self.result + + +def test_composition_fills_vehicle_over_the_stub_and_survives_a_failing_stage() -> None: + from vision_service.recognizer import StubRecognizer, WithVehicle + from vision_service.settings import Settings + + det = FakeDetector(VehicleResult(body_type="truck", confidence=0.77)) + rec = WithVehicle(StubRecognizer(Settings()), det) + res = rec.analyze(b"jpeg-bytes") + assert res.plate is None + assert res.vehicle == VehicleResult(body_type="truck", confidence=0.77) + assert res.model_version == "stub-0+fake-vehicle" + assert det.calls == [None] + + class Boom: + model_version = "boom" + + def detect(self, image_bytes: bytes, plate: BBox | None) -> VehicleResult | None: + raise RuntimeError("no model") + + rec2 = WithVehicle(StubRecognizer(Settings()), Boom()) + res2 = rec2.analyze(b"jpeg-bytes") + assert res2.vehicle is None + assert rec2.ready is True + assert "vehicle: RuntimeError: no model" in (rec2.error or "") + + +def test_app_reports_a_missing_model_file_and_keeps_serving() -> None: + from vision_service.app import app + + os.environ["VISION_VEHICLE_MODEL_PATH"] = "/nonexistent/yolox.onnx" + try: + with TestClient(app) as client: + health = client.get("/health").json() + assert health["ready"] is True # the plate stage (stub) is fine + assert "vehicle:" in (health["detail"] or "") + res = client.post("/analyze", content=b"x", headers={"content-type": "application/octet-stream"}) + assert res.status_code == 200 + assert res.json()["vehicle"] is None + finally: + os.environ.pop("VISION_VEHICLE_MODEL_PATH", None) diff --git a/apps/vision/vision_service/recognizer.py b/apps/vision/vision_service/recognizer.py index b4bf287..e342c58 100644 --- a/apps/vision/vision_service/recognizer.py +++ b/apps/vision/vision_service/recognizer.py @@ -19,6 +19,7 @@ from typing import Protocol from .schemas import AnalyzeResponse, BBox, PlateResult from .settings import Settings +from .vehicle import VehicleDetector, YoloxVehicleDetector class Recognizer(Protocol): @@ -165,10 +166,59 @@ class FastAlprRecognizer: ) +class WithVehicle: + """Composition: any plate recognizer + the vehicle stage. Runs the plate stage first + (its box picks WHICH vehicle), then fills `vehicle`. A failing vehicle stage is + logged into `error` and yields null — it must never cost the plate read.""" + + def __init__(self, inner: Recognizer, detector: VehicleDetector) -> None: + self._inner = inner + self._detector = detector + self.vehicle_error: str | None = None + + @property + def model_version(self) -> str: + return f"{self._inner.model_version}+{self._detector.model_version}" + + @property + def ready(self) -> bool: + return bool(self._inner.ready) + + @property + def error(self) -> str | None: + inner = getattr(self._inner, "error", None) + det = getattr(self._detector, "error", None) or self.vehicle_error + parts = [p for p in (inner, f"vehicle: {det}" if det else None) if p] + return "; ".join(parts) if parts else None + + def analyze(self, image_bytes: bytes) -> AnalyzeResponse: + started = time.perf_counter() + res = self._inner.analyze(image_bytes) + try: + vehicle = self._detector.detect(image_bytes, res.plate.bbox if res.plate else None) + except Exception as exc: # noqa: BLE001 - advisory stage, never fatal + self.vehicle_error = f"{type(exc).__name__}: {exc}" + vehicle = None + took_ms = (time.perf_counter() - started) * 1000.0 + return res.model_copy( + update={"vehicle": vehicle, "model_version": self.model_version, "took_ms": took_ms} + ) + + def build_recognizer(settings: Settings) -> Recognizer: """Factory: pick the recognizer from settings. Falls back to the stub if the real - one can't load, so the service always comes up (with ready=False surfaced).""" + one can't load, so the service always comes up (with ready=False surfaced). The + vehicle stage wraps whichever recognizer runs when a model path is configured.""" + rec: Recognizer if settings.recognizer == "fast_alpr": rec = FastAlprRecognizer(settings) - return rec - return StubRecognizer(settings) + else: + rec = StubRecognizer(settings) + if settings.vehicle_model_path: + detector = YoloxVehicleDetector( + settings.vehicle_model_path, + input_size=settings.vehicle_input_size, + min_confidence=settings.vehicle_min_confidence, + ) + return WithVehicle(rec, detector) + return rec diff --git a/apps/vision/vision_service/settings.py b/apps/vision/vision_service/settings.py index 18acefa..176db15 100644 --- a/apps/vision/vision_service/settings.py +++ b/apps/vision/vision_service/settings.py @@ -32,6 +32,16 @@ class Settings(BaseSettings): # Node side can fall back to the ticket path rather than trust it. min_confidence: float = 0.5 + # Vehicle stage (phase A — venue-modules.md §Vehicle category from vision): a YOLOX + # ONNX graph (Apache-2.0) run beside the plate recognizer. Unset = stage off (the + # response's `vehicle` stays null). Bake the file into the image (models/), never a + # path an operator can write (vision-service-hardening.md). + vehicle_model_path: str | None = None + vehicle_input_size: int = 640 + # Detection score floor for a vehicle box to count at all (the Node side applies the + # site's own, stricter threshold before it FLAGS anything). + vehicle_min_confidence: float = 0.4 + def get_settings() -> Settings: return Settings() diff --git a/apps/vision/vision_service/vehicle.py b/apps/vision/vision_service/vehicle.py new file mode 100644 index 0000000..3b1f309 --- /dev/null +++ b/apps/vision/vision_service/vehicle.py @@ -0,0 +1,247 @@ +"""Vehicle stage: a COCO object detector beside the plate recognizer (Job 2, phase A). + +Answers "what KIND of vehicle is in this entry frame?" for the Car Wash desk's category +suggestion (wiki/decisions/venue-modules.md §Vehicle category from vision). ADVISORY by +design: the Node server records it next to the plate, the desk pre-selects the site +category it maps to, the operator decides, a confident downgrade is flagged. Nothing is +ever gated on it, so a wrong or missing detection costs nothing but a suggestion. + +Model: YOLOX (Megvii, Apache-2.0) as an ONNX graph on the ONNX Runtime the plate stage +already uses — the licence rule that keeps Ultralytics (AGPL) out. COCO's vehicle classes +are car / motorcycle / bus / truck: enough to tell a van or a truck from a car, NOT enough +for SUV vs sedan — that is phase B (a body-type classifier on the pilot's own frames). +The detector's vehicle box is also the crop phase B will classify. + +Pure numpy/cv2 pre/post-processing, no torch: letterbox to the model's square input +(pad 114, no normalisation — YOLOX's exported graphs take raw 0–255 BGR), decode the +stride grids, class-agnostic NMS, map COCO ids to the shared vocabulary, pick ONE +vehicle: the one whose box holds the plate (when a plate was read), else the largest. +""" + +from __future__ import annotations + +import time +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Protocol + +from .schemas import BBox, VehicleResult + +# COCO-80 class index → the shared VEHICLE_CLASSES vocabulary (packages/shared). +COCO_VEHICLE_CLASSES: dict[int, str] = {2: "car", 3: "motorcycle", 5: "bus", 7: "truck"} + +# YOLOX feature strides; grids are input/stride per level (8400 anchors at 640). +_STRIDES = (8, 16, 32) + + +@dataclass(frozen=True) +class Detection: + body_type: str + confidence: float + x1: float + y1: float + x2: float + y2: float + + @property + def area(self) -> float: + return max(0.0, self.x2 - self.x1) * max(0.0, self.y2 - self.y1) + + def contains(self, x: float, y: float) -> bool: + return self.x1 <= x <= self.x2 and self.y1 <= y <= self.y2 + + +class VehicleDetector(Protocol): + """What the recognizer composition needs: frame bytes (+ the plate box) → a class.""" + + @property + def model_version(self) -> str: ... + + def detect(self, image_bytes: bytes, plate: BBox | None) -> VehicleResult | None: ... + + +# ---------------------------------------------------------------------------------- +# Pre/post-processing (pure functions — unit-tested on synthetic tensors) +# ---------------------------------------------------------------------------------- + + +def letterbox(frame: Any, size: int) -> tuple[Any, float]: + """Resize keeping aspect, pad bottom/right with 114 to size×size. Returns the CHW + float32 tensor (batch dim added) and the scale to map boxes back.""" + import cv2 + import numpy as np + + h, w = frame.shape[:2] + r = min(size / h, size / w) + nh, nw = int(round(h * r)), int(round(w * r)) + resized = cv2.resize(frame, (nw, nh), interpolation=cv2.INTER_LINEAR) + padded = np.full((size, size, 3), 114, dtype=np.uint8) + padded[:nh, :nw] = resized + tensor = padded.transpose(2, 0, 1)[None].astype(np.float32) + return np.ascontiguousarray(tensor), r + + +def decode(raw: Any, size: int) -> Any: + """YOLOX raw output [N, 5+classes] (batch squeezed) → same shape with xywh decoded + into pixel units of the letterboxed input. Rows are ordered stride 8, 16, 32.""" + import numpy as np + + out = raw.astype(np.float32).copy() + grids = [] + strides = [] + for s in _STRIDES: + n = size // s + ys, xs = np.meshgrid(np.arange(n), np.arange(n), indexing="ij") + grids.append(np.stack((xs, ys), axis=-1).reshape(-1, 2)) + strides.append(np.full((n * n, 1), s, dtype=np.float32)) + grid = np.concatenate(grids, axis=0).astype(np.float32) + stride = np.concatenate(strides, axis=0) + if out.shape[0] != grid.shape[0]: + raise ValueError(f"unexpected output rows {out.shape[0]} for input {size} (want {grid.shape[0]})") + out[:, :2] = (out[:, :2] + grid) * stride + out[:, 2:4] = np.exp(out[:, 2:4]) * stride + return out + + +def nms(boxes: Any, scores: Any, iou_threshold: float) -> list[int]: + """Greedy class-agnostic non-max suppression over xyxy boxes; returns kept indices.""" + import numpy as np + + if len(boxes) == 0: + return [] + order = scores.argsort()[::-1] + x1, y1, x2, y2 = boxes[:, 0], boxes[:, 1], boxes[:, 2], boxes[:, 3] + areas = np.clip(x2 - x1, 0, None) * np.clip(y2 - y1, 0, None) + keep: list[int] = [] + while order.size > 0: + i = int(order[0]) + keep.append(i) + if order.size == 1: + break + rest = order[1:] + xx1 = np.maximum(x1[i], x1[rest]) + yy1 = np.maximum(y1[i], y1[rest]) + xx2 = np.minimum(x2[i], x2[rest]) + yy2 = np.minimum(y2[i], y2[rest]) + inter = np.clip(xx2 - xx1, 0, None) * np.clip(yy2 - yy1, 0, None) + iou = inter / (areas[i] + areas[rest] - inter + 1e-9) + order = rest[iou <= iou_threshold] + return keep + + +def vehicles_from_output( + raw: Any, size: int, scale: float, min_confidence: float, iou_threshold: float = 0.45 +) -> list[Detection]: + """Full post-processing: decode → vehicle classes only → confidence floor → NMS → + boxes in ORIGINAL frame pixels.""" + import numpy as np + + dec = decode(raw, size) + cls_scores = dec[:, 5:] + cls_idx = cls_scores.argmax(axis=1) + score = dec[:, 4] * cls_scores[np.arange(len(dec)), cls_idx] + wanted = np.isin(cls_idx, list(COCO_VEHICLE_CLASSES)) & (score >= min_confidence) + if not wanted.any(): + return [] + d = dec[wanted] + s = score[wanted] + c = cls_idx[wanted] + boxes = np.stack( + (d[:, 0] - d[:, 2] / 2, d[:, 1] - d[:, 3] / 2, d[:, 0] + d[:, 2] / 2, d[:, 1] + d[:, 3] / 2), axis=1 + ) + keep = nms(boxes, s, iou_threshold) + out: list[Detection] = [] + for i in keep: + b = boxes[i] / scale + out.append( + Detection( + body_type=COCO_VEHICLE_CLASSES[int(c[i])], + confidence=float(s[i]), + x1=float(b[0]), + y1=float(b[1]), + x2=float(b[2]), + y2=float(b[3]), + ) + ) + return out + + +def pick_vehicle(detections: list[Detection], plate: BBox | None) -> Detection | None: + """ONE vehicle per frame: the box holding the plate's centre (the car that was read — + a lane frame can show the car behind too), else the largest box (nearest the camera).""" + if not detections: + return None + if plate is not None: + cx = (plate.x1 + plate.x2) / 2 + cy = (plate.y1 + plate.y2) / 2 + holders = [d for d in detections if d.contains(cx, cy)] + if holders: + return min(holders, key=lambda d: d.area) # the tightest box around the plate + return max(detections, key=lambda d: d.area) + + +# ---------------------------------------------------------------------------------- +# The ONNX Runtime detector +# ---------------------------------------------------------------------------------- + + +class YoloxVehicleDetector: + """YOLOX ONNX on onnxruntime (CPU). Loads once; a load failure is surfaced through + `error` and the stage simply yields no vehicle (never breaks the plate path).""" + + def __init__(self, model_path: str, input_size: int = 640, min_confidence: float = 0.4) -> None: + self._path = Path(model_path) + self._size = input_size + self._min_confidence = min_confidence + self._session = None + self._input_name = "images" + self._error: str | None = None + try: + import onnxruntime as ort + + opts = ort.SessionOptions() + opts.intra_op_num_threads = 2 # one frame per entry; leave cores to the lane + self._session = ort.InferenceSession( + str(self._path), sess_options=opts, providers=["CPUExecutionProvider"] + ) + self._input_name = self._session.get_inputs()[0].name + except Exception as exc: # noqa: BLE001 - not-ready, never fatal + self._error = f"{type(exc).__name__}: {exc}" + + @property + def model_version(self) -> str: + return f"yolox:{self._path.name}@{self._size}" + + @property + def ready(self) -> bool: + return self._session is not None + + @property + def error(self) -> str | None: + return self._error + + def detect(self, image_bytes: bytes, plate: BBox | None) -> VehicleResult | None: + if self._session is None: + return None + import cv2 + import numpy as np + + frame = cv2.imdecode(np.frombuffer(image_bytes, dtype=np.uint8), cv2.IMREAD_COLOR) + if frame is None: + return None + tensor, scale = letterbox(frame, self._size) + raw = self._session.run(None, {self._input_name: tensor})[0][0] + found = vehicles_from_output(raw, self._size, scale, self._min_confidence) + best = pick_vehicle(found, plate) + if best is None: + return None + return VehicleResult(body_type=best.body_type, confidence=round(best.confidence, 4)) + + +def time_detect( + detector: VehicleDetector, image_bytes: bytes, plate: BBox | None +) -> tuple[VehicleResult | None, float]: + """detect() with wall time in ms (for logs/benchmarks).""" + started = time.perf_counter() + result = detector.detect(image_bytes, plate) + return result, (time.perf_counter() - started) * 1000.0 diff --git a/docker-compose.yml b/docker-compose.yml index 780a92e..fad0807 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -55,6 +55,9 @@ services: environment: # Engine: stub (no models) by default; prod override sets fast_alpr. VISION_RECOGNIZER: ${VISION_RECOGNIZER:-stub} + # Vehicle stage (Car Wash category suggestion): the image bakes YOLOX-S at this path. + # Set the var to an EMPTY value in the stack env to switch the stage off. + VISION_VEHICLE_MODEL_PATH: ${VISION_VEHICLE_MODEL_PATH-/app/models/yolox_s.onnx} networks: - parking diff --git a/wiki/decisions/venue-modules.md b/wiki/decisions/venue-modules.md index 6f3dd00..acd6921 100644 --- a/wiki/decisions/venue-modules.md +++ b/wiki/decisions/venue-modules.md @@ -188,11 +188,13 @@ onto the site's own categories ("car, sedan, hatchback → Vetura"). and the read is at or above the threshold → one `anomaly` (`carwash.categoryDowngrade`, both categories, both prices, operator, snapshotId) and `downgrade_event_id` on the order. Equal, upgrade, unsure or unmapped reads flag nothing. The order is always created. -- **Model — NOT built.** No bundled recognizer produces `body_type` yet, so today the desk shows - nothing and nothing is flagged. Phase A = a COCO detector on Apache-2.0 ONNX weights (car / - truck / bus / motorcycle, plus the vehicle crop); Phase B = the body-type classifier trained on - the pilot's own frames — every wash order is a labelled frame (entry snapshot + the category a - person chose), so the dataset builds itself on park-2. Reports (discrepancies per operator per +- **Model — phase A built (same day).** YOLOX-S (Apache-2.0 ONNX) as a vehicle stage beside the + plate recognizer: car / truck / bus / motorcycle + the vehicle crop, ~250 ms per entry frame on + CPU, weights baked into the vision image. Details and measurements on [[opencv-anpr-service]] + §Vehicle body type. Phase B = the body-type classifier trained on the pilot's own frames — + every wash order is a labelled frame (entry snapshot + the category a person chose), so the + dataset builds itself on park-2. Until then a Vetura/SUV list sees every car as Vetura and no + downgrade fires; van/truck/bus/motorcycle do separate. Reports (discrepancies per operator per shift) wait for the first real reads. ## Car Wash — the pilot module (settled 2026-09-05) diff --git a/wiki/entities/opencv-anpr-service.md b/wiki/entities/opencv-anpr-service.md index b36fbc3..bb9686a 100644 --- a/wiki/entities/opencv-anpr-service.md +++ b/wiki/entities/opencv-anpr-service.md @@ -249,11 +249,36 @@ service's `/health` each tick and shows a **"Vision" chip** in the booth footer - Per-camera **opt-in** — ✅ **built**: `config.anpr === true` enables ANPR on a camera (set via the SetupWizard checkbox); ANPR then runs on that camera's entry/exit snapshot. -## Vehicle body type (advisory) — contract only, 2026-09-06 +## Vehicle body type (advisory) — the vehicle stage, phase A (2026-09-06) -`/analyze` may now populate `vehicle.body_type` + `vehicle.confidence` from the shared vocabulary +`/analyze` populates `vehicle.body_type` + `vehicle.confidence` from the shared vocabulary (car, sedan, hatchback, suv, minivan, pickup, van, truck, bus, motorcycle). Node records it beside the plate and the Car Wash desk pre-selects the category the site maps it to; the operator -decides, a confident downgrade is flagged, nothing is gated on it. No bundled recognizer emits -it yet — see [[venue-modules]] §Vehicle category from vision for the model plan (COCO detector -first, body-type classifier on own frames second). +decides, a confident downgrade is flagged, nothing is gated on it ([[venue-modules]] §Vehicle +category from vision). + +**Phase A = YOLOX-S (Megvii, Apache-2.0) as ONNX** on the same ONNX Runtime the plate stage uses +— the licence rule that keeps Ultralytics (AGPL) out. `vision_service/vehicle.py`: pure numpy/cv2 +letterbox (pad 114, raw 0–255 BGR — YOLOX's exported graphs are not normalised), stride-grid +decode, class-agnostic NMS, COCO `car/motorcycle/bus/truck` → the vocabulary, and ONE vehicle per +frame: the box holding the plate's centre when a plate was read (the car that was read, not the +one behind), else the largest box. `WithVehicle` in `recognizer.py` wraps whichever plate +recognizer runs (stub included, so the stage is testable without fast-alpr); a failing stage +yields `vehicle: null` and a `vehicle: …` note in `/health.detail` — it never costs the plate +read. Composed `model_version` reads `+yolox:yolox_s.onnx@640`. + +- **Config:** `VISION_VEHICLE_MODEL_PATH` (unset = stage off), `VISION_VEHICLE_INPUT_SIZE` (640), + `VISION_VEHICLE_MIN_CONFIDENCE` (0.4 — the detector's floor; the SITE threshold that decides a + flag lives in Setup → Car wash). The Docker image bakes the weights at `/app/models/yolox_s.onnx` + (best-effort curl at build; no network → stage off) and sets the path, so the air-gapped + appliance never fetches at runtime and no operator-writable path holds a model + ([[vision-service-hardening]]). Compose forwards the var; set it EMPTY in the stack env to + switch the stage off. Locally: curl the release file into `apps/vision/models/` (gitignored). +- **Measured on dev (2026-09-06), four real 2560×1440 entry frames from the DS-2CD1047G3H:** three + with a car → `car` at 0.83–0.88, ~240–330 ms each on the dev CPU with 2 intra-op threads; the + empty-lane frame with a person at the camera → no vehicle (correct: a person is not a class we + keep). One frame per entry, so the cost is invisible to the lane. +- **What it cannot do:** SUV vs sedan — COCO has one `car`. For a Vetura/SUV price list every car + maps to Vetura and no downgrade fires; vans, trucks, buses and motorcycles do separate. Phase B + (a body-type classifier on the pilot's own frames — every wash order is a labelled frame) is + what closes that gap; the detector's box is the crop it will classify. diff --git a/wiki/log.md b/wiki/log.md index 89d708e..f370c3e 100644 --- a/wiki/log.md +++ b/wiki/log.md @@ -3091,3 +3091,12 @@ category + threshold; the desk pre-selects the mapped category and shows the rea a confident, pricier-mapped read with a cheaper choice signs `anomaly carwash.categoryDowngrade` (both categories/prices, operator, snapshot) — never blocks. No recognizer emits body_type yet. Tests in carwash.test.ts. Updated [[venue-modules]] (As built), [[opencv-anpr-service]]. + +## [2026-09-06] ingest | Vision vehicle stage, phase A: YOLOX-S beside the plate recognizer +`vision_service/vehicle.py` (YOLOX ONNX on onnxruntime: letterbox, grid decode, NMS, COCO +car/motorcycle/bus/truck → vocabulary, one vehicle per frame — the box holding the plate, else the +largest) + `WithVehicle` composition over any plate recognizer; `VISION_VEHICLE_MODEL_PATH` (unset = +off), input size, detector floor; Dockerfile bakes yolox_s.onnx (best-effort curl) and sets the +path; compose forwards it (empty = off). Measured on four real dev entry frames: car at 0.83–0.88, +~240–330 ms, empty lane → none. Tests: tests/test_vehicle.py (pure post-processing + composition + +missing-model health). Updated [[opencv-anpr-service]], [[venue-modules]].