parking_solution/wiki/entities/opencv-anpr-service.md

type, tags, sources, updated, status

type	tags	sources	updated	status
entity	parking vision anpr anti-fraud service		2026-06-15	open

OpenCV ANPR / Vision Service

A local microservice that analyses camera snapshots: reads the licence plate (ANPR) and extracts vehicle attributes for verification. Built by us (decision 2026-06-15) to do recognition host-side on ordinary IP-camera snapshots, replacing the dedicated edge-AI lpr-camera. See decision vision-service.

Two jobs

1. Identity (ANPR). snapshot → { plate, confidence, bbox }. Feeds the existing IdentitySource = "lpr" (parking-session): the plate is a session/identity key and the way a plate-bound subscription is matched.
2. Verification (anti-fraud witness). snapshot → vehicle attributes — at minimum { make?, model?, colour, bodyType }, ideally a compact visual fingerprint (an embedding). This is the answer to plate-spoofing: a fraudster prints a registered/paid plate and drives in with a different car. Plate-reading alone can't catch that; comparing the vehicle seen at entry vs. exit (and vs. the subscription's known car) can. A plate that entered on a red hatchback but exits on a black SUV is a reconciliation anomaly — exactly the independent-witness role the append-only-event-chain flags as the unbuilt gap. See reconciliation.

The two jobs are why this is worth building rather than just plate-OCR: the service is both an identity source and an independent witness, the visual analogue of the whole system's "two records that must reconcile" thesis.

Architecture — separate localhost process

• A Python service (e.g. FastAPI) running on the appliance, called by the Node backend over localhost HTTP (POST /analyze with the JPEG bytes the camera driver already pulls — see lpr-camera "driver/storage boundary": Snapshot.bytes). Source lives in this monorepo at apps/vision/ (Turbo shim; uv-managed deps) — co-located source, separate process; see vision-service-packaging.
• Fully offline (offline-first): all inference is local, no cloud. Model weights ship on the appliance.
• Process isolation is deliberate — it keeps a heavy Python/native/AGPL stack out of the Node app's process and license surface (see licensing below), and gives it its own failure domain. If the service is down/slow, the host falls back (transient ticket path) rather than blocking the lane.
• Request/response (first cut):
  • POST /analyze → { plate: {text, confidence, bbox}|null, vehicle: {colour, bodyType, make?, model?, embedding?}, modelVersion, tookMs }
  • GET /health → readiness + model versions.
• The Node side wraps it behind an internal interface (like a device adapter) so the recognizer can be swapped without touching business logic.

Licensing — scoped AGPL exception (amends the standing rule)

The app is strictly MIT/Apache/BSD (technology-stack, standing-decisions). Accurate ANPR/vehicle models were assumed to be mostly AGPL (Ultralytics YOLO detectors, OpenALPR) or commercial — but the fast-alpr stack (above) is MIT end-to-end, so a permissive ANPR baseline now looks achievable (pending the weight-provenance caveat). The exception below still matters for the strongest models (Ultralytics YOLO) and for the vehicle-verification job. Decision (2026-06-15): allow AGPL inside this service only. It is a separate process, not linked into the app, so its obligations don't reach the Node/React codebase; the app's permissive guarantee is preserved. Recorded as an explicit exception in standing-decisions / vision-service.

• OpenCV core itself is Apache-2.0 (clean either way).
• AGPL note: if the appliance is ever offered as a network service to third parties, AGPL's network-use clause could require offering the service's source — relevant only if productised beyond the on-site appliance; flag at that point.

Recognizer evaluation — fast-alpr is the leading baseline (2026-06-19)

YOLO vs OpenCV is a category error — they're different pipeline layers, not competitors. ANPR is a pipeline: (1) plate detection (find the box → YOLO-family detector), (2) plate OCR (read the crop → a CRNN/CCT or OCR engine), (3) glue (capture/crop/deskew/draw → OpenCV, Apache-2.0, always present). So the real choice is which end-to-end recognizer, and OpenCV is used regardless as the image-handling toolkit.

Leading option: fast-alpr (v0.4.0, 15 Mar 2026). A thin orchestrator over two swappable stages, both on ONNX Runtime — which matches THIS service's decided architecture (separate localhost Python process, offline, swappable behind an interface) almost exactly:

Stage	Default model	Library	License
Plate detection	yolo-v9-t-384-license-plate-end2end	open-image-models	MIT
Plate OCR	cct-xs-v2-global-model	fast-plate-ocr	MIT

• MIT top-to-bottom (library and the published model weights), one maintainer (ankandrew) across all three repos. The detector is open-image-models' own YOLOv9 ONNX export — NOT the Ultralytics AGPL package — so fast-alpr is a permissive baseline that may not even need the scoped AGPL exception below. ⚠️ Caveat (verify before relying on it): a repo's LICENSE covers its code; redistributed model weights can carry separate provenance (YOLOv9 upstream is GPL-3.0; Ultralytics YOLO is AGPL). Confirm the weight training/provenance (model card) before treating "MIT weights" as settled for compliance — the AGPL-in-service exception is the safety net if it doesn't hold.
• CPU-only + fully offline. No runtime ships by default; pick a backend extra — fast-alpr[onnx] (CPU), or [onnx-gpu]/[onnx-openvino]/[onnx-directml]/[onnx-qnn] — which maps onto the "CPU now, small GPU/NPU later" compute question (bom, open-questions).
• Albanian/EU plates: fast-plate-ocr also has a European model trained on 40+ countries (newer than the default global model) — benchmark it against the default for AL accuracy.
• Modular, no lock-in: swap either stage via BaseDetector/BaseOCR (their docs plug in Tesseract). So fast-alpr is the baseline you keep while replacing one stage if needed.

Scope: fast-alpr is plate-only — it does Job 1 (ANPR) but NOT Job 2 (vehicle verification). The anti-spoofing vehicle-attribute/fingerprint stage is still ours to build — but since fast-alpr already standardizes on ONNX Runtime + a YOLO-family detector, the vehicle stage shares that runtime (the coherent outcome). Other options, weaker: OpenALPR (permissive but largely unmaintained, the old "permissive-only, weaker" path); Ultralytics YOLO + PaddleOCR (most accurate/tunable, but YOLO is AGPL → needs the in-service exception; most build effort — the "scale" path if fast-alpr's accuracy disappoints).

Recommendation: prototype with fast-alpr now (permissive, offline, ONNX, fits the decided shape); plan a YOLO-detector fine-tune + PaddleOCR only if production accuracy demands it. Choice kept open pending the weight-provenance check (the AL-plate benchmark below is now done).

Albanian-plate OCR benchmark — keep the default (2026-06-19)

Ran the four candidate fast-plate-ocr models through the full pipeline (YOLOv9 detect → OCR) on real AL plate photos (Wikimedia), CPU, scaffolded service:

OCR model	AA 558 EE	AA 687 KE	Speed	Note
cct-xs-v2-global-model (default)	✓ 0.999	✓ 1.000	33–39 ms	best accuracy + fastest; returns region=Albania
cct-s-v2-global-model	✓ 0.998	✓ 0.999	50–65 ms	as accurate, ~50% slower
global-plates-mobile-vit-v2-model	✓ 0.955	✓ 0.959	33–35 ms	fast, lower confidence
european-plates-mobile-vit-v2-model	✓ 0.784	✓ 0.766	38–46 ms	correct but much lower confidence; misread a synthetic AB123FG→AB123FO

Finding (overturns the prior assumption): the default cct-xs-v2-global-model is the best for Albania — most accurate AND fastest. The "European (40+ country)" model is worse here (~0.77 vs ~1.0 confidence, one synthetic misread), despite the "EU model → better for AL" intuition. So no config change: VISION_OCR_MODEL stays cct-xs-v2-global-model. Caveat: both test photos were clean head-on shots; real booth captures (angled, dirty, night, motion-blur) will lower absolute confidence — the min_confidence=0.5 floor (→ low_confidence → ticket-path fallback) covers that. The ranking should hold; re-benchmark on real on-site captures once the cameras are installed.

Anti-fraud / threat-model fit

• Plate spoofing (the motivating case): vehicle-attribute / fingerprint mismatch entry↔exit or vs. a subscription's registered car → anomaly. Doesn't block on its own (recognition is probabilistic) — it flags for reconciliation and is captured in the signed record.
• The recognition result and the source image both attach to the signed append-only-event-chain entry, so the evidence is tamper-evident even though recognition itself is host-side and fallible.
• Recognition is advisory, never the sole authority to open a barrier where money/access is at stake — confidence thresholds + fallback to ticket/manual; a low-confidence read must not strand a car (fail-state-safety).

Fitness for the entry/exit flows (assessment, 2026-06-19)

Asked after the scaffold + AL benchmark: is the service worthy to consume in the entry/exit flows? The benchmark settles accuracy (0.99+ on clean AL plates); "worthy" then turns on what authority the read is given — and the answer splits by role:

• ✅ Worthy NOW — as an ADVISORY identity source (Job 1). The flows are already built for a plate: a kind:"plate" device-events is a first-class identity today — exit-flow.ts signs source:"lpr" for it, and subscription-flow.ts matches a read plate against subscriptionPlates (subscription plate binding). So the service just produces the plate string a snapshot → /analyze → (if confident) a DeviceReadEvent{kind:"plate"} on the existing read bus. No flow rewrite — it feeds an existing input. Concretely worthy for: hands-free subscriber barrier open (plate-bound), and evidence enrichment (plate + image on the signed entry/exit for disputes).
• ⚠️ NOT worthy as the SOLE AUTHORITY to open a TRANSIENT barrier. Two threat-model reasons: (1) a plate is not a payment — a transient still needs a ticket + payment; letting a plate open the exit would be an unpaid-exit bypass. The min_confidence floor → low_confidence → ticket/manual fallback is the guard (already in the scaffold). (2) Plate-spoofing (a printed plate on a different car) — plate-only ANPR cannot catch it; that needs Job 2 (vehicle verification), which is NOT built. So plate-as-identity is convenience + evidence, never the lone reason a paid barrier opens. Consistent with "advisory, never sole authority" above.

Gaps before it's actually consumed (capable ≠ wired): (1) ✅ DONE — the Node→service VisionClient adapter (apps/server/src/vision-client.ts, localhost HTTP to /analyze + /health) now exists: opt-in (VISION_ENABLED, default off), fail-soft (any error/timeout/unreachable → null, never throws into the lane → ticket-path fallback), and re-applies the confidence floor (VISION_MIN_CONFIDENCE) so a low read is flagged advisory. Constructed in server.ts; verified end-to-end against the live service (Node → AA558EE 0.999, region=Albania). (2) ✅ DONE — trigger wiring (apps/server/src/vision-reader.ts). A VisionReader polls each opt-in camera (config.anpr === true, off by default) every VISION_POLL_MS, captures a snapshot → VisionClient.analyze → on a confident plate dispatches a DeviceReadEvent{kind:"plate"} through the same ReadDispatcher a physical reader uses (called directly to capture the outcome, like qr-reader.ts), so the subscription/exit flow consumes it unchanged. Guards: low-confidence reads are dropped (not an identity); a debounce (VISION_DEDUPE_MS) stops the same plate re-firing while a car sits in frame; an in-flight guard prevents overlapping recognizes; idle when vision is off or no camera opts in. Plate stays advisory + non-blocking — the exit flow still demands a payment, the subscription flow only matches a bound plate, and a refused read never holds a barrier.

Every confident read is PERSISTED (the ANPR audit trail, so a read is investigable): the snapshot bytes are stored in snapshots keyed by identity = plate — the SAME identity the flow signs its anomaly/event with — so GET /api/snapshots/by-identity/:plate (the booth event-detail modal's snapshot strip) shows the car's photo against that anomaly with no extra wiring; plus an unsigned device_events{kind:"read"} breadcrumb records plate / confidence / region / model / snapshotId / the dispatch outcome (accepted + reason) — a queryable log of every recognition and whether it matched, separate from the signed ledger. Verified end-to-end: a recognized AL plate with no open session was non-blocking → signed a exit.refused.noSession anomaly (identity=plate), stored a 555 KB snapshot under that plate, recorded the read breadcrumb with accepted:false, reason:"…no open session…", and by-identity returned the image — i.e. the refused read is fully investigable with its picture. Debounce held a re-seen plate to 1 emit over 7 polls. (3) field-accuracy unknown — re-benchmark/tune the threshold on real on-site captures (angle/night/dirt). (4) the weight-provenance check (open). Bottom line: consume it as a gated advisory identity source feeding the existing kind:"plate" path — not as sole authority — and Job 2 is still required for the anti-spoofing value. The adapter + the opt-in poll→read trigger are now both built and verified end-to-end; remaining is field tuning (3), the provenance check (4), the SetupWizard anpr toggle, and Job 2.

Open

• Recognizer choice — fast-alpr (MIT, YOLOv9+CCT on ONNX) is the baseline, AL-benchmarked: the default cct-xs-v2-global-model won over the EU model on real AL plates (table above). The one remaining open item is the model-weight-provenance check (the MIT-weights claim). A re-benchmark on real on-site captures (angled/night/dirty) is wanted once cameras are installed. See vision-service; AGPL still permitted in-service for the stronger fallback.
• Vehicle fingerprint: attribute classifier vs. embedding-similarity; what threshold makes a mismatch an anomaly without false-positiving on lighting/angle.
• Compute footprint on the appliance (CPU-only vs. a small GPU/NPU) — procurement input (bom, open-questions).
• Per-camera opt-in — ✅ mechanism built: config.anpr === true on a camera enables ANPR polling (the VisionReader). Remaining: expose the toggle in the SetupWizard (it's currently set in raw config) and decide sensible VISION_POLL_MS/VISION_DEDUPE_MS defaults per site.