parking_solution/wiki/concepts/app-logs.md

type, tags, sources, updated, status

type	tags	sources	updated	status
concept	parking observability diagnostics logging frontend backend		2026-07-08	open

Application logs (diagnostics) — the third stream

A third data stream, deliberately distinct from the two in event-streams-split:

Stream	Table	Signed?	Purpose
Business ledger	ledger_events	✅ ATECC608	money/accountability (append-only-event-chain)
Device telemetry	device_events	❌	hardware chatter (device-events)
App logs	app_logs	❌	operational/diagnostic logs (this page)

App logs answer "why did the booth misbehave?" — a question neither of the other streams should absorb (logs are neither business facts nor hardware telemetry). On an offline appliance there's no Sentry/Datadog to ship to, so the host is the log store: backend warnings/errors AND frontend errors land in one queryable table, viewable at the booth. Built 2026-06-19.

What's captured

• Backend warn / error / fatal — a pino stream tees these into app_logs (and still writes them to stdout, unchanged). info/debug/trace stay stdout-only — they'd bloat the DB. So every app.log.warn/error(...) already in the codebase is now persisted with no call-site change.
• Frontend errors (always): every failed API request (apiFetch's non-OK path → method, path, status, server error body — except 401, which is normal pre-login churn), every uncaught error (window.onerror), every unhandled promise rejection, and a top-level React ErrorBoundary (a render crash is reported as fatal instead of a white screen).
• console.warn / console.error — only forwarded when the client log level is debug/trace (off by default; they're noisy with third-party chatter). The high-signal sources above are always on. Toggle via VITE_LOG_LEVEL / setClientLogLevel().

Shape

app_logs: level (pino names), source (frontend|backend), message, context (one JSON column — the failed request, error name, component stack, anything), plus pulled-out httpStatus / path for cheap filtering, stack, userId, userAgent, createdAt. Indexed on created_at + level. Shared types: AppLogRecord / ClientLogInput / LogLevel in @parking/shared.

The API + the access split

• POST /api/logs — the frontend ships errors here. Any signed-in user may write (it's their own browser's diagnostics) — requireAuth, not a permission. CSRF still applies (it's a mutation). Accepts one entry or a { entries: [...] } batch (capped at 50). Deliberately never 4xx's on a malformed entry — a client erroring while reporting an error must not get a second error.
• GET /api/logs — read the store (level/source/since filters), gated by the new log:read permission (a new log resource in the dynamic local-jwt-auth grid). Admin holds it; it's grantable to a diagnostic role. Verified: a cashier without log:read gets 403 on GET but 204 on POST — the intended asymmetry.

Reliability invariants (a logger must never make things worse)

• No infinite loop. The frontend collector never logs the /api/logs request itself, and flushes via raw fetch/sendBeacon, not apiFetch (so a flush failure can't recurse into a new log). The backend LogService has a reentrancy guard — persisting a log can't emit a persisted log.
• Best-effort, never fatal. Every write is wrapped; a DB/logging failure is swallowed (it can't be logged — that's the recursion we guard). Diagnostics must never break the path they observe.
• Bounded. Frontend queue capped (drops oldest); message/stack/context clamped per row; ingest batch capped.
• Storm coalescing (2026-07-08). A line identical to the last persisted row (level+source+message+path) arriving within 5 min of its previous occurrence updates that row instead of inserting: context._repeat counts the fold, context._firstAt keeps the first occurrence, createdAt moves to the latest (so the storm stays at the top of the newest-first viewer, which badges it ×N). A continuous storm refreshes the window each hit, so it stays ONE row however long it rages. Motivation: the 2026-07-07 field incident — one unreachable controller (ENETUNREACH) produced hundreds of identical error rows per minute, evicting unrelated history (see button-light-indicator for the send-side fix: retry backoff + rate-limited logging). In-memory last-row cache only (a restart just starts a fresh row); if the row was pruned underneath, it falls through to a fresh insert.

Retention (offline appliance ⇒ must be bounded)

Pruned by age AND a row cap (a burst could blow past an age-only window): delete older than LOG_RETENTION_DAYS (default 60 — the operator's ≈2-month diagnostic window, 2026-07-04; was 30) and keep only the newest LOG_RETENTION_MAX_ROWS (default 50 000). Runs hourly (unref'd timer) + once at startup. Both env-configurable. Same "prunable, not precious" durability class as device_events — the opposite of the append-only ledger.

Also wipeable on demand: reset-db.mjs --diagnostics (new category 2026-07-08 — app_logs previously belonged to NO category and silently survived even --all; a drift guard in the script now refuses to run if any table is uncategorized). See local-dev-workflow.

Container (stdout) logs — the OTHER log store (2026-07-04)

docker logs is a separate, size-bounded store from app_logs — it holds everything (info/debug too), while app_logs keeps only warn+. Three knobs, all set 2026-07-04:

• Rotation: Docker's json-file driver rotates by SIZE, not time; the caps in docker-compose.prod.yml are sized to hold ≈2 months at observed booth rates (server 20 MB × 30, vision 20 MB × 10, proxy 10 MB × 5). docker logs reaches back only that far — revisit the caps if it shows under ~60 days. (Dev compose is uncapped — laptop concern only.)
• Human-readable lines: the pino logger stamps ISO-8601 UTC time (was epoch-ms) and level NAMES ("warn", was 40) via timestamp + formatters.level in server.ts. pinoDbStream accepts BOTH level encodings, so the app_logs tee survives either config.
• Level knob: LOG_LEVEL env (trace|debug|info|warn|error|fatal; default info) — a booth under diagnosis runs LOG_LEVEL=debug with no code change; warn+ persistence is unaffected (it filters independently in the tee).

The booth viewer

A Logs screen under Setup (/setup/logs, gated by log:read, sq+en) — filter by level/source/since, newest first, each row expands to the structured context + stack. Read-only (logs are evidence, never edited). Polls every 15 s (no WS — diagnostics aren't latency-critical). Sits alongside the other admin tabs in booth-console.

As-built (2026-06-19)

• packages/db: app_logs table + migration 0009_app_logs.sql (+ journal idx 9; seeds admin log:read). Applied to the live apps/server/parking.sqlite.
• @parking/shared: log resource + log:read permission; LogLevel/LogSource/AppLogRecord/ ClientLogInput/LOG_LEVEL_ORDER.
• apps/server: log-service.ts (LogService + pinoDbStream), routes/logs.ts, wired in server.ts (DB built before Fastify so the pino stream has the sink; prune timer).
• apps/web: lib/logger.ts (collector + global handlers), lib/ErrorBoundary.tsx, apiFetch hook, LogsViewer.tsx + route/nav, i18n.

Open

• No automated test (the standing harness gap) — though the ingest/read/gate path was verified by in-process app.inject smoke (login → POST 204 → GET 200 with the record; non-admin 403/204 split).
• Server API error strings stay English — unchanged here; this is about persisting logs, not localizing them. The localized-ledger-reason pattern (i18n) is the template if log display ever needs translation (currently the message is whatever the thrower wrote).
• Surfacing critical logs live — a fatal/error count badge on the booth footer over the existing /api/ws could flag problems without opening the viewer. Deferred.
• Correlation id — no request-id threads a frontend failed-request log to its backend log yet; add a x-request-id echo if cross-stream correlation is wanted.