docs(wiki): log-storm hardening + reset drift guard (2026-07-07 incident)
button-light-indicator: failure backoff + rate-limited logging rationale; app-logs: storm coalescing invariant + --diagnostics wipe; local-dev-workflow and appliance-provisioning §7d: new reset flag table + drift guard; log entry tying all three layers to the ENETUNREACH incident. Claude-Session: https://claude.ai/code/session_01Xcm6ikLgGoCxxHrxtjkk5V
This commit is contained in:
@@ -2,7 +2,7 @@
|
|||||||
type: concept
|
type: concept
|
||||||
tags: [parking, observability, diagnostics, logging, frontend, backend]
|
tags: [parking, observability, diagnostics, logging, frontend, backend]
|
||||||
sources: []
|
sources: []
|
||||||
updated: 2026-07-04
|
updated: 2026-07-08
|
||||||
status: open
|
status: open
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -62,6 +62,16 @@ column — the failed request, error name, component stack, anything), plus pull
|
|||||||
logged — that's the recursion we guard). Diagnostics must never break the path they observe.
|
logged — that's the recursion we guard). Diagnostics must never break the path they observe.
|
||||||
- **Bounded.** Frontend queue capped (drops oldest); message/stack/context clamped per row;
|
- **Bounded.** Frontend queue capped (drops oldest); message/stack/context clamped per row;
|
||||||
ingest batch capped.
|
ingest batch capped.
|
||||||
|
- **Storm coalescing (2026-07-08).** A line identical to the *last persisted row*
|
||||||
|
(level+source+message+path) arriving within **5 min** of its previous occurrence **updates that
|
||||||
|
row** instead of inserting: `context._repeat` counts the fold, `context._firstAt` keeps the first
|
||||||
|
occurrence, `createdAt` moves to the latest (so the storm stays at the top of the newest-first
|
||||||
|
viewer, which badges it `×N`). A *continuous* storm refreshes the window each hit, so it stays ONE
|
||||||
|
row however long it rages. Motivation: the 2026-07-07 field incident — one unreachable controller
|
||||||
|
(`ENETUNREACH`) produced hundreds of identical error rows per minute, evicting unrelated history
|
||||||
|
(see [[button-light-indicator]] for the send-side fix: retry backoff + rate-limited logging).
|
||||||
|
In-memory last-row cache only (a restart just starts a fresh row); if the row was pruned
|
||||||
|
underneath, it falls through to a fresh insert.
|
||||||
|
|
||||||
## Retention (offline appliance ⇒ must be bounded)
|
## Retention (offline appliance ⇒ must be bounded)
|
||||||
|
|
||||||
@@ -71,6 +81,10 @@ was 30) **and** keep only the newest `LOG_RETENTION_MAX_ROWS` (default 50 000).
|
|||||||
(unref'd timer) + once at startup. Both env-configurable. Same "prunable, not precious"
|
(unref'd timer) + once at startup. Both env-configurable. Same "prunable, not precious"
|
||||||
durability class as `device_events` — the opposite of the append-only ledger.
|
durability class as `device_events` — the opposite of the append-only ledger.
|
||||||
|
|
||||||
|
Also wipeable on demand: `reset-db.mjs --diagnostics` (new category 2026-07-08 — `app_logs`
|
||||||
|
previously belonged to NO category and silently survived even `--all`; a drift guard in the script
|
||||||
|
now refuses to run if any table is uncategorized). See [[local-dev-workflow]].
|
||||||
|
|
||||||
## Container (stdout) logs — the OTHER log store (2026-07-04)
|
## Container (stdout) logs — the OTHER log store (2026-07-04)
|
||||||
|
|
||||||
`docker logs` is a separate, size-bounded store from `app_logs` — it holds **everything**
|
`docker logs` is a separate, size-bounded store from `app_logs` — it holds **everything**
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
type: concept
|
type: concept
|
||||||
tags: [parking, device, indicator, radar, camera, aux-output, barrier-not-a-door, event-relay]
|
tags: [parking, device, indicator, radar, camera, aux-output, barrier-not-a-door, event-relay]
|
||||||
sources: []
|
sources: []
|
||||||
updated: 2026-06-28
|
updated: 2026-07-08
|
||||||
status: settled
|
status: settled
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -72,6 +72,16 @@ aux-output** capability.
|
|||||||
re-converges to the latest desired state. So the **final state is always authoritative** and a
|
re-converges to the latest desired state. So the **final state is always authoritative** and a
|
||||||
lost/stale packet self-corrects. This also de-dupes (it skips a send when `confirmedOn === desiredOn`),
|
lost/stale packet self-corrects. This also de-dupes (it skips a send when `confirmedOn === desiredOn`),
|
||||||
so the input stream never spams the controller.
|
so the input stream never spams the controller.
|
||||||
|
- **Failure backoff + rate-limited logging (2026-07-08).** The first serialized-worker cut re-pumped
|
||||||
|
*immediately* after a FAILED send (`confirmedOn` unchanged → converge again) — correct for a lost
|
||||||
|
packet, but an **unreachable controller** (`ENETUNREACH`, rejects instantly) turned it into a hot
|
||||||
|
loop: hundreds of identical error lines per minute into stdout AND [[app-logs]] (field incident
|
||||||
|
2026-07-07, park-buzi). Now a failed send arms a **retry backoff — 1 s doubling to a 30 s cap,
|
||||||
|
reset on success**; during the window `desiredOn` keeps tracking the truth table and the armed
|
||||||
|
retry converges to whatever it says when it fires (`#finalOff` waives the backoff for the one-shot
|
||||||
|
last-gasp OFF). Logging: only the **first** failure of a streak is logged, then **one summary per
|
||||||
|
minute** (`still failing (attempt N…)`), and a single `info` on recovery. The app_logs sink
|
||||||
|
additionally coalesces identical rows (see [[app-logs]]) as defense in depth.
|
||||||
- **Hot-reloads the config (no restart).** The lamp map is reconciled against the live device config
|
- **Hot-reloads the config (no restart).** The lamp map is reconciled against the live device config
|
||||||
at start AND before each event (mirroring [[device-status-monitoring|DeviceMonitor]], which re-reads
|
at start AND before each event (mirroring [[device-status-monitoring|DeviceMonitor]], which re-reads
|
||||||
the device set each tick) — adding/updating/dropping lamps. So a button light added or re-pointed in
|
the device set each tick) — adding/updating/dropping lamps. So a button light added or re-pointed in
|
||||||
@@ -102,7 +112,8 @@ Built 2026-06-24 for the first booth (button I1, radar I2, lamp on a spare relay
|
|||||||
a `radarAlert` event-relay (carrying its own `triggerInput`), so the operator can add arbitrary
|
a `radarAlert` event-relay (carrying its own `triggerInput`), so the operator can add arbitrary
|
||||||
event-driven blinkers (e.g. R4) without code changes; the 3-state machine itself is unchanged.
|
event-driven blinkers (e.g. R4) without code changes; the 3-state machine itself is unchanged.
|
||||||
Covered by `apps/server/src/button-light.test.ts` (the truth table, blink toggling asserted on the
|
Covered by `apps/server/src/button-light.test.ts` (the truth table, blink toggling asserted on the
|
||||||
device's *confirmed* state, fail-OFF, de-dupe, lamp-added-after-start reconcile, and two independent
|
device's *confirmed* state, fail-OFF, de-dupe, lamp-added-after-start reconcile, two independent
|
||||||
alert relays on one controller).
|
alert relays on one controller, and — since 2026-07-08 — backoff cadence on an unreachable
|
||||||
|
controller, log rate-limiting, and single-recovery-line + backoff-reset after success).
|
||||||
Related: [[hikvision-radar]], [[entry-double-press]], [[lpr-camera]], [[dingtian-relay]],
|
Related: [[hikvision-radar]], [[entry-double-press]], [[lpr-camera]], [[dingtian-relay]],
|
||||||
[[entry-exit-points]], [[barrier-not-a-door]].
|
[[entry-exit-points]], [[barrier-not-a-door]].
|
||||||
|
|||||||
@@ -71,10 +71,16 @@ RESET_ALLOWED=1 DATABASE_URL=/path node packages/db/scripts/reset-db.mjs --all
|
|||||||
| Flag | Wipes | Keeps |
|
| Flag | Wipes | Keeps |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| `--financial` | `ledger_events` (entry/exit/payment/void/shift/cash/anomaly), `device_events`, `snapshots`, subscription **instances** + credentials/plates, `blocklist` | users, devices, config, tariffs, subscription **plans** |
|
| `--financial` | `ledger_events` (entry/exit/payment/void/shift/cash/anomaly), `device_events`, `snapshots`, subscription **instances** + credentials/plates, `blocklist` | users, devices, config, tariffs, subscription **plans** |
|
||||||
| `--config` | `site_config`, `devices`, `setup_state` (→ re-runs first-run setup), tariffs + versions, subscription plans | everything else |
|
| `--config` | `site_config`, `devices`, `setup_state` (→ re-runs first-run setup), tariffs + versions + **drafts**, subscription plans | everything else |
|
||||||
| `--users` | `users`, `roles`, `role_permissions`, auth `sessions` | everything else |
|
| `--users` | `users`, `roles`, `role_permissions`, auth `sessions` | everything else |
|
||||||
|
| `--diagnostics` | `app_logs` (the unsigned [[app-logs]] store behind `/setup/logs`) | everything else |
|
||||||
| `--all` | every table (blank slate) | — |
|
| `--all` | every table (blank slate) | — |
|
||||||
|
|
||||||
|
**Drift guard** (2026-07-08): before doing anything, the script compares the category union against
|
||||||
|
`sqlite_master` and **refuses if any table is uncategorized** — `app_logs` and `tariff_drafts` had
|
||||||
|
silently survived every reset (including `--all`) because the hand-maintained table list lagged the
|
||||||
|
schema. A new table now forces a deliberate one-line categorization decision.
|
||||||
|
|
||||||
> **⚠ `--financial`/`--all` TRUNCATE the append-only, signed [[append-only-event-chain|ledger]].**
|
> **⚠ `--financial`/`--all` TRUNCATE the append-only, signed [[append-only-event-chain|ledger]].**
|
||||||
> That is the anti-fraud record; a *partial* delete would break the hash chain, so a financial reset
|
> That is the anti-fraud record; a *partial* delete would break the hash chain, so a financial reset
|
||||||
> wipes the whole ledger back to empty (re-seeding starts a NEW chain under the **same**
|
> wipes the whole ledger back to empty (re-seeding starts a NEW chain under the **same**
|
||||||
|
|||||||
@@ -381,8 +381,11 @@ entrypoint). `DATABASE_URL` in-container is **`/data/parking.sqlite`** (the `par
|
|||||||
# --financial ledger (entry/exit/payment/void/shift/cash/anomaly) + device_events + snapshots +
|
# --financial ledger (entry/exit/payment/void/shift/cash/anomaly) + device_events + snapshots +
|
||||||
# subscription INSTANCES/credentials/plates + blocklist. KEEPS users/devices/config/
|
# subscription INSTANCES/credentials/plates + blocklist. KEEPS users/devices/config/
|
||||||
# tariffs/subscription PLANS.
|
# tariffs/subscription PLANS.
|
||||||
# --config site_config, devices, setup_state (re-runs first-run setup), tariffs + versions, plans.
|
# --config site_config, devices, setup_state (re-runs first-run setup), tariffs + versions
|
||||||
# --users users, roles, role_permissions, auth sessions. --all every table.
|
# + drafts, plans.
|
||||||
|
# --users users, roles, role_permissions, auth sessions.
|
||||||
|
# --diagnostics app_logs (the /setup/logs store). --all every table.
|
||||||
|
# A drift guard refuses to run if the DB has a table no category covers (2026-07-08).
|
||||||
docker exec -it \
|
docker exec -it \
|
||||||
-e RESET_ALLOWED=1 \
|
-e RESET_ALLOWED=1 \
|
||||||
-e DATABASE_URL=/data/parking.sqlite \
|
-e DATABASE_URL=/data/parking.sqlite \
|
||||||
|
|||||||
+15
@@ -2525,3 +2525,18 @@ deferred (→ delivered by the lab fee breakdown); [[tariff]] lab section gained
|
|||||||
composer increment-guard paragraph; [[i18n]] now records the "25 Qer 14:30" date standard and the
|
composer increment-guard paragraph; [[i18n]] now records the "25 Qer 14:30" date standard and the
|
||||||
never-toLocaleString-for-dates rule; [[fleet-deployment-komodo]] gained the park-lab stack + tier
|
never-toLocaleString-for-dates rule; [[fleet-deployment-komodo]] gained the park-lab stack + tier
|
||||||
table (also logging the park-lab addition itself, which had slipped the log).
|
table (also logging the park-lab addition itself, which had slipped the log).
|
||||||
|
|
||||||
|
## [2026-07-08] update | Log-storm hardening + reset-db drift guard
|
||||||
|
|
||||||
|
Field incident 2026-07-07: an unreachable UHPPOTE (`ENETUNREACH 10.0.10.5:60000`) put the
|
||||||
|
button-light `#pump` worker in a zero-backoff hot loop — hundreds of identical `setAux failed`
|
||||||
|
error rows per minute into [[app-logs]]. Three-layer fix: (1) failed sends now arm a 1s→30s
|
||||||
|
exponential retry (reset on success), with only the first failure logged, one summary/minute
|
||||||
|
after, and one info on recovery ([[button-light-indicator]] §Implementation); (2) LogService
|
||||||
|
coalesces a row identical to the last (level+source+message+path, 5-min refreshing window) by
|
||||||
|
bumping `context._repeat` instead of inserting — the viewer badges `×N` ([[app-logs]]);
|
||||||
|
(3) the user's training reset had ALSO left logs behind: `app_logs` and `tariff_drafts` belonged
|
||||||
|
to no reset-db category, silently surviving even `--all`. Added `--diagnostics` (app_logs), put
|
||||||
|
tariff_drafts under `--config`, and a drift guard that refuses to run when any table is
|
||||||
|
uncategorized ([[local-dev-workflow]], [[appliance-provisioning]] §7d). 8 new tests
|
||||||
|
(3 button-light backoff, 5 coalescing); guard + both new wipes verified on a scratch DB.
|
||||||
|
|||||||
Reference in New Issue
Block a user