docs(camera): Hik event-push gotchas + dead-camera diagnostic method
Captures the hard-won findings from the field session: the WSL source-IP rewrite + skipSourceIpCheck fix, the boolean-as-string setup bug, the unreliable "Test" button, the latching httpBroken flag, and Notify- Surveillance-Center vs HTTP-Alarm-Server. Adds a "diagnose a non-pushing camera from its OWN state" runbook (alertStream heartbeat silence, SSH showStatus EventScribe:except, dmesg RTC/UBIFS, netstat outbound watch) and documents the verified-dead DS-2CD1043G2-LIU unit (defective event engine, survives factory reset -> RMA), with the pull+vision fallback. Claude-Session: https://claude.ai/code/session_01Xcm6ikLgGoCxxHrxtjkk5V
This commit is contained in:
@@ -86,3 +86,50 @@ Center**, then **Alarm Settings → Alarm Server**, makes the camera **HTTP-POST
|
||||
next step once the real payload is known. If the camera emits its own plate (`<plateNumber>`), we
|
||||
can use it as an advisory read directly; otherwise the server hands the attached/pulled frame to
|
||||
the [[opencv-anpr-service|vision service]] for ANPR.
|
||||
|
||||
### Gotchas learned the hard way (2026-06-22 field session)
|
||||
|
||||
Several traps surfaced trying to get a real camera to push. In order of how long each cost:
|
||||
|
||||
- **WSL rewrites the inbound source IP.** On the dev host (WSL mirrored mode), an inbound LAN packet
|
||||
arrives at our server with its **source rewritten to the host's own IP** (`10.0.10.203`), not the
|
||||
camera's. The source-IP guard then rejects every push as a mismatch. Fix: a per-device
|
||||
**`skipSourceIpCheck`** config flag (a Setup checkbox) that bypasses the IP guard — the signed
|
||||
ledger + optional Digest remain the real guards. Leave OFF on a normal LAN.
|
||||
- **The setup checkbox saved booleans as the STRING `"true"`.** The generic config-field form had no
|
||||
boolean renderer, so a `type:"boolean"` field fell through to a text input. Fixed (checkbox
|
||||
renderer); the server also coerces `"true"`/`1`/`yes`/`on` defensively.
|
||||
- **The camera's "Test" button proves almost nothing.** It does a TCP/connectivity probe and reports
|
||||
"service available" on ANY HTTP reply (even our 404) — it does **not** POST a real event to your
|
||||
URL. Only a real detection (or the ISAPI `httpHosts/<id>/test`) actually exercises the path.
|
||||
- **`httpBroken` latches.** Once the camera marks the host broken (from earlier failed deliveries),
|
||||
it stays `true` across reboots and won't retry. Clear it by **re-PUTting** the httpHost config
|
||||
(`PUT /ISAPI/Event/notification/httpHosts/1` with `<httpBroken>false</httpBroken>`).
|
||||
- **"Notify Surveillance Center" ≠ the HTTP Alarm Server** on some firmware (separate upload
|
||||
channels); and **basic Motion Detection may not POST to the httpHost** while smart/VCA events do.
|
||||
Always confirm the **Arming Schedule** covers the test time, too (a silent killer).
|
||||
|
||||
### Diagnosing a non-pushing camera — read its OWN state, don't guess
|
||||
|
||||
Status APIs lie (a `diskfull` error on a camera with **no disk** is a misleading label). The decisive
|
||||
checks, in order — they need no cooperation from our server:
|
||||
|
||||
1. **`GET /ISAPI/Event/notification/alertStream`** (Digest) — the camera's live event bus. A HEALTHY
|
||||
camera streams periodic multipart heartbeat frames even when idle, and an event line when one
|
||||
fires. **Total silence (no heartbeat) for 20s = the event subsystem is producing nothing** — the
|
||||
fault is internal, upstream of any linkage (HTTP host, FTP, alarm stream all die together).
|
||||
2. **SSH into the camera** (BusyBox "psh" restricted shell; default creds work): `showStatus` →
|
||||
look for **`EventScribe: except`** (event subsystem in an exception state); `dmesg` → UBIFS
|
||||
"recovery needed" on every volume (unclean flash) and **`fh rtc get time error ffffffff`** (dead
|
||||
RTC — a hardware fault; the clock reads 1970, which wedges time-keyed event logic). `ps` shows the
|
||||
`davinci` app. `df`/`printPart` are NOT available; partition sizes via the UBIFS lines in `dmesg`.
|
||||
3. **`netstat` on the camera** while triggering — watch for an OUTBOUND line `cam:port → server:3000`.
|
||||
None appearing = the camera never even attempts the POST.
|
||||
|
||||
> **Verified-dead unit (2026-06-22):** a `DS-2CD1043G2-LIU` (V5.8.10, 8 months old) failed all three:
|
||||
> silent alertStream, `EventScribe: except`, `diskfull` on `Event/triggers`, dead RTC, and **zero
|
||||
> outbound to `:3000`** across a 3-minute netstat watch — through reboot, basic reset, AND full
|
||||
> factory reset. Conclusion: **defective event engine (RMA)**, not our integration. Our endpoint was
|
||||
> verified correct throughout (it accepts + records a POST the instant one actually arrives). When a
|
||||
> camera's event push is dead, fall back to the **pull + [[opencv-anpr-service|vision]]** path, which
|
||||
> works with the same camera (it still serves snapshots fine).
|
||||
|
||||
+14
@@ -1384,3 +1384,17 @@ setup offers the backend push IP). NOT yet a barrier trigger or DeviceReadEvent
|
||||
read-bus/ANPR wiring is the next step once the real payload is captured (advisory-only rule still
|
||||
governs). Tests: hikvision-alarm.test.ts (6: vehicle XML summary, ANPR plate, raw JSON, wrong-IP
|
||||
404, disabled 404, unknown-device 404); server 109/109, build+lint 14/14. See [[lpr-camera]].
|
||||
|
||||
## [2026-06-22] debug | Hikvision event-push field session — fixes + a verified-dead camera
|
||||
Long session getting a real Hik camera to POST events. Server/integration fixes (committed): listen
|
||||
on ALL methods (camera probes with GET/etc, not just POST); record rejected pushes too (kind
|
||||
"alarm-rejected" + reason) so "nothing arrived" is never ambiguous; a GET /api/devices/hikvision/
|
||||
alarms read endpoint; per-device skipSourceIpCheck (WSL mirrored mode REWRITES the inbound source IP
|
||||
to the host's own, so the source-IP guard rejected every push); and a real checkbox renderer for
|
||||
type:"boolean" config fields (they were saving the STRING "true"). Then proved — via the camera's
|
||||
OWN state, not our server — that the specific DS-2CD1043G2-LIU unit has a DEAD event engine: silent
|
||||
alertStream (no heartbeat), EventScribe:except, diskfull on Event/triggers, dead RTC (rtc get time
|
||||
error / clock at 1970), and ZERO outbound to :3000 over a 3-min netstat watch, surviving reboot +
|
||||
basic reset + FULL factory reset. Verdict: defective camera (RMA), not our code. Captured the
|
||||
diagnostic method (alertStream silence / SSH showStatus / netstat) in [[lpr-camera]]. Fallback for a
|
||||
dead-push camera: pull + [[opencv-anpr-service|vision]] (the same camera still serves snapshots).
|
||||
|
||||
Reference in New Issue
Block a user