3 Commits

Author	SHA1	Message	Date
julian	be48da9baa	Implement Phase 1 tasks 1.9-1.11 (observability + integration test + Dockerfile/CI) ... src/observability/metrics.ts — full prom-client implementation. All 10 Phase 1 metrics registered (processor_consumer_reads_total, _records_total, _lag, _decode_errors_total, processor_position_writes_total {status}, _write_duration_seconds, processor_acks_total, processor_device_state_{size,evictions_total}) plus nodejs_* defaults. node:http server with /metrics, /healthz, /readyz. /readyz checks redis.status === 'ready' AND a 5s-cached SELECT 1 Postgres probe. processor_consumer_lag sampled every 10s via XINFO GROUPS, falling back to a no-op when the consumer group hasn't been created yet. src/main.ts — replaces the trace-logging shim with createMetrics() and startMetricsServer(); shutdown closes the metrics server before redis.quit() and pool.end(). test/metrics.test.ts — 22 unit tests: exposition format, every metric type behaviour, all four HTTP endpoint paths including /readyz 503 cases. test/pipeline.integration.test.ts — testcontainers Redis 7 + TimescaleDB latest-pg16. Four scenarios: happy path with bigint+Buffer attribute round-trip, idempotency on (device_id, ts), malformed payload stays in PEL (decode_errors_total increments), writer failure → retry (weaker variant per spec: stop Postgres before publish, restart, verify row appears). Skip-on-no-Docker pattern verified — exits 0 without Docker. Dockerfile — multi-stage matching tcp-ingestion. EXPOSE 9090 only, HEALTHCHECK on /readyz, image-source label points at processor repo. .gitea/workflows/build.yml — single-job workflow mirroring tcp-ingestion. Path filters cover src/, test/, build config, Dockerfile. Portainer webhook step uncommented for :main auto-deploy. compose.dev.yaml — local-build variant with Redis + TimescaleDB + processor-dev for verifying Dockerfile changes without the registry round-trip. README.md — fleshed out from stub: quick-start, Docker build, deployment note, env vars, tests (unit vs. integration), CI behavior. Flags the deploy-side change needed: deploy/compose.yaml needs a TimescaleDB service and a processor service entry added. Verification: typecheck, lint clean; 134 unit tests passing across 8 files (+22 from this batch). pnpm test:integration runs cleanly under the no-Docker skip pattern. Phase 1 is now complete. Service is pilot-ready.	2026-04-30 22:01:55 +02:00
julian	c314ba0902	Add planning documents for Phase 1 (throughput pipeline) and stub Phases 2-4 ... ROADMAP.md establishes status legend, architectural anchors pointing at the wiki, and seven non-negotiable design rules — most importantly the core/domain boundary that protects Phase 1 from Phase 2 churn, the schema-authority split (positions hypertable owned here; everything else owned by Directus), and idempotent-writes via (device_id, ts) ON CONFLICT. Phase 1 (throughput pipeline) is fully detailed across 11 task files: scaffold, core types + sentinel decoder, config + logging, Postgres hypertable, Redis Stream consumer, per-device LRU state, batched writer, main wiring, observability, integration test, Dockerfile + Gitea CI. Observability is in Phase 1 (not deferred) — lesson learned from tcp-ingestion task 1.10. Phases 2-4 are stub READMEs. Phase 2 (domain logic) blocks on Directus schema decisions and lists those open questions explicitly. Phase 3 (production hardening) and Phase 4 (future) sketch the task shape.	2026-04-30 21:16:59 +02:00
julian	1a4202f4d1	first commit	2026-04-30 20:48:33 +02:00