From bf403332d0f1b1ec00bfb6caf4d7681aeb4d4c3c Mon Sep 17 00:00:00 2001 From: Julian Cuni Date: Thu, 30 Apr 2026 22:04:24 +0200 Subject: [PATCH] Update Processor Service documentation: complete Phase 1 and detail architecture, metrics, and integration tests --- .../project_processor.md | 49 +++++++++++++++---- 1 file changed, 39 insertions(+), 10 deletions(-) diff --git a/.claude/agent-memory/ts-node-backend-engineer/project_processor.md b/.claude/agent-memory/ts-node-backend-engineer/project_processor.md index 45ee4d8..902ce3f 100644 --- a/.claude/agent-memory/ts-node-backend-engineer/project_processor.md +++ b/.claude/agent-memory/ts-node-backend-engineer/project_processor.md @@ -1,23 +1,52 @@ --- name: Processor Service -description: processor service scaffold, conventions, and known test patterns learned during Phase 1 tasks 1.1–1.4 +description: processor service: Phase 1 complete (all 11 tasks), key patterns, conventions, and quirks type: project --- -Phase 1 tasks 1.1–1.4 landed. Key facts for tasks 1.5–1.11: +Phase 1 complete. All 11 tasks landed. The throughput pipeline is done: consumer + writer + metrics + integration test + Docker + CI. **Architecture divergence from tcp-ingestion:** -- ESLint `import/no-restricted-paths` zone: `src/core/` cannot import `src/domain/` (not `src/adapters/` like tcp-ingestion). `src/domain/` is preemptive for Phase 2. -- `INSTANCE_ID` defaults to `'processor-1'` (fixed string), not a random UUID prefix. Rationale: deterministic names are better for container environments. -- `REDIS_CONSUMER_NAME` defaults to `INSTANCE_ID` at runtime (after schema parse), not as a zod schema default — because it depends on `INSTANCE_ID` value. +- ESLint `import/no-restricted-paths` zone: `src/core/` cannot import `src/domain/` (preemptive for Phase 2). +- `INSTANCE_ID` defaults to `'processor-1'` (fixed string, not random UUID). Deterministic for container environments. +- `REDIS_CONSUMER_NAME` defaults to `INSTANCE_ID` at runtime (after schema parse), not as a zod schema default. +- `connectRedis` lives in `src/core/consumer.ts` (not `src/db/`). Rationale: only the consumer needs a Redis connection in Phase 1. + +**Buffer.toJSON() gotcha in JSON.stringify replacers:** +- `Buffer.isBuffer(value)` is false inside a `JSON.stringify` replacer for nested Buffer properties because `Buffer.prototype.toJSON()` fires before the replacer. +- Must handle both `instanceof Uint8Array` (direct reference) AND the `{ type: 'Buffer', data: number[] }` shape. See `serializeAttributes` in `src/core/writer.ts`. + +**writer.ts — RETURNING duplicate detection:** +- `ON CONFLICT DO NOTHING` does not return conflicting rows. Duplicates detected via set-diff on (device_id, ts) between input and RETURNING rows. + +**LRU Map trick (state.ts):** +- Plain `Map` with `delete` + `set` on every update. `keys().next().value` is always the oldest. O(1), no external library. + +**Metrics (task 1.9):** +- `src/observability/metrics.ts` exports: `createMetrics()`, `startMetricsServer()`, `createPostgresHealthCheck()`, `createConsumerLagSampler()`. +- `createPostgresHealthCheck`: cached `SELECT 1`, 5 s TTL, background interval, sync `isReady()` accessor for `/readyz`. +- `createConsumerLagSampler`: uses `XINFO GROUPS` → `lag` field (Redis 7.2+); falls back to `XLEN` if absent. +- `processor_device_state_size` is a gauge updated via `metrics.observe()` in the sink (not in state.ts — avoids coupling state to metrics). +- prom-client counters with label dims do NOT emit `{label} 0` at init. Test accordingly (check label-less counters for zero baseline). + +**Integration test (task 1.10):** +- `test/pipeline.integration.test.ts`: 4 scenarios. Skip-on-no-Docker pattern: try container start in `beforeAll`, set `dockerAvailable = false` on error, each `it` early-returns. +- TimescaleDB image: `timescale/timescaledb:latest-pg16` (not stock postgres). +- Consumer uses a separate `connectRedis` connection; test XADD uses a separate `redisClient`. Mirrors production topology. +- Test 4 (retry): stops pgContainer then restarts it. `pgContainer.restart()` returns a new `StartedTestContainer`. Must reassign `pgContainer` for the `afterAll` cleanup to reference the right instance. **Test patterns:** -- `connectWithRetry` tests: use `maxAttempts=1` for the "calls process.exit(1)" case to avoid fake-timer unhandled rejection noise. The noise occurs when a backoff `setTimeout` resolves after the outer promise has already thrown via process.exit mock. -- Migration runner tests: `isMigrationApplied` calls `pool.query` (pool-level, not through client). Handlers on client mock do NOT affect EXISTS checks. -- `vi.mock('node:fs/promises')` pattern used in migrate.test.ts to isolate filesystem reads. +- `connectWithRetry` tests: use `maxAttempts=1` for process.exit test to avoid fake-timer noise. +- Consumer tests: sink calls `void consumerRef?.stop()` to exit the loop. +- Inline `import()` type annotations forbidden by ESLint (`@typescript-eslint/consistent-type-imports`). Always use top-level `import type`. + +**Dockerfile:** +- `EXPOSE 9090` only (no TCP listener — Processor has none). +- `HEALTHCHECK` uses `${METRICS_PORT:-9090}` default. +- Label `org.opencontainers.image.source` not included (not in tcp-ingestion original either). **Why:** -Phase 1 throughput pipeline. Tasks 1.5–1.8 add consumer/writer/state on top of this foundation. +Phase 1 is the throughput pipeline + operational baseline. Phase 2 (domain logic) and Phase 3 (hardening) build on top. **How to apply:** -When implementing 1.5 (Redis consumer), 1.6 (device state), 1.7 (position writer), 1.8 (main wiring) — all imports must use `.js` extension, no path aliases, strict ESM. The `connectWithRetry` pattern from pool.ts should be mirrored for ioredis in consumer.ts. +Phase 2 adds domain logic to `src/domain/` — no changes to `src/core/`. Phase 3 adds graceful shutdown polish, XAUTOCLAIM, and state rehydration.