processor/.planning/phase-1-throughput/README.md

# Phase 1 — Throughput pipeline

Implement a Node.js worker that joins a Redis Streams consumer group, decodes `Position` records, upserts them into a TimescaleDB hypertable, maintains per-device in-memory state, and ships with the operational baseline (Prometheus metrics, health/readiness endpoints, integration tests, Dockerfile, Gitea CI/CD pipeline).

## Outcome statement

When Phase 1 is done:

- The Processor connects to Redis and joins consumer group `processor` on stream `telemetry:t` (configurable). On startup it creates the group with `MKSTREAM` if missing.
- Every `Position` record published by `tcp-ingestion` lands as exactly one row in the `positions` hypertable, with `device_id`, `ts`, GPS fields, and the IO `attributes` bag preserved as `JSONB` (sentinel-decoded — bigint values become `numeric`, Buffer values become `bytea` or `text` per the spec in task 1.2).
- Per-device in-memory state (`last_position`, `last_seen`, `position_count_session`) is updated on every record and bounded by an LRU cap.
- `XACK` is sent only after the Postgres write succeeds. A crashed instance leaves work pending; on its next start it picks up via consumer-group resumption, and any other instance can claim its pending entries (full `XAUTOCLAIM` polish lives in Phase 3, but the basic resumption works in Phase 1).
- `GET /metrics` returns Prometheus exposition format with consumer lag, throughput, write-latency histogram, error counters. `GET /healthz` and `GET /readyz` cover liveness and readiness (Redis ready + Postgres ready).
- The service builds reproducibly via a Gitea Actions workflow, publishing a Docker image to the Gitea container registry tagged `:main` (and per-commit SHA tags later if needed).
- An integration test spins up Redis + Postgres via testcontainers, publishes a synthetic `Position` to the input stream, and verifies the resulting row in `positions`. End-to-end byte-level round-trip including bigint and Buffer sentinel reversal.

## Sequencing

```
1.1 Project scaffold
   ├─→ 1.2 Core types & contracts
   │      ├─→ 1.3 Configuration & logging
   │      ├─→ 1.4 Postgres connection & positions hypertable
   │      │      └─→ 1.7 Position writer (batched upsert)
   │      └─→ 1.5 Redis Stream consumer
   │             ├─→ 1.6 Per-device in-memory state
   │             └─→ 1.8 Main wiring & ACK semantics  (depends on 1.5, 1.6, 1.7)
   │                    └─→ 1.9 Observability
   │                           └─→ 1.10 Integration test
   │                                  └─→ 1.11 Dockerfile & CI
```

Tasks 1.5/1.6/1.7 can be developed in parallel after 1.4 lands. Task 1.10 (integration test) should land *before* 1.11 because the Dockerfile depends on knowing what `pnpm test` and `pnpm test:integration` will do.

## Files modified

Phase 1 produces this layout in `processor/`:

```
processor/
├── .gitea/workflows/build.yml
├── src/
│   ├── core/
│   │   ├── types.ts                # Position, DeviceState, Metrics
│   │   ├── consumer.ts             # XREADGROUP loop + claim handler
│   │   ├── writer.ts               # Postgres batched upsert
│   │   ├── state.ts                # in-memory device state with LRU
│   │   └── codec.ts                # sentinel decode (__bigint, __buffer_b64)
│   ├── db/
│   │   ├── pool.ts                 # pg.Pool factory
│   │   └── migrations/
│   │       └── 0001_positions.sql  # hypertable creation
│   ├── config/load.ts              # zod schema for env
│   ├── observability/
│   │   ├── logger.ts               # pino root logger
│   │   └── metrics.ts              # prom-client + HTTP server
│   └── main.ts
├── test/
│   ├── codec.test.ts
│   ├── state.test.ts
│   ├── consumer.test.ts            # mocked Redis behaviour
│   ├── writer.test.ts              # mocked pg behaviour
│   └── pipeline.integration.test.ts # testcontainers Redis + Postgres
├── Dockerfile
├── compose.dev.yaml
├── package.json
├── pnpm-lock.yaml
├── tsconfig.json
├── vitest.config.ts
├── vitest.integration.config.ts
├── .dockerignore
├── .gitignore
├── .prettierrc
├── eslint.config.js
└── README.md
```

## Tech stack (decided)

- **Node.js 22 LTS**, ESM-only.
- **TypeScript 5.x** with `strict: true`, `noUncheckedIndexedAccess: true`.
- **pnpm** for dependency management.
- **vitest** for tests (unit + integration split — same pattern as `tcp-ingestion`).
- **pino** for structured logging (ISO timestamps, string level labels — same config as `tcp-ingestion`).
- **prom-client** for Prometheus metrics.
- **ioredis** for Redis Streams (XREADGROUP, XACK, XAUTOCLAIM).
- **pg** (`pg` package, not `postgres.js`) for Postgres — battle-tested, simple Pool API.
- **zod** for environment-variable validation.
- **testcontainers** for integration tests (Redis 7 + TimescaleDB).

If an implementer wants to deviate, they must update the relevant task file first.

## Key design decisions inherited from `tcp-ingestion`

- **ESLint `import/no-restricted-paths`** — `src/core/` cannot import from `src/domain/` (the boundary that protects Phase 1 from Phase 2 churn). `src/db/` is shared.
- **Logger config** — `pino.stdTimeFunctions.isoTime` + level-as-string formatter. Lifecycle events at `info`; high-frequency per-record events at `debug` or `trace`.
- **Slim Dockerfile** — multi-stage with BuildKit cache mounts, `pnpm fetch` + `pnpm install --offline` in the build stage, `pnpm prune --prod` for runtime.
- **CI workflow** — single-job pattern matching `tcp-ingestion/.gitea/workflows/build.yml`. No `services:` block; no separate test container.