Implement Phase 1 tasks 1.1-1.4 (scaffold + core types + config + Postgres)

Scaffold mirrors tcp-ingestion conventions: ESM, strict TS, pnpm, vitest
with unit/integration split, ESLint flat config with no-floating-promises
+ no-misused-promises + import/no-restricted-paths (the new src/core/ →
src/domain/ boundary that protects Phase 1 from Phase 2 churn).

Core types in src/core/types.ts (Position, StreamRecord, DeviceState,
Metrics, AttributeValue) — Position is byte-equivalent to tcp-ingestion's
output. Codec in src/core/codec.ts implements sentinel reversal:
{__bigint:"..."} → bigint, {__buffer_b64:"..."} → Buffer, ISO timestamp
string → Date. CodecError surfaces malformed payload reasons with the
failing field named.

Config in src/config/load.ts (zod schema, all 13 env vars with defaults
and bounded numerics). Logger in src/observability/logger.ts matches
tcp-ingestion exactly: ISO timestamps, string level labels, pino-pretty
in development.

Postgres in src/db/: createPool with sane defaults and application_name,
connectWithRetry mirroring the ioredis retry pattern, a 30-line
migration runner using a schema_migrations table, and 0001_positions.sql
with the hypertable + (device_id, ts) unique index + ts DESC index.
Migration runner unit-tested against a mocked pg.Pool; the real
TimescaleDB round-trip is deferred to task 1.10 per spec.

Verification: typecheck, lint, build all clean; 73 unit tests passing
across 4 files. import/no-restricted-paths verified live by temporarily
adding a forbidden src/domain/ import.
This commit is contained in:
2026-04-30 21:35:16 +02:00
parent c314ba0902
commit 95efc23139
28 changed files with 7427 additions and 13 deletions
+5 -5
View File
@@ -40,17 +40,17 @@ These rules govern every task. Any deviation must be discussed and documented as
### Phase 1 — Throughput pipeline
**Status:** ⬜ Not started
**Status:** 🟨 In progress (1.11.4 done; 1.51.11 ahead)
**Outcome:** A Node.js Processor that joins a Redis Streams consumer group on `telemetry:t`, decodes each `Position` (including `__bigint`/`__buffer_b64` sentinel reversal), upserts it into a TimescaleDB `positions` hypertable, updates per-device in-memory state (last position, last seen), `XACK`s on successful write, and exposes Prometheus metrics + health/readiness HTTP endpoints. End-to-end pilot-quality service; no domain logic yet.
[**See `phase-1-throughput/README.md`**](./phase-1-throughput/README.md)
| # | Task | Status | Landed in |
|---|------|--------|-----------|
| 1.1 | [Project scaffold](./phase-1-throughput/01-project-scaffold.md) | | |
| 1.2 | [Core types & contracts](./phase-1-throughput/02-core-types.md) | | |
| 1.3 | [Configuration & logging](./phase-1-throughput/03-config-and-logging.md) | | |
| 1.4 | [Postgres connection & `positions` hypertable](./phase-1-throughput/04-postgres-schema.md) | | |
| 1.1 | [Project scaffold](./phase-1-throughput/01-project-scaffold.md) | 🟩 | *(pending commit SHA)* |
| 1.2 | [Core types & contracts](./phase-1-throughput/02-core-types.md) | 🟩 | *(pending commit SHA)* |
| 1.3 | [Configuration & logging](./phase-1-throughput/03-config-and-logging.md) | 🟩 | *(pending commit SHA)* |
| 1.4 | [Postgres connection & `positions` hypertable](./phase-1-throughput/04-postgres-schema.md) | 🟩 | *(pending commit SHA)* |
| 1.5 | [Redis Stream consumer (XREADGROUP)](./phase-1-throughput/05-stream-consumer.md) | ⬜ | — |
| 1.6 | [Per-device in-memory state](./phase-1-throughput/06-device-state.md) | ⬜ | — |
| 1.7 | [Position writer (batched upsert)](./phase-1-throughput/07-position-writer.md) | ⬜ | — |
@@ -1,7 +1,7 @@
# Task 1.1 — Project scaffold
**Phase:** 1 — Throughput pipeline
**Status:** ⬜ Not started
**Status:** 🟩 Done
**Depends on:** None
**Wiki refs:** `docs/wiki/entities/processor.md`
@@ -55,4 +55,4 @@ Initialize the Node.js / TypeScript project with the directory layout from the P
## Done
(Fill in once complete: commit SHA, brief notes.)
*(pending commit SHA)* — Scaffolded `package.json`, `tsconfig.json`, `tsconfig.test.json`, `eslint.config.js`, `.prettierrc`, `vitest.config.ts`, `vitest.integration.config.ts`, `.env.example`, `.gitignore`, `.dockerignore`, and `src/main.ts`. All tooling passes (`pnpm typecheck`, `pnpm lint`, `pnpm build`, `pnpm test`). Verified `import/no-restricted-paths` boundary rule fires on a temporary `src/core/``src/domain/` import. Divergence from tcp-ingestion: the restricted-paths zone targets `src/domain/` (Phase 2 boundary) instead of `src/adapters/` (tcp-ingestion boundary).
@@ -1,7 +1,7 @@
# Task 1.2 — Core types & contracts
**Phase:** 1 — Throughput pipeline
**Status:** ⬜ Not started
**Status:** 🟩 Done
**Depends on:** 1.1
**Wiki refs:** `docs/wiki/concepts/position-record.md`, `docs/wiki/concepts/io-element-bag.md`
@@ -63,4 +63,4 @@ Some Teltonika IO elements are u64 values that exceed `Number.MAX_SAFE_INTEGER`
## Done
(Fill in once complete: commit SHA, brief notes.)
*(pending commit SHA)* — Implemented `src/core/types.ts` (Position, StreamRecord, DeviceState, Metrics, AttributeValue) and `src/core/codec.ts` (decodePosition, CodecError). All sentinel reversal rules implemented: `__bigint` → bigint, `__buffer_b64` → Buffer, timestamp ISO string → Date. 26 test cases in `test/codec.test.ts` covering round-trips, u64-max, non-UTF-8 bytes, all error paths. Judgment call: `AttributeValue` extracted as a named type alias (not inline) to aid readability in downstream tasks.
@@ -1,7 +1,7 @@
# Task 1.3 — Configuration & logging
**Phase:** 1 — Throughput pipeline
**Status:** ⬜ Not started
**Status:** 🟩 Done
**Depends on:** 1.1
**Wiki refs:** `docs/wiki/entities/processor.md`
@@ -73,4 +73,4 @@ return pino({ level, base, timestamp: pino.stdTimeFunctions.isoTime, formatters
## Done
(Fill in once complete: commit SHA, brief notes.)
*(pending commit SHA)* — Implemented `src/config/load.ts` (zod schema, loadConfig) and `src/observability/logger.ts` (createLogger, pino-pretty in dev). 37 config test cases covering all defaults, missing required vars, URL protocol validation, and bounded numeric checks. Wired into `src/main.ts`. Divergence from tcp-ingestion config: `INSTANCE_ID` defaults to a fixed `'processor-1'` string rather than a random UUID prefix; rationale: operator-visible name is more useful than randomness in a containerised environment where the instance name can be set deterministically.
@@ -1,7 +1,7 @@
# Task 1.4 — Postgres connection & `positions` hypertable
**Phase:** 1 — Throughput pipeline
**Status:** ⬜ Not started
**Status:** 🟩 Done
**Depends on:** 1.1, 1.3
**Wiki refs:** `docs/wiki/entities/postgres-timescaledb.md`
@@ -86,4 +86,4 @@ Do **not** introduce a heavy framework (Knex, node-pg-migrate). The Processor ha
## Done
(Fill in once complete: commit SHA, brief notes.)
*(pending commit SHA)* — Implemented `src/db/pool.ts` (createPool, connectWithRetry), `src/db/migrate.ts` (runMigrations — 30-line runner), and `src/db/migrations/0001_positions.sql` (hypertable + unique index + ts-desc index). Unit tests use a mocked pg.Pool throughout; the real TimescaleDB round-trip is deferred to task 1.10 per spec. The "calls process.exit(1)" pool test uses `maxAttempts=1` to avoid fake-timer unhandled-rejection noise that surfaces when a backoff setTimeout resolves after the outer promise has already thrown.