processor

trm/processor

Fork 0

Commit Graph

Author	SHA1	Message	Date
julian	6a14eb1d01	Fill in commit SHA for Phase 1 tasks 1.1-1.4 in ROADMAP and task files	2026-04-30 21:36:23 +02:00
julian	95efc23139	Implement Phase 1 tasks 1.1-1.4 (scaffold + core types + config + Postgres) Scaffold mirrors tcp-ingestion conventions: ESM, strict TS, pnpm, vitest with unit/integration split, ESLint flat config with no-floating-promises + no-misused-promises + import/no-restricted-paths (the new src/core/ → src/domain/ boundary that protects Phase 1 from Phase 2 churn). Core types in src/core/types.ts (Position, StreamRecord, DeviceState, Metrics, AttributeValue) — Position is byte-equivalent to tcp-ingestion's output. Codec in src/core/codec.ts implements sentinel reversal: {__bigint:"..."} → bigint, {__buffer_b64:"..."} → Buffer, ISO timestamp string → Date. CodecError surfaces malformed payload reasons with the failing field named. Config in src/config/load.ts (zod schema, all 13 env vars with defaults and bounded numerics). Logger in src/observability/logger.ts matches tcp-ingestion exactly: ISO timestamps, string level labels, pino-pretty in development. Postgres in src/db/: createPool with sane defaults and application_name, connectWithRetry mirroring the ioredis retry pattern, a 30-line migration runner using a schema_migrations table, and 0001_positions.sql with the hypertable + (device_id, ts) unique index + ts DESC index. Migration runner unit-tested against a mocked pg.Pool; the real TimescaleDB round-trip is deferred to task 1.10 per spec. Verification: typecheck, lint, build all clean; 73 unit tests passing across 4 files. import/no-restricted-paths verified live by temporarily adding a forbidden src/domain/ import.	2026-04-30 21:35:59 +02:00
julian	c314ba0902	Add planning documents for Phase 1 (throughput pipeline) and stub Phases 2-4 ROADMAP.md establishes status legend, architectural anchors pointing at the wiki, and seven non-negotiable design rules — most importantly the core/domain boundary that protects Phase 1 from Phase 2 churn, the schema-authority split (positions hypertable owned here; everything else owned by Directus), and idempotent-writes via (device_id, ts) ON CONFLICT. Phase 1 (throughput pipeline) is fully detailed across 11 task files: scaffold, core types + sentinel decoder, config + logging, Postgres hypertable, Redis Stream consumer, per-device LRU state, batched writer, main wiring, observability, integration test, Dockerfile + Gitea CI. Observability is in Phase 1 (not deferred) — lesson learned from tcp-ingestion task 1.10. Phases 2-4 are stub READMEs. Phase 2 (domain logic) blocks on Directus schema decisions and lists those open questions explicitly. Phase 3 (production hardening) and Phase 4 (future) sketch the task shape.	2026-04-30 21:16:59 +02:00

Author

SHA1

Message

Date

julian

6a14eb1d01

Fill in commit SHA for Phase 1 tasks 1.1-1.4 in ROADMAP and task files

2026-04-30 21:36:23 +02:00

julian

95efc23139

Implement Phase 1 tasks 1.1-1.4 (scaffold + core types + config + Postgres)

Scaffold mirrors tcp-ingestion conventions: ESM, strict TS, pnpm, vitest
with unit/integration split, ESLint flat config with no-floating-promises
+ no-misused-promises + import/no-restricted-paths (the new src/core/ →
src/domain/ boundary that protects Phase 1 from Phase 2 churn).

Core types in src/core/types.ts (Position, StreamRecord, DeviceState,
Metrics, AttributeValue) — Position is byte-equivalent to tcp-ingestion's
output. Codec in src/core/codec.ts implements sentinel reversal:
{__bigint:"..."} → bigint, {__buffer_b64:"..."} → Buffer, ISO timestamp
string → Date. CodecError surfaces malformed payload reasons with the
failing field named.

Config in src/config/load.ts (zod schema, all 13 env vars with defaults
and bounded numerics). Logger in src/observability/logger.ts matches
tcp-ingestion exactly: ISO timestamps, string level labels, pino-pretty
in development.

Postgres in src/db/: createPool with sane defaults and application_name,
connectWithRetry mirroring the ioredis retry pattern, a 30-line
migration runner using a schema_migrations table, and 0001_positions.sql
with the hypertable + (device_id, ts) unique index + ts DESC index.
Migration runner unit-tested against a mocked pg.Pool; the real
TimescaleDB round-trip is deferred to task 1.10 per spec.

Verification: typecheck, lint, build all clean; 73 unit tests passing
across 4 files. import/no-restricted-paths verified live by temporarily
adding a forbidden src/domain/ import.

2026-04-30 21:35:59 +02:00

julian

c314ba0902

Add planning documents for Phase 1 (throughput pipeline) and stub Phases 2-4

ROADMAP.md establishes status legend, architectural anchors pointing at the
wiki, and seven non-negotiable design rules — most importantly the
core/domain boundary that protects Phase 1 from Phase 2 churn, the
schema-authority split (positions hypertable owned here; everything else
owned by Directus), and idempotent-writes via (device_id, ts) ON CONFLICT.

Phase 1 (throughput pipeline) is fully detailed across 11 task files:
scaffold, core types + sentinel decoder, config + logging, Postgres
hypertable, Redis Stream consumer, per-device LRU state, batched writer,
main wiring, observability, integration test, Dockerfile + Gitea CI.
Observability is in Phase 1 (not deferred) — lesson learned from
tcp-ingestion task 1.10.

Phases 2-4 are stub READMEs. Phase 2 (domain logic) blocks on Directus
schema decisions and lists those open questions explicitly. Phase 3
(production hardening) and Phase 4 (future) sketch the task shape.

2026-04-30 21:16:59 +02:00

3 Commits