julian
95efc23139
Scaffold mirrors tcp-ingestion conventions: ESM, strict TS, pnpm, vitest
with unit/integration split, ESLint flat config with no-floating-promises
+ no-misused-promises + import/no-restricted-paths (the new src/core/ →
src/domain/ boundary that protects Phase 1 from Phase 2 churn).
Core types in src/core/types.ts (Position, StreamRecord, DeviceState,
Metrics, AttributeValue) — Position is byte-equivalent to tcp-ingestion's
output. Codec in src/core/codec.ts implements sentinel reversal:
{__bigint:"..."} → bigint, {__buffer_b64:"..."} → Buffer, ISO timestamp
string → Date. CodecError surfaces malformed payload reasons with the
failing field named.
Config in src/config/load.ts (zod schema, all 13 env vars with defaults
and bounded numerics). Logger in src/observability/logger.ts matches
tcp-ingestion exactly: ISO timestamps, string level labels, pino-pretty
in development.
Postgres in src/db/: createPool with sane defaults and application_name,
connectWithRetry mirroring the ioredis retry pattern, a 30-line
migration runner using a schema_migrations table, and 0001_positions.sql
with the hypertable + (device_id, ts) unique index + ts DESC index.
Migration runner unit-tested against a mocked pg.Pool; the real
TimescaleDB round-trip is deferred to task 1.10 per spec.
Verification: typecheck, lint, build all clean; 73 unit tests passing
across 4 files. import/no-restricted-paths verified live by temporarily
adding a forbidden src/domain/ import.
50 lines
1.9 KiB
Bash
50 lines
1.9 KiB
Bash
# Environment variables for processor.
|
|
# Copy to .env and fill in values for local development.
|
|
# Required vars: REDIS_URL, POSTGRES_URL.
|
|
|
|
# Runtime environment: development | test | production
|
|
NODE_ENV=development
|
|
|
|
# Unique identifier for this service instance.
|
|
# Used in logs (base field), metrics labels, and as the default Redis consumer name.
|
|
# IMPORTANT: must be unique per running instance for safe consumer-group operation.
|
|
# If two instances share the same INSTANCE_ID they will also share REDIS_CONSUMER_NAME,
|
|
# which causes consumer-group split-brain — the stream will not progress correctly.
|
|
INSTANCE_ID=processor-1
|
|
|
|
# Log level: fatal | error | warn | info | debug | trace
|
|
LOG_LEVEL=info
|
|
|
|
# Redis connection URL — required; no default.
|
|
REDIS_URL=redis://localhost:6379
|
|
|
|
# Postgres / TimescaleDB connection URL — required; no default.
|
|
POSTGRES_URL=postgres://postgres:postgres@localhost:5432/trm
|
|
|
|
# Redis Stream name to consume from. Must match tcp-ingestion's REDIS_TELEMETRY_STREAM.
|
|
REDIS_TELEMETRY_STREAM=telemetry:t
|
|
|
|
# Redis consumer group name. All Processor instances join this group.
|
|
REDIS_CONSUMER_GROUP=processor
|
|
|
|
# Redis consumer name. Defaults to INSTANCE_ID.
|
|
# Override only when running multiple instances that should appear as distinct consumers
|
|
# in the group (e.g. when INSTANCE_ID is not set to a unique value per container).
|
|
# REDIS_CONSUMER_NAME=processor-1
|
|
|
|
# Port for Prometheus /metrics, /healthz, /readyz HTTP server.
|
|
METRICS_PORT=9090
|
|
|
|
# Max records fetched per XREADGROUP call.
|
|
BATCH_SIZE=100
|
|
|
|
# BLOCK timeout (ms) on XREADGROUP when the stream is empty. 0 = no blocking.
|
|
BATCH_BLOCK_MS=5000
|
|
|
|
# Max rows per Postgres INSERT batch.
|
|
WRITE_BATCH_SIZE=50
|
|
|
|
# Max devices kept in the per-device in-memory state map. LRU eviction beyond this cap.
|
|
# Size each entry at ~500 bytes → 10 000 devices ≈ 5 MB. Raise for large fleets.
|
|
DEVICE_STATE_LRU_CAP=10000
|