julian
95efc23139
Scaffold mirrors tcp-ingestion conventions: ESM, strict TS, pnpm, vitest
with unit/integration split, ESLint flat config with no-floating-promises
+ no-misused-promises + import/no-restricted-paths (the new src/core/ →
src/domain/ boundary that protects Phase 1 from Phase 2 churn).
Core types in src/core/types.ts (Position, StreamRecord, DeviceState,
Metrics, AttributeValue) — Position is byte-equivalent to tcp-ingestion's
output. Codec in src/core/codec.ts implements sentinel reversal:
{__bigint:"..."} → bigint, {__buffer_b64:"..."} → Buffer, ISO timestamp
string → Date. CodecError surfaces malformed payload reasons with the
failing field named.
Config in src/config/load.ts (zod schema, all 13 env vars with defaults
and bounded numerics). Logger in src/observability/logger.ts matches
tcp-ingestion exactly: ISO timestamps, string level labels, pino-pretty
in development.
Postgres in src/db/: createPool with sane defaults and application_name,
connectWithRetry mirroring the ioredis retry pattern, a 30-line
migration runner using a schema_migrations table, and 0001_positions.sql
with the hypertable + (device_id, ts) unique index + ts DESC index.
Migration runner unit-tested against a mocked pg.Pool; the real
TimescaleDB round-trip is deferred to task 1.10 per spec.
Verification: typecheck, lint, build all clean; 73 unit tests passing
across 4 files. import/no-restricted-paths verified live by temporarily
adding a forbidden src/domain/ import.
77 lines
4.8 KiB
Markdown
77 lines
4.8 KiB
Markdown
# Task 1.3 — Configuration & logging
|
||
|
||
**Phase:** 1 — Throughput pipeline
|
||
**Status:** 🟩 Done
|
||
**Depends on:** 1.1
|
||
**Wiki refs:** `docs/wiki/entities/processor.md`
|
||
|
||
## Goal
|
||
|
||
Validate environment variables on startup with `zod`, build the pino root logger with the same conventions as `tcp-ingestion` (ISO timestamps, string level labels, instance_id base field), and fail fast with a readable error message if config is invalid.
|
||
|
||
## Deliverables
|
||
|
||
- `src/config/load.ts` exporting:
|
||
- `loadConfig(): Config` — reads `process.env`, runs zod parse, returns a typed `Config`. Throws on invalid input with a multi-line message that names every invalid field.
|
||
- `Config` type derived from the zod schema.
|
||
- `src/observability/logger.ts` exporting:
|
||
- `createLogger({ level, nodeEnv, instanceId }): Logger` — pino root logger with base fields `service: 'processor'`, `instance_id`. ISO timestamps via `pino.stdTimeFunctions.isoTime`. Level formatter that emits `"level":"info"` not `"level":30`. In `nodeEnv === 'development'`, use the pino-pretty transport.
|
||
- `type Logger` re-exported from `pino`.
|
||
- Wire both into `src/main.ts`: `loadConfig()` → `createLogger()` → `logger.info('processor starting')` → exit 0 (still a stub; consumer wiring lands in 1.8).
|
||
|
||
## Specification
|
||
|
||
### Environment variables
|
||
|
||
| Var | Required | Default | Notes |
|
||
|---|---|---|---|
|
||
| `NODE_ENV` | no | `production` | `development` enables pino-pretty |
|
||
| `INSTANCE_ID` | no | `processor-1` | Used in metrics + log base field |
|
||
| `LOG_LEVEL` | no | `info` | `trace` / `debug` / `info` / `warn` / `error` |
|
||
| `REDIS_URL` | yes | — | e.g. `redis://redis:6379` |
|
||
| `POSTGRES_URL` | yes | — | e.g. `postgres://user:pass@db:5432/trm` |
|
||
| `REDIS_TELEMETRY_STREAM` | no | `telemetry:t` | Must match `tcp-ingestion`'s `REDIS_TELEMETRY_STREAM` |
|
||
| `REDIS_CONSUMER_GROUP` | no | `processor` | All Processor instances join this group |
|
||
| `REDIS_CONSUMER_NAME` | no | `${INSTANCE_ID}` | Unique per instance — defaults to instance id |
|
||
| `METRICS_PORT` | no | `9090` | HTTP server port for `/metrics`, `/healthz`, `/readyz` |
|
||
| `BATCH_SIZE` | no | `100` | Max records per `XREADGROUP` call |
|
||
| `BATCH_BLOCK_MS` | no | `5000` | `BLOCK` timeout on `XREADGROUP` when stream is empty |
|
||
| `WRITE_BATCH_SIZE` | no | `50` | Max rows per Postgres `INSERT` |
|
||
| `DEVICE_STATE_LRU_CAP` | no | `10000` | Max devices kept in memory; LRU eviction beyond this |
|
||
|
||
### Validation rules
|
||
|
||
- All defaults must be expressed in the zod schema with `.default(...)` so the parsed `Config` is fully typed and never has `undefined` for an optional field.
|
||
- Numeric env vars must be coerced from string and bounded: `BATCH_SIZE` 1–10000, `BATCH_BLOCK_MS` 0–60000, `WRITE_BATCH_SIZE` 1–1000, `DEVICE_STATE_LRU_CAP` 100–1_000_000.
|
||
- `REDIS_URL` and `POSTGRES_URL` must parse as URLs with the expected protocol (`redis:` or `rediss:`; `postgres:` or `postgresql:`).
|
||
- `LOG_LEVEL` must be one of pino's accepted levels.
|
||
|
||
### Logger conventions
|
||
|
||
Match `tcp-ingestion/src/observability/logger.ts` line for line where applicable. Future-you grepping across services should see the same shape:
|
||
|
||
```ts
|
||
const formatters = { level: (label: string) => ({ level: label }) };
|
||
|
||
if (nodeEnv === 'development') {
|
||
return pino({ level, base, timestamp: pino.stdTimeFunctions.isoTime, formatters,
|
||
transport: { target: 'pino-pretty', options: { colorize: true, translateTime: 'SYS:standard', ignore: 'pid,hostname' } } });
|
||
}
|
||
return pino({ level, base, timestamp: pino.stdTimeFunctions.isoTime, formatters });
|
||
```
|
||
|
||
## Acceptance criteria
|
||
|
||
- [ ] `pnpm test` covers config validation: missing required vars throw with the right message; invalid URLs throw; bounded numerics throw on out-of-range values.
|
||
- [ ] Running with valid env emits a single `processor starting` info log with `service=processor` and `instance_id=processor-1` base fields.
|
||
- [ ] Running with `NODE_ENV=development` produces colorized output via pino-pretty.
|
||
- [ ] Running with `NODE_ENV=production` produces JSON output with ISO `time` and string `level`.
|
||
|
||
## Risks / open questions
|
||
|
||
- `REDIS_CONSUMER_NAME` defaulting to `INSTANCE_ID` means `INSTANCE_ID` must be unique per instance for safe consumer-group operation. Document this in `.env.example` so operators don't accidentally run two instances with the same `INSTANCE_ID`.
|
||
|
||
## Done
|
||
|
||
*(pending commit SHA)* — Implemented `src/config/load.ts` (zod schema, loadConfig) and `src/observability/logger.ts` (createLogger, pino-pretty in dev). 37 config test cases covering all defaults, missing required vars, URL protocol validation, and bounded numeric checks. Wired into `src/main.ts`. Divergence from tcp-ingestion config: `INSTANCE_ID` defaults to a fixed `'processor-1'` string rather than a random UUID prefix; rationale: operator-visible name is more useful than randomness in a containerised environment where the instance name can be set deterministically.
|