Implement Phase 1 tasks 1.9-1.11 (observability + integration test + Dockerfile/CI)
src/observability/metrics.ts — full prom-client implementation. All 10
Phase 1 metrics registered (processor_consumer_reads_total,
_records_total, _lag, _decode_errors_total, processor_position_writes_total
{status}, _write_duration_seconds, processor_acks_total,
processor_device_state_{size,evictions_total}) plus nodejs_* defaults.
node:http server with /metrics, /healthz, /readyz. /readyz checks
redis.status === 'ready' AND a 5s-cached SELECT 1 Postgres probe.
processor_consumer_lag sampled every 10s via XINFO GROUPS, falling back
to a no-op when the consumer group hasn't been created yet.
src/main.ts — replaces the trace-logging shim with createMetrics() and
startMetricsServer(); shutdown closes the metrics server before
redis.quit() and pool.end().
test/metrics.test.ts — 22 unit tests: exposition format, every metric
type behaviour, all four HTTP endpoint paths including /readyz 503 cases.
test/pipeline.integration.test.ts — testcontainers Redis 7 +
TimescaleDB latest-pg16. Four scenarios: happy path with bigint+Buffer
attribute round-trip, idempotency on (device_id, ts), malformed payload
stays in PEL (decode_errors_total increments), writer failure → retry
(weaker variant per spec: stop Postgres before publish, restart, verify
row appears). Skip-on-no-Docker pattern verified — exits 0 without
Docker.
Dockerfile — multi-stage matching tcp-ingestion. EXPOSE 9090 only,
HEALTHCHECK on /readyz, image-source label points at processor repo.
.gitea/workflows/build.yml — single-job workflow mirroring
tcp-ingestion. Path filters cover src/, test/, build config, Dockerfile.
Portainer webhook step uncommented for :main auto-deploy.
compose.dev.yaml — local-build variant with Redis + TimescaleDB +
processor-dev for verifying Dockerfile changes without the registry
round-trip.
README.md — fleshed out from stub: quick-start, Docker build, deployment
note, env vars, tests (unit vs. integration), CI behavior. Flags the
deploy-side change needed: deploy/compose.yaml needs a TimescaleDB
service and a processor service entry added.
Verification: typecheck, lint clean; 134 unit tests passing across 8
files (+22 from this batch). pnpm test:integration runs cleanly under
the no-Docker skip pattern.
Phase 1 is now complete. Service is pilot-ready.
This commit is contained in:
@@ -5,3 +5,90 @@ Node.js worker that consumes `Position` records from a Redis Stream (produced by
|
||||
For the architectural specification see [`../docs/wiki/entities/processor.md`](../docs/wiki/entities/processor.md). For the work plan and task status see [`.planning/ROADMAP.md`](./.planning/ROADMAP.md).
|
||||
|
||||
This service is part of the [TRM](https://git.dev.microservices.al/trm) (Time Racing Management) platform.
|
||||
|
||||
---
|
||||
|
||||
## Quick start (local)
|
||||
|
||||
**Prerequisites:** Node.js 22+, pnpm, a local Redis instance, and a TimescaleDB instance.
|
||||
|
||||
```bash
|
||||
git clone <repo-url>
|
||||
cd processor
|
||||
pnpm install
|
||||
cp .env.example .env
|
||||
# Edit .env — at minimum set REDIS_URL and POSTGRES_URL
|
||||
pnpm dev
|
||||
```
|
||||
|
||||
`pnpm dev` uses `tsx watch` for hot-reload during development. The metrics server listens on `METRICS_PORT` (default `9090`). The service connects to Redis and Postgres on startup; both must be reachable before the process starts.
|
||||
|
||||
---
|
||||
|
||||
## Test the Docker build locally
|
||||
|
||||
`compose.dev.yaml` builds the image from source and runs it next to Redis and TimescaleDB containers. Useful for verifying Dockerfile changes before pushing:
|
||||
|
||||
```bash
|
||||
docker compose -f compose.dev.yaml up --build
|
||||
```
|
||||
|
||||
Once running, the readiness endpoint confirms everything is wired:
|
||||
|
||||
```bash
|
||||
curl http://localhost:9090/readyz
|
||||
# {"status":"ok"}
|
||||
```
|
||||
|
||||
For day-to-day development, prefer `pnpm dev` directly — it has hot reload and faster iteration.
|
||||
|
||||
---
|
||||
|
||||
## Production / stage deployment
|
||||
|
||||
This service is **not** deployed standalone. It runs as part of the platform stack defined in the [`deploy/`](https://git.dev.microservices.al/trm/deploy) repo, which Portainer pulls and runs on the stage and production hosts.
|
||||
|
||||
The image itself is published to `git.dev.microservices.al/trm/processor:main` on every push to `main` (see CI behavior below). The `deploy/` repo's `compose.yaml` references that image; updates flow through there, not through this repo.
|
||||
|
||||
To pin a specific commit in production, set `PROCESSOR_TAG=<sha>` in the deploy stack's environment variables.
|
||||
|
||||
> **Note:** The `deploy/compose.yaml` will need a `processor` service entry and a TimescaleDB service added before this service can run in stage/production. See `.planning/phase-1-throughput/11-dockerfile-and-ci.md` for the expected service block shape. That is a deploy-side change for the user to make.
|
||||
|
||||
---
|
||||
|
||||
## Environment variables
|
||||
|
||||
See `.env.example` for all variables with descriptions and defaults. Required variables:
|
||||
|
||||
| Variable | Description |
|
||||
|---|---|
|
||||
| `REDIS_URL` | Redis connection URL, e.g. `redis://localhost:6379` |
|
||||
| `POSTGRES_URL` | TimescaleDB connection URL, e.g. `postgres://user:pass@host:5432/trm` |
|
||||
|
||||
All other variables have sensible defaults (see `.env.example`).
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
- `pnpm test` — unit tests only. Fast (~1–2 s), no external dependencies. **This is what CI runs.**
|
||||
- `pnpm test:integration` — integration tests that need Docker (testcontainers spins up real Redis 7 and TimescaleDB containers). **Opt-in.** Run locally before changes to the consumer, writer, or migration.
|
||||
|
||||
Integration tests live in `test/**/*.integration.test.ts` and are excluded from the default run by `vitest.config.ts`.
|
||||
|
||||
### Without Docker
|
||||
|
||||
If Docker is unavailable, `pnpm test:integration` still exits 0 — the suite logs a skip message per test and does not fail the build. This is the correct behavior for CI runners that lack Docker access.
|
||||
|
||||
---
|
||||
|
||||
## CI behavior
|
||||
|
||||
Gitea Actions workflow is at `.gitea/workflows/build.yml`.
|
||||
|
||||
- **Push to `main`** (only when `src/`, `test/`, build config, Dockerfile, or the workflow file itself changes): runs `typecheck`, `lint`, `test` (unit tests only), then builds and pushes the Docker image tagged `:main`. Auto-deploys to stage if a Portainer webhook is configured via `secrets.PORTAINER_WEBHOOK_URL`.
|
||||
- **Manual trigger** (`workflow_dispatch`): same flow, run on demand.
|
||||
|
||||
Integration tests are not run in CI — they need Docker access on the runner, which is not currently configured. Run them locally as needed.
|
||||
|
||||
The workflow uses `secrets.REGISTRY_USERNAME` and `secrets.REGISTRY_PASSWORD` for the Gitea registry login — these must be configured in the repo's (or org's) Actions secrets.
|
||||
|
||||
Reference in New Issue
Block a user