processor/.planning/phase-3-hardening/05-retire-migration-runner.md

Task 3.5 — Retire processor migration runner

Phase: 3 — Production hardening Status: ⬜ Not started Depends on: Phase 1.5 ideally landed (avoid mid-flight code churn for the agent shipping the WS endpoint). No hard code dependency. Replaces: the original 3.5 sketch ("Migration advisory lock"). Once the processor doesn't run migrations, the lock concern is delegated to Directus's db-init runner — outside this service's surface. Wiki refs: docs/wiki/entities/processor.md §"Schema ownership vs. write access" (the line that needs to change), docs/wiki/entities/directus.md §"Schema management — snapshot/apply pipeline", docs/wiki/entities/postgres-timescaledb.md

Goal

Establish directus as the single owner of all DDL against the shared Postgres database. Retire the processor's migration runner. After this task, the only DDL paths are:

• trm/directus/db-init/*.sql (pre-schema: extensions, hypertables, raw tables Directus's snapshot-yaml format can't express).
• trm/directus/snapshots/schema.yaml (Directus-managed user collections).
• trm/directus/db-init-post/*.sql (post-schema: composite UNIQUE constraints on Directus-managed tables).

Processor exclusively does INSERT / SELECT / UPDATE. No CREATE, ALTER, CREATE EXTENSION, or any other DDL.

Context — why this exists

The current state has both services creating the positions hypertable and the faulty column:

• trm/processor/src/db/migrations/0001_positions.sql and 0002_positions_faulty.sql (processor's runner, from Phase 1 task 1.4 + the recent 1.5.5 prep).
• trm/directus/db-init/001_extensions.sql, 002_positions_hypertable.sql, 003_faulty_column.sql (directus's runner, added later when the destructive-apply incident showed positions had to exist before directus schema apply runs or it would get wiped).

Both runners are idempotent (IF NOT EXISTS, etc.) so the runtime collision is benign at the moment, but the architectural risks are real:

• Two sources of truth. Adding a column means editing two files in two repos; either one can drift silently.
• Schema divergence. A processor migration that adds a column the directus side doesn't know about is silently invisible to the admin UI.
• Two migrations_applied tables, which already caused the ghost-collection apply conflict earlier in Phase 1 of directus.
• Operator confusion. The wiki says "Directus owns the schema" but processor runs migrations — newcomers can't tell which is canonical.

The fix is the wiki's stated intent: directus owns DDL. Processor was the historical owner before directus's db-init story matured; the legacy runner survived the transition because nobody retired it.

Pre-flight (before deleting anything)

1. Confirm directus's db-init/ covers the full processor schema surface

Check that trm/directus/db-init/'s SQL is byte-equivalent (or semantically equivalent) to processor's migrations. As of writing, directus has:

• 001_extensions.sql — CREATE EXTENSION IF NOT EXISTS timescaledb + postgis.
• 002_positions_hypertable.sql — CREATE TABLE positions (...) + create_hypertable(...) + indexes.
• 003_faulty_column.sql — ALTER TABLE positions ADD COLUMN IF NOT EXISTS faulty ... + positions_device_ts_idx.

Processor has:

• src/db/migrations/0001_positions.sql — extensions + table + hypertable + positions_device_ts (UNIQUE on (device_id, ts)) + positions_ts (DESC).
• src/db/migrations/0002_positions_faulty.sql — faulty column + positions_device_ts_idx ((device_id, ts DESC)).

Diff the two before retiring. If processor's SQL has an index, column, or constraint directus's db-init/ doesn't, the deliverable starts with porting that diff into directus's db-init/ (and snapshotting if applicable). Specific things to verify:

• All four indexes exist in directus's db-init: positions_device_ts (UNIQUE), positions_ts, positions_device_ts_idx.
• Column types match exactly: device_id text, ts timestamptz, ingested_at timestamptz DEFAULT now(), etc.
• chunk_time_interval is INTERVAL '1 day' on both sides.
• The ON CONFLICT (device_id, ts) DO NOTHING upsert path requires the UNIQUE on (device_id, ts) — that's the positions_device_ts index, not positions_device_ts_idx. Both must exist.

2. Verify the directus db-init apply order is fixed

Per docs/wiki/entities/directus.md's 5-step boot pipeline:

1. db-init pre-schema   → positions hypertable, faulty column, timescaledb extension
2. directus bootstrap   → Directus system tables + first admin
3. directus schema apply → Directus-managed user collections
4. db-init post-schema  → composite UNIQUE constraints on user collections
5. pm2-runtime start    → server up at :8055

So when processor boots against a stage stack:

• directus container has run steps 1–4 (positions exists; everything else exists).
• processor container can connect and INSERT immediately.

Compose ordering. trm/deploy/compose.yaml's processor service should depends_on: directus with condition: service_healthy so processor doesn't try to read positions before directus's db-init has run on first deploy. Verify in this task.

Deliverables

trm/processor/

1. Delete src/db/migrations/0001_positions.sql.
2. Delete src/db/migrations/0002_positions_faulty.sql.
3. Delete the migrations directory if it's now empty.
4. Delete src/db/migrate.ts (or whatever the migration-runner module is named — the file that owns the migrations_applied table, the file walker, the pg_advisory_lock if any).
5. Update src/main.ts to remove the await migrate(...) step from boot. Postgres pool creation stays; migration call goes.
6. Update tests that exercise the migration runner — most likely deletes the corresponding test file. Integration tests that previously seeded the schema via migrate() either:
   • (a) Use directus's db-init/*.sql files directly (read them in beforeAll, execute against the testcontainer Postgres), or
   • (b) Carry a fixture SQL file in test/fixtures/ (the same approach Phase 1.5 task 1.5.6 already takes for its integration test). Pick (b) — it's already the established pattern.
7. Update Dockerfile to drop any migration-running step from the entrypoint (Phase 1's Dockerfile may not have this, but verify; the runtime container shouldn't carry the migrations directory if the runner is gone).
8. Update package.json dependencies — if pg-migrate or any migration-runner library was a Phase-1-only dep, remove it.
9. Update phase-1-throughput/04-postgres-schema.md's Done section with a note: "Migration runner retired in Phase 3 task 3.5 — see that task for context."
10. Update ROADMAP.md to reflect the retired runner under Phase 1's "what changed since landing" note.

trm/docs/wiki/

1. Update wiki/entities/processor.md — drop the "Schema ownership vs. write access" caveat that says "the positions hypertable is owned by processor's migration runner." Replace with a single sentence: "Processor never runs DDL. Schema is exclusively owned by Directus (snapshot.yaml + db-init/ for things the snapshot can't express)."
2. Update wiki/entities/directus.md — confirm the Schema-management section already lists db-init/ as covering everything (no edits unless current text implies a split).
3. Update wiki/entities/postgres-timescaledb.md — verify the writer-side documentation; remove any "split schema authority" framing.
4. Append docs/log.md with a note entry recording the retirement.

trm/deploy/

1. Verify compose.yaml's processor service has depends_on: directus: { condition: service_healthy }. Add if missing.
2. Confirm the deploy README doesn't mention the processor's migration runner anywhere.

Specification

What stays in src/db/

• pool.ts — Postgres Pool factory. Untouched.
• Connection helpers, query helpers (if any). Untouched.

What goes

• migrations/*.sql — gone.
• migrate.ts (the runner). Gone.
• migrations_applied table — directus's runner has its own; the processor's becomes orphaned but harmless. Don't drop it from existing databases. The retirement is a runtime change; the table is just unused. Phase 3 hardening's eventual OPERATIONS.md (task 3.7) can document a one-off DROP TABLE migrations_applied step for operators who want a clean schema.

Boot order on first deploy

1. postgres container starts → DB available.
2. directus container starts → runs 5-step boot pipeline.
   ├─ Step 1 (db-init pre-schema) creates positions hypertable + faulty column + extensions.
   ├─ Steps 2-4 set up Directus's own world.
   └─ Step 5 marks the container healthy.
3. processor container starts (depends_on: directus: service_healthy) → connects, finds positions, starts consuming.

If processor races directus on a fresh stack (no depends_on), it'll fail to find the positions table and crash-loop until directus catches up. depends_on: service_healthy makes the order deterministic.

Dev workflow

compose.dev.yaml in trm/processor (if it exists for processor-side dev) should depends_on: directus if running both. For pure-processor dev (no directus), the developer either:

• Runs directus's db-init/*.sql manually against their local Postgres before booting processor.
• Or copies the equivalent SQL into a one-off bootstrap script in processor/test/fixtures/.

Document the chosen path in processor/README.md.

What this task does NOT do

• Does not retire directus's snapshot-managed collections.
• Does not change Phase 1 or Phase 1.5 code paths beyond removing the migration runner step.
• Does not introduce a new migration tool. The fix is fewer moving parts, not different ones.

Acceptance criteria

• pnpm typecheck, pnpm lint, pnpm test clean.
• pnpm test:integration runs green — the integration test no longer relies on migrate(); it loads schema from a fixture SQL file instead.
• src/db/migrations/ directory is gone (or empty + gitignored).
• No migrate() call anywhere in the source tree.
• No migrations_applied references in processor source.
• Stage smoke against a fresh DB: redeploy the stack, watch directus boot through its 5 steps, watch processor connect and start writing positions. No errors.
• docs/wiki/entities/processor.md and directus.md agree: directus is the sole DDL owner.
• docs/log.md has a note entry recording the retirement.
• trm/deploy/compose.yaml's processor service has depends_on: directus: service_healthy.

Risks / open questions

• Existing prod databases. If anyone has deployed the processor's migrations on a real DB, the migrations_applied table is harmless but stale. Document a one-off cleanup query for operators (in OPERATIONS.md when 3.7 lands).
• Schema drift between processor's old migrations and directus's db-init. If the diff in pre-flight step 1 surfaces anything, that diff must land in directus's db-init/ before the processor's runner is retired. Order of operations matters: never delete the processor migration before the equivalent SQL is verified live in directus's runner.
• Test container schema setup. The integration test fixture has to mirror what directus actually creates. If directus's db-init/ changes in a way that breaks processor's read paths, the fixture and the read paths both need updating. Mitigation: the fixture file lives in test/fixtures/ and a comment at its top says "syncs with trm/directus/db-init/ — update both when schema changes."
• The original 3.5 ("Migration advisory lock") concern. Once processor doesn't run migrations, the advisory-lock concern is delegated to directus's runner. That's a directus concern; whether to add an advisory lock to directus's apply-db-init.sh is tracked as a follow-up in directus's own roadmap, not here.
• PostGIS usage in Phase 2. Processor's 0001_positions.sql enables PostGIS even though Phase 1 doesn't use it. Directus's db-init/001_extensions.sql does the same. Confirm in pre-flight; no change needed if the directus side already has it.

Done

(Filled in when the task lands.)