directus/.planning/phase-1-slice-1-schema/03-initial-migrations.md

Task 1.3 — Initial migrations

Phase: 1 — Slice 1 schema + deploy pipeline Status: ⬜ Not started Depends on: 1.2 Wiki refs: docs/wiki/entities/postgres-timescaledb.md, docs/wiki/concepts/position-record.md, docs/wiki/entities/processor.md (Faulty position handling)

Goal

Author the three Phase 1 migrations under db-init/: the TimescaleDB extension, the positions hypertable creation, and the faulty boolean column. Each is internally idempotent so that environments where they were applied ad-hoc (e.g. existing stage) absorb them as no-ops.

Deliverables

• db-init/001_extensions.sql:

  CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;
  

• db-init/002_positions_hypertable.sql — column shapes match processor/src/db/migrations/0001_positions.sql exactly. Processor is the writer; directus's migration must absorb cleanly into the schema processor produces. Do NOT diverge from the column types/nullability below without coordinating a processor-side migration first.

  CREATE TABLE IF NOT EXISTS positions (
    device_id    text             NOT NULL,
    ts           timestamptz      NOT NULL,
    ingested_at  timestamptz      NOT NULL DEFAULT now(),
    latitude     double precision NOT NULL,
    longitude    double precision NOT NULL,
    altitude     real             NOT NULL,
    angle        real             NOT NULL,
    speed        real             NOT NULL,
    satellites   smallint         NOT NULL,
    priority     smallint         NOT NULL,
    codec        text             NOT NULL,
    attributes   jsonb            NOT NULL
  );
  
  -- Hypertable: 1-day chunks, idiomatic for GPS telemetry at 1-60s intervals.
  SELECT create_hypertable(
    'positions', 'ts',
    if_not_exists => TRUE,
    chunk_time_interval => INTERVAL '1 day'
  );
  
  -- Two indexes: unique on (device_id, ts) for ON CONFLICT idempotency,
  -- separate descending ts index for global range scans.
  CREATE UNIQUE INDEX IF NOT EXISTS positions_device_ts ON positions (device_id, ts);
  CREATE INDEX IF NOT EXISTS positions_ts ON positions (ts DESC);
  

  No PRIMARY KEY — the unique index is idiomatic for TimescaleDB hypertables. End the file with a DO $$ ... $$ assertion block confirming the table exists, is registered as a hypertable, every column has the expected data_type from information_schema.columns, and both indexes are present. The assertion catches the case where stage already has the table with subtly different types.
• db-init/003_faulty_column.sql:

  ALTER TABLE positions
    ADD COLUMN IF NOT EXISTS faulty BOOLEAN NOT NULL DEFAULT FALSE;
  
  CREATE INDEX IF NOT EXISTS positions_faulty_idx
    ON positions (device_id, ts DESC) WHERE faulty = FALSE;
  

Specification

• Schema must match what processor writes. Cross-check column names, types, nullability against docs/wiki/concepts/position-record.md and the actual processor writer code (processor/src/db/migrations/0001_positions.sql). If any field differs, this task is blocked until directus-schema-draft and the processor's existing migration are reconciled — fix the divergence in the doc first, then this task.
• attributes is JSONB NOT NULL DEFAULT '{}' — never null, always an object. Keeps query plans simple.
• (device_id, ts) primary key — natural key, idempotent for the processor's ON CONFLICT DO NOTHING writer.
• Chunk interval = 7 days. Tunable later; 7 days is a reasonable default for hundreds of devices emitting at multi-Hz.
• Faulty index uses a partial-index WHERE faulty = FALSE. Optimizes the processor hot-path read which always filters faulty out. Operator queries that select faulty rows specifically use the broader (device_id, ts DESC) index.
• CASCADE on CREATE EXTENSION so that any dependent extensions install transparently. TimescaleDB has no required deps so CASCADE is a no-op for now, but harmless and future-proof.
• No IF EXISTS shortcuts that hide schema drift. The migrations are idempotent at the DDL level (IF NOT EXISTS), but if a column type already differs from what the file declares, the migration silently passes — leaving stage in an inconsistent state. Add a final DO $$ ... $$ block per file that asserts the table shape is what the migration intends:

  -- end of 002_positions_hypertable.sql
  DO $$ BEGIN
    IF NOT EXISTS (
      SELECT 1 FROM information_schema.columns
      WHERE table_name = 'positions' AND column_name = 'attributes' AND data_type = 'jsonb'
    ) THEN
      RAISE EXCEPTION 'positions.attributes is not JSONB — schema drift';
    END IF;
  END $$;
  

  One assertion per critical column shape. Catches the case where stage has the table but with subtly different types.

Acceptance criteria

• Against a fresh Postgres + TimescaleDB image, apply-db-init.sh runs all three files cleanly.
• \d positions shows the expected columns (including faulty).
• SELECT * FROM timescaledb_information.hypertables WHERE hypertable_name = 'positions'; returns one row.
• Indexes positions_device_ts (unique), positions_ts, and positions_faulty_idx (partial) all exist (\di+).
• Re-running the script is a no-op (verified via migrations_applied table contents).
• Against a Postgres that already has positions from a prior ad-hoc run, the migration absorbs it as a no-op (provided the existing schema matches; otherwise the assertion blocks deploy).
• Cross-checked against processor/src/db/migrations/0001_positions.sql — column names, types, indexes match.

Risks / open questions

• Existing stage Postgres may have a slightly different schema. Run pg_dump --schema-only -t positions on stage before this task lands and compare to the migration above. Reconcile differences in this file (or document them as known-divergent).
• Hypertable was created before — create_hypertable with if_not_exists should accept it, but the chunk interval can't be retroactively changed via this call. Both this migration and processor's use INTERVAL '1 day', so divergence is unlikely. If a stage env has a different interval, that's a non-blocking divergence (functional, just suboptimal). Don't try to migrate it via SQL; leave it as a follow-up.
• PostGIS double-init. Processor's 0001_positions.sql enables postgis already. On stage, postgis is therefore present once processor has run. Directus's 001_extensions.sql deliberately omits postgis (Plan A — defer to Phase 2 when geofences/SLZs/waypoints land in directus). Local dev environments without processor will see the "PostGIS isn't installed" warning during Directus boot; this is benign for Phase 1 (no geometry columns) and resolves automatically in Phase 2.

Done

Implementation landed and live-verified 2026-05-01. All acceptance criteria pass; one non-blocking observation about redundant indexes recorded below.

Files created at C:\Users\Administrator\projects\trm\directus\db-init\:

• 001_extensions.sql — timescaledb only (postgis intentionally deferred per Plan A).
• 002_positions_hypertable.sql — exact column-by-column match against processor/src/db/migrations/0001_positions.sql.
• 003_faulty_column.sql — operator-controlled flag + partial index on WHERE faulty = FALSE.

All three files end with DO $$ ... $$ assertion blocks that fail the boot if the resulting schema doesn't match expectations (catches the silent-IF NOT EXISTS-against-drifted-schema case).

8 task-spec divergences resolved (processor wins per the brief; spec was wrong):

1. Added ingested_at timestamptz NOT NULL DEFAULT now() (spec didn't list it; processor writes it).
2. Added codec text NOT NULL (spec didn't list it; processor writes it on every insert).
3. Changed altitude / angle / speed from DOUBLE PRECISION nullable → real NOT NULL (matches processor's float32 storage and never-null guarantee).
4. Changed satellites / priority to NOT NULL.
5. Removed attributes's DEFAULT '{}'::jsonb (processor always writes attributes; default is misleading).
6. Replaced PRIMARY KEY (device_id, ts) with CREATE UNIQUE INDEX positions_device_ts (idiomatic for TimescaleDB hypertables; matches processor).
7. Changed chunk interval from 7 days to 1 day (processor's choice; better for GPS at 1-60s intervals).
8. Replaced single positions_device_ts_idx with two indexes (positions_device_ts + positions_ts) per processor.

The deliverables section above has been updated to reflect what shipped. Future readers see the corrected spec, not the originally-wrong one.

Live-test commands:

docker compose -f compose.dev.yaml down -v        # wipe volumes
docker compose -f compose.dev.yaml build           # rebuild image with new SQL files baked in
docker compose -f compose.dev.yaml up -d db        # start db only

# First apply — expect 3 applied, 0 skipped, exit 0
docker compose -f compose.dev.yaml run --rm --no-deps \
  -e DB_HOST=db -e DB_PORT=5432 -e DB_USER=directus \
  -e DB_PASSWORD=directus -e DB_DATABASE=directus \
  -e DB_INIT_DIR=/directus/db-init \
  --entrypoint /directus/scripts/apply-db-init.sh \
  directus

# Re-run — expect 0 applied, 3 skipped, exit 0
docker compose -f compose.dev.yaml run --rm --no-deps \
  -e DB_HOST=db -e DB_PORT=5432 -e DB_USER=directus \
  -e DB_PASSWORD=directus -e DB_DATABASE=directus \
  -e DB_INIT_DIR=/directus/db-init \
  --entrypoint /directus/scripts/apply-db-init.sh \
  directus

# Inspect the resulting schema
docker compose -f compose.dev.yaml exec db psql -U directus -d directus -c "\d positions"
docker compose -f compose.dev.yaml exec db psql -U directus -d directus -c "\di+ positions_*"
docker compose -f compose.dev.yaml exec db psql -U directus -d directus -c "SELECT * FROM timescaledb_information.hypertables WHERE hypertable_name = 'positions';"

Live-verification results (2026-05-01):

• ✅ First apply: 3 applied, 0 skipped, exit 0. Migration 001 logged NOTICE: extension "timescaledb" already exists, skipping because the timescaledb-ha image pre-creates the extension at DB initialization — our IF NOT EXISTS correctly absorbs this.
• ✅ Re-run: 0 applied, 3 skipped, exit 0. The migrations_applied guard table works as designed.
• ✅ \d positions shows all 13 columns with correct types and nullability: device_id, ts, ingested_at, latitude, longitude, altitude, angle, speed, satellites, priority, codec, attributes, faulty. Defaults verified: ingested_at DEFAULT now(), faulty DEFAULT false.
• ✅ Hypertable registered (timescaledb_information.hypertables returned 1 row, 1 dimension, 0 chunks at zero data).
• ✅ Three expected indexes present: positions_device_ts (unique), positions_faulty_idx (partial WHERE faulty = false), positions_ts.

Non-blocking observation — auto-created positions_ts_idx redundancy:

\di+ positions_* returned 4 indexes, not 3. TimescaleDB's create_hypertable() defaults to auto-creating an index on the time partition column (positions_ts_idx ON (ts DESC)), which duplicates our explicit positions_ts ON (ts DESC). Both are functionally identical; cost is ~8KB extra per chunk across all chunks.

This is not a directus-side bug. Processor's migration (processor/src/db/migrations/0001_positions.sql) creates positions_ts explicitly without disabling the auto-index, so stage already has the redundancy. Our migration matches processor's pattern.

Cleanup path (Phase 3 hardening, not Phase 1):

SELECT create_hypertable(
  'positions', 'ts',
  if_not_exists => TRUE,
  chunk_time_interval => INTERVAL '1 day',
  create_default_indexes => FALSE   -- prevents auto-creation of positions_ts_idx
);

Both directus's and processor's migrations would need to apply this together (and a Phase 3 migration to drop the existing positions_ts_idx). Defer until Phase 3 since it's storage optimization, not correctness.