trm/directus
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Task 1.3 — Initial migrations

Browse Source 
Three SQL files under db-init/ create the schema processor writes
against. All three apply cleanly via apply-db-init.sh, are idempotent
on re-run, and end with assertion blocks that catch silent
schema drift.

001_extensions.sql — registers timescaledb on the directus database.
  PostGIS deferred to Phase 2 (per Plan A). The timescaledb-ha image
  pre-creates the extension at DB init, so the IF NOT EXISTS guard
  fires as a NOTICE — expected and harmless.

002_positions_hypertable.sql — positions hypertable, exact
  column-by-column match against processor/src/db/migrations/0001_positions.sql.

  Cross-checking against processor surfaced 8 divergences from the
  original task spec; processor wins in every case (it is the writer
  and is in production). The corrections:

    - added ingested_at timestamptz NOT NULL DEFAULT now()
    - added codec text NOT NULL
    - altitude/angle/speed: real NOT NULL (not DOUBLE PRECISION nullable)
    - satellites/priority: NOT NULL
    - removed attributes DEFAULT '{}'::jsonb (processor always writes)
    - replaced PRIMARY KEY with UNIQUE INDEX positions_device_ts
      (idiomatic for TimescaleDB hypertables)
    - chunk interval 1 day, not 7 days
    - two indexes (positions_device_ts + positions_ts), not one composite

  Without these corrections every processor INSERT would have failed
  with NOT NULL violations. Spec deliverables section updated to
  reflect the correct shape so future readers see the right schema.

003_faulty_column.sql — adds the operator-controlled faulty boolean
  flag plus the partial index positions_faulty_idx ON (device_id,
  ts DESC) WHERE faulty = FALSE. The column is set only via Directus
  admin (Phase 4 permissions); processor's writer never touches it.
  The partial index optimises the hot-path read pattern (every
  processor evaluator filters faulty = FALSE); operator queries that
  look at faulty rows specifically use the broader positions_device_ts
  index from 002.

Live-verified 2026-05-01:
  - First apply: 3 applied, 0 skipped, exit 0.
  - Re-run: 0 applied, 3 skipped, exit 0.
  - All 13 columns present with correct types/nullability/defaults.
  - Hypertable registered with 1-day chunk interval.
  - Three expected indexes present.

Non-blocking observation: TimescaleDB's create_hypertable()
auto-created a fourth index (positions_ts_idx) duplicating our
explicit positions_ts. Processor's migration has the same redundancy
so stage already lives with this. Cleanup path documented in the
task spec for Phase 3 hardening (create_default_indexes => FALSE
in the create_hypertable call).

ROADMAP marks 1.3 done; 1.4 next.
This commit is contained in:
Julian Çuni
2026-05-01 22:51:34 +02:00
parent dec2d190ce
commit 25a9731070
5 changed files with 514 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.planning/phase-1-slice-1-schema/03-initial-migrations.md
+101 -20
View File
@@ -15,32 +15,36 @@ Author the three Phase 1 migrations under `db-init/`: the TimescaleDB extension,
```sql
CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;
```
- `db-init/002_positions_hypertable.sql`:
- `db-init/002_positions_hypertable.sql` — column shapes match `processor/src/db/migrations/0001_positions.sql` exactly. Processor is the writer; directus's migration must absorb cleanly into the schema processor produces. Do NOT diverge from the column types/nullability below without coordinating a processor-side migration first.
```sql
CREATE TABLE IF NOT EXISTS positions (
device_id TEXT NOT NULL,
ts TIMESTAMPTZ NOT NULL,
latitude DOUBLE PRECISION NOT NULL,
longitude DOUBLE PRECISION NOT NULL,
altitude DOUBLE PRECISION,
angle SMALLINT,
speed SMALLINT,
satellites SMALLINT,
priority SMALLINT,
attributes JSONB NOT NULL DEFAULT '{}'::jsonb,
PRIMARY KEY (device_id, ts)
device_id text NOT NULL,
ts timestamptz NOT NULL,
ingested_at timestamptz NOT NULL DEFAULT now(),
latitude double precision NOT NULL,
longitude double precision NOT NULL,
altitude real NOT NULL,
angle real NOT NULL,
speed real NOT NULL,
satellites smallint NOT NULL,
priority smallint NOT NULL,
codec text NOT NULL,
attributes jsonb NOT NULL
);
-- Idempotent hypertable creation: if_not_exists => true
-- Hypertable: 1-day chunks, idiomatic for GPS telemetry at 1-60s intervals.
SELECT create_hypertable(
'positions', 'ts',
chunk_time_interval => INTERVAL '7 days',
if_not_exists => TRUE
if_not_exists => TRUE,
chunk_time_interval => INTERVAL '1 day'
);
CREATE INDEX IF NOT EXISTS positions_device_ts_idx
ON positions (device_id, ts DESC);
-- Two indexes: unique on (device_id, ts) for ON CONFLICT idempotency,
-- separate descending ts index for global range scans.
CREATE UNIQUE INDEX IF NOT EXISTS positions_device_ts ON positions (device_id, ts);
CREATE INDEX IF NOT EXISTS positions_ts ON positions (ts DESC);
```
No `PRIMARY KEY` — the unique index is idiomatic for TimescaleDB hypertables. End the file with a `DO $$ ... $$` assertion block confirming the table exists, is registered as a hypertable, every column has the expected `data_type` from `information_schema.columns`, and both indexes are present. The assertion catches the case where stage already has the table with subtly different types.
- `db-init/003_faulty_column.sql`:
```sql
ALTER TABLE positions
@@ -77,7 +81,7 @@ Author the three Phase 1 migrations under `db-init/`: the TimescaleDB extension,
- [ ] Against a fresh Postgres + TimescaleDB image, `apply-db-init.sh` runs all three files cleanly.
- [ ] `\d positions` shows the expected columns (including `faulty`).
- [ ] `SELECT * FROM timescaledb_information.hypertables WHERE hypertable_name = 'positions';` returns one row.
- [ ] Both indexes (`positions_device_ts_idx`, `positions_faulty_idx`) exist (`\di+`).
- [ ] Indexes `positions_device_ts` (unique), `positions_ts`, and `positions_faulty_idx` (partial) all exist (`\di+`).
- [ ] Re-running the script is a no-op (verified via `migrations_applied` table contents).
- [ ] Against a Postgres that *already* has `positions` from a prior ad-hoc run, the migration absorbs it as a no-op (provided the existing schema matches; otherwise the assertion blocks deploy).
- [ ] Cross-checked against `processor/src/db/migrations/0001_positions.sql` — column names, types, indexes match.
@@ -85,8 +89,85 @@ Author the three Phase 1 migrations under `db-init/`: the TimescaleDB extension,
## Risks / open questions
- **Existing stage Postgres may have a slightly different schema.** Run `pg_dump --schema-only -t positions` on stage before this task lands and compare to the migration above. Reconcile differences in this file (or document them as known-divergent).
- **Hypertable was created before — `create_hypertable` with `if_not_exists` should accept it, but the chunk interval can't be retroactively changed via this call.** If stage's chunk interval differs from `7 days`, that's a non-blocking divergence (functional, just suboptimal). Don't try to migrate it via SQL; leave it as a follow-up.
- **Hypertable was created before — `create_hypertable` with `if_not_exists` should accept it, but the chunk interval can't be retroactively changed via this call.** Both this migration and processor's use `INTERVAL '1 day'`, so divergence is unlikely. If a stage env has a different interval, that's a non-blocking divergence (functional, just suboptimal). Don't try to migrate it via SQL; leave it as a follow-up.
- **PostGIS double-init.** Processor's `0001_positions.sql` enables `postgis` already. On stage, postgis is therefore present once processor has run. Directus's `001_extensions.sql` deliberately omits postgis (Plan A — defer to Phase 2 when geofences/SLZs/waypoints land in directus). Local dev environments without processor will see the "PostGIS isn't installed" warning during Directus boot; this is benign for Phase 1 (no geometry columns) and resolves automatically in Phase 2.
## Done
(Fill in commit SHA + one-line note when this lands.)
**Implementation landed and live-verified 2026-05-01.** All acceptance criteria pass; one non-blocking observation about redundant indexes recorded below.
Files created at `C:\Users\Administrator\projects\trm\directus\db-init\`:
- `001_extensions.sql` — timescaledb only (postgis intentionally deferred per Plan A).
- `002_positions_hypertable.sql` — exact column-by-column match against `processor/src/db/migrations/0001_positions.sql`.
- `003_faulty_column.sql` — operator-controlled flag + partial index on `WHERE faulty = FALSE`.
All three files end with `DO $$ ... $$` assertion blocks that fail the boot if the resulting schema doesn't match expectations (catches the silent-`IF NOT EXISTS`-against-drifted-schema case).
**8 task-spec divergences resolved (processor wins per the brief; spec was wrong):**
1. Added `ingested_at timestamptz NOT NULL DEFAULT now()` (spec didn't list it; processor writes it).
2. Added `codec text NOT NULL` (spec didn't list it; processor writes it on every insert).
3. Changed `altitude` / `angle` / `speed` from `DOUBLE PRECISION nullable` → `real NOT NULL` (matches processor's float32 storage and never-null guarantee).
4. Changed `satellites` / `priority` to `NOT NULL`.
5. Removed `attributes`'s `DEFAULT '{}'::jsonb` (processor always writes attributes; default is misleading).
6. Replaced `PRIMARY KEY (device_id, ts)` with `CREATE UNIQUE INDEX positions_device_ts` (idiomatic for TimescaleDB hypertables; matches processor).
7. Changed chunk interval from `7 days` to `1 day` (processor's choice; better for GPS at 1-60s intervals).
8. Replaced single `positions_device_ts_idx` with two indexes (`positions_device_ts` + `positions_ts`) per processor.
The deliverables section above has been updated to reflect what shipped. Future readers see the corrected spec, not the originally-wrong one.
**Live-test commands:**
```bash
docker compose -f compose.dev.yaml down -v # wipe volumes
docker compose -f compose.dev.yaml build # rebuild image with new SQL files baked in
docker compose -f compose.dev.yaml up -d db # start db only
# First apply — expect 3 applied, 0 skipped, exit 0
docker compose -f compose.dev.yaml run --rm --no-deps \
-e DB_HOST=db -e DB_PORT=5432 -e DB_USER=directus \
-e DB_PASSWORD=directus -e DB_DATABASE=directus \
-e DB_INIT_DIR=/directus/db-init \
--entrypoint /directus/scripts/apply-db-init.sh \
directus
# Re-run — expect 0 applied, 3 skipped, exit 0
docker compose -f compose.dev.yaml run --rm --no-deps \
-e DB_HOST=db -e DB_PORT=5432 -e DB_USER=directus \
-e DB_PASSWORD=directus -e DB_DATABASE=directus \
-e DB_INIT_DIR=/directus/db-init \
--entrypoint /directus/scripts/apply-db-init.sh \
directus
# Inspect the resulting schema
docker compose -f compose.dev.yaml exec db psql -U directus -d directus -c "\d positions"
docker compose -f compose.dev.yaml exec db psql -U directus -d directus -c "\di+ positions_*"
docker compose -f compose.dev.yaml exec db psql -U directus -d directus -c "SELECT * FROM timescaledb_information.hypertables WHERE hypertable_name = 'positions';"
```
**Live-verification results (2026-05-01):**
- ✅ First apply: `3 applied, 0 skipped`, exit 0. Migration 001 logged `NOTICE: extension "timescaledb" already exists, skipping` because the timescaledb-ha image pre-creates the extension at DB initialization — our `IF NOT EXISTS` correctly absorbs this.
- ✅ Re-run: `0 applied, 3 skipped`, exit 0. The `migrations_applied` guard table works as designed.
- ✅ `\d positions` shows all 13 columns with correct types and nullability: device_id, ts, ingested_at, latitude, longitude, altitude, angle, speed, satellites, priority, codec, attributes, faulty. Defaults verified: `ingested_at DEFAULT now()`, `faulty DEFAULT false`.
- ✅ Hypertable registered (`timescaledb_information.hypertables` returned 1 row, 1 dimension, 0 chunks at zero data).
- ✅ Three expected indexes present: `positions_device_ts` (unique), `positions_faulty_idx` (partial WHERE faulty = false), `positions_ts`.
**Non-blocking observation — auto-created `positions_ts_idx` redundancy:**
`\di+ positions_*` returned 4 indexes, not 3. TimescaleDB's `create_hypertable()` defaults to auto-creating an index on the time partition column (`positions_ts_idx ON (ts DESC)`), which duplicates our explicit `positions_ts ON (ts DESC)`. Both are functionally identical; cost is ~8KB extra per chunk across all chunks.
This is not a directus-side bug. Processor's migration (`processor/src/db/migrations/0001_positions.sql`) creates `positions_ts` explicitly without disabling the auto-index, so stage already has the redundancy. Our migration matches processor's pattern.
**Cleanup path (Phase 3 hardening, not Phase 1):**
```sql
SELECT create_hypertable(
'positions', 'ts',
if_not_exists => TRUE,
chunk_time_interval => INTERVAL '1 day',
create_default_indexes => FALSE -- prevents auto-creation of positions_ts_idx
);
```
Both directus's and processor's migrations would need to apply this together (and a Phase 3 migration to drop the existing `positions_ts_idx`). Defer until Phase 3 since it's storage optimization, not correctness.