trm/directus
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
5 Commits 2 Branches 0 Tags
e22d9d489a02218443f314fa198e65d954f0853b
Commit Graph

5 Commits

Author SHA1 Message Date
julian e22d9d489a Tasks 1.6 + 1.7 — schema tooling + real entrypoint flow 
Two parallel tasks landing together. The boot pipeline is now wired
end-to-end: db-init → schema apply → directus bootstrap → pm2-runtime.
Live-verified by booting a fresh compose stack to a serving Directus
admin UI on :8055.

Task 1.6 — snapshot tooling:
- scripts/schema-snapshot.sh — host-side, dev-time. Verifies docker
  is on PATH and the directus compose service is running, runs
  `node /directus/cli.js schema snapshot --yes` inside the container,
  copies the YAML out to ./snapshots/schema.yaml. Used after admin-UI
  schema changes to capture the new state for git commit.
- scripts/schema-apply.sh — image-side, boot-time. Reads
  /directus/snapshots/schema.yaml, runs a dry-run preview, then
  applies. Gracefully skips when the snapshot is absent or whitespace-
  only (Phase 1 first-boot path before tasks 1.4/1.5 produce
  collections). SNAPSHOT_PATH env var override for CI flexibility.
- snapshots/README.md — lifecycle doc; warns against hand-editing.

Task 1.7 — real entrypoint flow:
- entrypoint.sh rewritten from Phase 1.1's placeholder to the
  4-step boot per ROADMAP design rule #3:
    1/4 db-init          → /directus/scripts/apply-db-init.sh
    2/4 schema apply     → /directus/scripts/schema-apply.sh
    3/4 directus bootstrap → node /directus/cli.js bootstrap
    4/4 directus start   → exec pm2-runtime start ecosystem.config.cjs
  set -euo pipefail halts boot on any step's non-zero exit. Each step
  emits a [entrypoint] log marker so an operator reading container
  logs sees which step failed.

Bug found and fixed during live verification:
- Both 1.6 scripts initially called bare `directus schema ...` as if
  the CLI were on PATH. Upstream directus/directus:11.17.4 does NOT
  expose `directus` on PATH — invocation is via `node /directus/cli.js`,
  same pattern as the entrypoint's bootstrap step. Both scripts
  corrected. Also added -T to docker compose exec in schema-snapshot.sh
  so the script works in non-TTY contexts (CI).

Phase 5 follow-up (non-blocking) flagged in 07's Done section: Directus
warns "Collection 'positions' doesn't have a primary key column and
will be ignored". The positions table uses UNIQUE INDEX (device_id, ts)
matching processor's pattern, not a PK constraint. Means positions is
not auto-registered as a Directus collection — fine for Phase 1, but
the operator faulty-flag workflow will need a custom endpoint or
manual collection registration in Phase 5.

ROADMAP marks 1.6 + 1.7 done. Phase 1 progress: 5/9 tasks complete
(1.1, 1.2, 1.3, 1.6, 1.7); 1.4, 1.5, 1.8, 1.9 remain.
 2026-05-02 09:40:53 +02:00
julian 25a9731070 Task 1.3 — Initial migrations 
Three SQL files under db-init/ create the schema processor writes
against. All three apply cleanly via apply-db-init.sh, are idempotent
on re-run, and end with assertion blocks that catch silent
schema drift.

001_extensions.sql — registers timescaledb on the directus database.
  PostGIS deferred to Phase 2 (per Plan A). The timescaledb-ha image
  pre-creates the extension at DB init, so the IF NOT EXISTS guard
  fires as a NOTICE — expected and harmless.

002_positions_hypertable.sql — positions hypertable, exact
  column-by-column match against processor/src/db/migrations/0001_positions.sql.

  Cross-checking against processor surfaced 8 divergences from the
  original task spec; processor wins in every case (it is the writer
  and is in production). The corrections:

    - added ingested_at timestamptz NOT NULL DEFAULT now()
    - added codec text NOT NULL
    - altitude/angle/speed: real NOT NULL (not DOUBLE PRECISION nullable)
    - satellites/priority: NOT NULL
    - removed attributes DEFAULT '{}'::jsonb (processor always writes)
    - replaced PRIMARY KEY with UNIQUE INDEX positions_device_ts
      (idiomatic for TimescaleDB hypertables)
    - chunk interval 1 day, not 7 days
    - two indexes (positions_device_ts + positions_ts), not one composite

  Without these corrections every processor INSERT would have failed
  with NOT NULL violations. Spec deliverables section updated to
  reflect the correct shape so future readers see the right schema.

003_faulty_column.sql — adds the operator-controlled faulty boolean
  flag plus the partial index positions_faulty_idx ON (device_id,
  ts DESC) WHERE faulty = FALSE. The column is set only via Directus
  admin (Phase 4 permissions); processor's writer never touches it.
  The partial index optimises the hot-path read pattern (every
  processor evaluator filters faulty = FALSE); operator queries that
  look at faulty rows specifically use the broader positions_device_ts
  index from 002.

Live-verified 2026-05-01:
  - First apply: 3 applied, 0 skipped, exit 0.
  - Re-run: 0 applied, 3 skipped, exit 0.
  - All 13 columns present with correct types/nullability/defaults.
  - Hypertable registered with 1-day chunk interval.
  - Three expected indexes present.

Non-blocking observation: TimescaleDB's create_hypertable()
auto-created a fourth index (positions_ts_idx) duplicating our
explicit positions_ts. Processor's migration has the same redundancy
so stage already lives with this. Cleanup path documented in the
task spec for Phase 3 hardening (create_default_indexes => FALSE
in the create_hypertable call).

ROADMAP marks 1.3 done; 1.4 next.
 2026-05-01 22:52:06 +02:00
julian dec2d190ce Task 1.2 — db-init runner script 
scripts/apply-db-init.sh implements the boot-time runner that walks
db-init/*.sql in numeric-prefix order, applies each via psql, and
records successful applications in a migrations_applied guard table
so re-runs are no-ops.

All 7 acceptance criteria pass live against the dev compose stack:
empty dir, missing env var, apply, idempotent re-run, checksum
mismatch, filename collision, broken SQL.

Two retroactive Dockerfile corrections folded in (exposed by the
first live-test attempt of 1.2's script):

1. apk add bash. The directus/directus:11.17.4 base is Alpine and
   ships ash via BusyBox, not bash. The script uses bash-specific
   features (associative arrays, [[ ]], mapfile, BASH_REMATCH) and
   fails at line 69 in sh.

2. .gitattributes added at repo root forcing LF on *.sh, *.sql,
   *.yaml, *.yml. Without it, Windows checkouts with core.autocrlf=true
   (the Git-for-Windows default) silently inject CRLF, causing
   "bad interpreter: /usr/bin/env bash^M" inside the Linux container.
   This failure mode only manifests in the container.

Both corrections are documented in 01-project-scaffold.md's Done
section; 02-db-init-runner.md's Done section captures the live-test
results, the corrected docker compose run --entrypoint commands, and
the gotcha about compose env defaults masking missing-env-var tests.

ROADMAP marks 1.2 done; 1.3 next.
 2026-05-01 22:35:17 +02:00
julian 387c3c4cfa Task 1.1 — Project scaffold 
Phase 1 task 1.1 lands. Directus 11.17.4 boots locally end-to-end
against a TimescaleDB+PostGIS container; admin UI serves at :8055,
admin bootstrap from env vars works, named volumes preserve data
across down/up cycles.

Scaffold:
- Dockerfile — FROM directus/directus:11.17.4. Pre-installs
  postgresql16-client (ahead of task 1.2's db-init runner needing psql).
  Bakes in /directus/snapshots, /directus/db-init, /directus/scripts,
  /directus/extensions, /directus/entrypoint.sh.
- compose.dev.yaml — db (timescale/timescaledb-ha:pg16.6-ts2.17.2-all)
  + directus (local build), healthchecks, named volumes
  directus-pg-data + directus-uploads.
- entrypoint.sh — placeholder using upstream's actual flow
  (node cli.js bootstrap && pm2-runtime start ecosystem.config.cjs);
  the real db-init -> schema apply -> start wrapper lands in task 1.7.
- package.json — scripts-only (dev, dev:down, dev:reset,
  schema:snapshot, schema:apply, db:init), no runtime deps.
- .env.example — sectioned, fully documented, KEY/SECRET marked
  required with generation hints.
- .gitignore, .dockerignore — match the processor service conventions.
- snapshots/, db-init/, scripts/, extensions/ — empty with .gitkeep,
  filled by later Phase 1 tasks (1.3, 1.6) and Phase 5.

Lessons locked in (against the empirical pnpm dev boot):
- timescale/timescaledb-ha:pg16-latest does NOT exist on Docker Hub.
  Pin a concrete version (we used pg16.6-ts2.17.2-all).
- This image's data directory is /home/postgres/pgdata/data, not
  /pgdata or /var/lib/postgresql/data. PGDATA env var and the volume
  mount must both target it.
- The -all variant bundles PostGIS binaries but the extension is not
  auto-created on the directus database; CREATE EXTENSION lands in
  Phase 2 alongside the geofences/SLZs/waypoints collections.
- The upstream image's CMD is bootstrap + pm2-runtime, not a simple
  cli.js start. Bypassing pm2 would lose crash recovery.

These corrections folded into 01-project-scaffold.md (deliverable line
+ Done section), 08-gitea-ci-dryrun.md (CI service tag), and the
inline comments in compose.dev.yaml so future implementers don't
re-discover them.

Status: ROADMAP marks 1.1 done, Phase 1 in progress, 1.2 next.
 2026-05-01 21:29:13 +02:00
julian a8e808e71c Scaffold directus service planning structure 
Initial commit. Establishes the .planning/ tree mirroring processor's
shape (ROADMAP.md as nav hub + per-phase folders with READMEs and
granular task files).

Six phases:

1. Slice 1 schema + deploy pipeline — what Rally Albania 2026 needs.
   Org catalog (orgs, users, vehicles, devices) + event participation
   (events, classes, entries, entry_crew, entry_devices). db-init/
   for the positions hypertable + faulty column. snapshot/apply
   tooling. Gitea CI dry-run. Dogfood seed of Rally Albania 2026.
   Nine task files with full Goal / Deliverables / Specification /
   Acceptance criteria / Risks / Done sections.

2. Course definition — stages, segments, geofences, waypoints, SLZs.
   PostGIS extension introduced here.

3. Timing & penalty tables — co-developed with processor Phase 2.
   entry_segment_starts, entry_crossings, entry_penalties,
   stage_results, penalty_formulas.

4. Permissions & policies — Directus 11 dynamic-filter Policies per
   logical role. Deployment-time work, deferred to keep early phases
   focused on the data model.

5. Custom extensions — TypeScript hooks/endpoints implementing the
   cross-plane workflows the schema implies (faulty-flag → Redis
   stream emit, stage-open materializer, etc.).

6. Future / optional — retroactivity preview UI, command-routing
   Flows, audit trails, federation rule import. Not committed.

Non-negotiable design rules captured in ROADMAP.md: schema authority
in Directus + snapshot-as-code + db-init for non-Directus DDL +
sequential idempotent migrations + entrypoint apply order + no
application logic in Flows + permissions deferred to Phase 4.

Architectural anchors point at the wiki at ../docs/wiki/ — the schema
draft, the Rally Albania 2025 source page, plus the existing
processor/postgres-timescaledb/live-channel pages. Each task file
calls out the wiki refs an implementing agent should read first.

README.md mirrors the processor service README structure: quick start,
local Docker test, prod/stage deployment notes, env vars, CI behavior.
 2026-05-01 20:42:44 +02:00