Add business-plane schema draft and ingest Rally Albania 2025 regs
Substantial design artifact + canonical-source ingest for the TRM business plane. Schema draft (synthesis): - wiki/synthesis/directus-schema-draft.md — working agreement for the multi-tenant schema. Pseudo multi-tenant under organizations; entries as the unit of timing; course definition (stages/segments/geofences/ waypoints/SLZs); penalty system "numbers in DB, math in code" with an evaluator registry and progressive bracket math; per-entry timing tables; per-stage start-order strategies (manual / previous_stage_clean_result / inverse_top_n_then_natural / inverse_of_overall) covering both Tirana 24h and Rally Albania patterns. Two role surfaces (org role vs racing role) called out explicitly. Decisions captured; Open questions reduced to one (geometry retroactivity engine, deferred to Phase 2.5). Source ingest: - raw/Regulations_2025.pdf + wiki/sources/rally-albania-regulations- 2025.md — formal ingest of the canonical Rally Albania 2025 rulebook. Section numbers preserved as §X.Y so the schema draft and future SPA work can cite precisely. Flagged follow-ups: the SLZ formula lives in the Supplementary Regulations (don't hardcode); M-7 numbering bug; unmodeled neutralization zones. Faulty-position flag (cross-plane operator workflow): - entities/postgres-timescaledb.md, entities/processor.md, concepts/position-record.md — operator-controlled boolean on the positions hypertable; processor filters WHERE faulty = false on every read; flagging triggers windowed recompute via the recompute:requests stream. Implementation strategy on entity pages: - entities/directus.md — Schema management section documenting the snapshots/ + db-init/ convention, container-startup apply pipeline. - entities/processor.md — Phase 2 long-lived branch model with PROCESSOR_PHASE_2_ENABLED flag-gating for incremental main merges; Phase 2.5 deferral note. Index and log updated.
This commit is contained in:
@@ -56,6 +56,17 @@ This means **direct database writes from [[processor]] are not visible** to Dire
|
||||
|
||||
See [[live-channel-architecture]] for the full design, including why this split is preferable to routing telemetry writes through [[directus]]'s API or running a bridging extension inside [[directus]].
|
||||
|
||||
## Schema management — snapshot/apply pipeline
|
||||
|
||||
Schema changes flow through Directus's native snapshot mechanism, kept under git. Two artifact directories:
|
||||
|
||||
- **`snapshots/schema.yaml`** — Directus collections, fields, relations. Generated locally with `directus schema snapshot`. Applied at container startup with `directus schema apply --yes`. Idempotent — applies only the diff against the running DB.
|
||||
- **`db-init/*.sql`** — schema Directus does not manage: the [[postgres-timescaledb]] positions hypertable, the `faulty` column, indexes that need PostGIS-specific syntax, or any DDL that predates Directus knowing about a collection. Numbered (`001_`, `002_`, …) and applied by a sidecar container or one-shot job ahead of `directus schema apply`. Tracked via a `migrations_applied` guard table to skip already-run files.
|
||||
|
||||
Local dev edits the schema in the admin UI, then snapshots before commit. CI builds the image with both directories baked in, spins a throwaway Postgres, and dry-runs `apply` to catch breakage before deploy. Production (Portainer) runs the same apply at container start; multi-env separation is a connection string, not different artifacts.
|
||||
|
||||
This treats `schema.yaml` as the source of truth and the admin UI as its editor. Don't hand-edit `schema.yaml`; round-trip through the UI to keep the format consistent.
|
||||
|
||||
## Phase 2 role
|
||||
|
||||
Directus owns the `commands` collection and is the **single auth surface** for outbound device commands. The SPA inserts command rows; a Directus Flow routes them via Redis to the Ingestion instance holding the device's socket. See [[phase-2-commands]].
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
title: PostgreSQL + TimescaleDB
|
||||
type: entity
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
updated: 2026-05-01
|
||||
sources: [gps-tracking-architecture]
|
||||
tags: [infrastructure, business-plane, database]
|
||||
---
|
||||
@@ -22,6 +22,12 @@ The durable storage layer. PostgreSQL with the TimescaleDB extension. Holds the
|
||||
|
||||
Schema is **defined and migrated through [[directus]]** — see that page for why. The Processor inserts rows respecting that schema; it does not create tables.
|
||||
|
||||
## Positions hypertable
|
||||
|
||||
Stores normalized [[position-record]] rows from [[processor]]. Beyond the wire-shape fields (device_id, timestamp, lat/lon/alt, angle, speed, satellites, priority, attributes), the hypertable carries one storage-only field:
|
||||
|
||||
- **`faulty boolean DEFAULT false`** — set by track operators via [[directus]] when a position is unrealistic (jumpy GPS, impossible speed/coordinate). The [[processor]]'s evaluators (peak-speed, crossing detection, recompute) filter `WHERE faulty = false` on every read of position data. Untouched at write time; mutated only through the operator workflow described in the schema draft.
|
||||
|
||||
## Operational note
|
||||
|
||||
The database is the **only single point of failure** in the architecture. Everything else is restartable, replaceable, or naturally redundant. Operational attention concentrates here:
|
||||
|
||||
@@ -47,6 +47,12 @@ In multi-instance deployments, each Processor reads the [[redis-streams]] stream
|
||||
|
||||
Per-model IO mappings live here, not in the Ingestion layer. Example: `{ "FMB920": { "16": "odometer_km", "240": "movement" } }`. This is the boundary set by the [[teltonika]] adapter — Ingestion produces raw IO maps; the Processor names and interprets them.
|
||||
|
||||
## Faulty position handling
|
||||
|
||||
The positions hypertable in [[postgres-timescaledb]] carries a `faulty boolean DEFAULT false` column that operators can flip through [[directus]] when a position is unrealistic. **All Processor read paths against position data filter `WHERE faulty = false`** — peak-speed evaluation inside SLZs, geofence crossing detection, waypoint pass detection, replay-based recompute. The flag is never set at write time; it's a post-hoc operator action.
|
||||
|
||||
When an operator flips the flag (set or unset), Directus emits a webhook → Redis Stream `recompute:requests`. The Processor consumes the request and re-evaluates `entry_penalties` whose evaluation window overlaps the flagged position's timestamp. Cost sits between formula recompute (cheap) and full geometry replay (expensive) — the affected window is bounded, but the inputs (peak speed, missed-waypoint count) must be re-derived from the now-filtered position stream rather than from snapshotted values.
|
||||
|
||||
## Scaling
|
||||
|
||||
Multiple Processor instances join a Redis Streams consumer group and split the load across device IDs. Consumer-group offsets ensure a crashed instance's work is picked up by the next one.
|
||||
@@ -54,3 +60,14 @@ Multiple Processor instances join a Redis Streams consumer group and split the l
|
||||
## Failure mode
|
||||
|
||||
Crash → consumer-group offsets ensure the next instance picks up where the last left off. In-memory state is rehydrated from the database. See [[failure-domains]].
|
||||
|
||||
## Development workflow — Phase 2 branch model
|
||||
|
||||
Phase 2 (geofence engine, evaluator registry, crossings/penalties/results writers) is a substantial body of work and lives on a long-lived **`phase-2`** branch rather than landing piecemeal on `main`. Conventions:
|
||||
|
||||
- **Rebase weekly** against `main`, not merge. Keeps history readable and avoids merge-commit clutter when the branch eventually lands.
|
||||
- **CI parity** — same workflow on `phase-2` PRs as on `main` PRs. Test coverage doesn't diverge across the branch boundary.
|
||||
- **Flag-gated incremental merges** — chunks that are self-contained (a single evaluator, the geofence detector) can land on `main` behind `PROCESSOR_PHASE_2_ENABLED=false`. Off in prod, on in stage. Lets the work merge before it's user-visible without keeping the entire feature on a side branch indefinitely.
|
||||
- **Single squash merge to retire the branch** — when Phase 2 is feature-complete enough to dogfood end-to-end, one squash merge retires the branch. Avoid death-by-a-thousand-merges.
|
||||
|
||||
Phase 2.5 (the geometry retroactivity engine in [[directus-schema-draft]]) follows the same pattern on its own branch when it starts; it is explicitly deferred until Phase 2 has shipped and the manual operator workflow for geometry edits has surfaced real pain points.
|
||||
|
||||
Reference in New Issue
Block a user