Bootstrap LLM-maintained wiki with TRM architecture knowledge
Initialize CLAUDE.md schema, index, and log; ingest three architecture sources (system overview, Teltonika ingestion design, official Teltonika data-sending protocols) into 7 entity pages, 8 concept pages, and 3 source pages with wikilink cross-references.
This commit is contained in:
@@ -0,0 +1,58 @@
|
||||
---
|
||||
title: Directus
|
||||
type: entity
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture, teltonika-ingestion-architecture]
|
||||
tags: [service, business-plane, api]
|
||||
---
|
||||
|
||||
# Directus
|
||||
|
||||
The **business plane**. Owns the relational schema, exposes it through auto-generated REST/GraphQL APIs, enforces role-based permissions, and provides the admin UI for back-office users.
|
||||
|
||||
## What Directus owns
|
||||
|
||||
- **Schema management** — collections, fields, relations, migrations.
|
||||
- **API generation** — REST and GraphQL endpoints, no boilerplate.
|
||||
- **Authentication and authorization** — users, roles, permissions, JWT issuance.
|
||||
- **Real-time** — WebSocket subscriptions on collections for live UIs.
|
||||
- **Workflow automation** — Flows for orchestrating side effects (notifications, integrations).
|
||||
- **Admin UI** — complete back-office interface for operators.
|
||||
|
||||
## What Directus is NOT
|
||||
|
||||
Not in the telemetry hot path. Does not accept device connections, run a geofence engine, or hold per-device runtime state. Mixing those responsibilities into the same process would couple deployment lifecycles and contaminate failure domains. See [[plane-separation]].
|
||||
|
||||
## Schema ownership vs. write access
|
||||
|
||||
Directus is the schema **owner** even though [[processor]] writes directly to the database. New tables, columns, and relations are defined through Directus. Reasons:
|
||||
|
||||
- Auto-generated admin UI and APIs are derived from the schema Directus knows about. Tables created outside Directus are invisible to it.
|
||||
- Permissions are configured per-collection in Directus.
|
||||
- Audit columns (created_at, updated_at, user_created) follow Directus conventions; bypassing them inconsistently leads to subtle UI bugs.
|
||||
|
||||
This is a normal Directus deployment pattern — it does not require sole write access, only schema authority.
|
||||
|
||||
## Extensions
|
||||
|
||||
Used for things that genuinely belong in the business layer:
|
||||
|
||||
- **Hooks** that react to data changes (e.g. on event-write, trigger a notification Flow).
|
||||
- **Custom endpoints** for permission-gated, audited operations that are not throughput-critical.
|
||||
- **Custom admin UI panels** for back-office workflows (data review, manual overrides, bulk ops).
|
||||
- **Flows** for declarative orchestration.
|
||||
|
||||
**Not** used for long-running listeners, persistent network sockets, or anything in the telemetry hot path.
|
||||
|
||||
## Real-time delivery
|
||||
|
||||
Directus's WebSocket subscriptions push live data to the [[react-spa]]. When [[processor]] writes a row, Directus broadcasts the change to subscribed clients. Sufficient for tens to low hundreds of concurrent subscribers. If fan-out becomes a bottleneck, a dedicated WebSocket gateway can read directly from [[redis-streams]] and push to clients, bypassing Directus for the live channel only — REST/GraphQL stays in Directus.
|
||||
|
||||
## Phase 2 role
|
||||
|
||||
Directus owns the `commands` collection and is the **single auth surface** for outbound device commands. The SPA inserts command rows; a Directus Flow routes them via Redis to the Ingestion instance holding the device's socket. See [[phase-2-commands]].
|
||||
|
||||
## Failure mode
|
||||
|
||||
Crash → telemetry continues to flow into the database; admin UI and SPA are unavailable; no telemetry is lost. See [[failure-domains]].
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
title: PostgreSQL + TimescaleDB
|
||||
type: entity
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture]
|
||||
tags: [infrastructure, business-plane, database]
|
||||
---
|
||||
|
||||
# PostgreSQL + TimescaleDB
|
||||
|
||||
The durable storage layer. PostgreSQL with the TimescaleDB extension. Holds the positions hypertable and all business schema owned by [[directus]].
|
||||
|
||||
## Writers
|
||||
|
||||
- **[[processor]]** — sole writer for high-volume telemetry (positions hypertable) and writer for derived business entities (events, violations, alerts).
|
||||
- **[[directus]]** — writes from the admin UI, custom endpoints, and Flows. Owns schema definition and migrations.
|
||||
|
||||
[[react-spa]] never writes (or reads) directly. [[tcp-ingestion]] does not touch the database.
|
||||
|
||||
## Schema authority
|
||||
|
||||
Schema is **defined and migrated through [[directus]]** — see that page for why. The Processor inserts rows respecting that schema; it does not create tables.
|
||||
|
||||
## Operational note
|
||||
|
||||
The database is the **only single point of failure** in the architecture. Everything else is restartable, replaceable, or naturally redundant. Operational attention concentrates here:
|
||||
|
||||
- Replication
|
||||
- Backups
|
||||
- Point-in-time recovery via TimescaleDB
|
||||
|
||||
## Scaling
|
||||
|
||||
- **Vertical** for write throughput.
|
||||
- **Read replicas** for analytics workloads.
|
||||
|
||||
## Deployment
|
||||
|
||||
Internal-only container. Persistence volume. Regular backups. Accessed only by [[directus]] and [[processor]].
|
||||
@@ -0,0 +1,47 @@
|
||||
---
|
||||
title: Processor
|
||||
type: entity
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture]
|
||||
tags: [service, telemetry-plane, domain-logic]
|
||||
---
|
||||
|
||||
# Processor
|
||||
|
||||
The service where domain logic lives. Consumes normalized telemetry from [[redis-streams]] and is responsible for per-device runtime state, applying domain rules, and writing durable state to [[postgres-timescaledb]].
|
||||
|
||||
## Responsibilities
|
||||
|
||||
- Maintain **per-device runtime state** — last position, derived metrics, current zone, accumulators.
|
||||
- Apply **domain rules** that turn raw telemetry into meaningful events.
|
||||
- Write **durable state** — both raw position history and any derived events.
|
||||
- Emit events for downstream consumers (Directus Flows, notification services, dashboards).
|
||||
|
||||
Where [[tcp-ingestion]] is about throughput and protocol correctness, the Processor is about correctness of meaning. It is the component most likely to evolve as requirements grow, which is why it is isolated from the sockets on one side and the API surface on the other.
|
||||
|
||||
## State management
|
||||
|
||||
- **Static reference data** (spatial assets, configurations) loaded at startup; refreshed on a known cadence or via explicit invalidation.
|
||||
- **Per-device state** held in memory keyed by device identifier (last seen, current segment, accumulators).
|
||||
- **Durable state** written asynchronously to the database.
|
||||
|
||||
The database is the source of truth for replay/analysis; in-memory state is the source of truth for the current decision. On restart, hot state is rehydrated from the DB — this is a recovery path, not a hot path.
|
||||
|
||||
## Database writes
|
||||
|
||||
- The Processor is the **only writer** for high-volume telemetry tables (e.g. the positions hypertable). [[directus]] does not insert positions; it reads them.
|
||||
- For derived business entities (events, violations, alerts), the Processor writes directly to tables [[directus]] also knows about. Schema is owned by Directus; the Processor inserts rows respecting that schema.
|
||||
- This keeps the hot write path off the Directus HTTP stack while still letting Directus expose the data through API and admin UI.
|
||||
|
||||
## IO element interpretation
|
||||
|
||||
Per-model IO mappings live here, not in the Ingestion layer. Example: `{ "FMB920": { "16": "odometer_km", "240": "movement" } }`. This is the boundary set by the [[teltonika]] adapter — Ingestion produces raw IO maps; the Processor names and interprets them.
|
||||
|
||||
## Scaling
|
||||
|
||||
Multiple Processor instances join a Redis Streams consumer group and split the load across device IDs. Consumer-group offsets ensure a crashed instance's work is picked up by the next one.
|
||||
|
||||
## Failure mode
|
||||
|
||||
Crash → consumer-group offsets ensure the next instance picks up where the last left off. In-memory state is rehydrated from the database. See [[failure-domains]].
|
||||
@@ -0,0 +1,57 @@
|
||||
---
|
||||
title: React SPA
|
||||
type: entity
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture]
|
||||
tags: [service, presentation-plane, frontend]
|
||||
---
|
||||
|
||||
# React SPA
|
||||
|
||||
The end-user experience for operators and external participants. Single React application, role-based views, served as a static bundle.
|
||||
|
||||
## Why a separate SPA (not just the Directus admin UI)
|
||||
|
||||
The Directus admin UI is generic CRUD over collections — right for back-office editing and operators who think in records, wrong for end users who think in domain concepts. A dedicated SPA delivers:
|
||||
|
||||
- **Domain-shaped UX** — screens organized around the user's mental model.
|
||||
- **Independent deployment** — front-end ships on its own cadence.
|
||||
- **Targeted access control** — public/partner-facing routes without exposing the admin surface.
|
||||
- **Mobile/offline tuning** — bundles tuned for the actual user environment.
|
||||
|
||||
## Single app, role-based views
|
||||
|
||||
One application serves multiple user types via role-based routing and conditional UI. All users authenticate through [[directus]]; the SPA receives a JWT, reads role, and renders appropriate navigation/screens. Splitting into multiple apps is only justified when user populations are genuinely disjoint (public site vs. authenticated console) or when bundle size for one audience harms another.
|
||||
|
||||
## Data access pattern
|
||||
|
||||
The SPA talks **exclusively** to Directus:
|
||||
|
||||
- REST/GraphQL via `@directus/sdk`.
|
||||
- WebSocket subscriptions via the same SDK.
|
||||
- JWT auth managed by the SDK; refresh handled transparently.
|
||||
|
||||
**Never** talks to the [[processor]], [[tcp-ingestion]], [[redis-streams]], or [[postgres-timescaledb]] directly. This boundary lets the back-end evolve internally and keeps the security model coherent — every request goes through Directus's permission system.
|
||||
|
||||
## Recommended stack
|
||||
|
||||
- **Vite + React + TypeScript**
|
||||
- **TanStack Router** — better TS support than React Router; optional file-based routing
|
||||
- **TanStack Query** — server state, caching, invalidation, optimistic updates
|
||||
- **@directus/sdk** — typed access + real-time
|
||||
- **MapLibre GL + react-map-gl** — open-source WebGL maps, no token needed
|
||||
- **shadcn/ui + Tailwind** — UI primitives
|
||||
- **Zustand** — client-only state (filters, UI prefs)
|
||||
- **react-hook-form + Zod** — forms and validation
|
||||
|
||||
Covers the spectrum from form-heavy admin screens to real-time map dashboards without architectural changes between them.
|
||||
|
||||
## Real-time rendering
|
||||
|
||||
- **Live maps with many markers**: React reconciler is not the bottleneck — drawing happens in WebGL via MapLibre, which manages features outside React's tree. The React layer manages subscriptions and feeds the map updates.
|
||||
- **High-frequency tabular updates** (live leaderboards, event feeds): split components so high-update areas re-render in isolation; use TanStack Query for live data; memoize at component boundaries that receive frequent updates.
|
||||
|
||||
## Failure mode
|
||||
|
||||
UI unavailable → back-end unaffected. See [[failure-domains]].
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
title: Redis Streams
|
||||
type: entity
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture, teltonika-ingestion-architecture]
|
||||
tags: [infrastructure, telemetry-plane, queue]
|
||||
---
|
||||
|
||||
# Redis Streams
|
||||
|
||||
The durable in-flight queue between [[tcp-ingestion]] and [[processor]]. Also the transport for Phase 2 outbound commands.
|
||||
|
||||
## What it provides
|
||||
|
||||
- **Buffering** — temporary slowness in [[processor]] does not push back on Ingestion sockets.
|
||||
- **Replayability** — Streams retain messages, so a Processor crash does not lose telemetry; consumer-group offsets resume from the last position.
|
||||
- **Horizontal scaling** — multiple Processor instances join a consumer group and split load across device IDs.
|
||||
|
||||
## Why Redis (and not Kafka/NATS)
|
||||
|
||||
Sufficient at current scale and adds minimal operational burden. NATS or Kafka are reasonable upgrades when **multi-region durability** or **very high throughput** become real concerns. Until then, Redis is the right choice.
|
||||
|
||||
## Phase 2 usage
|
||||
|
||||
Outbound commands ride on per-instance streams: `commands:outbound:{instance_id}`. Responses ride on `commands:responses`. Redis is the transport; the source of truth for commands is the Directus `commands` collection. See [[phase-2-commands]].
|
||||
|
||||
The connection registry (`connections:registry` hash) and per-instance heartbeats (`instance:heartbeat:{instance_id}` keys with `EX 90`) also live in Redis.
|
||||
|
||||
## Failure mode
|
||||
|
||||
Streams are persisted; restart resumes from disk. Complete Redis loss is recoverable from device retransmits and Processor checkpointing. See [[failure-domains]].
|
||||
|
||||
## Operational note
|
||||
|
||||
**Consumer lag is the canary metric** for the entire telemetry pipeline. Observability dashboards should make it prominent.
|
||||
|
||||
## Deployment
|
||||
|
||||
Internal-only container. Persistence enabled. Never exposed externally.
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
title: TCP Ingestion
|
||||
type: entity
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture, teltonika-ingestion-architecture]
|
||||
tags: [service, telemetry-plane]
|
||||
---
|
||||
|
||||
# TCP Ingestion
|
||||
|
||||
The service that maintains persistent TCP connections with GPS devices, parses vendor binary protocols, ACKs frames per protocol, and hands off normalized records to the [[redis-streams]] queue.
|
||||
|
||||
## Responsibility
|
||||
|
||||
Single concern: **protocol I/O**. Explicitly **not**:
|
||||
|
||||
- Apply business rules
|
||||
- Write to PostgreSQL
|
||||
- Perform geospatial computation
|
||||
- Serve any user-facing API
|
||||
|
||||
The narrow scope is what keeps the process fast, predictable, and safely restartable.
|
||||
|
||||
## Connection model
|
||||
|
||||
- Built around `net.createServer()` (Node.js) — each socket is an independent session.
|
||||
- Per-connection state is small: identifier (e.g. IMEI), parser instance, partial-frame buffer.
|
||||
- Devices reconnect automatically on network failure → connection loss is routine → service is trivially restartable.
|
||||
|
||||
## Vendor abstraction
|
||||
|
||||
Each device vendor (Teltonika, Queclink, Concox, etc.) ships its own binary protocol. Vendor-specific code is isolated behind a [[protocol-adapter]] interface:
|
||||
|
||||
- **Input**: byte stream from a TCP socket
|
||||
- **Output**: normalized [[position-record]] (`device_id`, `timestamp`, `lat`, `lon`, `speed`, `heading`, plus a free-form `attributes` bag)
|
||||
|
||||
Adding a new vendor = writing a new adapter. Nothing downstream changes.
|
||||
|
||||
## Handoff discipline
|
||||
|
||||
For every parsed frame:
|
||||
|
||||
1. Send protocol-required ACK to the device.
|
||||
2. Push normalized record to a Redis Stream.
|
||||
3. Return to reading the socket.
|
||||
|
||||
**The TCP handler never blocks on downstream work.** Backpressure is absorbed by the Stream; Ingestion keeps accepting and acknowledging. This is the discipline that keeps the system alive under load.
|
||||
|
||||
## Project layout
|
||||
|
||||
Lives at `tcp-ingestion/` — single Node.js/TypeScript project. Layout:
|
||||
|
||||
```
|
||||
tcp-ingestion/
|
||||
├── src/core/ # vendor-agnostic shell (no adapter imports)
|
||||
├── src/adapters/ # per-vendor adapters
|
||||
│ └── teltonika/ # see [[teltonika]]
|
||||
├── src/config/
|
||||
├── src/observability/
|
||||
└── test/fixtures/ # real packet captures per codec
|
||||
```
|
||||
|
||||
Three layout rules: `core/` never imports `adapters/`; adapters never import each other; each adapter folder is self-contained so it can be lifted into its own service later via `git mv`.
|
||||
|
||||
## Scaling shape
|
||||
|
||||
- Single Node.js process handles thousands of concurrent connections at typical telemetry rates.
|
||||
- Horizontal scaling: multiple instances behind a TCP-aware load balancer (HAProxy, NGINX stream module).
|
||||
- TCP guarantees session stickiness for the duration of the connection.
|
||||
- No shared state between instances required — per-device state lives entirely on the open socket.
|
||||
|
||||
The pattern ports cleanly to higher-throughput runtimes (Go, Elixir) if a future rewrite is warranted.
|
||||
|
||||
## Failure mode
|
||||
|
||||
Crash → devices reconnect → in-flight frames are retransmitted by the device per protocol → no data is lost beyond what was unacknowledged. See [[failure-domains]].
|
||||
|
||||
## Phase 2 addition
|
||||
|
||||
Each Ingestion instance will run a parallel **command consumer** reading from `commands:outbound:{instance_id}` and writing command frames to device sockets. The TCP read path is not blocked. See [[phase-2-commands]].
|
||||
@@ -0,0 +1,86 @@
|
||||
---
|
||||
title: Teltonika
|
||||
type: entity
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [teltonika-ingestion-architecture, teltonika-data-sending-protocols]
|
||||
tags: [vendor, gps-hardware, protocol]
|
||||
---
|
||||
|
||||
# Teltonika
|
||||
|
||||
Lithuanian GPS hardware vendor. First (and currently only) device family supported by [[tcp-ingestion]]. Device families seen include FMB / FMC / FMM / FMU.
|
||||
|
||||
## Protocol overview
|
||||
|
||||
Binary frames, self-describing via codec ID byte. Both **TCP and UDP transports** are supported by the protocol; our [[tcp-ingestion]] implementation is TCP-only by deliberate choice (connection-oriented reliability + persistent session for [[phase-2-commands|outbound commands]]).
|
||||
|
||||
Codec inventory (canonical, from the official Teltonika wiki):
|
||||
|
||||
| Codec | Hex ID | Direction | Purpose |
|
||||
|-------|--------|-----------|---------|
|
||||
| 8 | `0x08` | device → server | AVL telemetry; 1-byte IO IDs |
|
||||
| 8 Extended | `0x8E` | device → server | AVL telemetry; 2-byte IO IDs + variable-length IO (NX section) |
|
||||
| 16 | `0x10` | device → server | AVL telemetry; 2-byte IO IDs + Generation Type; AVL IDs > 255 |
|
||||
| 12 | `0x0C` | bidirectional | Server commands + device responses (ASCII text) |
|
||||
| 13 | `0x0D` | device → server | One-way command upload with timestamp |
|
||||
| 14 | `0x0E` | bidirectional | Like 12, IMEI-addressed; ACK `0x06` / nACK `0x11` |
|
||||
| 15 | `0x0F` | device → server | One-way; FMX6 professional devices via RS232 — out of scope |
|
||||
| 4 | `0x04` | SMS | 24-position daily SMS, bit-field compressed — out of scope |
|
||||
|
||||
See [[avl-data-format]] for the byte-level packet structure across the three telemetry codecs.
|
||||
|
||||
### Notes on individual codecs
|
||||
|
||||
- **Codec 8** — baseline AVL format. 8-bit IO element IDs. Older firmware.
|
||||
- **Codec 8 Extended** — 16-bit IO IDs and a **variable-length IO section (NX)** carrying length-prefixed values. Default on modern FMB/FMC/FMM/FMU devices.
|
||||
- **Codec 16** — adds a 1-byte **Generation Type** field per record (values 0–7: On Exit, On Entrance, On Both, Reserved, Hysteresis, On Change, Eventual, Periodical). Required for AVL IDs > 255 on FMB630/FM63XY firmware ≥ 00.03.xx.
|
||||
- **Codec 13** — Device→Server only. Used only when "Message Timestamp" is enabled in RS232 settings.
|
||||
- **Codec 14** — Device returns nACK (Type `0x11`) when the command's IMEI doesn't match. Available from FMB.Ver.03.25.04.Rev.00.
|
||||
- **Codec 15** — FMX6 professional devices only, via RS232 modes (TCP/UDP Ascii/Binary, ±Buffered). Includes both timestamp (4B, **seconds**) and IMEI (8B HEX). Not applicable to the deployed FMB/FMC/FMM/FMU fleet.
|
||||
|
||||
## Frame envelope (data codecs)
|
||||
|
||||
```
|
||||
┌──────────────┬──────────────┬───────────┬─────────────┬─────┐
|
||||
│ Preamble │ Data length │ Codec ID │ AVL data │ CRC │
|
||||
│ 4 bytes (0) │ 4 bytes │ 1 byte │ N bytes │ 4 B │
|
||||
└──────────────┴──────────────┴───────────┴─────────────┴─────┘
|
||||
```
|
||||
|
||||
CRC is CRC-16/IBM (lower 2 bytes carry the value; upper 2 are zero).
|
||||
|
||||
## Self-describing → model-agnostic parser
|
||||
|
||||
The codec ID announces framing; fixed AVL fields (timestamp, lat/lon/alt, angle, satellites, speed) are codec-defined and identical across models for a given codec; only the **IO element bag** varies between models. The bag is `{ id → value }` pairs the parser reads byte-correctly without knowing what each `id` means.
|
||||
|
||||
A brand-new Teltonika model will:
|
||||
|
||||
- Parse correctly at frame and codec layers (envelope, CRC, codec dispatch unchanged).
|
||||
- Produce a correct [[position-record]] (codec-defined fields).
|
||||
- Carry IO elements through as raw integer keys until a model-aware mapping is added in the [[processor]].
|
||||
|
||||
This is the load-bearing property of the [[protocol-adapter]] design.
|
||||
|
||||
## IMEI handshake
|
||||
|
||||
Device connects → sends 2-byte length + ASCII IMEI → server responds `0x01` to accept (or `0x00` to reject). Phase 1 always accepts.
|
||||
|
||||
## ACK protocol
|
||||
|
||||
After parsing a data frame, server sends 4-byte big-endian record count. Devices retransmit unacknowledged frames on the next session — this is the protocol-correct way to handle CRC failures (just don't ACK).
|
||||
|
||||
## Packet size limits
|
||||
|
||||
- Minimum AVL record: **45 bytes** (all IO elements disabled).
|
||||
- Maximum AVL record: **255 bytes**.
|
||||
- Maximum AVL packet: **512 bytes** for FMB640/FMB641/FMC640/FMM640; **1280 bytes** for other devices.
|
||||
|
||||
## Phase 1 vs Phase 2
|
||||
|
||||
| Phase | Codecs | Purpose | Status |
|
||||
|-------|--------|---------|--------|
|
||||
| Phase 1 (now) | 8, 8E, 16 | Device → server telemetry | In scope |
|
||||
| Phase 2 (later) | 12, 13, 14 | Server → device commands | Reserved |
|
||||
|
||||
Phase 1 covers essentially the entire deployed Teltonika telemetry fleet. Codec 15 and SMS-based protocols (Codec 4, binary SMS) are out of scope for now. See [[phase-2-commands]] for the deferred command-codec design.
|
||||
Reference in New Issue
Block a user