Bootstrap LLM-maintained wiki with TRM architecture knowledge
Initialize CLAUDE.md schema, index, and log; ingest three architecture sources (system overview, Teltonika ingestion design, official Teltonika data-sending protocols) into 7 entity pages, 8 concept pages, and 3 source pages with wikilink cross-references.
This commit is contained in:
@@ -0,0 +1,138 @@
|
||||
---
|
||||
title: AVL Data Format (Teltonika canonical)
|
||||
type: concept
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [teltonika-data-sending-protocols, teltonika-ingestion-architecture]
|
||||
tags: [teltonika, protocol, parser, data-model]
|
||||
---
|
||||
|
||||
# AVL Data Format
|
||||
|
||||
The canonical Teltonika AVL packet structure across codecs 8, 8E, and 16. This is the byte-level reference the [[teltonika]] adapter implements. See [[teltonika-data-sending-protocols]] for the official source and [[codec-dispatch]] for how the codec ID byte selects between the three formats.
|
||||
|
||||
## Outer envelope (TCP)
|
||||
|
||||
```
|
||||
┌────────────────┬──────────────────┬──────────┬────┬──────────┬────┬───────┐
|
||||
│ Preamble │ Data Field Length│ Codec ID │ N1 │ AVL Data │ N2 │ CRC-16│
|
||||
│ 4B = 0x00000000│ 4B │ 1B │ 1B │ X bytes │ 1B │ 4B │
|
||||
└────────────────┴──────────────────┴──────────┴────┴──────────┴────┴───────┘
|
||||
```
|
||||
|
||||
- **Data Field Length** is computed from the start of `Codec ID` through `N2` (not the whole packet).
|
||||
- **CRC-16/IBM** is computed over the same range. Lower 2 bytes of the 4-byte CRC field carry the value; upper 2 bytes are zero.
|
||||
- **N1 must equal N2** — record count repeated for integrity.
|
||||
- Server ACK is a **4-byte big-endian integer** equal to the number of records accepted. Mismatched count → device retransmits.
|
||||
|
||||
## AVL record (one of N records)
|
||||
|
||||
```
|
||||
[Timestamp 8B][Priority 1B][GPS Element 15B][IO Element X B]
|
||||
```
|
||||
|
||||
- **Timestamp**: UNIX milliseconds since epoch (UTC), big-endian.
|
||||
- **Priority**: enum — `0` Low, `1` High, `2` Panic.
|
||||
|
||||
### GPS Element (15B)
|
||||
|
||||
```
|
||||
[Longitude 4B][Latitude 4B][Altitude 2B][Angle 2B][Satellites 1B][Speed 2B]
|
||||
```
|
||||
|
||||
- **Lat/Lon**: signed 32-bit integer = `((d + m/60 + s/3600 + ms/3600000) × 10⁷)`, where d/m/s/ms are degrees/minutes/seconds/milliseconds. **Two's complement** — first bit = sign (0 positive, 1 negative).
|
||||
- **Altitude**: meters above sea level, signed 16-bit.
|
||||
- **Angle**: heading 0–360°, unsigned 16-bit.
|
||||
- **Satellites**: count of satellites in use, unsigned 8-bit.
|
||||
- **Speed**: km/h, unsigned 16-bit. **`0x0000` when GPS data is invalid** — distinct from "stationary."
|
||||
|
||||
## IO Element — codec-specific
|
||||
|
||||
The IO Element layout is where the three codecs diverge. The parser must dispatch on codec ID to read these correctly.
|
||||
|
||||
### Codec 8 (`0x08`) — 1-byte everything
|
||||
|
||||
```
|
||||
[Event IO ID 1B]
|
||||
[N total 1B]
|
||||
[N1 1B] [IO ID 1B][Value 1B] × N1
|
||||
[N2 1B] [IO ID 1B][Value 2B] × N2
|
||||
[N4 1B] [IO ID 1B][Value 4B] × N4
|
||||
[N8 1B] [IO ID 1B][Value 8B] × N8
|
||||
```
|
||||
|
||||
### Codec 8 Extended (`0x8E`) — 2-byte fields + variable-length
|
||||
|
||||
```
|
||||
[Event IO ID 2B]
|
||||
[N total 2B]
|
||||
[N1 2B] [IO ID 2B][Value 1B] × N1
|
||||
[N2 2B] [IO ID 2B][Value 2B] × N2
|
||||
[N4 2B] [IO ID 2B][Value 4B] × N4
|
||||
[N8 2B] [IO ID 2B][Value 8B] × N8
|
||||
[NX 2B] [IO ID 2B][Length 2B][Value <Length>B] × NX ← variable-length section
|
||||
```
|
||||
|
||||
The **NX section** is unique to 8E. Carries arbitrary-length values (e.g. ICCID-class data, BLE payloads). Each entry self-describes its length. The parser must be length-aware here — getting it wrong silently corrupts subsequent records.
|
||||
|
||||
### Codec 16 (`0x10`) — Generation Type, mixed widths
|
||||
|
||||
```
|
||||
[Event IO ID 2B]
|
||||
[Generation Type 1B] ← unique to Codec 16
|
||||
[N total 1B]
|
||||
[N1 1B] [IO ID 2B][Value 1B] × N1
|
||||
[N2 1B] [IO ID 2B][Value 2B] × N2
|
||||
[N4 1B] [IO ID 2B][Value 4B] × N4
|
||||
[N8 1B] [IO ID 2B][Value 8B] × N8
|
||||
```
|
||||
|
||||
- **No NX section** — Codec 16 does not include variable-size IO elements.
|
||||
- **Generation Type** values: `0`=On Exit, `1`=On Entrance, `2`=On Both, `3`=Reserved, `4`=Hysteresis, `5`=On Change, `6`=Eventual, `7`=Periodical.
|
||||
- Codec 16 is the channel for AVL IDs > 255 on FMB630/FM63XY.
|
||||
|
||||
### Side-by-side
|
||||
|
||||
| | Codec 8 | Codec 8 Extended | Codec 16 |
|
||||
|---|---------|------------------|----------|
|
||||
| Codec ID | 0x08 | 0x8E | 0x10 |
|
||||
| Event IO ID width | 1B | 2B | 2B |
|
||||
| N total / Nk count widths | 1B / 1B | 2B / 2B | 1B / 1B |
|
||||
| IO ID width | 1B | 2B | 2B |
|
||||
| Generation Type | — | — | 1B |
|
||||
| Variable-length IO (NX) | — | yes | — |
|
||||
| AVL IDs > 255 supported | no | yes | yes |
|
||||
|
||||
## Size limits
|
||||
|
||||
- **Minimum AVL record**: 45 bytes (all IO elements disabled).
|
||||
- **Maximum AVL record**: 255 bytes.
|
||||
- **Maximum AVL packet**: 512 bytes for FMB640/FMB641/FMC640/FMM640; **1280 bytes for other devices**.
|
||||
|
||||
The parser should treat oversized packets as malformed (drop the connection per the [[codec-dispatch]] unknown-codec policy — though this is a different malformation, the same loud-failure principle applies).
|
||||
|
||||
## UDP envelope
|
||||
|
||||
When the device runs over UDP instead of TCP:
|
||||
|
||||
```
|
||||
[UDP Channel Header 5B] [AVL Packet Header (1+2+15)B] [AVL Data Array — same as TCP]
|
||||
```
|
||||
|
||||
- **UDP Channel Header**: `Length 2B + Packet ID 2B + Not Usable Byte 1B`.
|
||||
- **AVL Packet Header**: `AVL Packet ID 1B + IMEI Length 2B = 0x000F + IMEI 15B`.
|
||||
- Server ACK over UDP is short: `[UDP Channel Header 5B][AVL Packet ID 1B][Number Accepted 1B]`.
|
||||
|
||||
The AVL Data Array (Codec ID byte + records + N2) is identical to TCP. **The Phase 1 implementation is TCP-only; UDP is documented here for completeness and future evaluation.**
|
||||
|
||||
## Mapping to [[position-record]]
|
||||
|
||||
The parser writes:
|
||||
|
||||
- `device_id` ← IMEI from the handshake (or from the UDP AVL Packet Header).
|
||||
- `timestamp` ← AVL record `Timestamp` (UNIX ms → `Date`).
|
||||
- `latitude`, `longitude`, `altitude`, `angle`, `speed`, `satellites` ← GPS Element fields, with two's-complement decoding for lat/lon and `Speed === 0x0000 ⇒ "GPS invalid"` retained as-is (the Processor decides how to surface it).
|
||||
- `priority` ← AVL record `Priority` (0/1/2).
|
||||
- `attributes` ← every IO element from N1/N2/N4/N8 (and NX for Codec 8E), keyed by **numeric IO ID as string**, value as `number | bigint | Buffer` per IO width. See [[io-element-bag]] for why naming/units stay out of the parser.
|
||||
|
||||
> Note: Codec 16's `Generation Type` and 8E's NX `Length` are not currently in the [[position-record]] shape. Generation Type is codec-defined (not model-defined) and may deserve a typed field — flagged as an open question on the source page.
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
title: Codec Dispatch (Registry)
|
||||
type: concept
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [teltonika-ingestion-architecture, teltonika-data-sending-protocols]
|
||||
tags: [teltonika, parser, design-seam]
|
||||
---
|
||||
|
||||
# Codec Dispatch
|
||||
|
||||
How the [[teltonika]] adapter routes incoming frames to the right parser, and the seam that makes [[phase-2-commands]] additive instead of a rewrite.
|
||||
|
||||
## The mechanism
|
||||
|
||||
The codec ID (byte 0 of the AVL data payload) indexes into a **flat registry** of handlers. Each handler implements:
|
||||
|
||||
```ts
|
||||
interface CodecHandler {
|
||||
codec_id: number
|
||||
handle(
|
||||
payload: Buffer,
|
||||
ctx: { imei: string; publish: (p: Position) => Promise<void> }
|
||||
): Promise<{ ack_count: number }>
|
||||
}
|
||||
```
|
||||
|
||||
Phase 1 registers handlers for IDs `0x08`, `0x8E`, and `0x10`. Phase 2 will register additional handlers for `0x0C`, `0x0D`, `0x0E` — these will use a different `ctx` shape (they need to write bytes back, not just `publish`), but the registry shape is identical.
|
||||
|
||||
Per-codec directionality matters for the Phase 2 handler shape:
|
||||
|
||||
| Codec | ID | Direction | Handler needs |
|
||||
|-------|-----|-----------|---------------|
|
||||
| 8 / 8E / 16 | `0x08` / `0x8E` / `0x10` | device → server | `publish(Position)` only |
|
||||
| 12 | `0x0C` | bidirectional | `respond(bytes)` for outbound + parse responses |
|
||||
| 13 | `0x0D` | device → server only | parse-only (no `respond`) |
|
||||
| 14 | `0x0E` | bidirectional | `respond(bytes)` + ACK type `0x06` / nACK type `0x11` |
|
||||
| 15 | `0x0F` | device → server only | out of scope (FMX6 RS232 only) |
|
||||
|
||||
Codec 13 and 15, despite living in the GPRS-message family, are *one-way* — handlers for them never write to the socket. This matters for handler-shape design: the Phase 2 outbound family is not "all command codecs," it's specifically `0x0C` and `0x0E`.
|
||||
|
||||
## Why a registry, not a switch
|
||||
|
||||
- Adding a new codec is a registration, not a code change in dispatch logic.
|
||||
- Phase 2 command codecs slot in alongside Phase 1 data codecs without modifying Phase 1 paths.
|
||||
- Codec parsers are **independent** — there is no shared base class. Their record shapes diverge in ways abstraction would obscure rather than help.
|
||||
|
||||
## The "session owns the socket; handler borrows it" rule
|
||||
|
||||
Phase 1 handlers receive payload + a `publish(Position)` callback and emit records via Redis. They never write to the socket directly — the session loop handles ACKs.
|
||||
|
||||
Phase 2 command handlers will need to **write to the socket** (to send commands). They borrow write access through a `respond(bytes: Buffer)` callback added to `ctx` for command codecs. The session retains socket ownership; handlers borrow write access through a narrow interface.
|
||||
|
||||
## Unknown codec policy
|
||||
|
||||
When the codec ID does not match a registered handler:
|
||||
|
||||
- `WARN` log entry with IMEI, offending codec ID, raw header bytes.
|
||||
- Socket is destroyed.
|
||||
- No ACK sent.
|
||||
- No attempt to "skip ahead" or guess record layout.
|
||||
|
||||
Reasoning: a Teltonika device sending an unrecognized codec is **misconfigured**, not subtly broken. Silently truncating its data — or worse, mis-parsing — produces records with plausible-looking but wrong coordinates. Loud failure beats quiet corruption.
|
||||
|
||||
The `teltonika_unknown_codec_total{codec_id}` counter is the canary for codec coverage drift.
|
||||
|
||||
## CRC failure policy
|
||||
|
||||
Different from unknown-codec. CRC mismatch = transient transmission issue:
|
||||
|
||||
- Frame is **not** ACK'd → device retransmits on next session.
|
||||
- `WARN` log with IMEI, expected CRC, computed CRC, frame length.
|
||||
- Connection stays open.
|
||||
|
||||
Repeated CRC failures from the same device in a short window indicate a deeper problem (firmware, line quality) — surface via metrics, not just logs.
|
||||
@@ -0,0 +1,38 @@
|
||||
---
|
||||
title: Failure Domains
|
||||
type: concept
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture]
|
||||
tags: [architecture, reliability]
|
||||
---
|
||||
|
||||
# Failure Domains
|
||||
|
||||
Each component of the platform fails independently. The architecture deliberately concentrates operational risk in one place — the database — and keeps everything else restartable, replaceable, or naturally redundant.
|
||||
|
||||
## Per-component failure behavior
|
||||
|
||||
| Component | Crash behavior | Data loss |
|
||||
|-----------|---------------|-----------|
|
||||
| [[tcp-ingestion]] | Devices reconnect; in-flight frames retransmitted by the device per protocol | None beyond unacknowledged frames |
|
||||
| [[redis-streams]] | Streams are persisted; restart resumes from disk | Recoverable from device retransmits + Processor checkpointing |
|
||||
| [[processor]] | Consumer-group offsets ensure next instance picks up; in-memory state rehydrated from DB | None |
|
||||
| [[directus]] | Telemetry continues to flow into DB; admin UI/SPA unavailable | None |
|
||||
| [[postgres-timescaledb]] | System stops accepting writes | The single point of failure |
|
||||
| [[react-spa]] | UI unavailable | N/A — no state owned |
|
||||
|
||||
## The discipline behind this
|
||||
|
||||
- **No component reaches across two plane boundaries** — see [[plane-separation]]. A failure in one plane cannot cascade through another.
|
||||
- **The TCP handler never blocks on downstream work.** Slow Processor or DB pressure is absorbed by [[redis-streams]], not by device sockets.
|
||||
- **Per-device session state lives only on the open socket** — Ingestion is trivially restartable.
|
||||
- **The Processor's hot state can always be rehydrated** from the DB.
|
||||
|
||||
## Operational consequence
|
||||
|
||||
The database gets careful operational attention — replication, backups, point-in-time recovery via TimescaleDB. Everything else can be restarted, redeployed, or scaled without ceremony.
|
||||
|
||||
## Canary metric
|
||||
|
||||
**Redis Streams consumer lag.** It reflects the health of the entire telemetry pipeline in one number.
|
||||
@@ -0,0 +1,35 @@
|
||||
---
|
||||
title: IO element bag
|
||||
type: concept
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [teltonika-ingestion-architecture]
|
||||
tags: [data-model, teltonika, principle]
|
||||
---
|
||||
|
||||
# IO element bag
|
||||
|
||||
The model-specific telemetry inside a Teltonika AVL record. Carried as `{ id → value }` pairs that [[tcp-ingestion]] reads byte-correctly without interpreting.
|
||||
|
||||
## The principle: pass through unchanged
|
||||
|
||||
The Ingestion service does **not** name, interpret, or filter IO elements. It produces a [[position-record]] whose `attributes` field is a verbatim representation of the IO bag, keyed by the numeric IO element ID as a string.
|
||||
|
||||
- No renaming.
|
||||
- No unit conversion.
|
||||
- No model lookup.
|
||||
|
||||
Per-model interpretation lives in the [[processor]], driven by configuration like `{ "FMB920": { "16": "odometer_km", "240": "movement" } }`.
|
||||
|
||||
## Why this boundary matters
|
||||
|
||||
1. **Model-specific interpretation belongs where the model is known.** Doing it in the parser would couple Ingestion to a registry of every device model in the fleet — exactly the coupling the [[protocol-adapter]] is designed to prevent.
|
||||
2. **A new model never breaks Ingestion.** A packet with `IO 1234` from an unknown device produces a parser that stores the raw value under `"1234"` and moves on. The position is still useful; the attribute is recoverable later once the model's mapping is imported.
|
||||
|
||||
## Structural implications
|
||||
|
||||
- 8-bit IO IDs in Codec 8.
|
||||
- 16-bit IO IDs in Codec 8E (Teltonika ran out of 8-bit slots) plus variable-length values.
|
||||
- Codec 16 introduces a per-record generation type but the IO bag itself behaves the same.
|
||||
|
||||
The parser handles all three identically with respect to interpretation: read bytes per the codec spec, store under string key, hand off.
|
||||
@@ -0,0 +1,123 @@
|
||||
---
|
||||
title: Phase 2 — Outbound Commands
|
||||
type: concept
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [teltonika-ingestion-architecture, teltonika-data-sending-protocols]
|
||||
tags: [teltonika, phase-2, commands, future]
|
||||
---
|
||||
|
||||
# Phase 2 — Outbound Commands
|
||||
|
||||
The deferred design for server-to-device commands using [[teltonika]] codecs 12 (`0x0C`) and 14 (`0x0E`). Codec 13 (`0x0D`) is one-way device→server and is not part of the outbound design; codec 15 (`0x0F`) is FMX6-only and out of scope. Specified now so Phase 1 code respects the seams; **no Phase 2 code ships until the platform actually needs to issue commands.**
|
||||
|
||||
## Codec selection: 12 vs 14
|
||||
|
||||
- **Codec 12** — generic command/response. Server sends with Type `0x05`; device responds with Type `0x06`. No IMEI in the frame; the device assumed to be the one on the other end of the socket.
|
||||
- **Codec 14** — IMEI-addressed command/response. Server sends with Type `0x05` and an 8-byte IMEI in HEX; device returns Type **`0x06` (ACK)** if its physical IMEI matches, **`0x11` (nACK)** if not. Available from FMB.Ver.03.25.04.Rev.00. Useful as a defense-in-depth check that the connection registry is routing to the device we think we are.
|
||||
|
||||
A reasonable default is Codec 12 for routine ops (the connection registry already guarantees we're talking to the right device's socket), with Codec 14 reserved for situations where IMEI reconfirmation matters (e.g. infrequent high-impact commands).
|
||||
|
||||
## Why deferred
|
||||
|
||||
Command codecs are a **distinct feature**, not an incremental codec, and require:
|
||||
|
||||
- A way to enqueue commands targeted at specific devices.
|
||||
- Routing to whichever Ingestion instance currently holds the device's connection.
|
||||
- Permissioned APIs upstream so commands cannot be issued by unauthorized callers.
|
||||
- Audit trails for every command issued and every response received.
|
||||
|
||||
None of this is needed to read telemetry. Building it speculatively would either ship dead code or, worse, ship half-built infrastructure mistaken for usable.
|
||||
|
||||
## End-to-end flow
|
||||
|
||||
```
|
||||
SPA ──HTTPS+JWT──▶ Directus ──XADD──▶ Redis Streams ──XREAD──▶ Ingestion ──▶ device
|
||||
│ │
|
||||
SPA ◀──WSS subscription── Directus ◀──hook on insert── commands:responses
|
||||
```
|
||||
|
||||
Five properties:
|
||||
|
||||
1. **Single auth surface** — [[directus]] enforces "can this user command this device?" Same machinery as every other write.
|
||||
2. **Commands are data before transport** — every command is a row in the `commands` collection before it hits Redis.
|
||||
3. **Symmetric to inbound telemetry** — same plane boundary, same seam, same operational tools.
|
||||
4. **Per-instance routing** via a connection registry mapping `imei → instance_id`.
|
||||
5. **Real-time status updates for free** — Directus WebSocket subscriptions on `commands` push delivery status to the SPA.
|
||||
|
||||
## Architectural posture
|
||||
|
||||
[[tcp-ingestion]] **does not expose user-facing HTTP endpoints**, in Phase 1 or Phase 2. All user-facing API surface is in [[directus]] (see [[plane-separation]]). Ingestion learns about commands by consuming its own Redis stream — never accepts inbound user-facing traffic.
|
||||
|
||||
## The `commands` collection (Directus)
|
||||
|
||||
Key fields: `id` (uuid, correlation ID), `target_imei`, `batch_id` (nullable, for fleet ops), `codec` (`12` or `14` — `13`/`15` are device-originated, not server-issued), `payload` (ASCII text), `status` (`pending` | `routed` | `delivered` | `responded` | `failed` | `nack` | `expired`), `requested_by`, timestamp fields, `response`, `failure_reason`, `expires_at` (default `requested_at + 5 min`). The `nack` status captures Codec 14's IMEI-mismatch case (Type `0x11`).
|
||||
|
||||
Permissions: writable by operator/admin roles; readable by requester + admin. The SPA inserts via SDK; Ingestion updates delivery status via a service token.
|
||||
|
||||
## Connection registry
|
||||
|
||||
Redis hash `connections:registry`, keyed by IMEI, valued by Ingestion instance ID:
|
||||
|
||||
- On handshake: `HSET connections:registry {imei} {instance_id}` + record IMEI in local `Set<string>`.
|
||||
- Every 30s: `SET instance:heartbeat:{instance_id} {now} EX 90`.
|
||||
- On socket close: `HDEL connections:registry {imei}`.
|
||||
- Graceful shutdown: `HDEL` all held IMEIs.
|
||||
|
||||
**Crash recovery via janitor.** Redis hashes don't support per-field TTL, so a registry janitor (Directus Flow or small process) runs every minute: for each `instance_id` in the registry, `EXISTS instance:heartbeat:{instance_id}` — if missing, scan the registry for entries pointing to it and `HDEL` them.
|
||||
|
||||
## Issuing commands
|
||||
|
||||
**Single device** — SPA inserts a `commands` row; Directus Flow on `items.create`:
|
||||
|
||||
1. Lookup `instance_id = HGET connections:registry {target_imei}`.
|
||||
2. If found: `XADD commands:outbound:{instance_id} ...`; status → `routed`.
|
||||
3. If not found: status stays `pending`; sweeper retries.
|
||||
|
||||
**Fleet** — SPA calls custom endpoint `POST /commands/batch`:
|
||||
|
||||
1. Validate (size, authorization).
|
||||
2. Generate `batch_id`.
|
||||
3. Transactional insert of N rows sharing `batch_id`.
|
||||
4. Per row: registry lookup + stream publish.
|
||||
5. Return `{ batch_id, command_ids }`.
|
||||
|
||||
The custom endpoint exists for fleet operations because transactional insert + routing fan-out is cleaner in code than in a Flow.
|
||||
|
||||
## Pending-command sweeper
|
||||
|
||||
Flow runs every 30s:
|
||||
|
||||
- `pending` rows where `expires_at > now()` → retry registry lookup; if device now online, publish + transition to `routed`.
|
||||
- `pending` or `routed` rows where `expires_at <= now()` → `expired` with `failure_reason`.
|
||||
|
||||
Also handles the case where Ingestion crashes after publish but before delivery — those rows sit in `routed` past `expires_at` and get expired. Operator re-issues. (A subtler retry option exists — re-route stale `routed` rows when the original instance has died — but is an enhancement, not v1.)
|
||||
|
||||
## Ingestion-side command consumer
|
||||
|
||||
Each Ingestion instance runs a parallel consumer reading `commands:outbound:{instance_id}` (XREADGROUP, COUNT 16, BLOCK 1000):
|
||||
|
||||
1. Lookup local `imei → socket` map. If gone: publish failure (`socket_closed`).
|
||||
2. Check `expires_at`. If past: publish failure (`expired_before_delivery`).
|
||||
3. Encode codec 12/13/14 frame from payload.
|
||||
4. Write bytes to the socket (via per-socket write queue to avoid interleaving with codec ACKs).
|
||||
5. Register pending-response entry keyed by `command_id` with timeout (default 30s).
|
||||
|
||||
The consumer never blocks the TCP read path.
|
||||
|
||||
## Response correlation
|
||||
|
||||
Teltonika's command codecs carry **no correlation ID** — the protocol assumes one outstanding command per connection. The Ingestion service enforces this; subsequent commands queue on the per-socket write queue.
|
||||
|
||||
When the device responds (Codec 12 with `Type = 0x06`), the codec dispatch routes to a response handler that publishes to `commands:responses`; a Directus hook (or small consumer) updates the row to `status = responded`. Timeout fires → `status = failed` with `reason = 'no_device_response'`; write queue is freed.
|
||||
|
||||
## What this requires of Phase 1
|
||||
|
||||
Phase 1 must respect these shapes so Phase 2 is purely additive:
|
||||
|
||||
- [[codec-dispatch]] is a registry keyed on codec ID byte — Phase 2 registers `0x0C`, `0x0D`, `0x0E`.
|
||||
- Session loop owns the socket; handlers borrow it via a `respond(bytes)` callback (Phase 1 handlers don't use it).
|
||||
- Per-device runtime state is local to the socket and the holding instance — no shared registry today.
|
||||
- The [[position-record]] shape and the inbound stream are unchanged. Outbound uses entirely separate streams (`commands:outbound:{instance_id}`, `commands:responses`) and a separate Directus collection.
|
||||
|
||||
When Phase 2 ships, no Phase 1 code is rewritten — the command consumer runs alongside.
|
||||
@@ -0,0 +1,37 @@
|
||||
---
|
||||
title: Plane Separation (Telemetry / Business / Presentation)
|
||||
type: concept
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture]
|
||||
tags: [architecture, principle]
|
||||
---
|
||||
|
||||
# Plane Separation
|
||||
|
||||
The system is organized as **three concentric concerns**, deliberately split along **data velocity** and **failure domain**.
|
||||
|
||||
## The three planes
|
||||
|
||||
1. **Telemetry plane** — [[tcp-ingestion]] + [[redis-streams]] + [[processor]]. Optimized for throughput, low latency, resilience to bursty input. Stateless or nearly so.
|
||||
2. **Business plane** — [[directus]] over [[postgres-timescaledb]]. Owns schema, API surface, permissions, back-office workflows.
|
||||
3. **Presentation plane** — [[react-spa]]. Consumes business-plane APIs and real-time subscriptions; never talks to the telemetry plane directly.
|
||||
|
||||
## How the boundaries are enforced
|
||||
|
||||
- **Redis Streams** is the boundary between Ingestion and the Processor side of the telemetry plane.
|
||||
- **The database + Directus API** is the boundary between the telemetry and business planes.
|
||||
- **Directus REST/GraphQL/WSS** is the boundary between business and presentation planes.
|
||||
|
||||
**No component reaches across two boundaries.** This is the rule that makes the architecture coherent.
|
||||
|
||||
## Why this matters
|
||||
|
||||
- Each component scales independently (Ingestion horizontally per TCP load; Directus horizontally per HTTP load; database vertically + read replicas).
|
||||
- Each component fails independently — see [[failure-domains]].
|
||||
- Adding new device vendors does not touch the business plane. Adding new front-end surfaces does not touch the telemetry plane.
|
||||
- Security model stays coherent: only Directus exposes user-facing APIs.
|
||||
|
||||
## The discipline that holds it together
|
||||
|
||||
The clearest articulation of the plane discipline: **the TCP handler never blocks on downstream work.** If the Processor or DB is slow, the queue absorbs pressure; Ingestion keeps accepting and acknowledging. Without this, the planes leak into each other under load.
|
||||
@@ -0,0 +1,62 @@
|
||||
---
|
||||
title: Position Record
|
||||
type: concept
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture, teltonika-ingestion-architecture, teltonika-data-sending-protocols]
|
||||
tags: [data-model, boundary-contract]
|
||||
---
|
||||
|
||||
# Position Record
|
||||
|
||||
The normalized record produced by [[tcp-ingestion]] and consumed by [[processor]]. The boundary contract between vendor adapters and the rest of the system.
|
||||
|
||||
## Shape (Teltonika reference)
|
||||
|
||||
```ts
|
||||
type Position = {
|
||||
device_id: string // IMEI from the handshake
|
||||
timestamp: Date // from the AVL record's GPS timestamp
|
||||
latitude: number
|
||||
longitude: number
|
||||
altitude: number
|
||||
angle: number // heading, 0-360
|
||||
speed: number // km/h
|
||||
satellites: number
|
||||
priority: 0 | 1 | 2 // 0=Low, 1=High, 2=Panic (Teltonika spec)
|
||||
attributes: {
|
||||
[io_id: string]: number | bigint | Buffer
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The system architecture doc lists a slimmer canonical form — `device_id`, `timestamp`, `lat`, `lon`, `speed`, `heading`, plus an attribute bag — and the Teltonika doc extends it with `altitude`, `satellites`, `priority`. Treat the Teltonika shape as the current concrete contract.
|
||||
|
||||
## The `attributes` bag — pass through unchanged
|
||||
|
||||
The Ingestion service does **not** name, interpret, filter, or unit-convert IO elements. `attributes` is a verbatim representation of the IO element bag from the AVL record, keyed by the numeric IO element ID **as a string**.
|
||||
|
||||
Two reasons:
|
||||
|
||||
- **Model-specific interpretation belongs where the model is known.** The [[processor]] configures per-model IO mappings; doing this in the parser would couple Ingestion to a registry of every device model in the fleet.
|
||||
- **A new model never breaks Ingestion.** A packet with `IO 1234` from a device whose config we have not yet imported gets stored under `"1234"` and the position is still useful; the attribute is recoverable later.
|
||||
|
||||
This is the property that makes the [[protocol-adapter]] model-agnostic.
|
||||
|
||||
## What's codec-defined vs. model-defined
|
||||
|
||||
For [[teltonika]]:
|
||||
|
||||
- **Codec-defined** (always present, same shape per codec): timestamp, lat/lon/alt, angle, satellites, speed, priority. These fill out the typed fields above.
|
||||
- **Model-defined** (varies between models, opaque to parser): everything in the IO bag.
|
||||
|
||||
### Codec-specific quirks
|
||||
|
||||
- **Speed = `0x0000` means GPS data is invalid**, not "stationary." The parser preserves this verbatim (speed = 0); the [[processor]] decides whether to surface it as "no GPS fix" or coalesce with the stationary case.
|
||||
- **Lat/Lon are two's complement** signed integers — first bit = sign. Parsing must be sign-aware.
|
||||
- **Codec 16 carries a Generation Type** (0–7: On Exit, On Entrance, On Both, Reserved, Hysteresis, On Change, Eventual, Periodical) that is codec-defined but not currently in the Position shape. Open question on whether to promote it to a typed field. See [[avl-data-format]].
|
||||
- **Codec 8 Extended NX section** carries variable-length IO values — these land in `attributes` as `Buffer`, alongside the fixed-width N1/N2/N4/N8 entries.
|
||||
|
||||
## Downstream contract
|
||||
|
||||
[[processor]] is responsible for naming IO elements (e.g. `"16"` → `"odometer_km"`), unit conversions, and any filtering. It writes the typed fields to the positions hypertable and may write derived/named attributes to other tables.
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
title: Protocol Adapter Pattern
|
||||
type: concept
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture, teltonika-ingestion-architecture]
|
||||
tags: [architecture, ingestion, vendor-abstraction]
|
||||
---
|
||||
|
||||
# Protocol Adapter Pattern
|
||||
|
||||
The strategy used by [[tcp-ingestion]] to keep vendor-specific code from leaking into the rest of the system.
|
||||
|
||||
## The interface
|
||||
|
||||
- **Input**: a stream of bytes from a TCP socket.
|
||||
- **Output**: normalized [[position-record]] values with a stable shape — `device_id`, `timestamp`, `lat`, `lon`, `speed`, `heading`, plus a free-form attribute bag for vendor-specific telemetry.
|
||||
|
||||
That's the entire boundary contract. Adding a new device vendor (Queclink, Concox, etc.) means writing a new adapter. **Nothing downstream of Ingestion changes** — not [[redis-streams]], not [[processor]], not the schema, not the SPA.
|
||||
|
||||
This is "the single most important property of [the Ingestion] layer for long-term maintenance."
|
||||
|
||||
## Layout rules
|
||||
|
||||
In `tcp-ingestion/`:
|
||||
|
||||
1. **`src/core/` never imports from `src/adapters/`.** The vendor-agnostic shell (TCP listeners, Redis publishing, configuration, observability) is built without knowing about any vendor.
|
||||
2. **Adapters never import from one another.** `adapters/teltonika/` and any future `adapters/queclink/` are independent. Shared utilities (e.g. length-prefix buffer accumulation) live in `core/`.
|
||||
3. **Each adapter folder is self-contained.** Lifting [[teltonika]] out into its own service later is a `git mv adapters/teltonika/ ../gps-ingest-teltonika/src/adapter/` plus a copy of `core/`. No untangling required.
|
||||
|
||||
## The deeper move: model-agnostic via self-description
|
||||
|
||||
For [[teltonika]] specifically, the adapter is also **model-agnostic** — what matters is the codec ID, not the device model. Fixed fields are codec-defined; model-specific telemetry is carried in an opaque IO bag the parser passes through verbatim. New models work without code changes. See [[teltonika]] and [[codec-dispatch]].
|
||||
|
||||
This is a property of the protocol, not the pattern, but it pairs well: the adapter pattern isolates vendors; the codec dispatch isolates models within a vendor.
|
||||
|
||||
## What stays out of adapters
|
||||
|
||||
Per the Teltonika reference:
|
||||
|
||||
- **Naming/interpreting telemetry** is a [[processor]] concern, not an adapter concern.
|
||||
- **Filtering** is also downstream — adapters produce raw records.
|
||||
- **Per-model configuration** lives in the Processor, not the adapter.
|
||||
|
||||
Adapters do one thing: turn vendor bytes into normalized records.
|
||||
Reference in New Issue
Block a user