Bootstrap LLM-maintained wiki with TRM architecture knowledge
Initialize CLAUDE.md schema, index, and log; ingest three architecture sources (system overview, Teltonika ingestion design, official Teltonika data-sending protocols) into 7 entity pages, 8 concept pages, and 3 source pages with wikilink cross-references.
This commit is contained in:
@@ -0,0 +1,115 @@
|
||||
# TRM Wiki — Schema & Conventions
|
||||
|
||||
This repo is an LLM-maintained wiki. The human curates sources and asks questions. You (the LLM) own all wiki pages: you write them, update them, and keep them consistent.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
docs/
|
||||
├── CLAUDE.md # this file — schema and workflows
|
||||
├── index.md # content catalog (you maintain)
|
||||
├── log.md # chronological activity log (you append to)
|
||||
├── raw/ # immutable source documents — READ ONLY
|
||||
└── wiki/ # all LLM-generated pages live here
|
||||
├── sources/ # one page per ingested source
|
||||
├── entities/ # people, orgs, products, places
|
||||
├── concepts/ # ideas, topics, themes
|
||||
└── synthesis/ # comparisons, analyses, overview pages
|
||||
```
|
||||
|
||||
Rules:
|
||||
- **Never modify anything in `raw/`.** Treat it as immutable source of truth.
|
||||
- **Never edit the wiki by hand is the user's expectation** — but the user may occasionally. If you see human edits, respect them.
|
||||
- All wiki pages are markdown with YAML frontmatter (see below).
|
||||
- Use Obsidian-style `[[wikilinks]]` for cross-references between wiki pages. Use standard markdown links for external URLs and for links into `raw/`.
|
||||
|
||||
## Page frontmatter
|
||||
|
||||
Every page in `wiki/` starts with frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: <page title>
|
||||
type: source | entity | concept | synthesis
|
||||
created: YYYY-MM-DD
|
||||
updated: YYYY-MM-DD
|
||||
sources: [<source-page-name>, ...] # which sources contribute to this page
|
||||
tags: [<tag>, ...]
|
||||
---
|
||||
```
|
||||
|
||||
For `type: source` pages, also include:
|
||||
```yaml
|
||||
source_path: raw/<filename>
|
||||
source_date: YYYY-MM-DD # date of the original document, if known
|
||||
source_kind: article | paper | transcript | note | image | other
|
||||
```
|
||||
|
||||
## Workflows
|
||||
|
||||
### Ingest (when user says "ingest X" or drops a new file in raw/)
|
||||
|
||||
1. Read the source from `raw/`.
|
||||
2. Briefly discuss key takeaways with the user before writing — confirm what to emphasize.
|
||||
3. Create `wiki/sources/<slug>.md` with:
|
||||
- Frontmatter (type: source).
|
||||
- One-paragraph TL;DR.
|
||||
- Key claims / facts as a bulleted list, each with enough context to stand alone.
|
||||
- Notable quotes (verbatim, with location if available).
|
||||
- Open questions or things to follow up on.
|
||||
4. For each entity or concept the source touches:
|
||||
- If a page exists in `wiki/entities/` or `wiki/concepts/`, update it: add new facts, link the new source in `sources:` frontmatter, flag contradictions with existing claims explicitly (don't silently overwrite).
|
||||
- If no page exists and the entity/concept is substantive, create one.
|
||||
5. Update `index.md` — add the new source row and any new entity/concept rows.
|
||||
6. Append an entry to `log.md`.
|
||||
|
||||
A single ingest typically touches 5–15 wiki files. That's expected.
|
||||
|
||||
### Query (when user asks a question)
|
||||
|
||||
1. Read `index.md` first to locate relevant pages.
|
||||
2. Read those pages (and follow wikilinks as needed).
|
||||
3. Answer with citations: link `[[page-name]]` for wiki pages, and reference the underlying source pages so the user can trace claims back to `raw/`.
|
||||
4. If the answer is substantive (a comparison, an analysis, a new connection), offer to file it into `wiki/synthesis/` as a new page. Don't file by default — ask first.
|
||||
5. If the wiki can't answer the question, say so plainly. Suggest what source would fill the gap.
|
||||
|
||||
### Lint (when user says "lint" or "health-check")
|
||||
|
||||
Scan the wiki for:
|
||||
- Contradictions between pages (same claim, different values).
|
||||
- Stale claims newer sources have superseded.
|
||||
- Orphan pages (no inbound wikilinks).
|
||||
- Concepts mentioned in 3+ pages but lacking their own page.
|
||||
- Missing cross-references (entity X mentioned on page Y but not linked).
|
||||
- Frontmatter drift (missing fields, wrong types).
|
||||
|
||||
Report findings as a checklist. Don't fix automatically — confirm with the user first.
|
||||
|
||||
## Log format
|
||||
|
||||
`log.md` is append-only. Each entry starts with a header line in this exact format so it's grep-able:
|
||||
|
||||
```
|
||||
## [YYYY-MM-DD] <op> | <short title>
|
||||
```
|
||||
|
||||
Where `<op>` is one of: `ingest`, `query`, `lint`, `synthesis`, `note`. Body underneath is 1–5 lines: what happened, what files changed.
|
||||
|
||||
Example:
|
||||
```
|
||||
## [2026-04-30] ingest | Acme Q1 earnings call transcript
|
||||
|
||||
Created wiki/sources/acme-q1-earnings.md. Updated [[Acme]], [[revenue-recognition]].
|
||||
Flagged contradiction with [[acme-2025-guidance]] re: margin outlook.
|
||||
```
|
||||
|
||||
## Index format
|
||||
|
||||
`index.md` is a content catalog organized by section: Sources, Entities, Concepts, Synthesis. Each row: `- [[page-name]] — one-line summary`. Keep summaries tight. Re-sort alphabetically within each section after edits.
|
||||
|
||||
## Style
|
||||
|
||||
- Wiki pages should be readable on their own. Don't write "as discussed above" — there's no above for someone landing via search.
|
||||
- Prefer short, declarative sentences over hedged prose. Where uncertainty matters, say so explicitly with a `> note:` callout.
|
||||
- Quote sparingly and verbatim. Paraphrase by default.
|
||||
- When new information conflicts with an existing page, do not silently overwrite. Add the new claim, mark the conflict, and let the user adjudicate.
|
||||
@@ -0,0 +1,34 @@
|
||||
# Index
|
||||
|
||||
Content catalog for the TRM wiki. Maintained by the LLM on every ingest. See [[CLAUDE]] for schema and conventions.
|
||||
|
||||
## Sources
|
||||
|
||||
- [[gps-tracking-architecture]] — System-level architecture: four-component platform, three planes, failure domains.
|
||||
- [[teltonika-ingestion-architecture]] — Internal Teltonika protocol adapter design; Phase 1 (8/8E/16) and Phase 2 (12/13/14) roadmap.
|
||||
- [[teltonika-data-sending-protocols]] — Official Teltonika canonical wiki; full codec inventory including Codec 15 and SMS protocols, UDP transport, ACK/nACK details.
|
||||
|
||||
## Entities
|
||||
|
||||
- [[directus]] — Business plane: schema owner, REST/GraphQL/WSS, admin UI, permissions, Flows.
|
||||
- [[postgres-timescaledb]] — Durable storage: positions hypertable + business schema. The system's only single point of failure.
|
||||
- [[processor]] — Domain-logic service consuming Redis Streams; per-device hot state in memory; sole writer for telemetry tables.
|
||||
- [[react-spa]] — End-user UI; talks exclusively to Directus; role-based views in a single bundle.
|
||||
- [[redis-streams]] — Durable in-flight queue between Ingestion and Processor; Phase 2 transport for outbound commands.
|
||||
- [[tcp-ingestion]] — Per-vendor TCP listener service; parses binary protocols and emits normalized records.
|
||||
- [[teltonika]] — GPS hardware vendor; Codec 8/8E/16 telemetry today, Codec 12/14 commands deferred (13/15 one-way, 15 out of scope).
|
||||
|
||||
## Concepts
|
||||
|
||||
- [[avl-data-format]] — Canonical Teltonika packet structure: envelope, AVL record, GPS element, IO element layouts per codec.
|
||||
- [[codec-dispatch]] — Flat registry keyed on codec ID; the seam that makes Phase 2 additive.
|
||||
- [[failure-domains]] — Independent component failure behavior; database is the only SPOF.
|
||||
- [[io-element-bag]] — The pass-through principle for model-specific telemetry inside AVL records.
|
||||
- [[phase-2-commands]] — Deferred design for server-to-device commands via Teltonika codecs 12/14.
|
||||
- [[plane-separation]] — Three-plane architecture (telemetry / business / presentation) split by data velocity and failure domain.
|
||||
- [[position-record]] — Boundary contract between vendor adapters and the rest of the system.
|
||||
- [[protocol-adapter]] — Vendor-abstraction interface (bytes in → normalized Position out) at the Ingestion layer.
|
||||
|
||||
## Synthesis
|
||||
|
||||
_None yet._
|
||||
@@ -0,0 +1,39 @@
|
||||
# Log
|
||||
|
||||
Chronological activity log. Append-only. Entry headers use the format `## [YYYY-MM-DD] <op> | <title>` so they can be grepped:
|
||||
|
||||
```
|
||||
grep "^## \[" log.md | tail -10
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## [2026-04-30] note | Wiki bootstrapped
|
||||
|
||||
Created CLAUDE.md (schema + workflows), index.md (empty catalog), and this log. Wiki directory structure (wiki/sources, wiki/entities, wiki/concepts, wiki/synthesis) will be created on first ingest.
|
||||
|
||||
## [2026-04-30] ingest | gps-tracking-architecture.md + teltonika-ingestion-architecture.md
|
||||
|
||||
Ingested both initial architecture docs in one pass. Created:
|
||||
- Source pages: [[gps-tracking-architecture]], [[teltonika-ingestion-architecture]].
|
||||
- Entity pages: [[tcp-ingestion]], [[processor]], [[directus]], [[react-spa]], [[redis-streams]], [[postgres-timescaledb]], [[teltonika]].
|
||||
- Concept pages: [[plane-separation]], [[protocol-adapter]], [[codec-dispatch]], [[position-record]], [[failure-domains]], [[phase-2-commands]].
|
||||
- Updated index.md with all 15 new pages.
|
||||
|
||||
No contradictions to flag — the two docs are coherent (the Teltonika doc explicitly cites and respects the system architecture). Open follow-ups: TRM business domain not yet captured; per-model IO dictionary location TBD; Phase 2 timing unspecified.
|
||||
|
||||
## [2026-04-30] ingest | Teltonika Data Sending Protocols (official wiki)
|
||||
|
||||
Ingested the canonical Teltonika spec covering all codec families. New additions:
|
||||
- Source page: [[teltonika-data-sending-protocols]].
|
||||
- New concept: [[avl-data-format]] — byte-level reference for codecs 8/8E/16, including UDP envelope.
|
||||
|
||||
Updates to existing pages (no contradictions; refinements + additions):
|
||||
- [[teltonika]] — added full codec table with hex IDs, Codec 15 (out of scope), Codec 14 ACK/nACK, packet size limits, UDP support note.
|
||||
- [[codec-dispatch]] — corrected hex IDs, added directionality table covering codecs 8–15.
|
||||
- [[position-record]] — concrete priority enum (0/1/2), two's-complement lat/lon note, Speed=0 means GPS invalid, Generation Type and NX section flagged.
|
||||
- [[phase-2-commands]] — clarified Codec 12 vs 14 selection, added `nack` status for Codec 14 IMEI-mismatch (Type `0x11`); noted 13/15 are not part of the outbound design.
|
||||
|
||||
Cleanup: removed stale duplicate concept files from earlier passes (system-planes.md, protocol-adapter-pattern.md, codec-dispatch-registry.md) — superseded by plane-separation.md, protocol-adapter.md, codec-dispatch.md respectively. Fixed dangling [[protocol-adapter-pattern]] link in [[io-element-bag]].
|
||||
|
||||
Open questions surfaced by the canonical doc: Codec 16 Generation Type — promote to typed [[position-record]] field? Codec 8E NX values land as `Buffer` in `attributes`; needs explicit fixture coverage. SMS-based protocols (Codec 4 + binary SMS) probably out of scope but worth a deliberate decision.
|
||||
@@ -0,0 +1,842 @@
|
||||
---
|
||||
title: "Teltonika Data Sending Protocols - Teltonika Telematics Wiki"
|
||||
source: "https://wiki.teltonika-gps.com/view/Teltonika_Data_Sending_Protocols#Codec_8"
|
||||
author:
|
||||
published:
|
||||
created: 2026-04-30
|
||||
description:
|
||||
tags:
|
||||
- "clippings"
|
||||
---
|
||||
[Main Page](https://wiki.teltonika-gps.com/view/Main_Page) > [General Information](https://wiki.teltonika-gps.com/view/General_Information) > **Teltonika Data Sending Protocols**
|
||||
|
||||
## Introduction
|
||||
|
||||
A codec is a device or computer program for encoding or decoding a digital data stream or signal. Codec is a portmanteau of coder-decoder. A codec encodes a data stream or a signal for transmission and storage, possibly in encrypted form, and the decoder function reverses the encoding for playback or editing.
|
||||
|
||||
Below you will see a table of all Codec types with IDs:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">Codec 8</th><th rowspan="1">Codec 8 Extended</th><th rowspan="1">Codec 16</th><th rowspan="1">Codec 12</th><th colspan="1">Codec 13</th><th rowspan="1">Codec 14</th></tr><tr><td>0x08</td><td>0x8E</td><td>0x10</td><td>0x0C</td><td>0x0D</td><td>0x0E</td></tr></tbody></table>
|
||||
|
||||
Also, there are using two data transport protocols: TCP and UDP. But it is not important which one will be used in Codec.
|
||||
|
||||
## Codec for device data sending
|
||||
|
||||
In this chapter, you will find information about every Codec protocol which are used for device data sending and the differences between them.
|
||||
|
||||
## Codec 8
|
||||
|
||||
- **Protocol Overview**
|
||||
|
||||
Codec8 – a main FM device protocol that is used for sending data to the server.
|
||||
|
||||
- **Codec 8 protocol sending over TCP**
|
||||
|
||||
TCP is a connection-oriented protocol that is used for communication between devices. The workings of this type of protocol is described below in the **communication with server** section.
|
||||
|
||||
- **AVL Data Packet**
|
||||
|
||||
The below table represents the AVL Data Packet structure:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">0x00000000 (Preamble)</th><th rowspan="1">Data Field Length</th><th rowspan="1">Codec ID</th><th rowspan="1">Number of Data 1</th><th colspan="1">AVL Data</th><th rowspan="1">Number of Data 2</th><th rowspan="1">CRC-16</th></tr><tr><td>4 bytes</td><td>4 bytes</td><td>1 byte</td><td>1 byte</td><td>X bytes</td><td>1 byte</td><td>4 bytes</td></tr></tbody></table>
|
||||
|
||||
**Preamble** – the packet starts with four zero bytes.
|
||||
**Data Field Length** – size is calculated starting from Codec ID to Number of Data 2.
|
||||
**Codec ID** – in Codec8 it is always `0x08`.
|
||||
**Number of Data 1** – a number that defines how many records are in the packet.
|
||||
**AVL Data** – actual data in the packet (more information below).
|
||||
**Number of Data 2** – a number that defines how many records are in the packet. This number must be the same as “Number of Data 1”.
|
||||
**CRC-16** – calculated from Codec ID to the Second Number of Data. CRC (Cyclic Redundancy Check) is an error-detecting code used to detect accidental changes to RAW data. For calculation we are using [CRC-16/IBM](https://wiki.teltonika-gps.com/view/Codec#CRC-16 "Codec").
|
||||
|
||||
**Note:** for [FMB640](https://wiki.teltonika-gps.com/view/FMB640 "FMB640"), [FMB641](https://wiki.teltonika-gps.com/view/FMB641 "FMB641"), [FMC640](https://wiki.teltonika-gps.com/view/FMC640 "FMC640"), and [FMM640](https://wiki.teltonika-gps.com/view/FMM640 "FMM640"), minimum AVL record size is 45 bytes (all IO elements disabled). The maximum AVL record size is 255 bytes. Maximum AVL packet size is 512 bytes. For other devices, the minimum AVL record size is 45 bytes (all IO elements disabled). Maximum AVL packet size is 1280 bytes.
|
||||
|
||||
- AVL Data
|
||||
|
||||
The below table represents the AVL Data structure.
|
||||
|
||||
<table><tbody><tr><th rowspan="1">Timestamp</th><th rowspan="1">Priority</th><th rowspan="1">GPS Element</th><th rowspan="1">IO Element</th></tr><tr><td>8 bytes</td><td>1 byte</td><td>15 bytes</td><td>X bytes</td></tr></tbody></table>
|
||||
|
||||
**Timestamp** – a difference, in milliseconds, between the current time and midnight, January 1970 UTC (UNIX time).
|
||||
**Priority** – a field that defines AVL data priority (more information below).
|
||||
**GPS Element** – location information of the AVL data (more information below).
|
||||
**IO Element** – additional configurable information from the device (more information below).
|
||||
|
||||
- Priority
|
||||
|
||||
The below table represents Priority values. Packet priority depends on device configuration and records sent.
|
||||
|
||||
<table><tbody><tr><th colspan="2">Priority</th></tr><tr><th rowspan="1">0</th><td>Low</td></tr><tr><th rowspan="1">1</th><td>High</td></tr><tr><th rowspan="1">2</th><td>Panic</td></tr></tbody></table>
|
||||
|
||||
- GPS element
|
||||
|
||||
The below table represents the GPS Element structure:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">Longitude</th><th rowspan="1">Latitude</th><th rowspan="1">Altitude</th><th rowspan="1">Angle</th><th rowspan="1">Satellites</th><th rowspan="1">Speed</th></tr><tr><td>4 bytes</td><td>4 bytes</td><td>2 bytes</td><td>2 bytes</td><td>1 byte</td><td>2 bytes</td></tr></tbody></table>
|
||||
|
||||
**Longitude** – east-west position.
|
||||
**Latitude** – north-south position.
|
||||
**Altitude** – meters above sea level.
|
||||
**Angle** – degrees from north pole.
|
||||
**Satellites** – number of satellites in use.
|
||||
**Speed** – speed calculated from satellites.
|
||||
|
||||
**Note:** Speed will be `0x0000` if GPS data is invalid.
|
||||
|
||||
Longitude and latitude are integer values built from degrees, minutes, seconds, and milliseconds by the formula:
|
||||
|
||||
Where:
|
||||
d – Degrees; m – Minutes; s – Seconds; ms – Milliseconds; p – Precision (10000000)
|
||||
If the longitude is in the west or latitude in the south, multiply the result by –1.
|
||||
|
||||
Note:
|
||||
To determine if the coordinate is negative, convert it to binary format and check the very first bit. If it is 0, the coordinate is positive. If it is 1, the coordinate is negative.
|
||||
|
||||
Example:
|
||||
Received value: `20 9C CA 80` converted to BIN: `00100000 10011100 11001010 10000000` first bit is 0, which means coordinate is positive converted to DEC: `547146368`. For more information see two‘s complement arithmetic.
|
||||
|
||||
- IO Element
|
||||
|
||||
<table><tbody><tr><th rowspan="1">Event IO ID</th><td>1 byte</td><td rowspan="26"></td><td rowspan="26"><b>Event IO ID</b> – if data is acquired on the event – this field defines which IO property has changed and generated an event. For example, when if the Ignition state changes and it generates an event, the Event IO ID will be <code>0xEF</code> (AVL ID: 239). If it’s not an eventual record – the value is 0.<br><p><b>N</b> – a total number of properties coming with record (N = N1 + N2 + N4 + N8).<br><b>N1</b> – number of properties, which length is 1 byte.<br><b>N2</b> – number of properties, which length is 2 bytes.<br><b>N4</b> – number of properties, which length is 4 bytes.<br><b>N8</b> – number of properties, which length is 8 bytes.<br><b>N’th IO ID</b> - AVL ID.<br><b>N’th IO Value</b> - AVL ID value.</p></td></tr><tr><th rowspan="1">N of Total IO</th><td>1 byte</td></tr><tr><th rowspan="1">N1 of One Byte IO</th><td>1 byte</td></tr><tr><th rowspan="1">1’st IO ID</th><td>1 byte</td></tr><tr><th rowspan="1">1’st IO Value</th><td>1 byte</td></tr><tr><td colspan="2">...</td></tr><tr><th rowspan="1">N1’th IO ID</th><td>1 byte</td></tr><tr><th rowspan="1">N1’th IO Value</th><td>1 byte</td></tr><tr><th rowspan="1">N2 of Two Bytes</th><td>1 byte</td></tr><tr><th rowspan="1">1’st IO ID</th><td>1 byte</td></tr><tr><th rowspan="1">1’st IO Value</th><td>2 bytes</td></tr><tr><td colspan="2">...</td></tr><tr><th rowspan="1">N2’th IO ID</th><td>1 byte</td></tr><tr><th rowspan="1">N2’th IO Value</th><td>2 bytes</td></tr><tr><th rowspan="1">N4 of Four Bytes</th><td>1 byte</td></tr><tr><th rowspan="1">1’st IO ID</th><td>1 byte</td></tr><tr><th rowspan="1">1’st IO Value</th><td>4 bytes</td></tr><tr><td colspan="2">...</td></tr><tr><th rowspan="1">N4’th IO ID</th><td>1 byte</td></tr><tr><th rowspan="1">N4’th IO Value</th><td>4 byte</td></tr><tr><th rowspan="1">N8 of Eight Bytes</th><td>1 byte</td></tr><tr><th rowspan="1">1’st IO ID</th><td>1 byte</td></tr><tr><th rowspan="1">1’st IO Value</th><td>8 byte</td></tr><tr><td colspan="2">...</td></tr><tr><th rowspan="1">N8’IO ID</th><td>1 byte</td></tr><tr><th rowspan="1">N8’IO Value</th><td>8 bytes</td></tr></tbody></table>
|
||||
|
||||
- **Communication with server**
|
||||
|
||||
First, when the module connects to the server, the module sends its IMEI. First comes a short identifying the number of bytes written and then goes IMEI as text (bytes).
|
||||
For example, IMEI `356307042441013` would be sent as `000F333536333037303432343431303133`.
|
||||
The first two bytes denote IMEI length. In this case `0x000F` means, that IMEI is 15 bytes long.
|
||||
After receiving IMEI, the server should determine if it would accept data from this module. If yes, server will reply to module `01`, if not - `00`. Note that confirmation should be sent as a binary packet. I.e. 1 byte `0x01` or `0x00`.
|
||||
Then the module starts to send the first AVL data packet. After the server receives a packet and parses it, the server must report to the module number of data received as an integer (four bytes).
|
||||
If the sent data number and the reported by the server don’t match module resends the sent data.
|
||||
|
||||
- Example:
|
||||
|
||||
The module connects to the server and sends IMEI:
|
||||
`000F333536333037303432343431303133`
|
||||
The server accepts the module:
|
||||
01
|
||||
The module sends data packet:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">AVL Data Packet Header</th><th rowspan="1">AVL Data Array</th><th rowspan="1">CRC-16</th></tr><tr><td>Four Zero Bytes – 0x00000000,<p>“AVL Data Array” length – 0x000000FE</p></td><td>Codec ID – 0x08,<p>Number of Data – <b>0x02</b><br>(Encoded using continuous bit stream. The last byte is padded to align to the byte boundary)</p></td><td>CRC of “AVL Data Array”</td></tr><tr><td>00000000000000FE</td><td>08 <b>02</b>...(data elements)...<b>02</b></td><td>00008612</td></tr></tbody></table>
|
||||
|
||||
Server acknowledges data reception (2 data elements): **`00000002`**
|
||||
|
||||
- **Examples**
|
||||
|
||||
The hexadecimal stream of AVL Data Packet receiving and response in these examples are given in the hexadecimal form. The different fields of packets are separated into different table columns for better readability and some of them are converted to ASCII values for better understanding.
|
||||
|
||||
**1'st example**
|
||||
Receiving one data record with each element property (1 byte, 2 bytes, 4 bytes, and 8 bytes).
|
||||
|
||||
Received data in the hexadecimal stream:
|
||||
`000000000000003608010000016B40D8EA30010000000000000000000000000000000105021503010101425E0F01F10000601A014E0000000000000000010000C7CF`
|
||||
|
||||
Parsed:
|
||||
|
||||
<table><tbody><tr><th colspan="3">AVL Data Packet</th></tr><tr><th colspan="2" rowspan="1">AVL Data Packet Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td rowspan="4"></td><td>Zero Bytes</td><td>00 00 00 00</td></tr><tr><td>Data Field Length</td><td>00 00 00 36</td></tr><tr><td>Codec ID</td><td>08</td></tr><tr><td>Number of Data 1 (Records)</td><td>01</td></tr><tr><td rowspan="24">AVL Data</td><td>Timestamp</td><td>00 00 01 6B 40 D8 EA 30 (GMT: Monday, June 10, 2019, 10:04:46 AM)</td></tr><tr><td>Priority</td><td>01</td></tr><tr><td>Longitude</td><td>00 00 00 00</td></tr><tr><td>Latitude</td><td>00 00 00 00</td></tr><tr><td>Altitude</td><td>00 00</td></tr><tr><td>Angle</td><td>00 00</td></tr><tr><td>Satellites</td><td>00</td></tr><tr><td>Speed</td><td>00 00</td></tr><tr><td>Event IO ID</td><td>01</td></tr><tr><td>N of Total ID</td><td>05</td></tr><tr><td>N1 of One Byte IO</td><td>02</td></tr><tr><td>1’st IO ID</td><td>15 (AVL ID: 21, Name: GSM Signal)</td></tr><tr><td>1’st IO Value</td><td>03</td></tr><tr><td>2’nd IO ID</td><td>01 (AVL ID: 1, Name: DIN1)</td></tr><tr><td>2’nd IO Value</td><td>01</td></tr><tr><td>N2 of Two Bytes IO</td><td>01</td></tr><tr><td>1’st IO ID</td><td>42 (AVL ID: 66, Name: External Voltage)</td></tr><tr><td>1’st IO Value</td><td>5E 0F</td></tr><tr><td>N4 of Four Bytes IO</td><td>01</td></tr><tr><td>1’st IO ID</td><td>F1 (AVL ID: 241, Name: Active GSM Operator)</td></tr><tr><td>1’st IO Value</td><td>00 00 60 1A</td></tr><tr><td>N8 of Eight Bytes IO</td><td>01</td></tr><tr><td>1’st IO ID</td><td>4E (AVL ID: 78, Name: iButton)</td></tr><tr><td>1’st IO Value</td><td>00 00 00 00 00 00 00 00</td></tr><tr><td rowspan="2"></td><td>Number of Data 2 (Number of Total Records)</td><td>01</td></tr><tr><td>CRC-16</td><td>00 00 C7 CF</td></tr></tbody></table>
|
||||
|
||||
Server response: `00000001`
|
||||
|
||||
**2'nd example**
|
||||
Receiving one data record with one or two different element properties (1 byte, 2 bytes).
|
||||
|
||||
Received data in the hexadecimal stream:
|
||||
`000000000000002808010000016B40D9AD80010000000000000000000000000000000103021503010101425E100000010000F22A`
|
||||
|
||||
Parsed:
|
||||
|
||||
<table><tbody><tr><th colspan="3">AVL Data Packet</th></tr><tr><th colspan="2" rowspan="1">AVL Data Packet Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td rowspan="4"></td><td>Zero Bytes</td><td>00 00 00 00</td></tr><tr><td>Data Field Length</td><td>00 00 00 28</td></tr><tr><td>Codec ID</td><td>08</td></tr><tr><td>Number of Data 1 (Records)</td><td>01</td></tr><tr><td rowspan="20">AVL Data</td><td>Timestamp</td><td>00 00 01 6B 40 D9 AD 80 (GMT: Monday, June 10, 2019, 10:05:36 AM)</td></tr><tr><td>Priority</td><td>01</td></tr><tr><td>Longitude</td><td>00 00 00 00</td></tr><tr><td>Latitude</td><td>00 00 00 00</td></tr><tr><td>Altitude</td><td>00 00</td></tr><tr><td>Angle</td><td>00 00</td></tr><tr><td>Satellites</td><td>00</td></tr><tr><td>Speed</td><td>00 00</td></tr><tr><td>Event IO ID</td><td>01</td></tr><tr><td>N of Total ID</td><td>03</td></tr><tr><td>N1 of One Byte IO</td><td>02</td></tr><tr><td>1’st IO ID</td><td>15 (AVL ID: 21, Name: GSM Signal)</td></tr><tr><td>1’st IO Value</td><td>03</td></tr><tr><td>2’nd IO ID</td><td>01 (AVL ID: 1, Name: DIN1)</td></tr><tr><td>2’nd IO Value</td><td>01</td></tr><tr><td>N2 of Two Bytes IO</td><td>01</td></tr><tr><td>1’st IO ID</td><td>42 (AVL ID: 66, Name: External Voltage)</td></tr><tr><td>1’st IO Value</td><td>5E 0F</td></tr><tr><td>N4 of Four Bytes IO</td><td>00</td></tr><tr><td>N8 of Eight Bytes IO</td><td>00</td></tr><tr><td rowspan="2"></td><td>Number of Data 2 (Number of Total Records)</td><td>01</td></tr><tr><td>CRC-16</td><td>00 00 F2 2A</td></tr></tbody></table>
|
||||
|
||||
Server response: `00000001`
|
||||
|
||||
**3'rd example**
|
||||
Receiving two or more data records with one or more different element properties.
|
||||
|
||||
Received data in the hexadecimal stream:
|
||||
`000000000000004308020000016B40D57B480100000000000000000000000000000001010101000000000000016B40D5C198010000000000000000000000000000000 101010101000000020000252C`
|
||||
|
||||
Parsed:
|
||||
|
||||
<table><tbody><tr><th colspan="3">AVL Data Packet</th></tr><tr><th colspan="2" rowspan="1">AVL Data Packet Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td rowspan="4"></td><td>Zero Bytes</td><td>00 00 00 00</td></tr><tr><td>Data Field Length</td><td>00 00 00 43</td></tr><tr><td>Codec ID</td><td>08</td></tr><tr><td>Number of Data 1 (Records)</td><td>02</td></tr><tr><td rowspan="16">AVL Data<p>(1'st record)</p></td><td>Timestamp</td><td>00 00 01 6B 40 D5 7B 48 (GMT: Monday, June 10, 2019, 10:01:01 AM)</td></tr><tr><td>Priority</td><td>01</td></tr><tr><td>Longitude</td><td>00 00 00 00</td></tr><tr><td>Latitude</td><td>00 00 00 00</td></tr><tr><td>Altitude</td><td>00 00</td></tr><tr><td>Angle</td><td>00 00</td></tr><tr><td>Satellites</td><td>00</td></tr><tr><td>Speed</td><td>00 00</td></tr><tr><td>Event IO ID</td><td>01</td></tr><tr><td>N of Total ID</td><td>01</td></tr><tr><td>N1 of One Byte IO</td><td>01</td></tr><tr><td>1’st IO ID</td><td>01 (AVL ID: 1, Name: DIN1)</td></tr><tr><td>1’st IO Value</td><td>00</td></tr><tr><td>N2 of Two Bytes IO</td><td>00</td></tr><tr><td>N4 of Four Bytes IO</td><td>00</td></tr><tr><td>N8 of Eight Bytes IO</td><td>00</td></tr><tr><td rowspan="16">AVL Data<p>(2'nd record)</p></td><td>Timestamp</td><td>00 00 01 6B 40 D5 C1 98 (GMT: Monday, June 10, 2019 10:01:19 AM)</td></tr><tr><td>Priority</td><td>01</td></tr><tr><td>Longitude</td><td>00 00 00 00</td></tr><tr><td>Latitude</td><td>00 00 00 00</td></tr><tr><td>Altitude</td><td>00 00</td></tr><tr><td>Angle</td><td>00 00</td></tr><tr><td>Satellites</td><td>00</td></tr><tr><td>Speed</td><td>00 00</td></tr><tr><td>Event IO ID</td><td>01</td></tr><tr><td>N of Total ID</td><td>01</td></tr><tr><td>N1 of One Byte IO</td><td>01</td></tr><tr><td>1’st IO ID</td><td>01 (AVL ID: 1, Name: DIN1)</td></tr><tr><td>1’st IO Value</td><td>01</td></tr><tr><td>N2 of Two Bytes IO</td><td>00</td></tr><tr><td>N4 of Four Bytes IO</td><td>00</td></tr><tr><td>N8 of Eight Bytes IO</td><td>00</td></tr><tr><td rowspan="2"></td><td>Number of Data 2 (Number of Total Records)</td><td>02</td></tr><tr><td>CRC-16</td><td>00 00 25 2C</td></tr></tbody></table>
|
||||
|
||||
Server response: `00000002`
|
||||
|
||||
- **Codec8 protocol sending over UDP**
|
||||
|
||||
Codec8 protocol \[over UDP\] is a transport layer protocol above UDP/IP to add reliability to plain UDP/IP using acknowledgment packets.
|
||||
|
||||
- **AVL Data Packet**
|
||||
|
||||
The packet structure is as follows:
|
||||
|
||||
<table><tbody><tr><th colspan="2">UDP Datagram</th></tr><tr><td>Example</td><td>2 bytes</td></tr><tr><td>Packet ID</td><td>2 bytes</td></tr><tr><td>Not Usable Byte</td><td>1 byte</td></tr><tr><td>Packet Payload</td><td>Variable</td></tr></tbody></table>
|
||||
|
||||
**Example** – packet length (excluding this field) in big ending byte order.
|
||||
**Packet ID** – packet ID unique for this channel.
|
||||
**Not Usable Byte** – not usable byte.
|
||||
**Packet payload** – data payload.
|
||||
|
||||
- Acknowledgment packet
|
||||
|
||||
The acknowledgment packet should have the same Packet ID as an acknowledged data packet and empty Data Payload. Acknowledgment should be sent in binary format.
|
||||
|
||||
<table><tbody><tr><th colspan="3">Acknowledgment Packet</th></tr><tr><th rowspan="1">Packet Length</th><th rowspan="1">Packet ID</th><th rowspan="1">Not Usable Byte</th></tr><tr><td>2 bytes</td><td>2 bytes</td><td>1 byte</td></tr></tbody></table>
|
||||
|
||||
**Packet Length** – packet length by sending/response data.
|
||||
**Packet ID** – same as in acknowledgment packet.
|
||||
**Not Usable Byte** – always will be `0x01`.
|
||||
|
||||
- Sending AVL Packet Payload using UDP channel
|
||||
|
||||
The below table represents the Sending Packet Payload structure.
|
||||
|
||||
<table><tbody><tr><th colspan="4">AVL data encapsulated in UDP channel packet</th></tr><tr><th rowspan="1">AVL Packet ID</th><th rowspan="1">IMEI Length</th><th rowspan="1">Module IMEI</th><th rowspan="1">AVL Data Array</th></tr><tr><td>1 byte</td><td>2 bytes</td><td>15 bytes</td><td>X bytes</td></tr></tbody></table>
|
||||
|
||||
**AVL Packet ID** – ID identifying this AVL packet.
|
||||
**IMEI Length** – always will be `0x000F`.
|
||||
**Module IMEI** – IMEI of a sending module encoded the same as with TCP.
|
||||
**AVL Data Array** – an array of encoded AVL data (same as TCP AVL Data Array).
|
||||
|
||||
- Server response Packet Payload using UDP channel
|
||||
|
||||
The below table represents the Server Response Packet Payload structure.
|
||||
|
||||
<table><tbody><tr><th colspan="2">Server Response to AVL Data Packet</th></tr><tr><th rowspan="1">AVL Packet ID</th><th rowspan="1">Number of Accepted AVL Elements</th></tr><tr><td>1 byte</td><td>1 byte</td></tr></tbody></table>
|
||||
|
||||
- **Communication with server**
|
||||
|
||||
The module sends the UDP channel packet with an encapsulated AVL data packet. The server sends the UDP channel packet with an encapsulated response module that validates the AVL Packet ID and the Number of accepted AVL elements. If the server response is not received with a valid AVL Packet ID within configured timeout, the module can retry sending.
|
||||
|
||||
- Example:
|
||||
|
||||
The module sends the data:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">UDP Channel Header</th><th rowspan="1">AVL Packet Header</th><th rowspan="1">AVL Data Array</th></tr><tr><td>Length – 0x00FE,<p>Packet ID – 0xCAFE<br>Not Usable Byte – 0x01</p></td><td>AVL Packet ID – 0xDD,<p>IMEI Length – 0x000F<br>IMEI – 0x313233343536373839303132333435 (Encoded using continuous bit stream. The last byte is padded to align to the byte boundary)</p></td><td>Codec ID – 0x08,<p>Number of Data – 0x02<br>(Encoded using continuous bit stream)</p></td></tr><tr><td>00FECAFE01</td><td>DD000F3133343536373839303132333435</td><td>0802…(data elements)…02</td></tr></tbody></table>
|
||||
|
||||
The server must respond with an acknowledgment:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">UDP Channel Header</th><th rowspan="1">AVL Packet Acknowledgment</th></tr><tr><td>Length – 0x0005,<p>Packet ID – 0xCAFE, Not Usable Byte – 0x01</p></td><td>AVL Packet ID – 0xDD,<p>Number of Accepted Data – 0x02</p></td></tr><tr><td>0005CAFE01</td><td>DD02</td></tr></tbody></table>
|
||||
|
||||
- **Example**
|
||||
|
||||
The hexadecimal stream of AVL Data Packet receiving and response in this example is given in the hexadecimal form. The different fields of the packet are separated into different table columns for better readability and some of them are converted to ASCII values for better understanding.
|
||||
|
||||
Received data in the hexadecimal stream:
|
||||
`003DCAFE0105000F33353230393330383634303336353508010000016B4F815B30010000000000000000000000000000000103021503010101425DBC000001`
|
||||
|
||||
Parsed:
|
||||
|
||||
<table><tbody><tr><th colspan="3">AVL Data Packet</th></tr><tr><th colspan="2" rowspan="1">AVL Data Packet Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td rowspan="3">UDP Channel Header</td><td>Length</td><td>00 3D</td></tr><tr><td>Packet ID</td><td>CA FE</td></tr><tr><td>Not usable byte</td><td>01</td></tr><tr><td rowspan="3">AVL Packet Header</td><td>AVL packet ID</td><td>05</td></tr><tr><td>IMEI Length</td><td>00 0F</td></tr><tr><td>IMEI</td><td>33 35 32 30 39 33 30 38 36 34 30 33 36 35 35</td></tr><tr><td rowspan="23">AVL Data Array</td><td>Codec ID</td><td>08</td></tr><tr><td>Number of Data 1 (Records)</td><td>01</td></tr><tr><td>Timestamp</td><td>00 00 01 6B 4F 81 5B 30 (GMT: Thursday, June 13, 2019, 6:23:26 AM)</td></tr><tr><td>Priority</td><td>01</td></tr><tr><td>Longitude</td><td>00 00 00 00</td></tr><tr><td>Latitude</td><td>00 00 00 00</td></tr><tr><td>Altitude</td><td>00 00</td></tr><tr><td>Angle</td><td>00 00</td></tr><tr><td>Satellites</td><td>00</td></tr><tr><td>Speed</td><td>00 00</td></tr><tr><td>Event IO ID</td><td>01</td></tr><tr><td>N of Total ID</td><td>03</td></tr><tr><td>N1 of One Byte IO</td><td>02</td></tr><tr><td>1’st IO ID</td><td>15 (AVL ID: 21, Name: GSM Signal)</td></tr><tr><td>1’st IO Value</td><td>03</td></tr><tr><td>2’nd IO ID</td><td>01 (AVL ID: 1, Name: DIN1)</td></tr><tr><td>2’nd IO Value</td><td>01</td></tr><tr><td>N2 of Two Bytes IO</td><td>01</td></tr><tr><td>1’st IO ID</td><td>42 (AVL ID: 66, Name: External Voltage)</td></tr><tr><td>1’st IO Value</td><td>5D BC</td></tr><tr><td>N4 of Four Bytes IO</td><td>00</td></tr><tr><td>N8 of EightBytes IO</td><td>00</td></tr><tr><td>Number of Data 2 (Number of Total Records)</td><td>01</td></tr></tbody></table>
|
||||
|
||||
The server response in the hexadecimal stream: `0005CAFE010501`
|
||||
|
||||
Parsed:
|
||||
|
||||
<table><tbody><tr><th colspan="3">Server Response to AVL Data Packet</th></tr><tr><th colspan="2" rowspan="1">Server Response Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td rowspan="3">UDP Channel Header</td><td>Length</td><td>00 05</td></tr><tr><td>Packet ID</td><td>CA FE</td></tr><tr><td>Not usable byte</td><td>01</td></tr><tr><td rowspan="2">AVL Packet Acknowledgment</td><td>AVL packet ID</td><td>05</td></tr><tr><td>Number of Accepted Data</td><td>01</td></tr></tbody></table>
|
||||
|
||||
## Codec 8 Extended
|
||||
|
||||
- **Protocols overview**
|
||||
|
||||
Codec8 Extended is used for FMBXXX family devices. This protocol looks familiar to Codec8 but they have some differences. The main differences between them are shown in below table:
|
||||
|
||||
<table><tbody><tr><th rowspan="1"></th><th rowspan="1">Codec8</th><th rowspan="1">Codec8 Extended</th></tr><tr><th rowspan="1">Codec ID</th><td>0x08</td><td>0x8E</td></tr><tr><th rowspan="1">AVL Data IO element length</th><td>1 byte</td><td>2 bytes</td></tr><tr><th rowspan="1">AVL Data IO element total IO count length</th><td>1 byte</td><td>2 bytes</td></tr><tr><th rowspan="1">AVL Data IO element IO count length</th><td>1 byte</td><td>2 bytes</td></tr><tr><th rowspan="1">AVL Data IO element AVL ID length</th><td>1 byte</td><td>2 bytes</td></tr><tr><th rowspan="1">Variable size IO elements</th><td>Does not include</td><td>Includes variable size elements</td></tr></tbody></table>
|
||||
|
||||
- **Codec 8 Extended protocol sending over TCP**
|
||||
- **AVL data packet**
|
||||
|
||||
The below table represents the AVL data packet structure:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">0x00000000 (Preamble)</th><th rowspan="1">Data Field Length</th><th rowspan="1">Codec ID</th><th rowspan="1">Number of Data 1</th><th colspan="1">AVL Data</th><th rowspan="1">Number of Data 2</th><th rowspan="1">CRC-16</th></tr><tr><td>4 bytes</td><td>4 bytes</td><td>1 byte</td><td>1 byte</td><td>X bytes</td><td>1 byte</td><td>4 bytes</td></tr></tbody></table>
|
||||
|
||||
**Preamble** – the packet starts with four zero bytes.
|
||||
**Data Field Length** – size is calculated starting from Codec ID to Number of Data 2.
|
||||
**Codec ID** – in Codec8 Extended it is always `0x8E`.
|
||||
**Number of Data 1** – a number that defines how many records are in the packet.
|
||||
**AVL Data** – actual data in the packet (more information below).
|
||||
**Number of Data 2** – a number that defines how many records are in the packet. This number must be the same as “Number of Data 1”.
|
||||
**CRC-16** – calculated from Codec ID to the Second Number of Data. CRC (Cyclic Redundancy Check) is an error-detecting code used to detect accidental changes to RAW data. For calculation we are using [CRC-16/IBM](https://wiki.teltonika-gps.com/view/Codec#CRC-16 "Codec").
|
||||
|
||||
**Note:** for [FMB640](https://wiki.teltonika-gps.com/view/FMB640 "FMB640"), [FMB641](https://wiki.teltonika-gps.com/view/FMB641 "FMB641"), [FMC640](https://wiki.teltonika-gps.com/view/FMC640 "FMC640"), and [FMM640](https://wiki.teltonika-gps.com/view/FMM640 "FMM640"), minimum AVL record size is 45 bytes (all IO elements disabled). The maximum AVL record size is 255 bytes. For other devices, the minimum AVL record size is 45 bytes (all IO elements disabled). Maximum AVL packet size is 1280 bytes.
|
||||
|
||||
- AVL Data
|
||||
|
||||
The below table represents the AVL Data structure:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">Timestamp</th><th rowspan="1">Priority</th><th rowspan="1">GPS Element</th><th rowspan="1">IO Element</th></tr><tr><td>8 bytes</td><td>1 byte</td><td>15 bytes</td><td>X bytes</td></tr></tbody></table>
|
||||
|
||||
**Timestamp** – a difference, in milliseconds, between the current time and midnight, January 1970 UTC (UNIX time).
|
||||
**Priority** – a field that defines AVL data priority (more information below).
|
||||
**GPS Element** – locational information of the AVL data (more information below).
|
||||
**IO Element** – additional configurable information from the device (more information below).
|
||||
|
||||
- Priority
|
||||
|
||||
The below table represents Priority values. Packet priority depends on device configuration and records sent.
|
||||
|
||||
<table><tbody><tr><th colspan="2">Priority</th></tr><tr><th rowspan="1">0</th><td>Low</td></tr><tr><th rowspan="1">1</th><td>High</td></tr><tr><th rowspan="1">2</th><td>Panic</td></tr></tbody></table>
|
||||
|
||||
- GPS element
|
||||
|
||||
The below table represents the GPS Element structure:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">Longitude</th><th rowspan="1">Latitude</th><th rowspan="1">Altitude</th><th rowspan="1">Angle</th><th rowspan="1">Satellites</th><th rowspan="1">Speed</th></tr><tr><td>4 bytes</td><td>4 bytes</td><td>2 bytes</td><td>2 bytes</td><td>1 byte</td><td>2 bytes</td></tr></tbody></table>
|
||||
|
||||
**Longitude** – east-west position.
|
||||
**Latitude** – north-south position.
|
||||
**Altitude** – meters above sea level.
|
||||
**Angle** – degrees from north pole.
|
||||
**Satellites** – number of satellites in use.
|
||||
**Speed** – speed calculated from satellites.
|
||||
|
||||
**Note:** Speed will be `0x0000` if GPS data is invalid.
|
||||
|
||||
Longitude and latitude are integer values built from degrees, minutes, seconds, and milliseconds by the formula:
|
||||
|
||||
Where:
|
||||
d – Degrees; m – Minutes; s – Seconds; ms – Milliseconds; p – Precision (10000000)
|
||||
If the longitude is in the west or latitude in the south, multiply the result by –1.
|
||||
|
||||
Note:
|
||||
To determine if the coordinate is negative, convert it to binary format and check the very first bit. If it is `0`, the coordinate is positive, if it is `1`, the coordinate is negative.
|
||||
|
||||
Example:
|
||||
Received value: `20 9C CA 80` converted to BIN: `00100000 10011100 11001010 10000000` first bit is 0, which means coordinate is positive converted to DEC: `547146368`. For more information see two‘s complement arithmetic.
|
||||
|
||||
- IO Element
|
||||
|
||||
<table><tbody><tr><th rowspan="1">Event IO ID</th><td>2 bytes</td><td rowspan="33"></td><td rowspan="33"><b>Event IO ID</b> – if data is acquired on the event – this field defines which IO property has changed and generated an event. For example, when if the Ignition state changes and it generates an event, the Event IO ID will be 0x00EF (AVL ID: 239). If it’s not an eventual record – the value is 0x0000.<br><p><b>N</b> – a total number of properties coming with record (N = N1 + N2 + N4 + N8).<br><b>N1</b> – number of properties, which length is 1 byte.<br><b>N2</b> – number of properties, which length is 2 bytes.<br><b>N4</b> – number of properties, which length is 4 bytes.<br><b>N8</b> – number of properties, which length is 8 bytes.<br><b>NX</b> – a number of properties, which length is defined by the length element. <b>N’th IO ID</b> - AVL ID.<br><b>N'th Lenght</b> - AVL ID value lenght.<br><b>N’th IO Value</b> - AVL ID value.<br></p></td></tr><tr><th rowspan="1">N of Total IO</th><td>2 bytes</td></tr><tr><th rowspan="1">N1 of One Byte IO</th><td>2 bytes</td></tr><tr><th rowspan="1">1’st IO ID</th><td>2 bytes</td></tr><tr><th rowspan="1">1’st IO Value</th><td>1 byte</td></tr><tr><td colspan="2">...</td></tr><tr><th rowspan="1">N1’th IO ID</th><td>2 bytes</td></tr><tr><th rowspan="1">N1’th IO Value</th><td>1 byte</td></tr><tr><th rowspan="1">N2 of Two Bytes</th><td>2 bytes</td></tr><tr><th rowspan="1">1’st IO ID</th><td>2 bytes</td></tr><tr><th rowspan="1">1’st IO Value</th><td>2 bytes</td></tr><tr><td colspan="2">...</td></tr><tr><th rowspan="1">N2’th IO ID</th><td>2 bytes</td></tr><tr><th rowspan="1">N2’th IO Value</th><td>2 bytes</td></tr><tr><th rowspan="1">N4 of Four Bytes</th><td>2 bytes</td></tr><tr><th rowspan="1">1’st IO ID</th><td>2 bytes</td></tr><tr><th rowspan="1">1’st IO Value</th><td>4 bytes</td></tr><tr><td colspan="2">...</td></tr><tr><th rowspan="1">N4’th IO ID</th><td>2 bytes</td></tr><tr><th rowspan="1">N4’th IO Value</th><td>4 byte</td></tr><tr><th rowspan="1">N8 of Eight Bytes</th><td>2 bytes</td></tr><tr><th rowspan="1">1’st IO ID</th><td>2 bytes</td></tr><tr><th rowspan="1">1’st IO Value</th><td>8 byte</td></tr><tr><td colspan="2">...</td></tr><tr><th rowspan="1">N8’IO ID</th><td>2 bytes</td></tr><tr><th rowspan="1">N8’IO Value</th><td>8 bytes</td></tr><tr><th rowspan="1">NX of X Byte IO</th><td>2 bytes</td></tr><tr><th rowspan="1">1’st IO ID</th><td>2 bytes</td></tr><tr><th rowspan="1">1’st IO Length</th><td>2 bytes</td></tr><tr><th rowspan="1">1’st IO Value</th><td>Defined by length</td></tr><tr><td colspan="2">...</td></tr><tr><th rowspan="1">NX’th IO ID</th><td>2 bytes</td></tr><tr><th rowspan="1">NX’th Length</th><td>2 bytes</td></tr><tr><th rowspan="1">NX’th Value</th><td>Defined by length</td></tr></tbody></table>
|
||||
|
||||
- **Communication with server**
|
||||
|
||||
Communication with the server is the same as with the Codec8 protocol, except in Codec8 Extended protocol Codec ID is 0x8E.
|
||||
|
||||
- Example:
|
||||
|
||||
The module connects to the server and sends IMEI:
|
||||
`000F333536333037303432343431303133`
|
||||
The server accepts the module:
|
||||
`01`
|
||||
The module sends data packet:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">AVL Data Packet Header</th><th rowspan="1">AVL Data Array</th><th rowspan="1">CRC-16</th></tr><tr><td>Four Zero Bytes – 0x00000000,<p>“AVL Data Array” length – 0x000000FE</p></td><td>Codec ID – 0x8E,<p>Number of Data – <b>0x02</b><br>(Encoded using continuous bit stream. The last byte is padded to align to the byte boundary)</p></td><td>CRC of “AVL Data Array”</td></tr><tr><td>00000000000000FE</td><td>8E <b>02</b>...(data elements)...<b>02</b></td><td>00008612</td></tr></tbody></table>
|
||||
|
||||
Server acknowledges data reception (2 data elements): **`00000002`**
|
||||
|
||||
- **Example**
|
||||
|
||||
The hexadecimal stream of AVL Data Packet receiving and response in this example is given in the hexadecimal form. The different fields of the packet are separated into different table columns for better readability and some of them are converted to ASCII values for better understanding.
|
||||
|
||||
Received data in the hexadecimal stream:
|
||||
`000000000000004A8E010000016B412CEE000100000000000000000000000000000000010005000100010100010011001D00010010015E2C880002000B000000003544C87 A000E000000001DD7E06A00000100002994`
|
||||
|
||||
Parsed data:
|
||||
|
||||
<table><tbody><tr><th colspan="3">AVL Data Packet</th></tr><tr><th colspan="2" rowspan="1">AVL Data Packet Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td rowspan="4"></td><td>Zero Bytes</td><td>00 00 00 00</td></tr><tr><td>Data Field Length</td><td>00 00 00 4A</td></tr><tr><td>Codec ID</td><td>8E</td></tr><tr><td>Number of Data 1 (Records)</td><td>01</td></tr><tr><td rowspan="25">AVL Data</td><td>Timestamp</td><td>00 00 01 6B 41 2C EE 00 (GMT: Monday, June 10, 2019, 11:36:32 AM)</td></tr><tr><td>Priority</td><td>01</td></tr><tr><td>Longitude</td><td>00 00 00 00</td></tr><tr><td>Latitude</td><td>00 00 00 00</td></tr><tr><td>Altitude</td><td>00 00</td></tr><tr><td>Angle</td><td>00 00</td></tr><tr><td>Satellites</td><td>00</td></tr><tr><td>Speed</td><td>00 00</td></tr><tr><td>Event IO ID</td><td>00 01</td></tr><tr><td>N of Total ID</td><td>00 05</td></tr><tr><td>N1 of One Byte IO</td><td>00 01</td></tr><tr><td>1’st IO ID</td><td>00 01 (AVL ID: 1, Name: DIN1)</td></tr><tr><td>1’st IO Value</td><td>01</td></tr><tr><td>N2 of Two Bytes IO</td><td>00 01</td></tr><tr><td>1’st IO ID</td><td>00 11 (AVL ID: 17, Name: Axis X)</td></tr><tr><td>1’st IO Value</td><td>00 1D</td></tr><tr><td>N4 of Four Bytes IO</td><td>00 01</td></tr><tr><td>1’st IO ID</td><td>00 10 (AVL ID: 16, Name: Total Odometer)</td></tr><tr><td>1’st IO Value</td><td>01 5E 2C 88</td></tr><tr><td>N8 of Eight Bytes IO</td><td>00 02</td></tr><tr><td>1’st IO ID</td><td>00 0B (AVL ID: 11, Name: ICCID1)</td></tr><tr><td>1’st IO Value</td><td>00 00 00 00 35 44 C8 7A</td></tr><tr><td>2’nd IO ID</td><td>00 0E (AVL ID: 14, Name: ICCID2)</td></tr><tr><td>2’nd IO Value</td><td>00 00 00 00 1D D7 E0 6A</td></tr><tr><td>NX of X Byte IO</td><td>00 00</td></tr><tr><td rowspan="2"></td><td>Number of Data 2 (Number of Total Records)</td><td>01</td></tr><tr><td>CRC-16</td><td>00 00 29 94</td></tr></tbody></table>
|
||||
|
||||
Server response: `00000001`
|
||||
|
||||
- **Codec8 Extended protocol sending over UDP**
|
||||
- **UDP channel protocol**
|
||||
|
||||
AVL data packet is the same as with Codec8, except Codec ID is changed to `0x8E`. AVL Data encoding was performed according to Codec8 Extended protocol.
|
||||
|
||||
- **Communication with server**
|
||||
|
||||
The module sends the UDP channel packet with an encapsulated AVL data packet. The server sends the UDP channel packet with an encapsulated response module that validates the AVL Packet ID and the Number of accepted AVL elements. If the server response is not received with a valid AVL Packet ID within configured timeout, the module can retry sending.
|
||||
|
||||
- Example:
|
||||
|
||||
The module sends the data:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">UDP Channel Header</th><th rowspan="1">AVL Packet Header</th><th rowspan="1">AVL Data Array</th></tr><tr><td>Length – 0x00FE,<p>Packet ID – 0xCAFE<br>Not Usable Byte – 0x01</p></td><td>AVL Packet ID – 0xDD,<p>IMEI Length – 0x000F<br>IMEI – 0x313233343536373839303132333435 (Encoded using continuous bit stream. The last byte is padded to align to the byte boundary)</p></td><td>Codec ID – 0x8E,<p>Number of Data – 0x02<br>(Encoded using continuous bit stream)</p></td></tr><tr><td>00FECAFE01</td><td>DD000F3133343536373839303132333435</td><td>8E02…(data elements)…02</td></tr></tbody></table>
|
||||
|
||||
The server must respond with an acknowledgment:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">UDP Channel Header</th><th rowspan="1">AVL Packet Acknowledgment</th></tr><tr><td>Length – 0x0005,<p>Packet ID – 0xCAFE, Not Usable Byte – 0x01</p></td><td>AVL Packet ID – 0xDD,<p>Number of Accepted Data – 0x02</p></td></tr><tr><td>0005CAFE01</td><td>DD02</td></tr></tbody></table>
|
||||
|
||||
- **Example**
|
||||
|
||||
The hexadecimal stream of AVL Data Packet receiving and response in this example is given in the hexadecimal form. The different fields of the packet are separated into different table columns for better readability and some of them are converted to ASCII values for better understanding.
|
||||
|
||||
Received data in the hexadecimal stream:
|
||||
`005FCAFE0107000F3335323039333038363430333635358E010000016B4F831C680100000000000000000000000000000000010005000100010100010011009D000100` `10015E2C880002000B000000003544C87A000E000000001DD7E06A000001`
|
||||
|
||||
Parsed:
|
||||
|
||||
<table><tbody><tr><th colspan="3">AVL Data Packet</th></tr><tr><th colspan="2" rowspan="1">AVL Data Packet Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td rowspan="3">UDP Channel Header</td><td>Length</td><td>00 5F</td></tr><tr><td>Packet ID</td><td>CA FE</td></tr><tr><td>Not usable byte</td><td>01</td></tr><tr><td rowspan="3">AVL Packet Header</td><td>AVL packet ID</td><td>07</td></tr><tr><td>IMEI Length</td><td>00 0F</td></tr><tr><td>IMEI</td><td>33 35 32 30 39 33 30 38 36 34 30 33 36 35 35</td></tr><tr><td rowspan="28">AVL Data Array</td><td>Codec ID</td><td>8E</td></tr><tr><td>Number of Data 1 (Records)</td><td>01</td></tr><tr><td>Timestamp</td><td>00 00 01 6B 4F 83 1C 68 (GMT: Thursday, June 13, 2019 6:25:21 AM)</td></tr><tr><td>Priority</td><td>01</td></tr><tr><td>Longitude</td><td>00 00 00 00</td></tr><tr><td>Latitude</td><td>00 00 00 00</td></tr><tr><td>Altitude</td><td>00 00</td></tr><tr><td>Angle</td><td>00 00</td></tr><tr><td>Satellites</td><td>00</td></tr><tr><td>Speed</td><td>00 00</td></tr><tr><td>Event IO ID</td><td>00 01</td></tr><tr><td>N of Total ID</td><td>00 05</td></tr><tr><td>N1 of One Byte IO</td><td>00 01</td></tr><tr><td>1’st IO ID</td><td>00 01 (AVL ID: 1, Name: DIN1)</td></tr><tr><td>1’st IO Value</td><td>00 01</td></tr><tr><td>N2 of Two Bytes IO</td><td>00 01</td></tr><tr><td>1’st IO ID</td><td>00 11 (AVL ID: 17, Name: Axis X)</td></tr><tr><td>1’st IO Value</td><td>00 1D</td></tr><tr><td>N4 of Four Bytes IO</td><td>00 01</td></tr><tr><td>1’st IO ID</td><td>00 10 (AVL ID: 16, Name: Total Odometer)</td></tr><tr><td>1’st IO Value</td><td>01 5E 2C 88</td></tr><tr><td>N8 of Eight Bytes IO</td><td>00 02</td></tr><tr><td>1’st IO ID</td><td>00 0B (AVL ID: 11, Name: ICCID1)</td></tr><tr><td>1’st IO Value</td><td>00 00 00 00 35 44 C8 7A</td></tr><tr><td>2’nd IO ID</td><td>00 0E (AVL ID: 14, Name: ICCID2)</td></tr><tr><td>2’nd IO Value</td><td>00 00 00 00 1D D7 E0 6A</td></tr><tr><td>NX of X Byte IO</td><td>00 00</td></tr><tr><td>Number of Data 2 (Records)</td><td>01</td></tr></tbody></table>
|
||||
|
||||
The server response in the hexadecimal stream: `0005CAFE010701`
|
||||
|
||||
Parsed:
|
||||
|
||||
<table><tbody><tr><th colspan="3">Server Response to AVL Data Packet</th></tr><tr><th colspan="2" rowspan="1">Server Response Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td rowspan="3">UDP Channel Header</td><td>Length</td><td>00 05</td></tr><tr><td>Packet ID</td><td>CA FE</td></tr><tr><td>Not usable byte</td><td>01</td></tr><tr><td rowspan="2">AVL Packet Acknowledgment</td><td>AVL packet ID</td><td>07</td></tr><tr><td>Number of Accepted Data</td><td>01</td></tr></tbody></table>
|
||||
|
||||
## Codec 16
|
||||
|
||||
- **Protocol overview**
|
||||
|
||||
Codec16 is using for [FMB630](https://wiki.teltonika-gps.com/view/FMB630 "FMB630") /FM63XY series devices. This protocol looks familiar like Codec8 but they have some differences. The main differences between them are shown in the table below:
|
||||
|
||||
<table><tbody><tr><th rowspan="1"></th><th rowspan="1">Codec8</th><th rowspan="1">Codec16</th></tr><tr><th rowspan="1">Codec ID</th><td>0x08</td><td>0x10</td></tr><tr><th rowspan="1">AVL Data IO element ID event length</th><td>1 byte</td><td>2 bytes</td></tr><tr><th rowspan="1">AVL Data IO element AVL ID length</th><td>1 byte</td><td>2 bytes</td></tr><tr><th rowspan="1">Generation Type</th><td>Not Using</td><td>Is Using</td></tr></tbody></table>
|
||||
|
||||
**Note:** Codec16 is supported from firmware – 00.03.xx and newer. ([FMB630](https://wiki.teltonika-gps.com/view/FMB630 "FMB630") /FM63XY) || AVL IDs that are higher than 255 will can be used only in the Codec16 protocol.
|
||||
|
||||
- **Codec 16 protocol sending over TCP**
|
||||
- **AVL data packet**
|
||||
|
||||
The below table represents the AVL data packet structure:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">0x00000000 (Preamble)</th><th rowspan="1">Data Field Length</th><th rowspan="1">Codec ID</th><th rowspan="1">Number of Data 1</th><th colspan="1">AVL Data</th><th rowspan="1">Number of Data 2</th><th rowspan="1">CRC-16</th></tr><tr><td>4 bytes</td><td>4 bytes</td><td>1 byte</td><td>1 byte</td><td>X bytes</td><td>1 byte</td><td>4 bytes</td></tr></tbody></table>
|
||||
|
||||
**Preamble** – the packet starts with four zero bytes.
|
||||
**Data Field Length** – size is calculated starting from Codec ID to Number of Data 2.
|
||||
**Codec ID** – in Codec16 it is always 0x10.
|
||||
**Number of Data 1** – a number that defines how many records are in the packet.
|
||||
**AVL Data** – actual data in the packet (more information below).
|
||||
**Number of Data 2** – a number that defines how many records are in the packet. This number must be the same as “Number of Data 1”.
|
||||
**CRC-16** – calculated from Codec ID to the Second Number of Data. CRC (Cyclic Redundancy Check) is an error-detecting code used to detect accidental changes to RAW data. For calculation we are using [CRC-16/IBM](https://wiki.teltonika-gps.com/view/Codec#CRC-16 "Codec").
|
||||
|
||||
**Note:** for [FMB630](https://wiki.teltonika-gps.com/view/FMB630 "FMB630") and FM63XY, the minimum AVL record size is 45 bytes (all IO elements disabled). The maximum AVL record size is 255 bytes.
|
||||
|
||||
- AVL Data
|
||||
|
||||
The below table represents the AVL Data structure:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">Timestamp</th><th rowspan="1">Priority</th><th rowspan="1">GPS Element</th><th rowspan="1">IO Element</th></tr><tr><td>8 bytes</td><td>1 byte</td><td>15 bytes</td><td>X bytes</td></tr></tbody></table>
|
||||
|
||||
**Timestamp** – a difference, in milliseconds, between the current time and midnight, January 1970 UTC (UNIX time).
|
||||
**Priority** – a field that defines AVL data priority (more information below).
|
||||
**GPS Element** – location information of the AVL data (more information below).
|
||||
**IO Element** – additional configurable information from the device (more information below).
|
||||
|
||||
- Priority
|
||||
|
||||
The below table represents Priority values. Packet priority depends on device configuration and records sent.
|
||||
|
||||
<table><tbody><tr><th colspan="2">Priority</th></tr><tr><th rowspan="1">0</th><td>Low</td></tr><tr><th rowspan="1">1</th><td>High</td></tr><tr><th rowspan="1">2</th><td>Panic</td></tr></tbody></table>
|
||||
|
||||
- GPS element
|
||||
|
||||
The below table represents the GPS Element structure:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">Longitude</th><th rowspan="1">Latitude</th><th rowspan="1">Altitude</th><th rowspan="1">Angle</th><th rowspan="1">Satellites</th><th rowspan="1">Speed</th></tr><tr><td>4 bytes</td><td>4 bytes</td><td>2 bytes</td><td>2 bytes</td><td>1 byte</td><td>2 bytes</td></tr></tbody></table>
|
||||
|
||||
**Longitude** – east-west position.
|
||||
**Latitude** – north-south position.
|
||||
**Altitude** – meters above sea level.
|
||||
**Angle** – degrees from north pole.
|
||||
**Satellites** – number of satellites in use.
|
||||
**Speed** – speed calculated from satellites.
|
||||
|
||||
**Note:** Speed will be `0x0000` if GPS data is invalid.
|
||||
|
||||
Longitude and latitude are integer values built from degrees, minutes, seconds, and milliseconds by the formula:
|
||||
|
||||
Where:
|
||||
d – Degrees; m – Minutes; s – Seconds; ms – Milliseconds; p – Precision (10000000)
|
||||
If the longitude is in the west or latitude in the south, multiply the result by –1.
|
||||
|
||||
Note:
|
||||
To determine if the coordinate is negative, convert it to binary format and check the very first bit. If it is `0`, the coordinate is positive, if it is `1`, the coordinate is negative.
|
||||
|
||||
Example:
|
||||
Received value: `20 9C CA 80` converted to BIN: `00100000 10011100 11001010 10000000` first bit is 0, which means coordinate is positive converted to DEC: `547146368`. For more information see two‘s complement arithmetic.
|
||||
|
||||
- IO Element
|
||||
|
||||
<table><tbody><tr><th rowspan="1">Event IO ID</th><td>2 bytes</td><td rowspan="27"></td><td rowspan="27"><b>Event IO ID</b> – if data is acquired on the event – this field defines which IO property has changed and generated an event. For example, when if the Ignition state changes and it generates an event, the Event IO ID will be 0xEF (AVL ID: 239). If it’s not an eventual record – the value is 0.<br><p><b>Generation type</b> - data event generation type. More information about it you can find here.<br><b>N</b> – a total number of properties coming with record (N = N1 + N2 + N4 + N8).<br><b>N1</b> – number of properties, which length is 1 byte.<br><b>N2</b> – number of properties, which length is 2 bytes.<br><b>N4</b> – number of properties, which length is 4 bytes.<br><b>N8</b> – number of properties, which length is 8 bytes.<br><b>N’th IO ID</b> - AVL ID.<br><b>N’th IO Value</b> - AVL ID value.<br></p></td></tr><tr><th rowspan="1">Generation Type</th><td>1 byte</td></tr><tr><th rowspan="1">N of Total IO</th><td>1 byte</td></tr><tr><th rowspan="1">N1 of One Byte IO</th><td>1 byte</td></tr><tr><th rowspan="1">1’st IO ID</th><td>2 bytes</td></tr><tr><th rowspan="1">1’st IO Value</th><td>1 byte</td></tr><tr><td colspan="2">...</td></tr><tr><th rowspan="1">N1’th IO ID</th><td>2 bytes</td></tr><tr><th rowspan="1">N1’th IO Value</th><td>1 byte</td></tr><tr><th rowspan="1">N2 of Two Bytes</th><td>1 byte</td></tr><tr><th rowspan="1">1’st IO ID</th><td>2 bytes</td></tr><tr><th rowspan="1">1’st IO Value</th><td>2 bytes</td></tr><tr><td colspan="2">...</td></tr><tr><th rowspan="1">N2’th IO ID</th><td>2 bytes</td></tr><tr><th rowspan="1">N2’th IO Value</th><td>2 bytes</td></tr><tr><th rowspan="1">N4 of Four Bytes</th><td>1 byte</td></tr><tr><th rowspan="1">1’st IO ID</th><td>2 bytes</td></tr><tr><th rowspan="1">1’st IO Value</th><td>4 bytes</td></tr><tr><td colspan="2">...</td></tr><tr><th rowspan="1">N4’th IO ID</th><td>2 bytes</td></tr><tr><th rowspan="1">N4’th IO Value</th><td>4 byte</td></tr><tr><th rowspan="1">N8 of Eight Bytes</th><td>1 byte</td></tr><tr><th rowspan="1">1’st IO ID</th><td>2 bytes</td></tr><tr><th rowspan="1">1’st IO Value</th><td>8 byte</td></tr><tr><td colspan="2">...</td></tr><tr><th rowspan="1">N8’IO ID</th><td>2 bytes</td></tr><tr><th rowspan="1">N8’IO Value</th><td>8 bytes</td></tr></tbody></table>
|
||||
|
||||
- Generation type
|
||||
|
||||
<table><tbody><tr><th rowspan="1">Value</th><th rowspan="1">Record Created</th></tr><tr><td>0</td><td>On Exit</td></tr><tr><td>1</td><td>On Entrance</td></tr><tr><td>2</td><td>On Both</td></tr><tr><td>3</td><td>Reserved</td></tr><tr><td>4</td><td>Hysteresis</td></tr><tr><td>5</td><td>On Change</td></tr><tr><td>6</td><td>Eventual</td></tr><tr><td>7</td><td>Periodical</td></tr></tbody></table>
|
||||
|
||||
- **Communication with server**
|
||||
|
||||
Communication with the server is the same as with Codec8 protocol, except in Codec16 protocol Codec ID is `0x10` and has generation type.
|
||||
|
||||
- Example:
|
||||
|
||||
The module connects to the server and sends IMEI:
|
||||
`000F333536333037303432343431303133`
|
||||
The server accepts the module:
|
||||
`01`
|
||||
The module sends data packet:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">AVL Data Packet Header</th><th rowspan="1">AVL Data Array</th><th rowspan="1">CRC-16</th></tr><tr><td>Four Zero Bytes – 0x00000000,<p>“AVL Data Array” length – 0x000000FE</p></td><td>Codec ID – 0x10,<p>Number of Data – <b>0x02</b><br>(Encoded using continuous bit stream. The last byte is padded to align to the byte boundary)</p></td><td>CRC of “AVL Data Array”</td></tr><tr><td>00000000000000FE</td><td>10 <b>02</b>...(data elements)...<b>02</b></td><td>00008612</td></tr></tbody></table>
|
||||
|
||||
Server acknowledges data reception (2 data elements): **`00000002`**
|
||||
|
||||
- **Example**
|
||||
|
||||
The hexadecimal stream of AVL Data Packet receiving and response in this example is given in the hexadecimal form. The different fields of the packet are separated into different table columns for better readability and some of them are converted to ASCII values for better understanding.
|
||||
|
||||
Received data in the hexadecimal stream:
|
||||
`000000000000005F10020000016BDBC7833000000000000000000000000000000000000B05040200010000030002000B00270042563A00000000016BDBC78718` `00000000000000000000000000000000000B05040200010000030002000B00260042563A00000200005FB3`
|
||||
|
||||
Parsed data:
|
||||
|
||||
<table><tbody><tr><th colspan="3">AVL Data Packet</th></tr><tr><th colspan="2" rowspan="1">AVL Data Packet Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td rowspan="4"></td><td>Zero Bytes</td><td>00 00 00 00</td></tr><tr><td>Data Field Length</td><td>00 00 00 5F</td></tr><tr><td>Codec ID</td><td>10</td></tr><tr><td>Number of Data 1 (Records)</td><td>02</td></tr><tr><td rowspan="23">AVL Data<p>(1'st record)</p></td><td>Timestamp</td><td>00 00 01 6B DB C7 83 30 (GMT: Wednesday, July 10, 2019, 12:06:54 PM)</td></tr><tr><td>Priority</td><td>01</td></tr><tr><td>Longitude</td><td>00 00 00 00</td></tr><tr><td>Latitude</td><td>00 00 00 00</td></tr><tr><td>Altitude</td><td>00 00</td></tr><tr><td>Angle</td><td>00 00</td></tr><tr><td>Satellites</td><td>00</td></tr><tr><td>Speed</td><td>00 00</td></tr><tr><td>Event IO ID</td><td>00 0B</td></tr><tr><td>Generation Type</td><td>05</td></tr><tr><td>N of Total ID</td><td>04</td></tr><tr><td>N1 of One Byte IO</td><td>02</td></tr><tr><td>1’st IO ID</td><td>00 01 (AVL ID: 1, Name: DIN1)</td></tr><tr><td>1’st IO Value</td><td>00</td></tr><tr><td>2’nd IO ID</td><td>00 03 (AVL ID: 3, Name: DIN3)</td></tr><tr><td>2’nd IO Value</td><td>00</td></tr><tr><td>N2 of Two Bytes IO</td><td>02</td></tr><tr><td>1’st IO ID</td><td>00 0B (AVL ID: 11, Name: ICCID1)</td></tr><tr><td>1’st IO Value</td><td>00 27</td></tr><tr><td>2’nd IO ID</td><td>00 42 (AVL ID: 66, Name: External Voltage)</td></tr><tr><td>2’nd IO Value</td><td>56 3A</td></tr><tr><td>N4 of Four Bytes IO</td><td>00</td></tr><tr><td>N8 of Eight Bytes IO</td><td>00</td></tr><tr><td rowspan="23">AVL Data<p>(2'nd record)</p></td><td>Timestamp</td><td>00 00 01 6B DB C7 87 18 (GMT: Wednesday, July 10, 2019, 12:06:55 PM)</td></tr><tr><td>Priority</td><td>01</td></tr><tr><td>Longitude</td><td>00 00 00 00</td></tr><tr><td>Latitude</td><td>00 00 00 00</td></tr><tr><td>Altitude</td><td>00 00</td></tr><tr><td>Angle</td><td>00 00</td></tr><tr><td>Satellites</td><td>00</td></tr><tr><td>Speed</td><td>00 00</td></tr><tr><td>Event IO ID</td><td>00 0B</td></tr><tr><td>Generation Type</td><td>05</td></tr><tr><td>N of Total ID</td><td>04</td></tr><tr><td>N1 of One Byte IO</td><td>02</td></tr><tr><td>1’st IO ID</td><td>00 01 (AVL ID: 1, Name: DIN1)</td></tr><tr><td>1’st IO Value</td><td>00</td></tr><tr><td>2’nd IO ID</td><td>00 03 (AVL ID: 3, Name: DIN3)</td></tr><tr><td>2’nd IO Value</td><td>00</td></tr><tr><td>N2 of Two Bytes IO</td><td>02</td></tr><tr><td>1’st IO ID</td><td>00 0B (AVL ID: 11, Name: ICCID1)</td></tr><tr><td>1’st IO Value</td><td>00 26</td></tr><tr><td>2’nd IO ID</td><td>00 42 (AVL ID: 66, Name: External Voltage)</td></tr><tr><td>2’nd IO Value</td><td>56 3A</td></tr><tr><td>N4 of Four Bytes IO</td><td>00</td></tr><tr><td>N8 of Eight Bytes IO</td><td>00</td></tr><tr><td rowspan="2"></td><td>Number of Data 2 (Number of Total Records)</td><td>02</td></tr><tr><td>CRC-16</td><td>00 00 5F B3</td></tr></tbody></table>
|
||||
|
||||
Server response: `00000002`
|
||||
|
||||
- **Codec16 protocol sending over UDP**
|
||||
- **UDP channel protocol**
|
||||
|
||||
AVL data packet is the same as with Codec8, except Codec ID is changed to `0x10`. AVL Data encoding is performed according to the Codec16 protocol.
|
||||
|
||||
- **Communication with server**
|
||||
|
||||
The module sends the UDP channel packet with an encapsulated AVL data packet. The server sends the UDP channel packet with an encapsulated response module that validates the AVL Packet ID and the Number of accepted AVL elements. If the server responds with a valid AVL Packet ID that is not received within configured timeout, the module can retry sending.
|
||||
|
||||
- Example:
|
||||
|
||||
The module sends the data:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">UDP Channel Header</th><th rowspan="1">AVL Packet Header</th><th rowspan="1">AVL Data Array</th></tr><tr><td>Length – 0x00FE,<p>Packet ID – 0xCAFE<br>Not Usable Byte – 0x01</p></td><td>AVL Packet ID – 0xDD,<p>IMEI Length – 0x000F<br>IMEI – 0x313233343536373839303132333435 (Encoded using continuous bit stream. The last byte is padded to align to the byte boundary)</p></td><td>Codec ID – 0x10,<p>Number of Data – 0x02<br>(Encoded using continuous bit stream)</p></td></tr><tr><td>00FECAFE01</td><td>DD000F3133343536373839303132333435</td><td>1002…(data elements)…02</td></tr></tbody></table>
|
||||
|
||||
The server must respond with an acknowledgment:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">UDP Channel Header</th><th rowspan="1">AVL Packet Acknowledgment</th></tr><tr><td>Length – 0x0005,<p>Packet ID – 0xCAFE, Not Usable Byte – 0x01</p></td><td>AVL Packet ID – 0xDD,<p>Number of Accepted Data – 0x02</p></td></tr><tr><td>0005CAFE01</td><td>DD02</td></tr></tbody></table>
|
||||
|
||||
- **Example**
|
||||
|
||||
The hexadecimal stream of AVL Data Packet receiving and response in this example is given in the hexadecimal form. The different fields of the packet are separated into different table columns for better readability and some of them are converted to ASCII values for better understanding.
|
||||
|
||||
Received data in the hexadecimal stream:
|
||||
`015BCAFE0101000F33353230393430383532333135393210070000015117E40FE80000000000000000000000000000000000EF05050400010000030000B4000` `0EF01010042111A000001`
|
||||
|
||||
Parsed:
|
||||
|
||||
<table><tbody><tr><th colspan="3">AVL Data Packet</th></tr><tr><th colspan="2" rowspan="1">AVL Data Packet Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td rowspan="3">UDP Channel Header</td><td>Length</td><td>01 5B</td></tr><tr><td>Packet ID</td><td>CA FE</td></tr><tr><td>Not usable byte</td><td>01</td></tr><tr><td rowspan="3">AVL Packet Header</td><td>AVL packet ID</td><td>07</td></tr><tr><td>IMEI Length</td><td>00 0F</td></tr><tr><td>IMEI</td><td>33 35 32 30 39 34 30 38 35 32 33 31 35 39 32</td></tr><tr><td rowspan="28">AVL Data Array</td><td>Codec ID</td><td>10</td></tr><tr><td>Number of Data 1 (Records)</td><td>01</td></tr><tr><td>Timestamp</td><td>00 00 01 51 17 E4 0F E8 (GMT: Wednesday, November 18, 2015, 12:00:01 AM)</td></tr><tr><td>Priority</td><td>00</td></tr><tr><td>Longitude</td><td>00 00 00 00</td></tr><tr><td>Latitude</td><td>00 00 00 00</td></tr><tr><td>Altitude</td><td>00 00</td></tr><tr><td>Angle</td><td>00 00</td></tr><tr><td>Satellites</td><td>00</td></tr><tr><td>Speed</td><td>00 00</td></tr><tr><td>Event IO ID</td><td>00 EF</td></tr><tr><td>Generation type</td><td>05</td></tr><tr><td>N of Total ID</td><td>05</td></tr><tr><td>N1 of One Byte IO</td><td>04</td></tr><tr><td>1’st IO ID</td><td>00 01 (AVL ID: 1, Name: DIN1)</td></tr><tr><td>1’st IO Value</td><td>00</td></tr><tr><td>2’nd IO ID</td><td>00 03 (AVL ID: 3, Name: DIN3)</td></tr><tr><td>2’nd IO Value</td><td>00</td></tr><tr><td>3’rd IO ID</td><td>00 B4 (AVL ID: 180, Name: DOUT2)</td></tr><tr><td>3’rd IO Value</td><td>00</td></tr><tr><td>4’th IO ID</td><td>00 EF (AVL ID: 239, Name: Ignition)</td></tr><tr><td>4’th IO Value</td><td>00</td></tr><tr><td>N2 of Two Bytes IO</td><td>01</td></tr><tr><td>1’st IO ID</td><td>42 (AVL ID: 66, Name: External Voltage)</td></tr><tr><td>1’st IO Value</td><td>11 1A</td></tr><tr><td>N4 of Four Bytes IO</td><td>00</td></tr><tr><td>N8 of Eight Bytes IO</td><td>00</td></tr><tr><td>Number of Data 2 (Number of Total Records)</td><td>01</td></tr></tbody></table>
|
||||
|
||||
The server response in the hexadecimal stream: `0005CAFE010701`
|
||||
|
||||
Parsed:
|
||||
|
||||
<table><tbody><tr><th colspan="3">Server Response to AVL Data Packet</th></tr><tr><th colspan="2" rowspan="1">Server Response Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td rowspan="3">UDP Channel Header</td><td>Length</td><td>00 05</td></tr><tr><td>Packet ID</td><td>CA FE</td></tr><tr><td>Not usable byte</td><td>01</td></tr><tr><td rowspan="2">AVL Packet Acknowledgment</td><td>AVL packet ID</td><td>07</td></tr><tr><td>Number of Accepted Data</td><td>01</td></tr></tbody></table>
|
||||
|
||||
## Differences between Codec 8, Codec 8 Extended and Codec 16
|
||||
|
||||
In the table below you will see differences between Codec8, Codec8 Extended, and Codec16.
|
||||
|
||||
<table><tbody><tr><th rowspan="1"></th><th rowspan="1">Codec8</th><th rowspan="1">Codec8 Extended</th><th rowspan="1">Codec16</th></tr><tr><th rowspan="1">Codec ID</th><td>0x08</td><td>0x8E</td><td>0x10</td></tr><tr><th rowspan="1">AVL Data IO element length</th><td>1 byte</td><td>2 bytes</td><td>2 bytes</td></tr><tr><th rowspan="1">AVL Data IO element total IO count length</th><td>1 byte</td><td>2 bytes</td><td>2 bytes</td></tr><tr><th rowspan="1">Generation Type</th><td>Is Using</td><td>Not Using</td><td>Is Using</td></tr><tr><th rowspan="1">AVL Data IO element IO count length</th><td>1 byte</td><td>2 bytes</td><td>1 byte</td></tr><tr><th rowspan="1">AVL Data IO element AVL ID length</th><td>1 byte</td><td>2 bytes</td><td>2 bytes</td></tr><tr><th rowspan="1">Variable size IO elements</th><td>Does not include</td><td>Includes variable size elements</td><td>Does not include</td></tr></tbody></table>
|
||||
|
||||
## Codec for communication over GPRS messages
|
||||
|
||||
In this chapter, you will find information about every Codec protocol which are used for communication over GPRS messages and the differences between them.
|
||||
|
||||
## Codec 12
|
||||
|
||||
- **About Codec12**
|
||||
|
||||
Codec12 is the original and main Teltonika protocol for device-server communication over GPRS messages. Codec12 GPRS commands can be used for sending configuration, debug, digital outputs control commands, or other (special purpose commands on special firmware versions). This protocol is also necessary for using [FMB630](https://wiki.teltonika-gps.com/view/FMB630 "FMB630") / [FM6300](https://wiki.teltonika-gps.com/view/FM6300 "FM6300") /FM5300/FM5500/FM4200 features like Garmin, LCD communication, and COM TCP Link Mode.
|
||||
|
||||
- **GPRS command session**
|
||||
|
||||
The following figure shows how the GRPS command session is started over TCP.
|
||||
 First, the Teltonika device opens the GPRS session and sends AVL data to the server (refer to device protocols). Once all records are sent and the correct sent data array acknowledgment is received by the device then GPRS commands in Hex can be sent to the device.
|
||||
The ACK (acknowledgment of IMEI from server) is a one-byte constant 0x01. The acknowledgment of each data array send from the device is four bytes integer – a number of received records.
|
||||
Note, that the GPRS session should remain active between the device and server, while GPRS commands are sent. For this reason, active datalink timeout (global parameters in device configuration) is recommended to be set to 259200 (maximum value).
|
||||
|
||||
- **General Codec12 message structure**
|
||||
|
||||
The following diagram shows the basic structure of Codec12 messages.
|
||||
|
||||
**Command message structure:**
|
||||
|
||||
<table><tbody><tr><th rowspan="1">0x00000000 (Preamble)</th><th rowspan="1">Data Size</th><th rowspan="1">Codec ID</th><th rowspan="1">Command Quantity 1</th><th colspan="1">Type (0x05)</th><th rowspan="1">Command Size</th><th rowspan="1">Command</th><th rowspan="1">Command Quantity 2</th><th rowspan="1">CRC-16</th></tr><tr><td>4 bytes</td><td>4 bytes</td><td>1 byte</td><td>1 byte</td><td>1 byte</td><td>4 bytes</td><td>X bytes</td><td>1 byte</td><td>4 bytes</td></tr></tbody></table>
|
||||
|
||||
**Response message structure:**
|
||||
|
||||
<table><tbody><tr><th rowspan="1">0x00000000 (Preamble)</th><th rowspan="1">Data Size</th><th rowspan="1">Codec ID</th><th rowspan="1">Response Quantity 1</th><th colspan="1">Type (0x06)</th><th rowspan="1">Response Size</th><th rowspan="1">Response</th><th rowspan="1">Response Quantity 2</th><th rowspan="1">CRC-16</th></tr><tr><td>4 bytes</td><td>4 bytes</td><td>1 byte</td><td>1 byte</td><td>1 byte</td><td>4 bytes</td><td>X bytes</td><td>1 byte</td><td>4 bytes</td></tr></tbody></table>
|
||||
|
||||
**Preamble** - the packet starts with four zero bytes.
|
||||
**Data Size** - size is calculated from the Codec ID field to the second command or response quantity field.
|
||||
**Codec ID** - in Codec12 it is always `0x0C`.
|
||||
**Command/Response Quantity 1** - it is ignored when parsing the message.
|
||||
**Type** - it can be 0x05 to denote command or 0x06 to denote response.
|
||||
**Command/Response Size** – command or response length.
|
||||
**Command/Response** – command or response in HEX.
|
||||
**Command/Response Quantity 2** - a byte that defines how many records (commands or responses) are in the packet. This byte will not be parsed but it’s recommended that it should contain the same value as Command/Response Quantity 1.
|
||||
**CRC-16** – calculated from Codec ID to the Command Quantity 2. CRC (Cyclic Redundancy Check) is an error-detecting code used to detect accidental changes to RAW data. For calculation we are using [CRC-16/IBM](https://wiki.teltonika-gps.com/view/Codec#CRC-16 "Codec").
|
||||
|
||||
Note that the difference between commands and responses is the message type field: `0x05` means command and `0x06` means response.
|
||||
|
||||
- **Command coding table**
|
||||
|
||||
Command has to be converted from ASCII characters (char) to hexadecimal (HEX):
|
||||
|
||||
|
||||
- **Command parsing example**
|
||||
|
||||
The hexadecimal stream of command and answer in this example is given in the hexadecimal form. The different fields of the message are separated into different table columns for better readability and understanding.
|
||||
|
||||
- **GPRS commands examples**
|
||||
|
||||
The hexadecimal stream of GPRS command and answer in these examples are given in the hexadecimal form. The different fields of messages are separated into different table columns for better readability and some of them are converted to ASCII values for better understanding.
|
||||
|
||||
**1'st example:** Sending *[getinfo](https://wiki.teltonika-gps.com/view/FMB_getinfo "FMB getinfo")* SMS command via GPRS Codec12
|
||||
|
||||
Server request in the hexadecimal stream:
|
||||
`000000000000000F0C010500000007676574696E666F0100004312`
|
||||
|
||||
Parsed:
|
||||
|
||||
<table><tbody><tr><th colspan="2">Server Command</th></tr><tr><th rowspan="1">Server Command Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td>Zero Bytes</td><td>00 00 00 00</td></tr><tr><td>Data Size</td><td>00 00 00 0F</td></tr><tr><td>Codec ID</td><td>0C</td></tr><tr><td>Command Quantity 1</td><td>01</td></tr><tr><td>Command Type</td><td>05</td></tr><tr><td>Command Size</td><td>00 00 00 07</td></tr><tr><td>Command</td><td>67 65 74 69 6E 66 6F</td></tr><tr><td>Command Quantity 2</td><td>01</td></tr><tr><td>CRC-16</td><td>00 00 43 12</td></tr></tbody></table>
|
||||
|
||||
Note that Server Command converted from HEX to ASCII means *[getinfo](https://wiki.teltonika-gps.com/view/FMB_getinfo "FMB getinfo")*
|
||||
|
||||
Device response in the hexadecimal stream:
|
||||
`00000000000000900C010600000088494E493A323031392F372F323220373A3232205254433A323031392F372F323220373A3533205253543A32204552523A` `312053523A302042523A302043463A302046473A3020464C3A302054553A302F302055543A3020534D533A30204E4F4750533A303A3330204750533A312053` `41543A302052533A332052463A36352053463A31204D443A30010000C78F`
|
||||
|
||||
Parsed:
|
||||
|
||||
<table><tbody><tr><th colspan="2">Device Answer</th></tr><tr><th rowspan="1">Device Answer Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td>Zero Bytes</td><td>00 00 00 00</td></tr><tr><td>Data Size</td><td>00 00 00 90</td></tr><tr><td>Codec ID</td><td>0C</td></tr><tr><td>Response Quantity 1</td><td>01</td></tr><tr><td>Response Type</td><td>06</td></tr><tr><td>Response Size</td><td>00 00 00 88</td></tr><tr><td>Response</td><td>49 4E 49 3A 32 30 31 39 2F 37 2F 32 32 20 37 3A 32 32 20 52 54 43 3A 32 30 31 39 2F 37 2F 32 32 20 37 3A 35 33 20 52 53 54 3A 32 20 45 52 52 3A 31 20 53 52 3A 30 20 42 52 3A 30 20 43 46 3A 30 20 46 47 3A 30 20 46 4C 3A 30 20 54 55 3A 30 2F 30 20 55 54 3A 30 20 53 4D 53 3A 30 20 4E 4F 47 50 53 3A 30 3A 33 30 20 47 50 53 3A 31 20 53 41 54 3A 30 20 52 53 3A 33 20 52 46 3A 36 35 20 53 46 3A 31 20 4D 44 3A 30</td></tr><tr><td>Response Quantity 2</td><td>01</td></tr><tr><td>CRC-16</td><td>00 00 C7 8F</td></tr></tbody></table>
|
||||
|
||||
Note that Device Response converted from HEX to ASCII means:
|
||||
*INI:2019/7/22 7:22 RTC:2019/7/22 7:53 RST:2 ERR:1 SR:0 BR:0 CF:0 FG:0 FL:0 TU:0/0 UT:0 [SMS:0](sms:0) NOGPS:0:30 GPS:1 SAT:0 RS:3 RF:65 SF:1 MD:0*
|
||||
|
||||
**2'nd example:** Sending *[getio](https://wiki.teltonika-gps.com/view/FMB_getio "FMB getio")* SMS command via GPRS Codec12
|
||||
|
||||
Server request in the hexadecimal stream:
|
||||
`000000000000000D0C010500000005676574696F01000000CB`
|
||||
|
||||
Parsed:
|
||||
|
||||
<table><tbody><tr><th colspan="2">Server Command</th></tr><tr><th rowspan="1">Server Command Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td>Zero Bytes</td><td>00 00 00 00</td></tr><tr><td>Data Size</td><td>00 00 00 0D</td></tr><tr><td>Codec ID</td><td>0C</td></tr><tr><td>Command Quantity 1</td><td>01</td></tr><tr><td>Command Type</td><td>05</td></tr><tr><td>Command Size</td><td>00 00 00 05</td></tr><tr><td>Command</td><td>67 65 74 69 6F</td></tr><tr><td>Command Quantity 2</td><td>01</td></tr><tr><td>CRC-16</td><td>00 00 00 CB</td></tr></tbody></table>
|
||||
|
||||
Note that Server Command converted from HEX to ASCII means *[getio](https://wiki.teltonika-gps.com/view/FMB_getio "FMB getio")*
|
||||
|
||||
Device response in the hexadecimal stream:
|
||||
`00000000000000370C01060000002F4449313A31204449323A30204449333A302041494E313A302041494E323A313639323420444F313A3020444F323A3101000066E3`
|
||||
|
||||
Parsed:
|
||||
|
||||
<table><tbody><tr><th colspan="2">Device Answer</th></tr><tr><th rowspan="1">Device Answer Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td>Zero Bytes</td><td>00 00 00 00</td></tr><tr><td>Data Size</td><td>00 00 00 37</td></tr><tr><td>Codec ID</td><td>0C</td></tr><tr><td>Response Quantity 1</td><td>01</td></tr><tr><td>Response Type</td><td>06</td></tr><tr><td>Response Size</td><td>00 00 00 2F</td></tr><tr><td>Response</td><td>44 49 31 3A 31 20 44 49 32 3A 30 20 44 49 33 3A 30 20 41 49 4E 31 3A 30 20 41 49 4E 32 3A 31 36 39 32 34 20 44 4F 31 3A 30 20 44 4F 32 3A 31</td></tr><tr><td>Response Quantity 2</td><td>01</td></tr><tr><td>CRC-16</td><td>00 00 66 E3</td></tr></tbody></table>
|
||||
|
||||
Note that Device Response converted from HEX to ASCII means:
|
||||
*DI1:1 DI2:0 DI3:0 AIN1:0 AIN2:16924 DO1:0 DO2:1*
|
||||
|
||||
- **Communication with server**
|
||||
|
||||
The GSM/GPRS commands can be sent from a terminal program. We recommend using Hercules (in TCP server mode). Simply write the command into the Hercules Send field, check the HEX box and click Send button. Note that the TCP server must be listening on a specified port (see Port field and Listen button below).
|
||||
|
||||
|
||||
|
||||
- **FMXX and Codec12 functionality**
|
||||
- **Garmin**
|
||||
|
||||
All information is provided in the “FMXX and Garmin development.pdf” document.
|
||||
|
||||
- **COM TCP Link Mode**
|
||||
|
||||
All information is provided in the “FMxx TCP Link mode test instructions.pdf” document.
|
||||
|
||||
## Codec 13
|
||||
|
||||
- **About Codec13**
|
||||
|
||||
Codec13 is the original Teltonika protocol for device-server communication over GPRS messages and it is based on the Codec12 protocol. The main differences of Codec13 are that timestamp is used in messages and communication is one way only (Codec13 is used for Device -> Server sending).
|
||||
|
||||
- **General Codec13 message structure**
|
||||
|
||||
The following diagram shows the basic structure of Codec 13 messages:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">0x00000000 (Preamble)</th><th rowspan="1">Data Size</th><th rowspan="1">Codec ID</th><th rowspan="1">Command Quantity 1</th><th colspan="1">Type</th><th rowspan="1">Command Size</th><th rowspan="1">Timestamp</th><th rowspan="1">Command</th><th rowspan="1">Command Quantity 2</th><th rowspan="1">CRC-16</th></tr><tr><td>4 bytes</td><td>4 bytes</td><td>1 byte</td><td>1 byte</td><td>1 byte</td><td>4 bytes</td><td>4 bytes</td><td>X bytes</td><td>1 byte</td><td>4 bytes</td></tr></tbody></table>
|
||||
|
||||
**Preamble** – the packet starts with a preamble field (four zero bytes).
|
||||
**Data Size** – size is calculated from the Codec ID field to the second Command Quantity field.
|
||||
**Codec ID** – in Codec13 it is always `0x0D`.
|
||||
**Command Quantity 1** – `0x01`, it is ignored when parsing the message.
|
||||
**Command Type** – it is always `0x06` since the packet is direction is FM->Server.
|
||||
**Command Size** – command size field includes the size of the timestamp too, so it is equal to the size of the payload + the size of the timestamp.
|
||||
**Timestamp** – a difference, in milliseconds, between the current time and midnight, January 1970 UTC (UNIX time).
|
||||
**Command** – actual received data.
|
||||
**Command Quantity 2** – a byte that defines how many records (commands) are in the packet. This byte will not be parsed but it’s recommended that it should contain the same value as Command/Response Quantity 1.
|
||||
**CRC-16** – calculated from Codec ID to the Second Number of Data. CRC (Cyclic Redundancy Check) is an error-detecting code used to detect accidental changes to RAW data. For calculation we are using [CRC-16/IBM](https://wiki.teltonika-gps.com/view/Codec#CRC-16 "Codec").
|
||||
|
||||
**Note:** Codec13 packets are used only when the “Message Timestamp” parameter in RS232 settings is enabled.
|
||||
|
||||
## Codec 14
|
||||
|
||||
- **About Codec14**
|
||||
|
||||
Codec14 is the original Teltonika protocol for device-server communication over GPRS messages and it is based on the Codec12 protocol.
|
||||
The main difference of Codec14 is that the device will answer the GPRS command if the device's physical IMEI number matches the specified IMEI number in the GPRS command.
|
||||
|
||||
Codec14 GPRS commands can be used for sending configuration, debug, digital outputs control commands, or other (special purpose commands on special firmware versions).
|
||||
|
||||
- **FMB firmware requirements**
|
||||
|
||||
Implemented in base firmware from FMB.Ver.03.25.04.Rev.00 and newer.
|
||||
|
||||
- **General Codec14 message structure**
|
||||
|
||||
The following diagram shows the basic structure of Codec14 messages.
|
||||
|
||||
**Command message structure**
|
||||
|
||||
<table><tbody><tr><th rowspan="1">0x00000000 (preamble)</th><th rowspan="1">Data size</th><th rowspan="1">0x0E (Codec ID)</th><th rowspan="1">Command quantity</th><th colspan="1">0x05<br>(Message type)</th><th rowspan="1">Command size + IMEI size (8 bytes)</th><th rowspan="1">IMEI (HEX)</th><th rowspan="1">Command</th><th rowspan="1">Command quantity</th><th rowspan="1">CRC-16</th></tr><tr><td>4 bytes</td><td>4 bytes</td><td>1 bytes</td><td>1 bytes</td><td>1 bytes</td><td>4 bytes</td><td>8 bytes</td><td>X bytes</td><td>1 bytes</td><td>4 bytes</td></tr></tbody></table>
|
||||
|
||||
**Response message structure**
|
||||
|
||||
<table><tbody><tr><th rowspan="1">0x00000000 (preamble)</th><th rowspan="1">Data size</th><th rowspan="1">0x0E (Codec ID)</th><th rowspan="1">Response quantity</th><th colspan="1">0x06 / 0x11 (Message type)</th><th rowspan="1">Response size + IMEI size (8 bytes)</th><th rowspan="1">IMEI (HEX)</th><th rowspan="1">Response</th><th rowspan="1">Response quantity</th><th rowspan="1">CRC-16</th></tr><tr><td>4 bytes</td><td>4 bytes</td><td>1 bytes</td><td>1 bytes</td><td>1 bytes</td><td>4 bytes</td><td>8 bytes</td><td>X bytes</td><td>1 bytes</td><td>4 bytes</td></tr></tbody></table>
|
||||
|
||||
**Preamble** – the packet starts with four zero bytes.
|
||||
**Data Size** – size is calculated from the Codec ID field to the second command or response quantity field.
|
||||
**Codec ID** – in Codec14 it is always `0x0E`.
|
||||
**Command/Response Quantity 1** – it is ignored when parsing the message.
|
||||
**Type** – if it is a request command from the server it has to contain 0x05. The response type field will contain `0x06` if it’s ACK or `0x11` if it’s nACK.
|
||||
*Explanation:* If command message IMEI is equal to actual device IMEI, received command will be executed and response will be sent with ACK (`0x06`) message type field value. If the command message IMEI doesn’t match the actual device IMEI, the received command won’t be executed and a response to the server will be sent with nACK (`0x11`) message type field value.
|
||||
**Command/Response Size** – command or response length.
|
||||
*Note:* make sure that size is IMEI size 8 + actual command size. The minimal value is 8 because Codec14 always contains IMEI and it’s 8 bytes.
|
||||
**IMEI (HEX)** – it is send as HEX value. For example, if the device IMEI is 123456789123456 then the IMEI data field will contain `0x0123456789123456` value.
|
||||
**Command/Response** – command or response in HEX.
|
||||
**Command/Response Quantity 2** - a byte that defines how many records (commands or responses) are in the packet. This byte will not be parsed but it’s recommended that it should contain the same value as Command/Response Quantity 1.
|
||||
**CRC-16** – calculated from Codec ID to the Second Number of Data. CRC (Cyclic Redundancy Check) is an error-detecting code used to detect accidental changes to RAW data. For calculation we are using [CRC-16/IBM](https://wiki.teltonika-gps.com/view/Codec#CRC-16 "Codec").
|
||||
|
||||
- **GPRS in Codec14 examples**
|
||||
|
||||
The hexadecimal stream of the GPRS command and answer in this example is given in the hexadecimal form. The different fields of the message are separated into different table columns for better readability and some of them are converted to ASCII values for better understanding.
|
||||
|
||||
Sending *[getver](https://wiki.teltonika-gps.com/view/FMB_getver "FMB getver")* SMS command via GPRS Codec14:
|
||||
|
||||
Server requests in Hexadecimal stream:
|
||||
`00000000000000160E01050000000E0352093081452251676574766572010000D2C1`
|
||||
|
||||
Parsed:
|
||||
|
||||
<table><tbody><tr><th colspan="2">Server Command</th></tr><tr><th rowspan="1">Server Command Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td>Zero Bytes</td><td>00 00 00 00</td></tr><tr><td>Data Size</td><td>00 00 00 16</td></tr><tr><td>Codec ID</td><td>0E</td></tr><tr><td>Command Quantity 1</td><td>01</td></tr><tr><td>Command Type</td><td>05</td></tr><tr><td>Command Size</td><td>00 00 00 0E</td></tr><tr><td>IMEI</td><td>03 52 09 30 81 45 22 51</td></tr><tr><td>Command</td><td>67 65 74 76 65 72</td></tr><tr><td>Command Quantity 2</td><td>01</td></tr><tr><td>CRC-16</td><td>00 00 D2 C1</td></tr></tbody></table>
|
||||
|
||||
Note that Server Command converted from HEX to ASCII means *[getver](https://wiki.teltonika-gps.com/view/FMB_getver "FMB getver")*
|
||||
|
||||
Device ACK response in the hexadecimal stream:
|
||||
`00000000000000AB0E0106000000A303520930814522515665723A30332E31382E31345F3034204750533A41584E5F352E31305F333333332048773A464D42313230` `204D6F643A313520494D45493A33353230393330383134353232353120496E69743A323031382D31312D323220373A313320557074696D653A3137323334204D4143` `3A363042444430303136323631205350433A312830292041584C3A30204F42443A3020424C3A312E362042543A340100007AAE`
|
||||
|
||||
Parsed:
|
||||
|
||||
<table><tbody><tr><th colspan="2">Device Answer</th></tr><tr><th rowspan="1">Device Answer Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td>Zero Bytes</td><td>00 00 00 00</td></tr><tr><td>Data Size</td><td>00 00 00 37</td></tr><tr><td>Codec ID</td><td>0E</td></tr><tr><td>Response Quantity 1</td><td>01</td></tr><tr><td>Response Type</td><td>06</td></tr><tr><td>Response Size</td><td>00 00 00 A3</td></tr><tr><td>IMEI</td><td>03 52 09 30 81 45 22 51</td></tr><tr><td>Response</td><td>56 65 72 3A 30 33 2E 31 38 2E 31 34 5F 30 34 20 47 50 53 3A 41 58 4E 5F 35 2E 31 30 5F 33 33 33 33 20 48 77 3A 46 4D 42 31 32 30 20 4D 6F 64 3A 31 35 20 49 4D 45 49 3A 33 35 32 30 39 33 30 38 31 34 35 32 32 35 31 20 49 6E 69 74 3A 32 30 31 38 2D 31 31 2D 32 32 20 37 3A 31 33 20 55 70 74 69 6D 65 3A 31 37 32 33 34 20 4D 41 43 3A 36 30 42 44 44 30 30 31 36 32 36 31 20 53 50 43 3A 31 28 30 29 20 41 58 4C 3A 30 20 4F 42 44 3A 30 20 42 4C 3A 31 2E 36 20 42 54 3A 34</td></tr><tr><td>Response Quantity 2</td><td>01</td></tr><tr><td>CRC-16</td><td>00 00 7A AE</td></tr></tbody></table>
|
||||
|
||||
Note that Device Response converted from HEX to ASCII means:
|
||||
*Ver:03.18.14\_04 GPS:AXN\_5.10\_3333 Hw:FMB120 Mod:15 IMEI:352093081452251 Init:2018-11-22 7:13 Uptime:17234 MAC:60BDD0016261 SPC:1(0) AXL:0 OBD:0 BL:1.6 BT:4*
|
||||
|
||||
Device nACK response in the hexadecimal stream:
|
||||
`00000000000000100E011100000008035209308145246801000032AC`
|
||||
|
||||
Parsed:
|
||||
|
||||
<table><tbody><tr><th colspan="2">Device Answer</th></tr><tr><th rowspan="1">Device Answer Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td>Zero Bytes</td><td>00 00 00 00</td></tr><tr><td>Data Size</td><td>00 00 00 10</td></tr><tr><td>Codec ID</td><td>0E</td></tr><tr><td>Response Quantity 1</td><td>01</td></tr><tr><td>Response Type</td><td>11</td></tr><tr><td>Response Size</td><td>00 00 00 08</td></tr><tr><td>IMEI</td><td>03 52 09 30 81 45 24 68</td></tr><tr><td>Response Quantity 2</td><td>01</td></tr><tr><td>CRC-16</td><td>00 00 32 AC</td></tr></tbody></table>
|
||||
|
||||
## Codec 15
|
||||
|
||||
- **Protocol Overview**
|
||||
|
||||
Codec 15 relies on the Codec12 protocol and is employed when both message timestamp and device IMEI are enabled. It serves as the original Teltonika protocol for communication from the device to the server via GPRS messages. This protocol is exclusively applicable to FMX6 professional devices.
|
||||
|
||||
Codec15 is available in RS232 modes:
|
||||
|
||||
1\. TCP/UDP Ascii
|
||||
2\. TCP/UDP Binary
|
||||
3\. TCP/UDP Ascii Buffered
|
||||
4\. TCP/UDP Binary Buffered.
|
||||
|
||||
- **Structure of Codec 15 messages**
|
||||
|
||||
<table><tbody><tr><th rowspan="1">0x00000000 (Preamble)</th><th rowspan="1">Data Size</th><th rowspan="1">0x0F (Codec ID)</th><th rowspan="1">Command quantity</th><th colspan="1">Message type</th><th rowspan="1">Command size + timestamp + imei</th><th rowspan="1">Timestamp</th><th rowspan="1">IMEI</th><th rowspan="1">Command</th><th rowspan="1">Command quantity</th><th rowspan="1">CRC - 16</th></tr><tr><td>4 bytes</td><td>4 bytes</td><td>1 byte</td><td>1 byte</td><td>1 bytes</td><td>4 byte</td><td>4 bytes</td><td>8 bytes</td><td>X bytes</td><td>1 bytes</td><td>4 bytes</td></tr></tbody></table>
|
||||
|
||||
- **Structure explanation**
|
||||
|
||||
**Preamble** - four zero bytes.
|
||||
**Data size** - size is calculated from codec id(0x0F) field to the second command quantity field.
|
||||
**Codec ID** - in Codec 15 it is always 0x0F.
|
||||
**Command quantity** - a number which defines how many commands are in the packet.
|
||||
**Message type** - this value is configurable in RS232 settings box.
|
||||
**Command size + Timestamp + IMEI** - it is equal to size of payload + size of timestamp + size of imei.
|
||||
**Timestamp** – data record creation time in seconds(Unix timestamp).
|
||||
**IMEI** - send as HEX value. Example if device IMEI is 123456789123456 then IMEI data field will contain 0x0123456789123456 value.
|
||||
**Command field** - actual received data.
|
||||
**Command quantity** - a number which defines how many commands are in the packet.
|
||||
**CRC field** - calculated from Codec ID to the Second Number of Data.
|
||||
|
||||
- **Codec 15 examples**
|
||||
|
||||
Device sends message „Hello\\n“ via GPRS Codec15:
|
||||
|
||||
```
|
||||
000000000000001b0f010b00000013654b65a4012345678912345648656c6c6f210a01000093d6
|
||||
```
|
||||
|
||||
<table><tbody><tr><th colspan="2">Parsed command</th></tr><tr><th rowspan="1">Command Part</th><th rowspan="1">HEX Code Part</th></tr><tr><td>Zero Bytes</td><td>00 00 00 00</td></tr><tr><td>Data Size</td><td>00 00 00 1B</td></tr><tr><td>Codec ID</td><td>0F</td></tr><tr><td>Quantity of commands</td><td>01</td></tr><tr><td>Command type</td><td>0B</td></tr><tr><td>Command Size</td><td>00 00 00 13</td></tr><tr><td>Timestamp</td><td>65 4B 65 A4</td></tr><tr><td>IMEI</td><td>01 23 45 67 89 12 34 56</td></tr><tr><td>Command</td><td>48 65 6c 6c 6f 21 0a</td></tr><tr><td>Quantity of commands</td><td>01</td></tr><tr><td>CRC-16</td><td>00 00 93 D6</td></tr></tbody></table>
|
||||
|
||||
CRC: 0x 000093d6 The algorithm to calculate CRC is CRC-16 (also known as CRC-16-IBM). All the fields from codec ID to second command/response quantity field are used to calculate CRC.
|
||||
|
||||
## Differences between Codec 12, Codec 13, Codec 14 and Codec 15
|
||||
|
||||
In the table below you will see differences between Codec12, Codec13, Codec14 and Codec 15.
|
||||
|
||||
<table><tbody><tr><th rowspan="1"></th><th rowspan="1">Codec12</th><th rowspan="1">Codec13</th><th rowspan="1">Codec14</th><th rowspan="1">Codec15</th></tr><tr><th rowspan="1">Communication</th><td>Server - Device Communication</td><td>One-way (Device -> Server communication)</td><td>Server - Device Communication</td><td>One-way (Device -> Server communication)</td></tr><tr><th rowspan="1">Codec ID</th><td>0x0C</td><td>0x0D</td><td>0x0E</td><td>0x0F</td></tr><tr><th rowspan="1">Response Message Type</th><td>0x06</td><td>-</td><td>0x06 (if it is ACK) or 0x11 (if it is nACK)</td><td>-</td></tr><tr><th rowspan="1">Command / Response size</th><td>Only Command/Response</td><td>Only Command</td><td>Command/Response + IMEI</td><td>Command + IMEI</td></tr><tr><th rowspan="1">Timestamp</th><td>Not Using</td><td>Is Using</td><td>Not Using</td><td>Is Using</td></tr><tr><th rowspan="1">IMEI</th><td>Not Using</td><td>Not Using</td><td>Is Using</td><td>Is Using</td></tr></tbody></table>
|
||||
|
||||
## 24 Position SMS Data Protocol
|
||||
|
||||
24-hour SMS is usually sent once every day and contains GPS data for the last 24 hours. The TP-DCS field of this SMS should indicate that message contains 8-bit data (i.e. TP-DCS can be `0x04`).
|
||||
Note, that 24 position data protocol is used only with subscribed SMS. Event SMS uses standard AVL data protocol.
|
||||
|
||||
- **Encoding**
|
||||
|
||||
To be able to compress 24 GPS data entries into one SMS (140 octets), the data is encoded extensively using bit fields. The data packet can be interpreted as a bitstream, where all bits are numbered as follows:
|
||||
|
||||
<table><tbody><tr><th rowspan="1">Byte 1</th><th rowspan="1">Byte 2</th><th rowspan="1">Byte 3</th><th rowspan="1">Byte 4...</th></tr><tr><td>Bits 0 - 7</td><td>Bits 8 - 15</td><td>Bits 16 - 24</td><td>Bits 25 -...</td></tr></tbody></table>
|
||||
|
||||
Bits in a byte are numbered starting from the least significant bit. A field of 25 bits would consist of bits 0 to 24 where 0 is the least significant bit and bit 24 – is the most significant bit.
|
||||
|
||||
- **Structure**
|
||||
|
||||
Below in the tables, you will see the SMS Data Structure:
|
||||
|
||||
<table><tbody><tr><th colspan="3">SMS Data Structure</th></tr><tr><td>8</td><td>Codec ID</td><td>Codec ID = 4 (0x04)</td></tr><tr><td>35</td><td>Timestamp</td><td>Time corresponding to the first (oldest) GPS data element, represented in seconds elapsed from 2000.01.01 00:00 EET.</td></tr><tr><td>5</td><td>ElementCount</td><td>Number of GPS data elements</td></tr></tbody></table>
|
||||
|
||||
<table><tbody><tr><th colspan="4">SMS Data Structure</th></tr><tr><td rowspan="3">ElementCount *</td><td></td><td>GPSDataElement</td><td>GPS data elements</td></tr><tr><td></td><td>Byte - align padding</td><td>Padding bits to align to 8 - bits boundary represented in seconds elapsed from 2000.01.01 00:00 EET.</td></tr><tr><td>64</td><td>IMEI</td><td>IMEI of sending device as 8 byte long integer</td></tr></tbody></table>
|
||||
|
||||
The time of only the first GPS data element is specified in the Timestamp field. The time corresponding to each further element can be computed as elementTime = Timestamp + (1 hour \* elementNumber).
|
||||
|
||||
<table><tbody><tr><th colspan="5" rowspan="1">GPS Data Element</th></tr><tr><th rowspan="1"></th><th rowspan="1"></th><th rowspan="1">Size (bits)</th><th colspan="1">Field</th><th rowspan="1">Description</th></tr><tr><td></td><td></td><td>1</td><td>ValidElement</td><td>ValidElement = 1 – there is a valid Gps Data Element following,<p>ValidElement = 0 – no element at this position</p></td></tr><tr><td rowspan="5">ValidElement == 1</td><td></td><td>1</td><td>DifferentialCoords</td><td>Format of following data</td></tr><tr><td rowspan="2">DifferentialCoords == 1</td><td>14</td><td>LongitudeDiff</td><td>Difference from previous element‘s longitude.<p>LongitudeDiff = prevLongitude – Longitude + 213 – 1</p></td></tr><tr><td>14</td><td>LatitudeDiff</td><td>Difference from previous element‘s latitude<p>LatitudeDiff = prevLatitude – Latitude + 213 – 1</p></td></tr><tr><td rowspan="2">DifferentialCoords == 0</td><td>21</td><td>Longitude</td><td>Longitude = {(LongDegMult + 18 * 108) * (221 – 1)} over {36*108}</td></tr><tr><td>20</td><td>Latitude</td><td>Latitude = (LatDegMult + 9*108) * (220 – 1) over {18*108}</td></tr><tr><td></td><td></td><td>8</td><td>Speed</td><td>Speed in km/h</td></tr></tbody></table>
|
||||
|
||||
**Longitude** - longitude field value of GPSDataElement
|
||||
**Latitude** - latitude field value of GPSDataElement
|
||||
**LongDegMult** - longitude in degrees multiplied by 107 (integer part)
|
||||
**LatDegMult** - latitude in degrees multiplied by 107 (integer part)
|
||||
**prevLongitude** - longitude field value of previous GPSDataElemen
|
||||
**prevLatitude** - latitude field value of previous GPSDataElement
|
||||
|
||||
- **Decoding GPS position**
|
||||
|
||||
When decoding GPS data with DifferentialCoords = 1, Latitude and Longitude values can be computed as follows: Longitude = prevLongitude – LongitudeDiff + 213 – 1, Latitude = prevLatitude – LatitudeDiff + 213 – 1.
|
||||
If there were no previous non-differential positions, differential coordinates should be computed assuming prevLongitude = prevLatitude = 0.
|
||||
When Longitude and Latitude values are known, longitude and latitude representation in degrees can be computed as follows:
|
||||
|
||||
|
||||
|
||||
- **SMS Events**
|
||||
|
||||
When configured to generate an SMS event user will get this SMS upon event:
|
||||
*<Year/Month/Day> <Hour:Minute:Second> P:<profile\_nr> <SMS Text> Val:<Event Value> Lon:<longitude> Lat:<latitude> Q:<HDOP>*
|
||||
|
||||
Example:
|
||||
*2016./04/11 12:00:00 P:3 Digital Input 1 Val:1 Lon:51.12258 Lat: 25.7461 Q:0.6*
|
||||
|
||||
## Sending data using SMS
|
||||
|
||||
This type of data sent is used for FMBXXX devices which can be configured in [SMS Data Sending settings](https://wiki.teltonika-gps.com/view/FMB120_SMS/Call_settings#SMS_Data_Sending "FMB120 SMS/Call settings").
|
||||
|
||||
- **Data sending via SMS**
|
||||
|
||||
AVL data or events can be sent encapsulated in binary SMS. The TP-DCS field of these SMS should indicate that message contains 8-bit data (for example TP-DCS can be `0x04`).
|
||||
|
||||
<table><tbody><tr><th colspan="2">SMS data (TP-UD)</th></tr><tr><th rowspan="1">AVL data array</th><th rowspan="1">IMEI</th></tr><tr><td>X bytes</td><td>8 bytes</td></tr></tbody></table>
|
||||
|
||||
**AVL data array** – array of encoded AVL data.
|
||||
**IMEI** – IMEI of sending module encoded as a big-endian 8-byte long number.
|
||||
|
||||
## CRC-16
|
||||
|
||||
CRC (Cyclic Redundancy Check) is an error-detecting code used to detect accidental changes to RAW data. The algorithm on how to calculate CRC-16 (also known as CRC-16/IBM) you will find below.
|
||||
|
||||
@@ -0,0 +1,366 @@
|
||||
# GPS Tracking Platform — Architecture Overview
|
||||
|
||||
**Document type:** Architecture reference
|
||||
**Scope:** System-level design for a real-time GPS telemetry platform
|
||||
**Audience:** Engineering, infrastructure, future contributors
|
||||
|
||||
---
|
||||
|
||||
## 1. Purpose and scope
|
||||
|
||||
This document describes the high-level architecture of a real-time GPS tracking platform built around four cooperating components:
|
||||
|
||||
- A **TCP Ingestion** service that accepts persistent connections from GPS hardware and parses vendor-specific binary protocols.
|
||||
- A **Processor** service that consumes parsed telemetry, applies domain logic, and writes durable state.
|
||||
- A **Directus** instance that owns the relational schema, exposes data through REST/GraphQL APIs, and provides an admin UI for back-office users.
|
||||
- A **React** single-page application that delivers the end-user experience for operators and external participants.
|
||||
|
||||
The architecture is deliberately split along the lines of **data velocity** and **failure domain**. Hot-path telemetry is isolated from business APIs; live state is decoupled from durable state; user interfaces are separated from data ownership. The result is a system where each component can scale, fail, and be deployed independently, and where adding new device vendors or new front-end surfaces does not require touching the core.
|
||||
|
||||
This document does not cover business logic, domain modeling, or specific operational scenarios — it is intentionally generic and focused on the structural shape of the system.
|
||||
|
||||
---
|
||||
|
||||
## 2. System overview
|
||||
|
||||
```
|
||||
┌────────────────────────────┐
|
||||
│ GPS Devices (TCP) │
|
||||
└──────────────┬─────────────┘
|
||||
│ binary frames
|
||||
▼
|
||||
┌────────────────────────────┐
|
||||
│ TCP Ingestion │
|
||||
│ - Vendor protocol parsers │
|
||||
│ - Frame ACKs to devices │
|
||||
│ - Normalized Position │
|
||||
└──────────────┬─────────────┘
|
||||
│ enqueue
|
||||
▼
|
||||
┌────────────────────────────┐
|
||||
│ Redis Streams │
|
||||
│ (durable in-flight queue) │
|
||||
└──────────────┬─────────────┘
|
||||
│ consume
|
||||
▼
|
||||
┌────────────────────────────┐
|
||||
│ Processor │
|
||||
│ - Per-device state │
|
||||
│ - Domain rules │
|
||||
│ - Derived events │
|
||||
└──────────────┬─────────────┘
|
||||
│ writes
|
||||
▼
|
||||
┌──────────────────────────────────────┐
|
||||
│ PostgreSQL + TimescaleDB │
|
||||
│ (positions hypertable, business │
|
||||
│ schema owned by Directus) │
|
||||
└──────┬───────────────────────┬───────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌───────────────────┐ ┌───────────────────┐
|
||||
│ Directus │ │ Direct DB writes │
|
||||
│ REST / GraphQL │ │ from Processor │
|
||||
│ WebSockets │ │ (positions, │
|
||||
│ Admin UI │ │ events) │
|
||||
└─────────┬─────────┘ └───────────────────┘
|
||||
│ HTTPS / WSS
|
||||
▼
|
||||
┌───────────────────┐
|
||||
│ React SPA │
|
||||
│ (operators & │
|
||||
│ participants) │
|
||||
└───────────────────┘
|
||||
```
|
||||
|
||||
The system is best understood as **three concentric concerns**:
|
||||
|
||||
1. **Telemetry plane** — TCP Ingestion + Redis Streams + Processor. Optimized for throughput, low latency, and resilience to bursty input. Stateless or nearly so.
|
||||
2. **Business plane** — Directus over PostgreSQL/TimescaleDB. Owns the schema, the API surface, the permissions model, and back-office workflows.
|
||||
3. **Presentation plane** — React SPA. Consumes the business plane's APIs and real-time subscriptions; never talks to the telemetry plane directly.
|
||||
|
||||
The boundary between planes is enforced by the queue (Redis Streams) on one side and the database/API on the other. No component reaches across two boundaries.
|
||||
|
||||
---
|
||||
|
||||
## 3. TCP Ingestion
|
||||
|
||||
### 3.1 Responsibility
|
||||
|
||||
The Ingestion service exists to do one thing well: maintain persistent TCP connections with GPS devices, parse incoming binary frames, acknowledge them according to the vendor protocol, and hand off normalized records to the rest of the system.
|
||||
|
||||
It deliberately does **not**:
|
||||
|
||||
- Apply business rules
|
||||
- Write to the primary database
|
||||
- Perform geospatial computation
|
||||
- Serve any user-facing API
|
||||
|
||||
This narrow scope is what allows the Ingestion process to remain fast, predictable, and safely restartable.
|
||||
|
||||
### 3.2 Connection model
|
||||
|
||||
GPS hardware typically opens a long-lived TCP connection and streams telemetry frames over it for hours or days at a time. The Ingestion service is built around `net.createServer()` (or equivalent in another runtime) and treats each socket as an independent session. Per-connection state is small: an identifier (e.g. IMEI), a parser instance, and a buffer for partial frames.
|
||||
|
||||
Devices reconnect automatically on network failure, so the Ingestion service treats connection loss as routine — there is no need to preserve sessions across restarts. This makes the service trivially restartable, which in turn makes deployments and crashes cheap.
|
||||
|
||||
### 3.3 Vendor abstraction
|
||||
|
||||
Each device vendor (Teltonika, Queclink, Concox, etc.) ships its own binary protocol. To prevent vendor-specific code from leaking into the rest of the system, the Ingestion layer defines a **protocol adapter** interface:
|
||||
|
||||
- **Input:** a stream of bytes from a TCP socket
|
||||
- **Output:** normalized `Position` records with a stable shape (`device_id`, `timestamp`, `latitude`, `longitude`, `speed`, `heading`, plus a free-form attribute bag for vendor-specific telemetry)
|
||||
|
||||
Adding a new device family means writing a new adapter. Nothing downstream of the Ingestion service changes. This is the single most important property of this layer for long-term maintenance.
|
||||
|
||||
### 3.4 Handoff
|
||||
|
||||
Once a frame is parsed and normalized, the Ingestion service:
|
||||
|
||||
1. Sends the protocol-required ACK back to the device.
|
||||
2. Pushes the normalized record onto a Redis Stream.
|
||||
3. Returns to reading the socket.
|
||||
|
||||
The TCP handler **never blocks on downstream work**. If the Processor falls behind or the database is slow, the Stream absorbs the pressure; the Ingestion path keeps accepting and acknowledging frames. This single discipline is the difference between a system that survives a bad day and one that doesn't.
|
||||
|
||||
### 3.5 Scaling shape
|
||||
|
||||
A single Node.js process comfortably handles thousands of concurrent connections at typical telemetry rates. When a single process becomes insufficient — whether due to CPU, file descriptor limits, or operational preference — the service scales horizontally:
|
||||
|
||||
- Multiple Ingestion instances run behind a TCP-aware load balancer (HAProxy, NGINX stream module).
|
||||
- Each device's connection is sticky for the duration of the session (TCP guarantees this naturally).
|
||||
- No shared state between Ingestion instances is required, because per-device state lives entirely on the open socket.
|
||||
|
||||
This is the same scaling pattern used in higher-throughput runtimes (Go, Elixir) and ports cleanly if a future rewrite is ever warranted.
|
||||
|
||||
---
|
||||
|
||||
## 4. Processor
|
||||
|
||||
### 4.1 Responsibility
|
||||
|
||||
The Processor is where domain logic lives. It consumes normalized telemetry from Redis Streams and is responsible for:
|
||||
|
||||
- Maintaining **per-device runtime state** (last position, derived metrics, current zone, etc.)
|
||||
- Applying **domain rules** that turn raw telemetry into meaningful events
|
||||
- Writing **durable state** to the database — both the raw position history and any derived events
|
||||
- Emitting events that downstream consumers (Directus Flows, notification services, dashboards) can react to
|
||||
|
||||
Where Ingestion is about throughput and protocol correctness, the Processor is about correctness of meaning. It is the component most likely to evolve as requirements grow, which is why it is isolated from the sockets on one side and the API surface on the other.
|
||||
|
||||
### 4.2 State management
|
||||
|
||||
Because devices report at high frequency, the Processor keeps **hot state in memory**. Reaching for the database on every incoming record would be wasteful and slow. The pattern is:
|
||||
|
||||
- Static reference data (e.g. spatial assets, configurations) is loaded at startup and refreshed on a known cadence or via explicit invalidation.
|
||||
- Per-device state (last seen, current segment, accumulators) is held in memory keyed by device identifier.
|
||||
- Durable state (position history, derived events, audit trail) is written asynchronously to the database.
|
||||
|
||||
The database is the source of truth for replay and analysis; the in-memory state is the source of truth for the current decision being made. If the Processor restarts, it can rehydrate from the database — but this is a recovery path, not a hot path.
|
||||
|
||||
### 4.3 Decoupling via the queue
|
||||
|
||||
Redis Streams between Ingestion and Processor provides three things:
|
||||
|
||||
- **Buffering** — temporary slowness in the Processor does not push back on the Ingestion sockets.
|
||||
- **Replayability** — Streams retain messages, so a Processor crash does not lose telemetry; it picks up from its last consumer-group position.
|
||||
- **Horizontal scaling** — multiple Processor instances can join a consumer group and split the load across device IDs.
|
||||
|
||||
Redis is sufficient at this scale and adds minimal operational burden. NATS or Kafka are reasonable upgrades when multi-region durability or very high throughput become real concerns; until then, Redis is the right choice.
|
||||
|
||||
### 4.4 Writing to the database
|
||||
|
||||
The Processor writes directly to PostgreSQL/TimescaleDB. It is the **only writer** for high-volume telemetry tables (e.g. the positions hypertable). Directus does not insert positions; it reads them.
|
||||
|
||||
For derived business entities (events, violations, alerts), the Processor writes to tables that Directus also knows about. The schema is owned by Directus — defined and migrated through it — but the Processor inserts rows directly using a database connection. This keeps the hot write path off the Directus HTTP stack, while still letting Directus expose the data through its API and admin UI.
|
||||
|
||||
---
|
||||
|
||||
## 5. Directus
|
||||
|
||||
### 5.1 Role in the system
|
||||
|
||||
Directus is the **business plane**. It owns the relational schema, exposes it through auto-generated REST and GraphQL APIs, enforces role-based permissions, and provides the admin UI for back-office users.
|
||||
|
||||
This includes:
|
||||
|
||||
- **Schema management** — collections, fields, relations, migrations
|
||||
- **API generation** — REST and GraphQL endpoints, no boilerplate
|
||||
- **Authentication and authorization** — users, roles, permissions, JWT issuance
|
||||
- **Real-time** — WebSocket subscriptions on collections for live UIs
|
||||
- **Workflow automation** — Flows for orchestrating side effects (notifications, integrations)
|
||||
- **Admin UI** — a complete back-office interface for operators
|
||||
|
||||
Directus is **not** in the telemetry hot path. It does not accept device connections, run the geofence engine, or hold per-device runtime state. Mixing those responsibilities into the same process would couple deployment lifecycles and contaminate failure domains.
|
||||
|
||||
### 5.2 Why Directus owns the schema
|
||||
|
||||
Even though the Processor writes directly to the database, Directus is treated as the **owner** of the schema. New tables, columns, and relations are defined through Directus. This matters because:
|
||||
|
||||
- The admin UI and APIs are auto-generated from the schema Directus knows about. Tables created outside Directus are invisible to it.
|
||||
- Permissions are configured per-collection in Directus. Tables it does not manage cannot be permission-controlled through the standard mechanism.
|
||||
- Audit and metadata columns (created_at, updated_at, user_created, etc.) follow Directus conventions; bypassing them inconsistently leads to subtle UI bugs.
|
||||
|
||||
The Processor inserts rows into Directus-owned tables, but it respects the schema as defined. This is a normal Directus deployment pattern — Directus does not require sole write access to its database, only schema authority.
|
||||
|
||||
### 5.3 Extension surface
|
||||
|
||||
Directus extensions are used for things that genuinely belong in the business layer:
|
||||
|
||||
- **Hooks** that react to data changes (e.g. when an event is written, trigger a notification Flow).
|
||||
- **Custom endpoints** for operations that need permissions, audit, and orchestration but are not throughput-critical.
|
||||
- **Custom admin UI panels** for back-office workflows that benefit from being inside the Directus admin (e.g. data review, manual overrides, bulk operations).
|
||||
- **Flows** for declarative orchestration — when X happens, do Y, Z, and notify W.
|
||||
|
||||
Extensions are **not** used for long-running listeners, persistent network sockets, or anything in the telemetry hot path. Those belong in dedicated services.
|
||||
|
||||
### 5.4 Real-time delivery
|
||||
|
||||
Directus's WebSocket subscriptions are used to push live data to the React SPA. When the Processor writes a new row (a position, an event), Directus broadcasts the change to subscribed clients. This is sufficient for moderate fan-out scenarios — tens to low hundreds of concurrent subscribers.
|
||||
|
||||
If real-time fan-out becomes a bottleneck (many clients each subscribing to many streams), a dedicated WebSocket gateway can be introduced that reads directly from Redis Streams and pushes to clients, bypassing Directus for the live channel only. The REST/GraphQL surface remains in Directus. This is a future evolution, not a day-one concern.
|
||||
|
||||
---
|
||||
|
||||
## 6. React SPA
|
||||
|
||||
### 6.1 Why a separate SPA
|
||||
|
||||
The Directus admin UI is designed for data managers — generic CRUD over collections and fields. It is the right tool for back-office editing and for operators who think in records. It is the wrong tool for end users who think in domain concepts.
|
||||
|
||||
A separate React SPA delivers:
|
||||
|
||||
- **Domain-shaped UX** — screens organized around the user's mental model, not the database schema.
|
||||
- **Independent deployment** — the front-end can ship on its own cadence without touching Directus.
|
||||
- **Targeted access control** — public-facing or partner-facing routes can be served without exposing the admin surface.
|
||||
- **Mobile and offline considerations** — designs and bundles can be tuned for the actual user environment, separate from the desktop-oriented admin UI.
|
||||
|
||||
### 6.2 Single app, role-based views
|
||||
|
||||
Unless there is a strong reason to split, a single React application serves multiple user types via role-based routing and conditional UI. All users authenticate through Directus; the SPA receives a JWT, reads the user's role, and renders the appropriate navigation and screens.
|
||||
|
||||
Splitting into multiple apps is only justified when the user populations are genuinely disjoint (e.g. a public marketing-style site versus an authenticated operator console) or when bundle size for one audience would meaningfully harm another.
|
||||
|
||||
### 6.3 Data access pattern
|
||||
|
||||
The SPA talks exclusively to Directus. It uses:
|
||||
|
||||
- **REST or GraphQL** for queries and mutations, via the official Directus SDK.
|
||||
- **WebSocket subscriptions** for live data, via the same SDK.
|
||||
- **JWT authentication** managed by the SDK; refresh handled transparently.
|
||||
|
||||
The SPA never talks to the Processor, Ingestion, Redis, or the database directly. This boundary is what allows the back-end to evolve internally without breaking the front-end, and what keeps the security model coherent — every request goes through Directus's permission system.
|
||||
|
||||
### 6.4 Suggested stack
|
||||
|
||||
A pragmatic, well-supported stack for this kind of application:
|
||||
|
||||
- **Vite + React + TypeScript** — fast builds, strong typing, broad ecosystem.
|
||||
- **TanStack Router** for routing — better TypeScript support than React Router, file-based routing optional.
|
||||
- **TanStack Query** for server state — caching, invalidation, background refresh, optimistic updates.
|
||||
- **@directus/sdk** for typed Directus access and real-time subscriptions.
|
||||
- **MapLibre GL** with **react-map-gl** for live map views — open source, WebGL-based, no token requirements.
|
||||
- **shadcn/ui + Tailwind** for UI primitives — fast to assemble, consistent, professional.
|
||||
- **Zustand** for client-only state that does not fit the server-state model (filters, UI prefs).
|
||||
- **react-hook-form + Zod** for forms and validation.
|
||||
|
||||
This stack covers the spectrum from form-heavy admin screens to real-time map dashboards without architectural changes between them.
|
||||
|
||||
### 6.5 Real-time rendering considerations
|
||||
|
||||
For live map views with many moving markers, the React reconciler is not the bottleneck — the actual rendering happens in WebGL via MapLibre, which manages its own layer of features outside React's tree. The React layer is responsible for managing subscription state and feeding the map with updates; the map handles drawing.
|
||||
|
||||
For high-frequency tabular updates (live leaderboards, event feeds), the standard React patterns apply: split components so that high-update areas re-render in isolation, use `TanStack Query` to manage live data, and consider memoization at component boundaries that receive frequent updates.
|
||||
|
||||
---
|
||||
|
||||
## 7. Cross-cutting concerns
|
||||
|
||||
### 7.1 Data flow summary
|
||||
|
||||
The end-to-end path of a single telemetry record:
|
||||
|
||||
1. Device sends a binary frame over TCP.
|
||||
2. Ingestion parses the frame, ACKs the device, pushes a normalized record to a Redis Stream.
|
||||
3. Processor consumes the record, updates in-memory device state, applies domain rules, writes the position to the database, and emits any derived events.
|
||||
4. Directus exposes the new data through its API and broadcasts changes to subscribed clients.
|
||||
5. React SPA receives the update via a WebSocket subscription and renders it.
|
||||
|
||||
End-to-end latency under healthy conditions is dominated by the Processor's logic and the database write — typically tens of milliseconds. Under load, Redis Streams absorbs bursts without back-pressuring the device-facing sockets.
|
||||
|
||||
### 7.2 Failure domains
|
||||
|
||||
Each component fails independently:
|
||||
|
||||
- **Ingestion crash** — devices reconnect, in-flight frames are retransmitted by the device per protocol, no data is lost beyond what was unacknowledged.
|
||||
- **Redis loss** — Streams are persisted; restart resumes from disk. A complete Redis loss is recoverable from device retransmits and from Processor checkpointing.
|
||||
- **Processor crash** — Stream consumer-group offsets ensure the next instance picks up where the last left off. In-memory state is rehydrated from the database.
|
||||
- **Directus crash** — telemetry continues to flow into the database. The admin UI and SPA are unavailable, but no telemetry is lost.
|
||||
- **Database loss** — the system stops accepting writes. This is the only single point of failure and is addressed through standard PostgreSQL operational practices (replication, backups, point-in-time recovery via TimescaleDB).
|
||||
|
||||
The architecture deliberately makes the database the **only** part of the system that requires careful operational attention. Everything else is restartable, replaceable, or naturally redundant.
|
||||
|
||||
### 7.3 Deployment topology
|
||||
|
||||
A typical Docker-based deployment:
|
||||
|
||||
- `gps-ingest` — exposes a TCP port on the host (TCP-aware reverse proxies are used only when TLS termination or sharding is needed).
|
||||
- `gps-processor` — internal container, no exposed ports, reads from Redis.
|
||||
- `redis` — internal container, persistence enabled.
|
||||
- `postgres` (with TimescaleDB extension) — persistence volume, regular backups.
|
||||
- `directus` — behind an HTTP reverse proxy (e.g. Nginx Proxy Manager) with TLS.
|
||||
- `react-spa` — static build served behind the same HTTP reverse proxy.
|
||||
|
||||
Components scale independently. Ingestion and Processor scale horizontally for throughput; Directus scales horizontally for HTTP load; the database scales vertically and through read replicas for analytics workloads.
|
||||
|
||||
### 7.4 Observability
|
||||
|
||||
Each component emits its own telemetry:
|
||||
|
||||
- **Ingestion** — connection counts, frame rates, parse errors, ACK latencies.
|
||||
- **Processor** — consumer lag, processing latency per record, rule outcomes, in-memory state size.
|
||||
- **Directus** — request latencies, error rates, active subscriptions.
|
||||
- **SPA** — basic client-side error reporting.
|
||||
|
||||
A standard observability stack (Prometheus + Grafana, or a managed equivalent) suffices. The Redis Stream consumer lag is the single most important metric for spotting trouble — it reflects the health of the entire telemetry pipeline in one number.
|
||||
|
||||
### 7.5 Security boundaries
|
||||
|
||||
- Devices speak plain TCP by default; if confidentiality is required, a TLS-terminating proxy fronts the Ingestion service, or the protocol's own encryption is used where supported.
|
||||
- Redis is internal-only and never exposed.
|
||||
- The database is internal-only and accessed only by Directus and the Processor.
|
||||
- Directus enforces all user-facing authentication and authorization. The SPA holds JWTs; refresh is handled by the SDK.
|
||||
- The admin UI is access-controlled by Directus role and can be restricted at the proxy level (IP allow-listing, separate hostname) for additional defense in depth.
|
||||
|
||||
No component other than Directus exposes a user-facing API. This keeps the security model coherent and auditable.
|
||||
|
||||
---
|
||||
|
||||
## 8. Evolution and future considerations
|
||||
|
||||
The architecture is intentionally conservative — it favors well-understood components and clear boundaries over novelty. It supports growth along several axes without structural changes:
|
||||
|
||||
- **More device vendors** — add protocol adapters in the Ingestion layer.
|
||||
- **More throughput** — scale Ingestion and Processor horizontally; partition Redis Streams; add read replicas to the database.
|
||||
- **More users** — scale the SPA (CDN-served static bundle) and Directus instances; introduce a dedicated WebSocket gateway if subscription fan-out becomes a bottleneck.
|
||||
- **More domain complexity** — extend the Processor; add Directus Flows for orchestration; introduce additional consumer services on Redis Streams for parallel concerns (analytics, archival, third-party integrations).
|
||||
- **Multi-region** — replace Redis with NATS or Kafka; introduce regional Ingestion and Processor clusters writing to a central or regional database tier.
|
||||
|
||||
Structural changes (e.g. replacing Node.js with Go for the Ingestion layer, splitting the SPA into multiple apps, introducing a separate analytics pipeline) are possible but not required by the design. They become considerations when specific bottlenecks or operational concerns make them worthwhile, not before.
|
||||
|
||||
---
|
||||
|
||||
## 9. Summary
|
||||
|
||||
The platform is built around four components with clean boundaries:
|
||||
|
||||
| Component | Concern | Scales by | Failure impact |
|
||||
|-----------|---------|-----------|----------------|
|
||||
| TCP Ingestion | Protocol I/O | Horizontal sharding | Devices reconnect; brief retransmit |
|
||||
| Processor | Domain logic | Consumer groups | Queue absorbs; resume from offset |
|
||||
| Directus | Business API & admin | Horizontal HTTP | Admin/UI unavailable; telemetry unaffected |
|
||||
| React SPA | Presentation | CDN/static | UI unavailable; back-end unaffected |
|
||||
|
||||
Telemetry flows in one direction: devices → Ingestion → Streams → Processor → database → Directus → SPA. Each hop is decoupled from the next, each component owns a single concern, and each can be restarted, scaled, or replaced without touching the others.
|
||||
|
||||
This shape is the foundation. Domain logic, business workflows, and user-facing features are built on top of it without disturbing it.
|
||||
@@ -0,0 +1,498 @@
|
||||
# Teltonika Ingestion — Architecture Reference
|
||||
|
||||
**Document type:** Architecture reference
|
||||
**Scope:** Design of the Teltonika protocol adapter inside the TCP Ingestion service
|
||||
**Audience:** Engineering, future contributors
|
||||
**Related:** [`gps-tracking-architecture.md`](./gps-tracking-architecture.md) — system-level overview
|
||||
**Project location:** [`tcp-ingestion/`](../tcp-ingestion/) — repository root for this service
|
||||
|
||||
---
|
||||
|
||||
## 1. Purpose and scope
|
||||
|
||||
This document describes how the TCP Ingestion service handles **Teltonika** GPS hardware. It defines the protocol surface the service must support, how the parser is structured, and the boundary between framing concerns (which belong here) and interpretation concerns (which belong in the Processor).
|
||||
|
||||
The design goal is simple and load-bearing: **ingest telemetry from any Teltonika device, including models we have never seen, without code changes to the parser.** This is achievable because the Teltonika protocol is self-describing — the codec is announced in the frame header, and the device-specific telemetry is carried as an opaque key-value bag that the Ingestion layer is not responsible for interpreting.
|
||||
|
||||
This document does **not** cover:
|
||||
|
||||
- The vendor-agnostic Ingestion shell (covered in the system architecture).
|
||||
- IO element semantics per device model (a Processor concern, driven by configuration).
|
||||
- Business logic, geofencing, or downstream consumers.
|
||||
|
||||
---
|
||||
|
||||
## 2. Project location and layout
|
||||
|
||||
This service lives at the repository path **`tcp-ingestion/`** — a single Node.js/TypeScript project that contains both the vendor-agnostic shell (TCP listeners, Redis publishing, configuration loading, observability) and all per-vendor protocol adapters. Today the only adapter is Teltonika; future vendors are added as sibling folders under `adapters/`.
|
||||
|
||||
All file paths in this document are relative to `tcp-ingestion/` unless explicitly prefixed otherwise. When a section refers to `adapters/teltonika/codec/data/`, the absolute path is `tcp-ingestion/adapters/teltonika/codec/data/`.
|
||||
|
||||
The recommended layout:
|
||||
|
||||
```
|
||||
tcp-ingestion/
|
||||
├── src/
|
||||
│ ├── core/ # vendor-agnostic, no adapter imports
|
||||
│ │ ├── types.ts # Position type, Codec/Adapter interfaces
|
||||
│ │ ├── publish.ts # Redis Streams publisher
|
||||
│ │ ├── registry.ts # codec-ID → handler registry
|
||||
│ │ ├── session.ts # generic socket session loop
|
||||
│ │ └── server.ts # net.createServer bootstrap
|
||||
│ ├── adapters/
|
||||
│ │ └── teltonika/
|
||||
│ │ ├── index.ts # adapter entry — wires codecs into the registry
|
||||
│ │ ├── handshake.ts # IMEI handshake (length-prefixed ASCII)
|
||||
│ │ ├── frame.ts # outer envelope: preamble, length, codec ID, CRC
|
||||
│ │ ├── crc.ts # CRC-16/IBM
|
||||
│ │ └── codec/
|
||||
│ │ ├── data/ # Phase 1 — device → server telemetry
|
||||
│ │ │ ├── codec8.ts
|
||||
│ │ │ ├── codec8e.ts
|
||||
│ │ │ └── codec16.ts
|
||||
│ │ └── command/ # Phase 2 — server → device commands
|
||||
│ │ ├── codec12.ts
|
||||
│ │ ├── codec13.ts
|
||||
│ │ └── codec14.ts
|
||||
│ ├── config/
|
||||
│ │ └── load.ts # port/adapter bindings from env
|
||||
│ ├── observability/
|
||||
│ │ ├── logger.ts
|
||||
│ │ └── metrics.ts # Prometheus exporter
|
||||
│ └── main.ts # process entry point
|
||||
├── test/
|
||||
│ └── fixtures/
|
||||
│ └── teltonika/ # hex captures + expected Position[] (§5.6)
|
||||
│ ├── codec8/
|
||||
│ ├── codec8e/
|
||||
│ └── codec16/
|
||||
├── Dockerfile
|
||||
├── package.json
|
||||
├── tsconfig.json
|
||||
└── README.md
|
||||
```
|
||||
|
||||
Three rules govern this layout:
|
||||
|
||||
1. **`core/` never imports from `adapters/`.** The shell is vendor-agnostic by construction. If a piece of code in `core/` needs to know about Teltonika, it belongs in `adapters/teltonika/` instead.
|
||||
2. **Adapters never import from one another.** `adapters/teltonika/` and any future `adapters/queclink/` are independent. Shared utilities (e.g. length-prefix buffer accumulation) live in `core/`, not in a sibling adapter.
|
||||
3. **The Teltonika folder is self-contained.** Lifting Teltonika out into its own service later (see the system architecture's discussion of monolith-vs-services) is a `git mv adapters/teltonika/ ../gps-ingest-teltonika/src/adapter/` plus a copy of `core/`. No untangling required.
|
||||
|
||||
The Phase 2 command codecs (`codec/command/`) are listed here so the directory structure is allocated up front. The folder is empty in Phase 1; nothing imports from it. This is deliberate — when Phase 2 lands, contributors do not need to invent a location for the new code or risk drift from the parsers in `codec/data/`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Scope of protocol support
|
||||
|
||||
Teltonika defines two families of codecs:
|
||||
|
||||
- **Data-sending codecs** — device → server AVL telemetry (positions and IO elements).
|
||||
- **GPRS command codecs** — bidirectional command channel (server-issued configuration commands and device responses).
|
||||
|
||||
The Ingestion service supports these in two phases:
|
||||
|
||||
| Phase | Codecs | Purpose | Status |
|
||||
|-------|--------|---------|--------|
|
||||
| **Phase 1 — Now** | Codec 8, Codec 8 Extended (8E), Codec 16 | Device-to-server telemetry ingestion | In scope |
|
||||
| **Phase 2 — Later** | Codec 12, Codec 13, Codec 14 | Server-to-device commands (configuration, remote actions) | Reserved, not implemented |
|
||||
|
||||
Phase 1 covers essentially the entire deployed Teltonika telemetry fleet. Phase 2 is a distinct feature with different security implications (writing to devices, not just reading from them) and is intentionally deferred until the platform actually needs to issue commands. The architecture leaves room for it without requiring it now — see [§8. Forward compatibility for command codecs](#8-forward-compatibility-for-command-codecs).
|
||||
|
||||
---
|
||||
|
||||
## 4. Why the parser is model-agnostic
|
||||
|
||||
The Teltonika protocol is structured so that the device's **model** is irrelevant to the parser. What the parser cares about is the **codec ID**, which is announced in the first byte of every AVL data payload.
|
||||
|
||||
Every Teltonika TCP AVL data frame has the same outer envelope:
|
||||
|
||||
```
|
||||
┌──────────────┬──────────────┬───────────┬─────────────┬─────┐
|
||||
│ Preamble │ Data length │ Codec ID │ AVL data │ CRC │
|
||||
│ 4 bytes (0) │ 4 bytes │ 1 byte │ N bytes │ 4 B │
|
||||
└──────────────┴──────────────┴───────────┴─────────────┴─────┘
|
||||
```
|
||||
|
||||
The codec ID byte selects the parser for the AVL data section. Inside an AVL record, the fixed fields (timestamp, latitude, longitude, altitude, angle, satellites, speed) are always present in the same shape for a given codec; only the **IO element bag** varies between device models, and the bag is carried as `{ id → value }` pairs that the parser reads byte-correctly without needing to know what each `id` means.
|
||||
|
||||
This means a brand-new Teltonika model the team has never seen will:
|
||||
|
||||
- Parse correctly at the frame and codec layers (envelope, CRC, codec dispatch all unchanged).
|
||||
- Produce a correct `Position` record (lat, lon, speed, timestamp are codec-defined, not model-defined).
|
||||
- Carry its IO elements through as raw integer keys until a model-aware mapping is added downstream.
|
||||
|
||||
The Ingestion service's job ends at the normalized `Position`. Naming and interpreting IO elements is explicitly a Processor concern, driven by per-model configuration.
|
||||
|
||||
---
|
||||
|
||||
## 5. Design principles
|
||||
|
||||
The following six principles govern the Teltonika adapter. They are listed in priority order — when they conflict, earlier principles win.
|
||||
|
||||
### 5.1 Implement Codec 8, 8E, and 16 — the closed set for telemetry
|
||||
|
||||
Phase 1 ships with three codec parsers and no others on the data-sending side:
|
||||
|
||||
- **Codec 8** — the baseline AVL data format. 8-bit IO element IDs. Older firmware.
|
||||
- **Codec 8 Extended (8E)** — same record shape as Codec 8, but 16-bit IO element IDs (Teltonika ran out of 8-bit slots) and variable-length IO values. This is the default on most modern FMB/FMC/FMM/FMU devices.
|
||||
- **Codec 16** — adds a generation type per record. Less common but appears in some configurations.
|
||||
|
||||
These three cover the deployed Teltonika fleet for telemetry purposes. The set is closed: Teltonika releases new data-sending codecs rarely, and when they do, support is added by registering a new parser keyed on the codec ID byte — the rest of the pipeline is unaffected.
|
||||
|
||||
The parser dispatch is a flat lookup, not an inheritance hierarchy. There is no "shared base codec" — codec parsers are independent because their record shapes diverge in ways that abstraction would obscure rather than help.
|
||||
|
||||
### 5.2 Defer Codec 12, 13, and 14 — server-to-device commands
|
||||
|
||||
The command codecs are out of scope for Phase 1:
|
||||
|
||||
- **Codec 12** — server-issued commands and device responses (text strings).
|
||||
- **Codec 13** — same as 12, with timestamps.
|
||||
- **Codec 14** — same as 12, addressed by IMEI (multi-device server scenarios).
|
||||
|
||||
These are deferred because they are a **distinct feature, not an incremental codec**. Supporting them requires:
|
||||
|
||||
- A way to enqueue commands targeted at specific devices.
|
||||
- Routing those commands to whichever Ingestion instance currently holds the device's connection.
|
||||
- Permissioned APIs upstream so commands cannot be issued by unauthorized callers.
|
||||
- Audit trails for every command issued and every response received.
|
||||
|
||||
None of this is needed to read telemetry. Building it speculatively would either ship dead code or, worse, ship half-built infrastructure that someone later mistakes for usable. The Phase 1 service ignores command codecs entirely; Phase 2 will introduce them as a deliberate addition. See [§8. Forward compatibility for command codecs](#8-forward-compatibility-for-command-codecs) for the seams that make this addition cheap.
|
||||
|
||||
### 5.3 Pass the IO map through unchanged
|
||||
|
||||
The Ingestion service does **not** name, interpret, or filter IO elements. It produces records of this shape:
|
||||
|
||||
```ts
|
||||
type Position = {
|
||||
device_id: string // IMEI from the handshake
|
||||
timestamp: Date // from the AVL record's GPS timestamp
|
||||
latitude: number
|
||||
longitude: number
|
||||
altitude: number
|
||||
angle: number // heading, 0-360
|
||||
speed: number // km/h
|
||||
satellites: number
|
||||
priority: number // codec-defined priority field
|
||||
attributes: {
|
||||
[io_id: string]: number | bigint | Buffer
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`attributes` is a verbatim representation of the IO element bag from the AVL record, keyed by the numeric IO element ID as a string. No renaming. No unit conversion. No model lookup.
|
||||
|
||||
This is deliberate. Two reasons:
|
||||
|
||||
- **Model-specific interpretation belongs where the model is known.** The Processor configures per-model IO mappings (e.g. `{ "FMB920": { "16": "odometer_km", "240": "movement" } }`); doing this in the parser would couple Ingestion to a registry of every device model in the fleet.
|
||||
- **A new model never breaks Ingestion.** If we receive a packet with `IO 1234` from a device whose config we have not yet imported, the parser stores the raw value under `"1234"` and moves on. The position is still useful; the attribute is recoverable later.
|
||||
|
||||
Adapters do not own the IO dictionary. They produce raw IO maps.
|
||||
|
||||
### 5.4 Log unknown codec IDs and drop the connection
|
||||
|
||||
When a packet's codec ID byte does not match a parser the service knows about, the connection is closed without sending an ACK. Specifically:
|
||||
|
||||
- A `WARN`-level log entry is emitted with the IMEI, the offending codec ID, and the raw header bytes.
|
||||
- The socket is destroyed.
|
||||
- No partial attempt is made to "skip ahead" or guess the record layout.
|
||||
|
||||
The reasoning: a Teltonika device sending an unrecognized codec is **misconfigured**, not subtly broken. Silently truncating its data — or worse, mis-parsing it — produces records with plausible-looking but wrong coordinates. A loud failure (the device reconnects, fails again, shows up in logs) is strictly better than a quiet corruption.
|
||||
|
||||
Phase 2 will widen the recognized set to include 12/13/14 on the same socket; until then, those codec IDs are treated like any other unknown value.
|
||||
|
||||
### 5.5 Validate the CRC; NACK on mismatch
|
||||
|
||||
Every AVL data frame ends with a 4-byte CRC-16/IBM (the lower 2 bytes carry the CRC; the upper 2 are zero). The parser computes the CRC over the data section and compares.
|
||||
|
||||
On mismatch:
|
||||
|
||||
- The frame is **not** ACK'd. Teltonika devices retransmit unacknowledged data on the next session, so a missing ACK is the correct way to ask for a resend.
|
||||
- A `WARN` log entry records the IMEI, expected CRC, computed CRC, and frame length.
|
||||
- The connection remains open — a single corrupt frame is a transient transmission issue, not a reason to drop the session.
|
||||
|
||||
Repeated CRC failures from the same device within a short window indicate a deeper problem (firmware bug, line-quality issue) and should be surfaced through metrics, not just logs. See [§7. Observability](#7-observability).
|
||||
|
||||
### 5.6 Maintain a fixture suite of real packet captures
|
||||
|
||||
Binary protocol bugs are silent. A wrong byte offset produces wrong coordinates, not an exception. The parser cannot rely on type checking, schema validation, or runtime errors to catch its own mistakes — the only reliable defense is regression tests against known-good captures.
|
||||
|
||||
The fixture suite consists of:
|
||||
|
||||
- **Hex dumps of real frames** from devices we have ingested in production, one per codec (8, 8E, 16) at minimum, ideally several per codec covering different IO element shapes.
|
||||
- **Hex dumps from Teltonika's official protocol documentation** — these are the canonical reference captures and should always pass.
|
||||
- **Synthetic edge cases** — empty IO bags, maximum-size IO values, multi-record frames, frames near the length limit.
|
||||
|
||||
Each fixture is paired with the expected parsed output (a `Position[]`). Tests run on every CI build. New captures are added whenever a new device model is onboarded or a parser bug is fixed in production — the fix and the regression fixture are always committed together.
|
||||
|
||||
A fixture suite is not optional infrastructure. It is the only place the parser's correctness is actually verified.
|
||||
|
||||
---
|
||||
|
||||
## 6. The TCP session lifecycle
|
||||
|
||||
For reference, the full happy-path session flow under Phase 1:
|
||||
|
||||
1. **Device connects** to the Teltonika port (per-vendor port, see system architecture §3).
|
||||
2. **IMEI handshake**:
|
||||
- Device sends 2 bytes of length followed by the ASCII IMEI.
|
||||
- Server responds with `0x01` to accept (or `0x00` to reject; current Phase 1 always accepts).
|
||||
3. **AVL data loop** — repeatedly:
|
||||
- Read 4-byte preamble (must be `0x00000000`; anything else is a framing error).
|
||||
- Read 4-byte data length.
|
||||
- Read `length` bytes of payload.
|
||||
- Read 4-byte CRC.
|
||||
- Validate CRC. On mismatch, do not ACK; log; continue reading.
|
||||
- Inspect byte 0 of payload — the codec ID.
|
||||
- Dispatch to the matching codec parser.
|
||||
- For each AVL record produced, build a normalized `Position` and `publish` it to the Redis Stream.
|
||||
- Send 4-byte big-endian record-count ACK back to the device.
|
||||
4. **Connection close** — device-initiated or due to network failure. The session ends; the device will reconnect when it has more data. No state is preserved across sessions.
|
||||
|
||||
Phase 2 will add a parallel branch after step 3's codec dispatch: codec IDs 12/13/14 will route to a command-codec handler instead of the AVL parser. The framing envelope (steps 1, 2, and the read part of 3) is shared.
|
||||
|
||||
---
|
||||
|
||||
## 7. Observability
|
||||
|
||||
Per-adapter metrics for Teltonika:
|
||||
|
||||
- **`teltonika_connections_active`** — gauge of currently open device sessions.
|
||||
- **`teltonika_handshake_total{result="accepted|rejected|malformed"}`** — IMEI handshake outcomes.
|
||||
- **`teltonika_frames_total{codec="8|8E|16|unknown", result="ok|crc_fail|truncated"}`** — frame-level outcomes, partitioned by codec.
|
||||
- **`teltonika_records_published_total{codec}`** — AVL records emitted to the Redis Stream.
|
||||
- **`teltonika_parse_duration_seconds{codec}`** — histogram of per-frame parse time.
|
||||
- **`teltonika_unknown_codec_total{codec_id}`** — counter of dropped connections due to unrecognized codec IDs. A non-zero rate here is an alert: it means devices are configured for codecs the service does not support.
|
||||
|
||||
The "unknown codec" counter is the canary for codec coverage drift — if Teltonika ships a new codec or a customer reconfigures devices to use one we have not implemented, this metric will surface it immediately.
|
||||
|
||||
---
|
||||
|
||||
## 8. Forward compatibility for command codecs
|
||||
|
||||
Phase 2 adds outbound command support (codecs 12, 13, 14). The Phase 1 design accommodates this without requiring upfront work, by respecting three seams:
|
||||
|
||||
### 8.1 The codec dispatch is a registry, not a switch
|
||||
|
||||
The codec ID byte indexes into a registered set of handlers. Each handler implements the same interface:
|
||||
|
||||
```ts
|
||||
interface CodecHandler {
|
||||
codec_id: number
|
||||
handle(
|
||||
payload: Buffer,
|
||||
ctx: { imei: string; publish: (p: Position) => Promise<void> }
|
||||
): Promise<{ ack_count: number }>
|
||||
}
|
||||
```
|
||||
|
||||
Phase 1 registers handlers for IDs 8, 8E (`0x8E`), and 16. Phase 2 registers additional handlers for 12, 13, 14 — these handlers will use a different shape of `ctx` (they need to send bytes back, not just `publish`), but the registry shape is the same.
|
||||
|
||||
### 8.2 The session owns the socket; the codec handler borrows it
|
||||
|
||||
Today, codec handlers receive the parsed payload and emit `Position` records via `publish`. They do not write to the socket directly — the session loop handles ACKs.
|
||||
|
||||
Phase 2's command handlers will need to **write to the socket** (to send commands to the device). This is supported by passing a `respond(bytes: Buffer)` callback into the handler context for command codecs, alongside `publish`. The session retains ownership of the socket; handlers borrow write access through a narrow interface.
|
||||
|
||||
### 8.3 Connection-to-device routing is a future-Phase-2 concern
|
||||
|
||||
To send a command to a specific device, the platform must route the command to whichever Ingestion instance currently holds that device's open socket. Phase 1 does not need this (no commands exist). Phase 2 will introduce a connection registry (likely a Redis hash mapping `imei → instance_id`) and a command queue per instance. None of this affects Phase 1 code; it is additive.
|
||||
|
||||
The Phase 1 design intentionally keeps the per-device session state local to the socket (no shared state, no registry). When Phase 2 lands, the registry is added alongside, not woven through.
|
||||
|
||||
---
|
||||
|
||||
## 9. Phase 2 — Outbound commands
|
||||
|
||||
This section specifies the command-delivery design that Phase 2 will introduce. It is included now so that Phase 1 code can respect the seams; no Phase 2 code is shipped until the feature is actually needed.
|
||||
|
||||
### 9.1 Architectural posture
|
||||
|
||||
The Ingestion service **does not expose user-facing HTTP endpoints**, in Phase 1 or Phase 2. This is non-negotiable: the system architecture (`gps-tracking-architecture.md` §5.1, §7.5) places all user-facing API surface inside Directus, and any deviation would split the auth surface, couple deployment lifecycles, and contaminate failure domains.
|
||||
|
||||
Commands originate at the SPA, are authorized and persisted by Directus, and reach the Ingestion instance that holds the device's socket via Redis. The Ingestion service only learns about commands by consuming its own stream — it never accepts inbound user-facing traffic.
|
||||
|
||||
End-to-end flow:
|
||||
|
||||
```
|
||||
SPA ──HTTPS+JWT──▶ Directus ──XADD──▶ Redis Streams ──XREAD──▶ Ingestion
|
||||
│ │
|
||||
│ │ writes Codec 12 frame
|
||||
│ │ to device socket
|
||||
│ │
|
||||
SPA ◀──WSS subscription── Directus ◀──hook on insert── commands:responses (XADD by Ingestion)
|
||||
```
|
||||
|
||||
Five properties to hold onto:
|
||||
|
||||
1. **Single auth surface.** Directus enforces "can this user issue commands to this device?" — the same machinery that authorizes every other write in the system. The Ingestion service treats anything that arrives on its command stream as already-authorized.
|
||||
2. **Commands are data before they are transport.** Every command is a row in a `commands` collection before it is delivered. The Redis stream is the transport, not the source of truth.
|
||||
3. **Symmetric to inbound telemetry.** Inbound: device → Ingestion → Redis → Processor → Directus → SPA. Outbound: SPA → Directus → Redis → Ingestion → device. Same plane boundary, same seam, same operational tools.
|
||||
4. **Per-instance routing via a connection registry.** Only the instance holding the device's socket can deliver the command, so the registry maps `imei → instance_id` and the issuer publishes to that instance's stream specifically.
|
||||
5. **Real-time status updates for free.** Directus's WebSocket subscriptions on the `commands` collection push delivery status to the SPA without any new transport.
|
||||
|
||||
### 9.2 The `commands` collection
|
||||
|
||||
Owned by Directus. The schema:
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `id` | uuid | Primary key. Used as the correlation ID across the system. |
|
||||
| `target_imei` | string | The device the command is bound for. |
|
||||
| `batch_id` | uuid (nullable) | Set when the command is part of a fleet operation; null for single-device commands. |
|
||||
| `codec` | enum (`12`, `13`, `14`) | Which Teltonika command codec to use. Defaults to `12`. |
|
||||
| `payload` | string | The command text (Teltonika command codecs carry ASCII strings — e.g. `"setdigout 1"`). |
|
||||
| `status` | enum | `pending`, `routed`, `delivered`, `responded`, `failed`, `expired`. |
|
||||
| `requested_by` | user fk | Set automatically by Directus from the JWT. |
|
||||
| `requested_at` | timestamp | Insert time. |
|
||||
| `routed_at` | timestamp | When the issuer published to Redis. |
|
||||
| `delivered_at` | timestamp | When the Ingestion instance wrote bytes to the socket. |
|
||||
| `responded_at` | timestamp | When the device's response arrived. |
|
||||
| `response` | text | The device's response string, if any. |
|
||||
| `failure_reason` | string | Set on `failed` or `expired`. Free text. |
|
||||
| `expires_at` | timestamp | When the command should be abandoned if not delivered. Default: `requested_at + 5 minutes`. |
|
||||
|
||||
Permissions: the `commands` collection is writable by roles that have command-issuing privileges (operator, admin) and readable by the requester plus admin roles. The SPA inserts via the Directus SDK; nothing else writes to this collection except the Ingestion service updating delivery status (via a service token with restricted scope).
|
||||
|
||||
The collection is a hypertable candidate if command volume becomes high — but at expected rates (occasional operator actions, periodic fleet pushes), a regular table with an index on `(target_imei, requested_at desc)` is sufficient.
|
||||
|
||||
### 9.3 The connection registry
|
||||
|
||||
A Redis hash, keyed by IMEI, valued by Ingestion instance identifier:
|
||||
|
||||
```
|
||||
HSET connections:registry {imei} {instance_id}
|
||||
EXPIRE connections:registry 0 # the hash itself does not expire
|
||||
```
|
||||
|
||||
Per-entry freshness is maintained by **heartbeat**, not by hash-level TTL (Redis does not support per-field TTLs on hashes). Each Ingestion instance:
|
||||
|
||||
- On device handshake (after IMEI is known), writes `HSET connections:registry {imei} {instance_id}` and records the IMEI in a local `Set<string>` of held devices.
|
||||
- Every 30 seconds, refreshes a per-instance heartbeat key: `SET instance:heartbeat:{instance_id} {now} EX 90`.
|
||||
- On socket close, `HDEL connections:registry {imei}` and removes the IMEI from the local set.
|
||||
- On graceful shutdown, `HDEL` for every held IMEI.
|
||||
|
||||
Crash recovery — when an instance dies without cleanup, its registry entries are stale. A small **registry janitor** (a periodic Directus Flow, or a dedicated lightweight process) runs every minute:
|
||||
|
||||
- For each `instance_id` referenced in `connections:registry`, check `EXISTS instance:heartbeat:{instance_id}`.
|
||||
- If the heartbeat is missing, the instance is dead. Scan the registry for entries pointing to it and `HDEL` them.
|
||||
|
||||
Devices held by the dead instance will reconnect (devices are configured to retry on disconnect; system architecture §3.2), and the new instance will write a fresh registry entry on handshake. Pending commands targeting those devices will be re-routed when the device reappears (see §9.5).
|
||||
|
||||
### 9.4 Issuing a command
|
||||
|
||||
Two paths, depending on whether the command is single-device or fleet:
|
||||
|
||||
**Single device** — the SPA inserts a row into `commands` via the Directus SDK. A Directus Flow on `items.create` for the `commands` collection:
|
||||
|
||||
1. Looks up the instance: `instance_id = HGET connections:registry {target_imei}`.
|
||||
2. If found:
|
||||
- `XADD commands:outbound:{instance_id} * command_id {id} target_imei {target_imei} codec {codec} payload {payload} expires_at {expires_at}`.
|
||||
- Updates the row to `status = routed`, `routed_at = now()`.
|
||||
3. If not found (device is offline):
|
||||
- Updates the row to `status = pending`. The command will be retried by the **pending-command sweeper** (§9.5).
|
||||
|
||||
**Fleet (list of devices)** — the SPA calls a custom Directus endpoint, `POST /commands/batch`, which:
|
||||
|
||||
1. Validates the batch (max size, target devices the user is authorized for).
|
||||
2. Generates a `batch_id`.
|
||||
3. In a single transaction, inserts N rows into `commands` with the shared `batch_id`.
|
||||
4. After commit, for each row, performs the registry lookup and stream publish as in the single-device path.
|
||||
5. Returns `{ batch_id, command_ids: [...] }` to the SPA.
|
||||
|
||||
The custom endpoint exists for fleet operations specifically because the transactional insert + the routing fan-out is cleaner in code than in a Flow. Single-device commands stay on the Flow path; the endpoint is a layer above for batch ergonomics.
|
||||
|
||||
The SPA tracks progress by subscribing to `commands` filtered by `batch_id` (or `id`) — Directus pushes WebSocket updates as the rows transition through statuses.
|
||||
|
||||
### 9.5 The pending-command sweeper
|
||||
|
||||
When a device is offline at command time, the row sits in `status = pending`. A sweeper Flow runs every 30 seconds:
|
||||
|
||||
1. Selects rows where `status = pending` and `expires_at > now()`.
|
||||
2. For each row, attempts the registry lookup again.
|
||||
3. If the device is now online, publishes to the instance's stream and transitions `status → routed`.
|
||||
4. Selects rows where `status IN ('pending', 'routed')` and `expires_at <= now()`, transitions them to `status = expired` with `failure_reason = 'device offline'` (for pending) or `'no delivery confirmation'` (for routed).
|
||||
|
||||
The sweeper is also what handles the case where an Ingestion instance crashes after publishing-to-stream but before delivering — those rows sit in `routed` past their `expires_at` and get expired. The next time the device reconnects, the operator can re-issue.
|
||||
|
||||
(There's a subtler retry option — re-route stale `routed` rows when their original instance has died according to the registry janitor — but it's an enhancement, not a Phase 2 requirement. Expire-and-let-the-operator-retry is acceptable v1 behavior and avoids accidental duplicate delivery.)
|
||||
|
||||
### 9.6 The Ingestion-side command consumer
|
||||
|
||||
Each Ingestion instance, alongside its TCP listener, runs a **command consumer** that reads from `commands:outbound:{instance_id}`:
|
||||
|
||||
```
|
||||
loop:
|
||||
msgs = XREADGROUP commands:outbound:{instance_id} GROUP ingest {instance_id} COUNT 16 BLOCK 1000
|
||||
for msg in msgs:
|
||||
handle_command(msg)
|
||||
XACK commands:outbound:{instance_id} ingest msg.id
|
||||
```
|
||||
|
||||
`handle_command` does:
|
||||
|
||||
1. Look up the in-memory map of `imei → socket` for this instance. If the device has disconnected since the registry lookup at issue time, publish a failure to `commands:responses` with `reason = 'socket_closed'` and stop.
|
||||
2. Check `expires_at`. If already expired, publish a failure with `reason = 'expired_before_delivery'`.
|
||||
3. Encode the Codec 12/13/14 frame from the payload (see §9.7).
|
||||
4. Write the frame bytes to the socket. Record `delivered_at` by publishing an interim status.
|
||||
5. Register a pending-response entry keyed by `command_id` with a timeout (default: 30 seconds).
|
||||
6. Return to the read loop.
|
||||
|
||||
The consumer never blocks the TCP read path — it runs in parallel. The shared resource (the socket) is accessed through a per-socket write queue so that command writes do not interleave with codec ACKs in a way that confuses the device.
|
||||
|
||||
### 9.7 Encoding command frames (Codec 12 reference)
|
||||
|
||||
For Codec 12, the over-the-wire frame is:
|
||||
|
||||
```
|
||||
┌──────────┬──────────┬───────┬──────┬──────────┬─────────┬───────┬──────┐
|
||||
│ Preamble │ Length │ Codec │ Cmd │ Cmd size │ Cmd │ Count │ CRC │
|
||||
│ 4 bytes │ 4 bytes │ 0x0C │ Qty │ 4 bytes │ N bytes │ 1 byte│ 4 B │
|
||||
│ (zero) │ │ │ 1 B │ │ (ASCII) │ │ │
|
||||
└──────────┴──────────┴───────┴──────┴──────────┴─────────┴───────┴──────┘
|
||||
▲ ▲
|
||||
└─── usually 0x05 ─────────────────┘
|
||||
(Type = command from server)
|
||||
```
|
||||
|
||||
Codec 13 adds a 4-byte timestamp; Codec 14 prepends an 8-byte IMEI. The encoder is a small set of pure functions in the `adapters/teltonika/codec/command/` folder, mirroring the AVL parsers in `adapters/teltonika/codec/data/`. They are tested against captures from Teltonika's documentation and from real device exchanges, exactly like the data codec parsers (§5.6).
|
||||
|
||||
### 9.8 Response correlation
|
||||
|
||||
When the device responds to a command, it sends a Codec 12 frame back over the same socket with `Type = 0x06` (response from device). The session's codec dispatch routes this to the command response handler.
|
||||
|
||||
The handler:
|
||||
|
||||
1. Reads the response payload (ASCII string).
|
||||
2. Looks up the pending command for this socket. Teltonika's command codecs do not carry an explicit correlation ID — the protocol assumes one outstanding command per connection at a time. The Ingestion service enforces this: only one command may be in-flight per socket. Subsequent commands for the same device queue on the per-socket write queue and are sent only after the previous response arrives or its timeout fires.
|
||||
3. Publishes to `commands:responses`: `XADD commands:responses * command_id {id} response {text} responded_at {now}`.
|
||||
4. Cancels the pending-response timeout.
|
||||
|
||||
A Directus hook on the `commands:responses` stream (or a dedicated small consumer that updates Directus via SDK) updates the row to `status = responded`, fills `response` and `responded_at`, and Directus broadcasts the change to subscribed SPA clients.
|
||||
|
||||
If the timeout fires without a response, the consumer publishes `command_id, reason = 'no_device_response'` to `commands:responses`, the row transitions to `status = failed`, and the per-socket write queue is freed for the next command.
|
||||
|
||||
### 9.9 What this adds to Phase 1 code
|
||||
|
||||
Concretely, the Phase 1 implementation must respect these shapes so Phase 2 is a pure addition:
|
||||
|
||||
- **Codec dispatch is a registry** keyed on codec ID byte (already specified in §8.1). Phase 2 registers handlers for `0x0C`, `0x0D`, `0x0E`.
|
||||
- **The session loop owns the socket; handlers borrow it** through a `respond(bytes)` callback (already specified in §8.2). Phase 1 handlers simply do not use it.
|
||||
- **Per-device runtime state is local to the socket and the holding instance.** No shared state today (§8.3). Phase 2 adds the connection registry and per-instance command stream alongside, not woven through.
|
||||
- **The `Position` shape and the inbound stream are unchanged.** Outbound commands use entirely separate streams (`commands:outbound:{instance_id}`, `commands:responses`) and a separate Directus collection. There is no conflation with the telemetry path.
|
||||
|
||||
When Phase 2 ships, no Phase 1 code is rewritten — Phase 1 continues to do exactly what it does today, with the command consumer running alongside it.
|
||||
|
||||
---
|
||||
|
||||
## 10. Summary
|
||||
|
||||
The Teltonika adapter handles model diversity by leaning on the protocol's own self-description. The codec ID announces the framing; the IO bag carries model-specific telemetry without the parser interpreting it. Six principles keep the implementation honest:
|
||||
|
||||
1. **Implement Codec 8, 8E, 16** — the closed set for telemetry today.
|
||||
2. **Defer Codec 12, 13, 14** — command codecs reserved for Phase 2; design leaves room without building it.
|
||||
3. **Pass the IO map through unchanged** — naming and interpretation are downstream concerns.
|
||||
4. **Drop on unknown codec IDs** — loud failure beats silent corruption.
|
||||
5. **Validate the CRC; NACK on mismatch** — devices retransmit on missing ACKs.
|
||||
6. **Maintain a fixture suite** — the only real defense against silent binary-protocol bugs.
|
||||
|
||||
Together, these mean any Teltonika device shipping codec 8, 8E, or 16 is supported on day one, and the path to outbound commands later is a clean addition rather than a rewrite.
|
||||
@@ -0,0 +1,138 @@
|
||||
---
|
||||
title: AVL Data Format (Teltonika canonical)
|
||||
type: concept
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [teltonika-data-sending-protocols, teltonika-ingestion-architecture]
|
||||
tags: [teltonika, protocol, parser, data-model]
|
||||
---
|
||||
|
||||
# AVL Data Format
|
||||
|
||||
The canonical Teltonika AVL packet structure across codecs 8, 8E, and 16. This is the byte-level reference the [[teltonika]] adapter implements. See [[teltonika-data-sending-protocols]] for the official source and [[codec-dispatch]] for how the codec ID byte selects between the three formats.
|
||||
|
||||
## Outer envelope (TCP)
|
||||
|
||||
```
|
||||
┌────────────────┬──────────────────┬──────────┬────┬──────────┬────┬───────┐
|
||||
│ Preamble │ Data Field Length│ Codec ID │ N1 │ AVL Data │ N2 │ CRC-16│
|
||||
│ 4B = 0x00000000│ 4B │ 1B │ 1B │ X bytes │ 1B │ 4B │
|
||||
└────────────────┴──────────────────┴──────────┴────┴──────────┴────┴───────┘
|
||||
```
|
||||
|
||||
- **Data Field Length** is computed from the start of `Codec ID` through `N2` (not the whole packet).
|
||||
- **CRC-16/IBM** is computed over the same range. Lower 2 bytes of the 4-byte CRC field carry the value; upper 2 bytes are zero.
|
||||
- **N1 must equal N2** — record count repeated for integrity.
|
||||
- Server ACK is a **4-byte big-endian integer** equal to the number of records accepted. Mismatched count → device retransmits.
|
||||
|
||||
## AVL record (one of N records)
|
||||
|
||||
```
|
||||
[Timestamp 8B][Priority 1B][GPS Element 15B][IO Element X B]
|
||||
```
|
||||
|
||||
- **Timestamp**: UNIX milliseconds since epoch (UTC), big-endian.
|
||||
- **Priority**: enum — `0` Low, `1` High, `2` Panic.
|
||||
|
||||
### GPS Element (15B)
|
||||
|
||||
```
|
||||
[Longitude 4B][Latitude 4B][Altitude 2B][Angle 2B][Satellites 1B][Speed 2B]
|
||||
```
|
||||
|
||||
- **Lat/Lon**: signed 32-bit integer = `((d + m/60 + s/3600 + ms/3600000) × 10⁷)`, where d/m/s/ms are degrees/minutes/seconds/milliseconds. **Two's complement** — first bit = sign (0 positive, 1 negative).
|
||||
- **Altitude**: meters above sea level, signed 16-bit.
|
||||
- **Angle**: heading 0–360°, unsigned 16-bit.
|
||||
- **Satellites**: count of satellites in use, unsigned 8-bit.
|
||||
- **Speed**: km/h, unsigned 16-bit. **`0x0000` when GPS data is invalid** — distinct from "stationary."
|
||||
|
||||
## IO Element — codec-specific
|
||||
|
||||
The IO Element layout is where the three codecs diverge. The parser must dispatch on codec ID to read these correctly.
|
||||
|
||||
### Codec 8 (`0x08`) — 1-byte everything
|
||||
|
||||
```
|
||||
[Event IO ID 1B]
|
||||
[N total 1B]
|
||||
[N1 1B] [IO ID 1B][Value 1B] × N1
|
||||
[N2 1B] [IO ID 1B][Value 2B] × N2
|
||||
[N4 1B] [IO ID 1B][Value 4B] × N4
|
||||
[N8 1B] [IO ID 1B][Value 8B] × N8
|
||||
```
|
||||
|
||||
### Codec 8 Extended (`0x8E`) — 2-byte fields + variable-length
|
||||
|
||||
```
|
||||
[Event IO ID 2B]
|
||||
[N total 2B]
|
||||
[N1 2B] [IO ID 2B][Value 1B] × N1
|
||||
[N2 2B] [IO ID 2B][Value 2B] × N2
|
||||
[N4 2B] [IO ID 2B][Value 4B] × N4
|
||||
[N8 2B] [IO ID 2B][Value 8B] × N8
|
||||
[NX 2B] [IO ID 2B][Length 2B][Value <Length>B] × NX ← variable-length section
|
||||
```
|
||||
|
||||
The **NX section** is unique to 8E. Carries arbitrary-length values (e.g. ICCID-class data, BLE payloads). Each entry self-describes its length. The parser must be length-aware here — getting it wrong silently corrupts subsequent records.
|
||||
|
||||
### Codec 16 (`0x10`) — Generation Type, mixed widths
|
||||
|
||||
```
|
||||
[Event IO ID 2B]
|
||||
[Generation Type 1B] ← unique to Codec 16
|
||||
[N total 1B]
|
||||
[N1 1B] [IO ID 2B][Value 1B] × N1
|
||||
[N2 1B] [IO ID 2B][Value 2B] × N2
|
||||
[N4 1B] [IO ID 2B][Value 4B] × N4
|
||||
[N8 1B] [IO ID 2B][Value 8B] × N8
|
||||
```
|
||||
|
||||
- **No NX section** — Codec 16 does not include variable-size IO elements.
|
||||
- **Generation Type** values: `0`=On Exit, `1`=On Entrance, `2`=On Both, `3`=Reserved, `4`=Hysteresis, `5`=On Change, `6`=Eventual, `7`=Periodical.
|
||||
- Codec 16 is the channel for AVL IDs > 255 on FMB630/FM63XY.
|
||||
|
||||
### Side-by-side
|
||||
|
||||
| | Codec 8 | Codec 8 Extended | Codec 16 |
|
||||
|---|---------|------------------|----------|
|
||||
| Codec ID | 0x08 | 0x8E | 0x10 |
|
||||
| Event IO ID width | 1B | 2B | 2B |
|
||||
| N total / Nk count widths | 1B / 1B | 2B / 2B | 1B / 1B |
|
||||
| IO ID width | 1B | 2B | 2B |
|
||||
| Generation Type | — | — | 1B |
|
||||
| Variable-length IO (NX) | — | yes | — |
|
||||
| AVL IDs > 255 supported | no | yes | yes |
|
||||
|
||||
## Size limits
|
||||
|
||||
- **Minimum AVL record**: 45 bytes (all IO elements disabled).
|
||||
- **Maximum AVL record**: 255 bytes.
|
||||
- **Maximum AVL packet**: 512 bytes for FMB640/FMB641/FMC640/FMM640; **1280 bytes for other devices**.
|
||||
|
||||
The parser should treat oversized packets as malformed (drop the connection per the [[codec-dispatch]] unknown-codec policy — though this is a different malformation, the same loud-failure principle applies).
|
||||
|
||||
## UDP envelope
|
||||
|
||||
When the device runs over UDP instead of TCP:
|
||||
|
||||
```
|
||||
[UDP Channel Header 5B] [AVL Packet Header (1+2+15)B] [AVL Data Array — same as TCP]
|
||||
```
|
||||
|
||||
- **UDP Channel Header**: `Length 2B + Packet ID 2B + Not Usable Byte 1B`.
|
||||
- **AVL Packet Header**: `AVL Packet ID 1B + IMEI Length 2B = 0x000F + IMEI 15B`.
|
||||
- Server ACK over UDP is short: `[UDP Channel Header 5B][AVL Packet ID 1B][Number Accepted 1B]`.
|
||||
|
||||
The AVL Data Array (Codec ID byte + records + N2) is identical to TCP. **The Phase 1 implementation is TCP-only; UDP is documented here for completeness and future evaluation.**
|
||||
|
||||
## Mapping to [[position-record]]
|
||||
|
||||
The parser writes:
|
||||
|
||||
- `device_id` ← IMEI from the handshake (or from the UDP AVL Packet Header).
|
||||
- `timestamp` ← AVL record `Timestamp` (UNIX ms → `Date`).
|
||||
- `latitude`, `longitude`, `altitude`, `angle`, `speed`, `satellites` ← GPS Element fields, with two's-complement decoding for lat/lon and `Speed === 0x0000 ⇒ "GPS invalid"` retained as-is (the Processor decides how to surface it).
|
||||
- `priority` ← AVL record `Priority` (0/1/2).
|
||||
- `attributes` ← every IO element from N1/N2/N4/N8 (and NX for Codec 8E), keyed by **numeric IO ID as string**, value as `number | bigint | Buffer` per IO width. See [[io-element-bag]] for why naming/units stay out of the parser.
|
||||
|
||||
> Note: Codec 16's `Generation Type` and 8E's NX `Length` are not currently in the [[position-record]] shape. Generation Type is codec-defined (not model-defined) and may deserve a typed field — flagged as an open question on the source page.
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
title: Codec Dispatch (Registry)
|
||||
type: concept
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [teltonika-ingestion-architecture, teltonika-data-sending-protocols]
|
||||
tags: [teltonika, parser, design-seam]
|
||||
---
|
||||
|
||||
# Codec Dispatch
|
||||
|
||||
How the [[teltonika]] adapter routes incoming frames to the right parser, and the seam that makes [[phase-2-commands]] additive instead of a rewrite.
|
||||
|
||||
## The mechanism
|
||||
|
||||
The codec ID (byte 0 of the AVL data payload) indexes into a **flat registry** of handlers. Each handler implements:
|
||||
|
||||
```ts
|
||||
interface CodecHandler {
|
||||
codec_id: number
|
||||
handle(
|
||||
payload: Buffer,
|
||||
ctx: { imei: string; publish: (p: Position) => Promise<void> }
|
||||
): Promise<{ ack_count: number }>
|
||||
}
|
||||
```
|
||||
|
||||
Phase 1 registers handlers for IDs `0x08`, `0x8E`, and `0x10`. Phase 2 will register additional handlers for `0x0C`, `0x0D`, `0x0E` — these will use a different `ctx` shape (they need to write bytes back, not just `publish`), but the registry shape is identical.
|
||||
|
||||
Per-codec directionality matters for the Phase 2 handler shape:
|
||||
|
||||
| Codec | ID | Direction | Handler needs |
|
||||
|-------|-----|-----------|---------------|
|
||||
| 8 / 8E / 16 | `0x08` / `0x8E` / `0x10` | device → server | `publish(Position)` only |
|
||||
| 12 | `0x0C` | bidirectional | `respond(bytes)` for outbound + parse responses |
|
||||
| 13 | `0x0D` | device → server only | parse-only (no `respond`) |
|
||||
| 14 | `0x0E` | bidirectional | `respond(bytes)` + ACK type `0x06` / nACK type `0x11` |
|
||||
| 15 | `0x0F` | device → server only | out of scope (FMX6 RS232 only) |
|
||||
|
||||
Codec 13 and 15, despite living in the GPRS-message family, are *one-way* — handlers for them never write to the socket. This matters for handler-shape design: the Phase 2 outbound family is not "all command codecs," it's specifically `0x0C` and `0x0E`.
|
||||
|
||||
## Why a registry, not a switch
|
||||
|
||||
- Adding a new codec is a registration, not a code change in dispatch logic.
|
||||
- Phase 2 command codecs slot in alongside Phase 1 data codecs without modifying Phase 1 paths.
|
||||
- Codec parsers are **independent** — there is no shared base class. Their record shapes diverge in ways abstraction would obscure rather than help.
|
||||
|
||||
## The "session owns the socket; handler borrows it" rule
|
||||
|
||||
Phase 1 handlers receive payload + a `publish(Position)` callback and emit records via Redis. They never write to the socket directly — the session loop handles ACKs.
|
||||
|
||||
Phase 2 command handlers will need to **write to the socket** (to send commands). They borrow write access through a `respond(bytes: Buffer)` callback added to `ctx` for command codecs. The session retains socket ownership; handlers borrow write access through a narrow interface.
|
||||
|
||||
## Unknown codec policy
|
||||
|
||||
When the codec ID does not match a registered handler:
|
||||
|
||||
- `WARN` log entry with IMEI, offending codec ID, raw header bytes.
|
||||
- Socket is destroyed.
|
||||
- No ACK sent.
|
||||
- No attempt to "skip ahead" or guess record layout.
|
||||
|
||||
Reasoning: a Teltonika device sending an unrecognized codec is **misconfigured**, not subtly broken. Silently truncating its data — or worse, mis-parsing — produces records with plausible-looking but wrong coordinates. Loud failure beats quiet corruption.
|
||||
|
||||
The `teltonika_unknown_codec_total{codec_id}` counter is the canary for codec coverage drift.
|
||||
|
||||
## CRC failure policy
|
||||
|
||||
Different from unknown-codec. CRC mismatch = transient transmission issue:
|
||||
|
||||
- Frame is **not** ACK'd → device retransmits on next session.
|
||||
- `WARN` log with IMEI, expected CRC, computed CRC, frame length.
|
||||
- Connection stays open.
|
||||
|
||||
Repeated CRC failures from the same device in a short window indicate a deeper problem (firmware, line quality) — surface via metrics, not just logs.
|
||||
@@ -0,0 +1,38 @@
|
||||
---
|
||||
title: Failure Domains
|
||||
type: concept
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture]
|
||||
tags: [architecture, reliability]
|
||||
---
|
||||
|
||||
# Failure Domains
|
||||
|
||||
Each component of the platform fails independently. The architecture deliberately concentrates operational risk in one place — the database — and keeps everything else restartable, replaceable, or naturally redundant.
|
||||
|
||||
## Per-component failure behavior
|
||||
|
||||
| Component | Crash behavior | Data loss |
|
||||
|-----------|---------------|-----------|
|
||||
| [[tcp-ingestion]] | Devices reconnect; in-flight frames retransmitted by the device per protocol | None beyond unacknowledged frames |
|
||||
| [[redis-streams]] | Streams are persisted; restart resumes from disk | Recoverable from device retransmits + Processor checkpointing |
|
||||
| [[processor]] | Consumer-group offsets ensure next instance picks up; in-memory state rehydrated from DB | None |
|
||||
| [[directus]] | Telemetry continues to flow into DB; admin UI/SPA unavailable | None |
|
||||
| [[postgres-timescaledb]] | System stops accepting writes | The single point of failure |
|
||||
| [[react-spa]] | UI unavailable | N/A — no state owned |
|
||||
|
||||
## The discipline behind this
|
||||
|
||||
- **No component reaches across two plane boundaries** — see [[plane-separation]]. A failure in one plane cannot cascade through another.
|
||||
- **The TCP handler never blocks on downstream work.** Slow Processor or DB pressure is absorbed by [[redis-streams]], not by device sockets.
|
||||
- **Per-device session state lives only on the open socket** — Ingestion is trivially restartable.
|
||||
- **The Processor's hot state can always be rehydrated** from the DB.
|
||||
|
||||
## Operational consequence
|
||||
|
||||
The database gets careful operational attention — replication, backups, point-in-time recovery via TimescaleDB. Everything else can be restarted, redeployed, or scaled without ceremony.
|
||||
|
||||
## Canary metric
|
||||
|
||||
**Redis Streams consumer lag.** It reflects the health of the entire telemetry pipeline in one number.
|
||||
@@ -0,0 +1,35 @@
|
||||
---
|
||||
title: IO element bag
|
||||
type: concept
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [teltonika-ingestion-architecture]
|
||||
tags: [data-model, teltonika, principle]
|
||||
---
|
||||
|
||||
# IO element bag
|
||||
|
||||
The model-specific telemetry inside a Teltonika AVL record. Carried as `{ id → value }` pairs that [[tcp-ingestion]] reads byte-correctly without interpreting.
|
||||
|
||||
## The principle: pass through unchanged
|
||||
|
||||
The Ingestion service does **not** name, interpret, or filter IO elements. It produces a [[position-record]] whose `attributes` field is a verbatim representation of the IO bag, keyed by the numeric IO element ID as a string.
|
||||
|
||||
- No renaming.
|
||||
- No unit conversion.
|
||||
- No model lookup.
|
||||
|
||||
Per-model interpretation lives in the [[processor]], driven by configuration like `{ "FMB920": { "16": "odometer_km", "240": "movement" } }`.
|
||||
|
||||
## Why this boundary matters
|
||||
|
||||
1. **Model-specific interpretation belongs where the model is known.** Doing it in the parser would couple Ingestion to a registry of every device model in the fleet — exactly the coupling the [[protocol-adapter]] is designed to prevent.
|
||||
2. **A new model never breaks Ingestion.** A packet with `IO 1234` from an unknown device produces a parser that stores the raw value under `"1234"` and moves on. The position is still useful; the attribute is recoverable later once the model's mapping is imported.
|
||||
|
||||
## Structural implications
|
||||
|
||||
- 8-bit IO IDs in Codec 8.
|
||||
- 16-bit IO IDs in Codec 8E (Teltonika ran out of 8-bit slots) plus variable-length values.
|
||||
- Codec 16 introduces a per-record generation type but the IO bag itself behaves the same.
|
||||
|
||||
The parser handles all three identically with respect to interpretation: read bytes per the codec spec, store under string key, hand off.
|
||||
@@ -0,0 +1,123 @@
|
||||
---
|
||||
title: Phase 2 — Outbound Commands
|
||||
type: concept
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [teltonika-ingestion-architecture, teltonika-data-sending-protocols]
|
||||
tags: [teltonika, phase-2, commands, future]
|
||||
---
|
||||
|
||||
# Phase 2 — Outbound Commands
|
||||
|
||||
The deferred design for server-to-device commands using [[teltonika]] codecs 12 (`0x0C`) and 14 (`0x0E`). Codec 13 (`0x0D`) is one-way device→server and is not part of the outbound design; codec 15 (`0x0F`) is FMX6-only and out of scope. Specified now so Phase 1 code respects the seams; **no Phase 2 code ships until the platform actually needs to issue commands.**
|
||||
|
||||
## Codec selection: 12 vs 14
|
||||
|
||||
- **Codec 12** — generic command/response. Server sends with Type `0x05`; device responds with Type `0x06`. No IMEI in the frame; the device assumed to be the one on the other end of the socket.
|
||||
- **Codec 14** — IMEI-addressed command/response. Server sends with Type `0x05` and an 8-byte IMEI in HEX; device returns Type **`0x06` (ACK)** if its physical IMEI matches, **`0x11` (nACK)** if not. Available from FMB.Ver.03.25.04.Rev.00. Useful as a defense-in-depth check that the connection registry is routing to the device we think we are.
|
||||
|
||||
A reasonable default is Codec 12 for routine ops (the connection registry already guarantees we're talking to the right device's socket), with Codec 14 reserved for situations where IMEI reconfirmation matters (e.g. infrequent high-impact commands).
|
||||
|
||||
## Why deferred
|
||||
|
||||
Command codecs are a **distinct feature**, not an incremental codec, and require:
|
||||
|
||||
- A way to enqueue commands targeted at specific devices.
|
||||
- Routing to whichever Ingestion instance currently holds the device's connection.
|
||||
- Permissioned APIs upstream so commands cannot be issued by unauthorized callers.
|
||||
- Audit trails for every command issued and every response received.
|
||||
|
||||
None of this is needed to read telemetry. Building it speculatively would either ship dead code or, worse, ship half-built infrastructure mistaken for usable.
|
||||
|
||||
## End-to-end flow
|
||||
|
||||
```
|
||||
SPA ──HTTPS+JWT──▶ Directus ──XADD──▶ Redis Streams ──XREAD──▶ Ingestion ──▶ device
|
||||
│ │
|
||||
SPA ◀──WSS subscription── Directus ◀──hook on insert── commands:responses
|
||||
```
|
||||
|
||||
Five properties:
|
||||
|
||||
1. **Single auth surface** — [[directus]] enforces "can this user command this device?" Same machinery as every other write.
|
||||
2. **Commands are data before transport** — every command is a row in the `commands` collection before it hits Redis.
|
||||
3. **Symmetric to inbound telemetry** — same plane boundary, same seam, same operational tools.
|
||||
4. **Per-instance routing** via a connection registry mapping `imei → instance_id`.
|
||||
5. **Real-time status updates for free** — Directus WebSocket subscriptions on `commands` push delivery status to the SPA.
|
||||
|
||||
## Architectural posture
|
||||
|
||||
[[tcp-ingestion]] **does not expose user-facing HTTP endpoints**, in Phase 1 or Phase 2. All user-facing API surface is in [[directus]] (see [[plane-separation]]). Ingestion learns about commands by consuming its own Redis stream — never accepts inbound user-facing traffic.
|
||||
|
||||
## The `commands` collection (Directus)
|
||||
|
||||
Key fields: `id` (uuid, correlation ID), `target_imei`, `batch_id` (nullable, for fleet ops), `codec` (`12` or `14` — `13`/`15` are device-originated, not server-issued), `payload` (ASCII text), `status` (`pending` | `routed` | `delivered` | `responded` | `failed` | `nack` | `expired`), `requested_by`, timestamp fields, `response`, `failure_reason`, `expires_at` (default `requested_at + 5 min`). The `nack` status captures Codec 14's IMEI-mismatch case (Type `0x11`).
|
||||
|
||||
Permissions: writable by operator/admin roles; readable by requester + admin. The SPA inserts via SDK; Ingestion updates delivery status via a service token.
|
||||
|
||||
## Connection registry
|
||||
|
||||
Redis hash `connections:registry`, keyed by IMEI, valued by Ingestion instance ID:
|
||||
|
||||
- On handshake: `HSET connections:registry {imei} {instance_id}` + record IMEI in local `Set<string>`.
|
||||
- Every 30s: `SET instance:heartbeat:{instance_id} {now} EX 90`.
|
||||
- On socket close: `HDEL connections:registry {imei}`.
|
||||
- Graceful shutdown: `HDEL` all held IMEIs.
|
||||
|
||||
**Crash recovery via janitor.** Redis hashes don't support per-field TTL, so a registry janitor (Directus Flow or small process) runs every minute: for each `instance_id` in the registry, `EXISTS instance:heartbeat:{instance_id}` — if missing, scan the registry for entries pointing to it and `HDEL` them.
|
||||
|
||||
## Issuing commands
|
||||
|
||||
**Single device** — SPA inserts a `commands` row; Directus Flow on `items.create`:
|
||||
|
||||
1. Lookup `instance_id = HGET connections:registry {target_imei}`.
|
||||
2. If found: `XADD commands:outbound:{instance_id} ...`; status → `routed`.
|
||||
3. If not found: status stays `pending`; sweeper retries.
|
||||
|
||||
**Fleet** — SPA calls custom endpoint `POST /commands/batch`:
|
||||
|
||||
1. Validate (size, authorization).
|
||||
2. Generate `batch_id`.
|
||||
3. Transactional insert of N rows sharing `batch_id`.
|
||||
4. Per row: registry lookup + stream publish.
|
||||
5. Return `{ batch_id, command_ids }`.
|
||||
|
||||
The custom endpoint exists for fleet operations because transactional insert + routing fan-out is cleaner in code than in a Flow.
|
||||
|
||||
## Pending-command sweeper
|
||||
|
||||
Flow runs every 30s:
|
||||
|
||||
- `pending` rows where `expires_at > now()` → retry registry lookup; if device now online, publish + transition to `routed`.
|
||||
- `pending` or `routed` rows where `expires_at <= now()` → `expired` with `failure_reason`.
|
||||
|
||||
Also handles the case where Ingestion crashes after publish but before delivery — those rows sit in `routed` past `expires_at` and get expired. Operator re-issues. (A subtler retry option exists — re-route stale `routed` rows when the original instance has died — but is an enhancement, not v1.)
|
||||
|
||||
## Ingestion-side command consumer
|
||||
|
||||
Each Ingestion instance runs a parallel consumer reading `commands:outbound:{instance_id}` (XREADGROUP, COUNT 16, BLOCK 1000):
|
||||
|
||||
1. Lookup local `imei → socket` map. If gone: publish failure (`socket_closed`).
|
||||
2. Check `expires_at`. If past: publish failure (`expired_before_delivery`).
|
||||
3. Encode codec 12/13/14 frame from payload.
|
||||
4. Write bytes to the socket (via per-socket write queue to avoid interleaving with codec ACKs).
|
||||
5. Register pending-response entry keyed by `command_id` with timeout (default 30s).
|
||||
|
||||
The consumer never blocks the TCP read path.
|
||||
|
||||
## Response correlation
|
||||
|
||||
Teltonika's command codecs carry **no correlation ID** — the protocol assumes one outstanding command per connection. The Ingestion service enforces this; subsequent commands queue on the per-socket write queue.
|
||||
|
||||
When the device responds (Codec 12 with `Type = 0x06`), the codec dispatch routes to a response handler that publishes to `commands:responses`; a Directus hook (or small consumer) updates the row to `status = responded`. Timeout fires → `status = failed` with `reason = 'no_device_response'`; write queue is freed.
|
||||
|
||||
## What this requires of Phase 1
|
||||
|
||||
Phase 1 must respect these shapes so Phase 2 is purely additive:
|
||||
|
||||
- [[codec-dispatch]] is a registry keyed on codec ID byte — Phase 2 registers `0x0C`, `0x0D`, `0x0E`.
|
||||
- Session loop owns the socket; handlers borrow it via a `respond(bytes)` callback (Phase 1 handlers don't use it).
|
||||
- Per-device runtime state is local to the socket and the holding instance — no shared registry today.
|
||||
- The [[position-record]] shape and the inbound stream are unchanged. Outbound uses entirely separate streams (`commands:outbound:{instance_id}`, `commands:responses`) and a separate Directus collection.
|
||||
|
||||
When Phase 2 ships, no Phase 1 code is rewritten — the command consumer runs alongside.
|
||||
@@ -0,0 +1,37 @@
|
||||
---
|
||||
title: Plane Separation (Telemetry / Business / Presentation)
|
||||
type: concept
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture]
|
||||
tags: [architecture, principle]
|
||||
---
|
||||
|
||||
# Plane Separation
|
||||
|
||||
The system is organized as **three concentric concerns**, deliberately split along **data velocity** and **failure domain**.
|
||||
|
||||
## The three planes
|
||||
|
||||
1. **Telemetry plane** — [[tcp-ingestion]] + [[redis-streams]] + [[processor]]. Optimized for throughput, low latency, resilience to bursty input. Stateless or nearly so.
|
||||
2. **Business plane** — [[directus]] over [[postgres-timescaledb]]. Owns schema, API surface, permissions, back-office workflows.
|
||||
3. **Presentation plane** — [[react-spa]]. Consumes business-plane APIs and real-time subscriptions; never talks to the telemetry plane directly.
|
||||
|
||||
## How the boundaries are enforced
|
||||
|
||||
- **Redis Streams** is the boundary between Ingestion and the Processor side of the telemetry plane.
|
||||
- **The database + Directus API** is the boundary between the telemetry and business planes.
|
||||
- **Directus REST/GraphQL/WSS** is the boundary between business and presentation planes.
|
||||
|
||||
**No component reaches across two boundaries.** This is the rule that makes the architecture coherent.
|
||||
|
||||
## Why this matters
|
||||
|
||||
- Each component scales independently (Ingestion horizontally per TCP load; Directus horizontally per HTTP load; database vertically + read replicas).
|
||||
- Each component fails independently — see [[failure-domains]].
|
||||
- Adding new device vendors does not touch the business plane. Adding new front-end surfaces does not touch the telemetry plane.
|
||||
- Security model stays coherent: only Directus exposes user-facing APIs.
|
||||
|
||||
## The discipline that holds it together
|
||||
|
||||
The clearest articulation of the plane discipline: **the TCP handler never blocks on downstream work.** If the Processor or DB is slow, the queue absorbs pressure; Ingestion keeps accepting and acknowledging. Without this, the planes leak into each other under load.
|
||||
@@ -0,0 +1,62 @@
|
||||
---
|
||||
title: Position Record
|
||||
type: concept
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture, teltonika-ingestion-architecture, teltonika-data-sending-protocols]
|
||||
tags: [data-model, boundary-contract]
|
||||
---
|
||||
|
||||
# Position Record
|
||||
|
||||
The normalized record produced by [[tcp-ingestion]] and consumed by [[processor]]. The boundary contract between vendor adapters and the rest of the system.
|
||||
|
||||
## Shape (Teltonika reference)
|
||||
|
||||
```ts
|
||||
type Position = {
|
||||
device_id: string // IMEI from the handshake
|
||||
timestamp: Date // from the AVL record's GPS timestamp
|
||||
latitude: number
|
||||
longitude: number
|
||||
altitude: number
|
||||
angle: number // heading, 0-360
|
||||
speed: number // km/h
|
||||
satellites: number
|
||||
priority: 0 | 1 | 2 // 0=Low, 1=High, 2=Panic (Teltonika spec)
|
||||
attributes: {
|
||||
[io_id: string]: number | bigint | Buffer
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The system architecture doc lists a slimmer canonical form — `device_id`, `timestamp`, `lat`, `lon`, `speed`, `heading`, plus an attribute bag — and the Teltonika doc extends it with `altitude`, `satellites`, `priority`. Treat the Teltonika shape as the current concrete contract.
|
||||
|
||||
## The `attributes` bag — pass through unchanged
|
||||
|
||||
The Ingestion service does **not** name, interpret, filter, or unit-convert IO elements. `attributes` is a verbatim representation of the IO element bag from the AVL record, keyed by the numeric IO element ID **as a string**.
|
||||
|
||||
Two reasons:
|
||||
|
||||
- **Model-specific interpretation belongs where the model is known.** The [[processor]] configures per-model IO mappings; doing this in the parser would couple Ingestion to a registry of every device model in the fleet.
|
||||
- **A new model never breaks Ingestion.** A packet with `IO 1234` from a device whose config we have not yet imported gets stored under `"1234"` and the position is still useful; the attribute is recoverable later.
|
||||
|
||||
This is the property that makes the [[protocol-adapter]] model-agnostic.
|
||||
|
||||
## What's codec-defined vs. model-defined
|
||||
|
||||
For [[teltonika]]:
|
||||
|
||||
- **Codec-defined** (always present, same shape per codec): timestamp, lat/lon/alt, angle, satellites, speed, priority. These fill out the typed fields above.
|
||||
- **Model-defined** (varies between models, opaque to parser): everything in the IO bag.
|
||||
|
||||
### Codec-specific quirks
|
||||
|
||||
- **Speed = `0x0000` means GPS data is invalid**, not "stationary." The parser preserves this verbatim (speed = 0); the [[processor]] decides whether to surface it as "no GPS fix" or coalesce with the stationary case.
|
||||
- **Lat/Lon are two's complement** signed integers — first bit = sign. Parsing must be sign-aware.
|
||||
- **Codec 16 carries a Generation Type** (0–7: On Exit, On Entrance, On Both, Reserved, Hysteresis, On Change, Eventual, Periodical) that is codec-defined but not currently in the Position shape. Open question on whether to promote it to a typed field. See [[avl-data-format]].
|
||||
- **Codec 8 Extended NX section** carries variable-length IO values — these land in `attributes` as `Buffer`, alongside the fixed-width N1/N2/N4/N8 entries.
|
||||
|
||||
## Downstream contract
|
||||
|
||||
[[processor]] is responsible for naming IO elements (e.g. `"16"` → `"odometer_km"`), unit conversions, and any filtering. It writes the typed fields to the positions hypertable and may write derived/named attributes to other tables.
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
title: Protocol Adapter Pattern
|
||||
type: concept
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture, teltonika-ingestion-architecture]
|
||||
tags: [architecture, ingestion, vendor-abstraction]
|
||||
---
|
||||
|
||||
# Protocol Adapter Pattern
|
||||
|
||||
The strategy used by [[tcp-ingestion]] to keep vendor-specific code from leaking into the rest of the system.
|
||||
|
||||
## The interface
|
||||
|
||||
- **Input**: a stream of bytes from a TCP socket.
|
||||
- **Output**: normalized [[position-record]] values with a stable shape — `device_id`, `timestamp`, `lat`, `lon`, `speed`, `heading`, plus a free-form attribute bag for vendor-specific telemetry.
|
||||
|
||||
That's the entire boundary contract. Adding a new device vendor (Queclink, Concox, etc.) means writing a new adapter. **Nothing downstream of Ingestion changes** — not [[redis-streams]], not [[processor]], not the schema, not the SPA.
|
||||
|
||||
This is "the single most important property of [the Ingestion] layer for long-term maintenance."
|
||||
|
||||
## Layout rules
|
||||
|
||||
In `tcp-ingestion/`:
|
||||
|
||||
1. **`src/core/` never imports from `src/adapters/`.** The vendor-agnostic shell (TCP listeners, Redis publishing, configuration, observability) is built without knowing about any vendor.
|
||||
2. **Adapters never import from one another.** `adapters/teltonika/` and any future `adapters/queclink/` are independent. Shared utilities (e.g. length-prefix buffer accumulation) live in `core/`.
|
||||
3. **Each adapter folder is self-contained.** Lifting [[teltonika]] out into its own service later is a `git mv adapters/teltonika/ ../gps-ingest-teltonika/src/adapter/` plus a copy of `core/`. No untangling required.
|
||||
|
||||
## The deeper move: model-agnostic via self-description
|
||||
|
||||
For [[teltonika]] specifically, the adapter is also **model-agnostic** — what matters is the codec ID, not the device model. Fixed fields are codec-defined; model-specific telemetry is carried in an opaque IO bag the parser passes through verbatim. New models work without code changes. See [[teltonika]] and [[codec-dispatch]].
|
||||
|
||||
This is a property of the protocol, not the pattern, but it pairs well: the adapter pattern isolates vendors; the codec dispatch isolates models within a vendor.
|
||||
|
||||
## What stays out of adapters
|
||||
|
||||
Per the Teltonika reference:
|
||||
|
||||
- **Naming/interpreting telemetry** is a [[processor]] concern, not an adapter concern.
|
||||
- **Filtering** is also downstream — adapters produce raw records.
|
||||
- **Per-model configuration** lives in the Processor, not the adapter.
|
||||
|
||||
Adapters do one thing: turn vendor bytes into normalized records.
|
||||
@@ -0,0 +1,58 @@
|
||||
---
|
||||
title: Directus
|
||||
type: entity
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture, teltonika-ingestion-architecture]
|
||||
tags: [service, business-plane, api]
|
||||
---
|
||||
|
||||
# Directus
|
||||
|
||||
The **business plane**. Owns the relational schema, exposes it through auto-generated REST/GraphQL APIs, enforces role-based permissions, and provides the admin UI for back-office users.
|
||||
|
||||
## What Directus owns
|
||||
|
||||
- **Schema management** — collections, fields, relations, migrations.
|
||||
- **API generation** — REST and GraphQL endpoints, no boilerplate.
|
||||
- **Authentication and authorization** — users, roles, permissions, JWT issuance.
|
||||
- **Real-time** — WebSocket subscriptions on collections for live UIs.
|
||||
- **Workflow automation** — Flows for orchestrating side effects (notifications, integrations).
|
||||
- **Admin UI** — complete back-office interface for operators.
|
||||
|
||||
## What Directus is NOT
|
||||
|
||||
Not in the telemetry hot path. Does not accept device connections, run a geofence engine, or hold per-device runtime state. Mixing those responsibilities into the same process would couple deployment lifecycles and contaminate failure domains. See [[plane-separation]].
|
||||
|
||||
## Schema ownership vs. write access
|
||||
|
||||
Directus is the schema **owner** even though [[processor]] writes directly to the database. New tables, columns, and relations are defined through Directus. Reasons:
|
||||
|
||||
- Auto-generated admin UI and APIs are derived from the schema Directus knows about. Tables created outside Directus are invisible to it.
|
||||
- Permissions are configured per-collection in Directus.
|
||||
- Audit columns (created_at, updated_at, user_created) follow Directus conventions; bypassing them inconsistently leads to subtle UI bugs.
|
||||
|
||||
This is a normal Directus deployment pattern — it does not require sole write access, only schema authority.
|
||||
|
||||
## Extensions
|
||||
|
||||
Used for things that genuinely belong in the business layer:
|
||||
|
||||
- **Hooks** that react to data changes (e.g. on event-write, trigger a notification Flow).
|
||||
- **Custom endpoints** for permission-gated, audited operations that are not throughput-critical.
|
||||
- **Custom admin UI panels** for back-office workflows (data review, manual overrides, bulk ops).
|
||||
- **Flows** for declarative orchestration.
|
||||
|
||||
**Not** used for long-running listeners, persistent network sockets, or anything in the telemetry hot path.
|
||||
|
||||
## Real-time delivery
|
||||
|
||||
Directus's WebSocket subscriptions push live data to the [[react-spa]]. When [[processor]] writes a row, Directus broadcasts the change to subscribed clients. Sufficient for tens to low hundreds of concurrent subscribers. If fan-out becomes a bottleneck, a dedicated WebSocket gateway can read directly from [[redis-streams]] and push to clients, bypassing Directus for the live channel only — REST/GraphQL stays in Directus.
|
||||
|
||||
## Phase 2 role
|
||||
|
||||
Directus owns the `commands` collection and is the **single auth surface** for outbound device commands. The SPA inserts command rows; a Directus Flow routes them via Redis to the Ingestion instance holding the device's socket. See [[phase-2-commands]].
|
||||
|
||||
## Failure mode
|
||||
|
||||
Crash → telemetry continues to flow into the database; admin UI and SPA are unavailable; no telemetry is lost. See [[failure-domains]].
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
title: PostgreSQL + TimescaleDB
|
||||
type: entity
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture]
|
||||
tags: [infrastructure, business-plane, database]
|
||||
---
|
||||
|
||||
# PostgreSQL + TimescaleDB
|
||||
|
||||
The durable storage layer. PostgreSQL with the TimescaleDB extension. Holds the positions hypertable and all business schema owned by [[directus]].
|
||||
|
||||
## Writers
|
||||
|
||||
- **[[processor]]** — sole writer for high-volume telemetry (positions hypertable) and writer for derived business entities (events, violations, alerts).
|
||||
- **[[directus]]** — writes from the admin UI, custom endpoints, and Flows. Owns schema definition and migrations.
|
||||
|
||||
[[react-spa]] never writes (or reads) directly. [[tcp-ingestion]] does not touch the database.
|
||||
|
||||
## Schema authority
|
||||
|
||||
Schema is **defined and migrated through [[directus]]** — see that page for why. The Processor inserts rows respecting that schema; it does not create tables.
|
||||
|
||||
## Operational note
|
||||
|
||||
The database is the **only single point of failure** in the architecture. Everything else is restartable, replaceable, or naturally redundant. Operational attention concentrates here:
|
||||
|
||||
- Replication
|
||||
- Backups
|
||||
- Point-in-time recovery via TimescaleDB
|
||||
|
||||
## Scaling
|
||||
|
||||
- **Vertical** for write throughput.
|
||||
- **Read replicas** for analytics workloads.
|
||||
|
||||
## Deployment
|
||||
|
||||
Internal-only container. Persistence volume. Regular backups. Accessed only by [[directus]] and [[processor]].
|
||||
@@ -0,0 +1,47 @@
|
||||
---
|
||||
title: Processor
|
||||
type: entity
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture]
|
||||
tags: [service, telemetry-plane, domain-logic]
|
||||
---
|
||||
|
||||
# Processor
|
||||
|
||||
The service where domain logic lives. Consumes normalized telemetry from [[redis-streams]] and is responsible for per-device runtime state, applying domain rules, and writing durable state to [[postgres-timescaledb]].
|
||||
|
||||
## Responsibilities
|
||||
|
||||
- Maintain **per-device runtime state** — last position, derived metrics, current zone, accumulators.
|
||||
- Apply **domain rules** that turn raw telemetry into meaningful events.
|
||||
- Write **durable state** — both raw position history and any derived events.
|
||||
- Emit events for downstream consumers (Directus Flows, notification services, dashboards).
|
||||
|
||||
Where [[tcp-ingestion]] is about throughput and protocol correctness, the Processor is about correctness of meaning. It is the component most likely to evolve as requirements grow, which is why it is isolated from the sockets on one side and the API surface on the other.
|
||||
|
||||
## State management
|
||||
|
||||
- **Static reference data** (spatial assets, configurations) loaded at startup; refreshed on a known cadence or via explicit invalidation.
|
||||
- **Per-device state** held in memory keyed by device identifier (last seen, current segment, accumulators).
|
||||
- **Durable state** written asynchronously to the database.
|
||||
|
||||
The database is the source of truth for replay/analysis; in-memory state is the source of truth for the current decision. On restart, hot state is rehydrated from the DB — this is a recovery path, not a hot path.
|
||||
|
||||
## Database writes
|
||||
|
||||
- The Processor is the **only writer** for high-volume telemetry tables (e.g. the positions hypertable). [[directus]] does not insert positions; it reads them.
|
||||
- For derived business entities (events, violations, alerts), the Processor writes directly to tables [[directus]] also knows about. Schema is owned by Directus; the Processor inserts rows respecting that schema.
|
||||
- This keeps the hot write path off the Directus HTTP stack while still letting Directus expose the data through API and admin UI.
|
||||
|
||||
## IO element interpretation
|
||||
|
||||
Per-model IO mappings live here, not in the Ingestion layer. Example: `{ "FMB920": { "16": "odometer_km", "240": "movement" } }`. This is the boundary set by the [[teltonika]] adapter — Ingestion produces raw IO maps; the Processor names and interprets them.
|
||||
|
||||
## Scaling
|
||||
|
||||
Multiple Processor instances join a Redis Streams consumer group and split the load across device IDs. Consumer-group offsets ensure a crashed instance's work is picked up by the next one.
|
||||
|
||||
## Failure mode
|
||||
|
||||
Crash → consumer-group offsets ensure the next instance picks up where the last left off. In-memory state is rehydrated from the database. See [[failure-domains]].
|
||||
@@ -0,0 +1,57 @@
|
||||
---
|
||||
title: React SPA
|
||||
type: entity
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture]
|
||||
tags: [service, presentation-plane, frontend]
|
||||
---
|
||||
|
||||
# React SPA
|
||||
|
||||
The end-user experience for operators and external participants. Single React application, role-based views, served as a static bundle.
|
||||
|
||||
## Why a separate SPA (not just the Directus admin UI)
|
||||
|
||||
The Directus admin UI is generic CRUD over collections — right for back-office editing and operators who think in records, wrong for end users who think in domain concepts. A dedicated SPA delivers:
|
||||
|
||||
- **Domain-shaped UX** — screens organized around the user's mental model.
|
||||
- **Independent deployment** — front-end ships on its own cadence.
|
||||
- **Targeted access control** — public/partner-facing routes without exposing the admin surface.
|
||||
- **Mobile/offline tuning** — bundles tuned for the actual user environment.
|
||||
|
||||
## Single app, role-based views
|
||||
|
||||
One application serves multiple user types via role-based routing and conditional UI. All users authenticate through [[directus]]; the SPA receives a JWT, reads role, and renders appropriate navigation/screens. Splitting into multiple apps is only justified when user populations are genuinely disjoint (public site vs. authenticated console) or when bundle size for one audience harms another.
|
||||
|
||||
## Data access pattern
|
||||
|
||||
The SPA talks **exclusively** to Directus:
|
||||
|
||||
- REST/GraphQL via `@directus/sdk`.
|
||||
- WebSocket subscriptions via the same SDK.
|
||||
- JWT auth managed by the SDK; refresh handled transparently.
|
||||
|
||||
**Never** talks to the [[processor]], [[tcp-ingestion]], [[redis-streams]], or [[postgres-timescaledb]] directly. This boundary lets the back-end evolve internally and keeps the security model coherent — every request goes through Directus's permission system.
|
||||
|
||||
## Recommended stack
|
||||
|
||||
- **Vite + React + TypeScript**
|
||||
- **TanStack Router** — better TS support than React Router; optional file-based routing
|
||||
- **TanStack Query** — server state, caching, invalidation, optimistic updates
|
||||
- **@directus/sdk** — typed access + real-time
|
||||
- **MapLibre GL + react-map-gl** — open-source WebGL maps, no token needed
|
||||
- **shadcn/ui + Tailwind** — UI primitives
|
||||
- **Zustand** — client-only state (filters, UI prefs)
|
||||
- **react-hook-form + Zod** — forms and validation
|
||||
|
||||
Covers the spectrum from form-heavy admin screens to real-time map dashboards without architectural changes between them.
|
||||
|
||||
## Real-time rendering
|
||||
|
||||
- **Live maps with many markers**: React reconciler is not the bottleneck — drawing happens in WebGL via MapLibre, which manages features outside React's tree. The React layer manages subscriptions and feeds the map updates.
|
||||
- **High-frequency tabular updates** (live leaderboards, event feeds): split components so high-update areas re-render in isolation; use TanStack Query for live data; memoize at component boundaries that receive frequent updates.
|
||||
|
||||
## Failure mode
|
||||
|
||||
UI unavailable → back-end unaffected. See [[failure-domains]].
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
title: Redis Streams
|
||||
type: entity
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture, teltonika-ingestion-architecture]
|
||||
tags: [infrastructure, telemetry-plane, queue]
|
||||
---
|
||||
|
||||
# Redis Streams
|
||||
|
||||
The durable in-flight queue between [[tcp-ingestion]] and [[processor]]. Also the transport for Phase 2 outbound commands.
|
||||
|
||||
## What it provides
|
||||
|
||||
- **Buffering** — temporary slowness in [[processor]] does not push back on Ingestion sockets.
|
||||
- **Replayability** — Streams retain messages, so a Processor crash does not lose telemetry; consumer-group offsets resume from the last position.
|
||||
- **Horizontal scaling** — multiple Processor instances join a consumer group and split load across device IDs.
|
||||
|
||||
## Why Redis (and not Kafka/NATS)
|
||||
|
||||
Sufficient at current scale and adds minimal operational burden. NATS or Kafka are reasonable upgrades when **multi-region durability** or **very high throughput** become real concerns. Until then, Redis is the right choice.
|
||||
|
||||
## Phase 2 usage
|
||||
|
||||
Outbound commands ride on per-instance streams: `commands:outbound:{instance_id}`. Responses ride on `commands:responses`. Redis is the transport; the source of truth for commands is the Directus `commands` collection. See [[phase-2-commands]].
|
||||
|
||||
The connection registry (`connections:registry` hash) and per-instance heartbeats (`instance:heartbeat:{instance_id}` keys with `EX 90`) also live in Redis.
|
||||
|
||||
## Failure mode
|
||||
|
||||
Streams are persisted; restart resumes from disk. Complete Redis loss is recoverable from device retransmits and Processor checkpointing. See [[failure-domains]].
|
||||
|
||||
## Operational note
|
||||
|
||||
**Consumer lag is the canary metric** for the entire telemetry pipeline. Observability dashboards should make it prominent.
|
||||
|
||||
## Deployment
|
||||
|
||||
Internal-only container. Persistence enabled. Never exposed externally.
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
title: TCP Ingestion
|
||||
type: entity
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture, teltonika-ingestion-architecture]
|
||||
tags: [service, telemetry-plane]
|
||||
---
|
||||
|
||||
# TCP Ingestion
|
||||
|
||||
The service that maintains persistent TCP connections with GPS devices, parses vendor binary protocols, ACKs frames per protocol, and hands off normalized records to the [[redis-streams]] queue.
|
||||
|
||||
## Responsibility
|
||||
|
||||
Single concern: **protocol I/O**. Explicitly **not**:
|
||||
|
||||
- Apply business rules
|
||||
- Write to PostgreSQL
|
||||
- Perform geospatial computation
|
||||
- Serve any user-facing API
|
||||
|
||||
The narrow scope is what keeps the process fast, predictable, and safely restartable.
|
||||
|
||||
## Connection model
|
||||
|
||||
- Built around `net.createServer()` (Node.js) — each socket is an independent session.
|
||||
- Per-connection state is small: identifier (e.g. IMEI), parser instance, partial-frame buffer.
|
||||
- Devices reconnect automatically on network failure → connection loss is routine → service is trivially restartable.
|
||||
|
||||
## Vendor abstraction
|
||||
|
||||
Each device vendor (Teltonika, Queclink, Concox, etc.) ships its own binary protocol. Vendor-specific code is isolated behind a [[protocol-adapter]] interface:
|
||||
|
||||
- **Input**: byte stream from a TCP socket
|
||||
- **Output**: normalized [[position-record]] (`device_id`, `timestamp`, `lat`, `lon`, `speed`, `heading`, plus a free-form `attributes` bag)
|
||||
|
||||
Adding a new vendor = writing a new adapter. Nothing downstream changes.
|
||||
|
||||
## Handoff discipline
|
||||
|
||||
For every parsed frame:
|
||||
|
||||
1. Send protocol-required ACK to the device.
|
||||
2. Push normalized record to a Redis Stream.
|
||||
3. Return to reading the socket.
|
||||
|
||||
**The TCP handler never blocks on downstream work.** Backpressure is absorbed by the Stream; Ingestion keeps accepting and acknowledging. This is the discipline that keeps the system alive under load.
|
||||
|
||||
## Project layout
|
||||
|
||||
Lives at `tcp-ingestion/` — single Node.js/TypeScript project. Layout:
|
||||
|
||||
```
|
||||
tcp-ingestion/
|
||||
├── src/core/ # vendor-agnostic shell (no adapter imports)
|
||||
├── src/adapters/ # per-vendor adapters
|
||||
│ └── teltonika/ # see [[teltonika]]
|
||||
├── src/config/
|
||||
├── src/observability/
|
||||
└── test/fixtures/ # real packet captures per codec
|
||||
```
|
||||
|
||||
Three layout rules: `core/` never imports `adapters/`; adapters never import each other; each adapter folder is self-contained so it can be lifted into its own service later via `git mv`.
|
||||
|
||||
## Scaling shape
|
||||
|
||||
- Single Node.js process handles thousands of concurrent connections at typical telemetry rates.
|
||||
- Horizontal scaling: multiple instances behind a TCP-aware load balancer (HAProxy, NGINX stream module).
|
||||
- TCP guarantees session stickiness for the duration of the connection.
|
||||
- No shared state between instances required — per-device state lives entirely on the open socket.
|
||||
|
||||
The pattern ports cleanly to higher-throughput runtimes (Go, Elixir) if a future rewrite is warranted.
|
||||
|
||||
## Failure mode
|
||||
|
||||
Crash → devices reconnect → in-flight frames are retransmitted by the device per protocol → no data is lost beyond what was unacknowledged. See [[failure-domains]].
|
||||
|
||||
## Phase 2 addition
|
||||
|
||||
Each Ingestion instance will run a parallel **command consumer** reading from `commands:outbound:{instance_id}` and writing command frames to device sockets. The TCP read path is not blocked. See [[phase-2-commands]].
|
||||
@@ -0,0 +1,86 @@
|
||||
---
|
||||
title: Teltonika
|
||||
type: entity
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [teltonika-ingestion-architecture, teltonika-data-sending-protocols]
|
||||
tags: [vendor, gps-hardware, protocol]
|
||||
---
|
||||
|
||||
# Teltonika
|
||||
|
||||
Lithuanian GPS hardware vendor. First (and currently only) device family supported by [[tcp-ingestion]]. Device families seen include FMB / FMC / FMM / FMU.
|
||||
|
||||
## Protocol overview
|
||||
|
||||
Binary frames, self-describing via codec ID byte. Both **TCP and UDP transports** are supported by the protocol; our [[tcp-ingestion]] implementation is TCP-only by deliberate choice (connection-oriented reliability + persistent session for [[phase-2-commands|outbound commands]]).
|
||||
|
||||
Codec inventory (canonical, from the official Teltonika wiki):
|
||||
|
||||
| Codec | Hex ID | Direction | Purpose |
|
||||
|-------|--------|-----------|---------|
|
||||
| 8 | `0x08` | device → server | AVL telemetry; 1-byte IO IDs |
|
||||
| 8 Extended | `0x8E` | device → server | AVL telemetry; 2-byte IO IDs + variable-length IO (NX section) |
|
||||
| 16 | `0x10` | device → server | AVL telemetry; 2-byte IO IDs + Generation Type; AVL IDs > 255 |
|
||||
| 12 | `0x0C` | bidirectional | Server commands + device responses (ASCII text) |
|
||||
| 13 | `0x0D` | device → server | One-way command upload with timestamp |
|
||||
| 14 | `0x0E` | bidirectional | Like 12, IMEI-addressed; ACK `0x06` / nACK `0x11` |
|
||||
| 15 | `0x0F` | device → server | One-way; FMX6 professional devices via RS232 — out of scope |
|
||||
| 4 | `0x04` | SMS | 24-position daily SMS, bit-field compressed — out of scope |
|
||||
|
||||
See [[avl-data-format]] for the byte-level packet structure across the three telemetry codecs.
|
||||
|
||||
### Notes on individual codecs
|
||||
|
||||
- **Codec 8** — baseline AVL format. 8-bit IO element IDs. Older firmware.
|
||||
- **Codec 8 Extended** — 16-bit IO IDs and a **variable-length IO section (NX)** carrying length-prefixed values. Default on modern FMB/FMC/FMM/FMU devices.
|
||||
- **Codec 16** — adds a 1-byte **Generation Type** field per record (values 0–7: On Exit, On Entrance, On Both, Reserved, Hysteresis, On Change, Eventual, Periodical). Required for AVL IDs > 255 on FMB630/FM63XY firmware ≥ 00.03.xx.
|
||||
- **Codec 13** — Device→Server only. Used only when "Message Timestamp" is enabled in RS232 settings.
|
||||
- **Codec 14** — Device returns nACK (Type `0x11`) when the command's IMEI doesn't match. Available from FMB.Ver.03.25.04.Rev.00.
|
||||
- **Codec 15** — FMX6 professional devices only, via RS232 modes (TCP/UDP Ascii/Binary, ±Buffered). Includes both timestamp (4B, **seconds**) and IMEI (8B HEX). Not applicable to the deployed FMB/FMC/FMM/FMU fleet.
|
||||
|
||||
## Frame envelope (data codecs)
|
||||
|
||||
```
|
||||
┌──────────────┬──────────────┬───────────┬─────────────┬─────┐
|
||||
│ Preamble │ Data length │ Codec ID │ AVL data │ CRC │
|
||||
│ 4 bytes (0) │ 4 bytes │ 1 byte │ N bytes │ 4 B │
|
||||
└──────────────┴──────────────┴───────────┴─────────────┴─────┘
|
||||
```
|
||||
|
||||
CRC is CRC-16/IBM (lower 2 bytes carry the value; upper 2 are zero).
|
||||
|
||||
## Self-describing → model-agnostic parser
|
||||
|
||||
The codec ID announces framing; fixed AVL fields (timestamp, lat/lon/alt, angle, satellites, speed) are codec-defined and identical across models for a given codec; only the **IO element bag** varies between models. The bag is `{ id → value }` pairs the parser reads byte-correctly without knowing what each `id` means.
|
||||
|
||||
A brand-new Teltonika model will:
|
||||
|
||||
- Parse correctly at frame and codec layers (envelope, CRC, codec dispatch unchanged).
|
||||
- Produce a correct [[position-record]] (codec-defined fields).
|
||||
- Carry IO elements through as raw integer keys until a model-aware mapping is added in the [[processor]].
|
||||
|
||||
This is the load-bearing property of the [[protocol-adapter]] design.
|
||||
|
||||
## IMEI handshake
|
||||
|
||||
Device connects → sends 2-byte length + ASCII IMEI → server responds `0x01` to accept (or `0x00` to reject). Phase 1 always accepts.
|
||||
|
||||
## ACK protocol
|
||||
|
||||
After parsing a data frame, server sends 4-byte big-endian record count. Devices retransmit unacknowledged frames on the next session — this is the protocol-correct way to handle CRC failures (just don't ACK).
|
||||
|
||||
## Packet size limits
|
||||
|
||||
- Minimum AVL record: **45 bytes** (all IO elements disabled).
|
||||
- Maximum AVL record: **255 bytes**.
|
||||
- Maximum AVL packet: **512 bytes** for FMB640/FMB641/FMC640/FMM640; **1280 bytes** for other devices.
|
||||
|
||||
## Phase 1 vs Phase 2
|
||||
|
||||
| Phase | Codecs | Purpose | Status |
|
||||
|-------|--------|---------|--------|
|
||||
| Phase 1 (now) | 8, 8E, 16 | Device → server telemetry | In scope |
|
||||
| Phase 2 (later) | 12, 13, 14 | Server → device commands | Reserved |
|
||||
|
||||
Phase 1 covers essentially the entire deployed Teltonika telemetry fleet. Codec 15 and SMS-based protocols (Codec 4, binary SMS) are out of scope for now. See [[phase-2-commands]] for the deferred command-codec design.
|
||||
@@ -0,0 +1,44 @@
|
||||
---
|
||||
title: GPS Tracking Platform — Architecture Overview
|
||||
type: source
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [gps-tracking-architecture]
|
||||
tags: [architecture, system-design, overview]
|
||||
source_path: raw/gps-tracking-architecture.md
|
||||
source_kind: note
|
||||
---
|
||||
|
||||
# GPS Tracking Platform — Architecture Overview
|
||||
|
||||
## TL;DR
|
||||
|
||||
System-level reference for a real-time GPS telemetry platform built around four cooperating components: a TCP Ingestion service, a Processor, a Directus instance, and a React SPA. The architecture splits the system along **data velocity** and **failure domain**: a hot telemetry plane (Ingestion → Redis Streams → Processor), a business plane (Directus over PostgreSQL/TimescaleDB), and a presentation plane (React SPA). Each component scales, fails, and deploys independently.
|
||||
|
||||
## Key claims
|
||||
|
||||
- The system is structured as **three concentric planes** — telemetry, business, presentation. Boundaries are enforced by Redis Streams on one side and the database/API on the other; no component reaches across two boundaries. See [[plane-separation]].
|
||||
- **TCP Ingestion** maintains persistent device connections, parses vendor binary protocols, ACKs frames, and pushes normalized records to a Redis Stream. It does not apply business rules, write to Postgres, or serve user APIs. See [[tcp-ingestion]].
|
||||
- **Vendor abstraction** lives at the Ingestion layer via a protocol adapter interface (bytes in → normalized `Position` out). Adding a new device family means writing a new adapter; nothing downstream changes. See [[protocol-adapter]].
|
||||
- **The TCP handler never blocks on downstream work.** If the Processor or DB is slow, the Stream absorbs pressure; Ingestion keeps accepting and acknowledging. This single discipline is "the difference between a system that survives a bad day and one that doesn't."
|
||||
- **Processor** owns domain logic: hot per-device state in memory, durable writes to PostgreSQL/TimescaleDB, derived events. It is the **only writer** for high-volume telemetry tables (positions hypertable). See [[processor]].
|
||||
- **Directus** owns the schema, REST/GraphQL APIs, permissions, WebSockets, admin UI, and Flows — but is **not** in the telemetry hot path. The Processor writes directly to Directus-owned tables; schema authority stays with Directus. See [[directus]].
|
||||
- **React SPA** is the only user-facing surface. It talks exclusively to Directus (REST/GraphQL/WSS via the official SDK); it never touches Ingestion, Processor, Redis, or the database. See [[react-spa]].
|
||||
- **Redis Streams** between Ingestion and Processor provides buffering, replayability (consumer-group offsets), and horizontal scaling. NATS or Kafka are reasonable upgrades for multi-region or very high throughput; Redis is the right choice at current scale. See [[redis-streams]].
|
||||
- **Failure domains are isolated.** Ingestion crash → devices reconnect. Processor crash → resume from offset. Directus crash → telemetry continues; UI offline. The database is the only single point of failure. See [[failure-domains]].
|
||||
- **Recommended SPA stack**: Vite + React + TS, TanStack Router, TanStack Query, @directus/sdk, MapLibre GL + react-map-gl, shadcn/ui + Tailwind, Zustand, react-hook-form + Zod.
|
||||
- **Observability**: Redis Stream consumer lag is the single most important metric — it reflects the health of the entire telemetry pipeline in one number.
|
||||
|
||||
## Notable quotes
|
||||
|
||||
> "The architecture is deliberately split along the lines of **data velocity** and **failure domain**."
|
||||
|
||||
> "The TCP handler **never blocks on downstream work**. … This single discipline is the difference between a system that survives a bad day and one that doesn't."
|
||||
|
||||
> "The architecture deliberately makes the database the **only** part of the system that requires careful operational attention. Everything else is restartable, replaceable, or naturally redundant."
|
||||
|
||||
## Open questions / follow-ups
|
||||
|
||||
- No specific business domain modeled here ("intentionally generic"). Future sources should clarify the actual TRM domain (events, alerts, geofencing rules).
|
||||
- WebSocket fan-out limits cited as "tens to low hundreds" of subscribers — what's the expected scale for TRM?
|
||||
- Multi-region is mentioned as a possible future evolution. Is it on the roadmap?
|
||||
@@ -0,0 +1,121 @@
|
||||
---
|
||||
title: Teltonika Data Sending Protocols (official wiki)
|
||||
type: source
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [teltonika-data-sending-protocols]
|
||||
tags: [teltonika, protocol, canonical, reference]
|
||||
source_path: raw/Teltonika Data Sending Protocols - Teltonika Telematics Wiki.md
|
||||
source_url: https://wiki.teltonika-gps.com/view/Teltonika_Data_Sending_Protocols
|
||||
source_kind: article
|
||||
---
|
||||
|
||||
# Teltonika Data Sending Protocols
|
||||
|
||||
Official Teltonika Telematics wiki page documenting all codec families: data-sending (8, 8E, 16), GPRS command (12, 13, 14, **15**), and SMS-based (24-position, binary SMS). Canonical reference; supersedes our internal summaries where they conflict.
|
||||
|
||||
## TL;DR
|
||||
|
||||
The complete codec inventory: **Codec 8 (0x08), 8 Extended (0x8E), 16 (0x10)** for device-to-server telemetry; **Codec 12 (0x0C), 13 (0x0D), 14 (0x0E), 15 (0x0F)** for GPRS-message-based device/server communication; plus **Codec 4 (0x04)** for 24-position SMS and a separate binary-SMS data-sending protocol. All telemetry codecs work over both TCP and UDP. See [[avl-data-format]] for the canonical frame structure and [[teltonika]] for the per-codec summary.
|
||||
|
||||
## What's new vs. our prior wiki
|
||||
|
||||
This source corrects/extends earlier pages with several concrete details we hadn't captured:
|
||||
|
||||
- **Codec 16 hex ID = 0x10** (not just decimal 16).
|
||||
- **UDP is officially supported** for codecs 8/8E/16 — separate envelope with UDP Channel Header (Length 2B + Packet ID 2B + Not Usable Byte 1B), AVL Packet Header (AVL Packet ID 1B + IMEI Length 2B + IMEI 15B), then the AVL Data Array. Server ACK over UDP carries (AVL Packet ID 1B + Number of Accepted 1B), much shorter than the TCP 4-byte big-endian count.
|
||||
- **Codec 8 Extended has variable-size IO elements** (NX section), separate from N1/N2/N4/N8 — IO ID 2B + Length 2B + Value (variable).
|
||||
- **Codec 16 Generation Type** is a 1-byte field with 8 defined values: 0=On Exit, 1=On Entrance, 2=On Both, 3=Reserved, 4=Hysteresis, 5=On Change, 6=Eventual, 7=Periodical.
|
||||
- **Priority is an enum** with values 0=Low, 1=High, 2=Panic.
|
||||
- **Codec 14 has a nACK response type** `0x11` (not just the `0x06` ACK we previously documented). nACK is sent when the command's IMEI does not match the device's actual IMEI. Available from firmware FMB.Ver.03.25.04.Rev.00.
|
||||
- **Codec 13** is **Device→Server only** (one-way), Type always `0x06`, includes a 4-byte timestamp prepended to the command. Used only when the "Message Timestamp" parameter in RS232 settings is enabled.
|
||||
- **Codec 15 (0x0F)** is **new to our wiki** — one-way Device→Server, FMX6 professional devices, RS232 modes (TCP/UDP Ascii/Binary, ±Buffered). Includes both timestamp (4B, **seconds**, not ms) and IMEI (8B HEX). Out of scope for our deployed FMB/FMC/FMM/FMU fleet.
|
||||
- **Packet size limits**: min AVL record 45B (all IO elements disabled); max AVL record 255B; **max AVL packet 512B for FMB640/FMB641/FMC640/FMM640, 1280B for other devices**.
|
||||
- **AVL ID range**: AVL IDs > 255 are only available in Codec 16 (and, by inference from the spec, Codec 8E with its 2-byte IO ID width).
|
||||
- **Coordinate encoding**: Lat/Lon are 4-byte integers built from `((d + m/60 + s/3600 + ms/3600000) × 10⁷)` per axis. Negative coords use **two's complement** — first bit = sign. Speed = `0x0000` when GPS data is invalid.
|
||||
- **CRC scope** is precisely "from Codec ID to second Number of Data" (we had said roughly "the data section"). CRC is CRC-16/IBM, lower 2 bytes of the 4-byte field carry the value.
|
||||
- **Codec 12 GPRS command session prerequisite**: device must have sent AVL data and received a correct ACK before commands can be sent over the same socket. Recommended `Active datalink timeout` is 259200 (max) so the session stays open.
|
||||
|
||||
## Key claims (canonical)
|
||||
|
||||
### Codec table
|
||||
|
||||
| Codec | Hex ID | Direction | Purpose |
|
||||
|-------|--------|-----------|---------|
|
||||
| 8 | 0x08 | device → server | AVL telemetry; 1-byte IO IDs |
|
||||
| 8 Extended | 0x8E | device → server | AVL telemetry; 2-byte IO IDs + variable-length IO |
|
||||
| 16 | 0x10 | device → server | AVL telemetry; 2-byte IO IDs + Generation Type; supports IO IDs > 255 |
|
||||
| 12 | 0x0C | bidirectional | Server commands + device responses (ASCII text) |
|
||||
| 13 | 0x0D | device → server | One-way command upload with timestamp |
|
||||
| 14 | 0x0E | bidirectional | Like 12, addressed by IMEI; ACK 0x06 / nACK 0x11 |
|
||||
| 15 | 0x0F | device → server | One-way; timestamp + IMEI; FMX6 RS232 modes only |
|
||||
| 4 | 0x04 | SMS (one-way) | 24-position daily SMS, bit-field compressed |
|
||||
|
||||
### AVL outer envelope (TCP, codecs 8/8E/16)
|
||||
|
||||
```
|
||||
[Preamble 4B = 0x00000000][DataFieldLength 4B][CodecID 1B][N1 1B][AVL Data X B][N2 1B][CRC 4B]
|
||||
```
|
||||
|
||||
- `DataFieldLength` = bytes from `CodecID` through `N2`.
|
||||
- `CRC-16/IBM` calculated over the same range.
|
||||
- `N1` MUST equal `N2` (record count, repeated).
|
||||
- TCP server ACK = **4-byte big-endian integer** = number of records accepted.
|
||||
|
||||
### AVL record (all three telemetry codecs)
|
||||
|
||||
```
|
||||
[Timestamp 8B (UNIX ms)][Priority 1B][GPS Element 15B][IO Element X B]
|
||||
```
|
||||
|
||||
- **GPS Element**: `Longitude 4B + Latitude 4B + Altitude 2B + Angle 2B + Satellites 1B + Speed 2B` — total 15 bytes.
|
||||
- **Priority**: 0=Low, 1=High, 2=Panic.
|
||||
- IO Element layout differs per codec — see [[avl-data-format]].
|
||||
|
||||
### IMEI handshake
|
||||
|
||||
Device → server: `2-byte length` + ASCII IMEI (e.g. `000F` + 15 ASCII bytes for a 15-digit IMEI).
|
||||
Server → device: `0x01` (accept) or `0x00` (reject).
|
||||
|
||||
### UDP envelope (data codecs)
|
||||
|
||||
```
|
||||
[Length 2B][Packet ID 2B][Not Usable Byte 1B] | [AVL Packet ID 1B][IMEI Length 2B = 0x000F][IMEI 15B] | [AVL Data Array (same as TCP)]
|
||||
```
|
||||
|
||||
UDP server ACK: `[Length 2B = 0x0005][Packet ID][Not Usable 1B] | [AVL Packet ID 1B][Number of Accepted 1B]`.
|
||||
|
||||
### Codec 12 message structure
|
||||
|
||||
```
|
||||
[Preamble 4B][DataSize 4B][CodecID 0x0C][CmdQty1 1B][Type 1B][CmdSize 4B][Command X B][CmdQty2 1B][CRC 4B]
|
||||
```
|
||||
|
||||
- `Type`: `0x05` for command (server → device), `0x06` for response (device → server).
|
||||
- `CRC-16/IBM` from `CodecID` through `CmdQty2`.
|
||||
- Command/Response field is ASCII text encoded as HEX (e.g. `getinfo` → `67 65 74 69 6E 66 6F`).
|
||||
|
||||
### Codec 14 specifics
|
||||
|
||||
```
|
||||
[...][CodecID 0x0E][CmdQty 1B][Type 1B][Size 4B = command + 8 IMEI][IMEI HEX 8B][Command X B][CmdQty 1B][CRC 4B]
|
||||
```
|
||||
|
||||
- IMEI is encoded as 8-byte HEX (e.g. IMEI `123456789123456` → `01 23 45 67 89 12 34 56`).
|
||||
- **Type 0x06 = ACK** (IMEI matched, command executed); **Type 0x11 = nACK** (IMEI mismatch, command rejected).
|
||||
- Available from FMB.Ver.03.25.04.Rev.00.
|
||||
|
||||
## Notable quotes
|
||||
|
||||
> "Note that the GPRS session should remain active between the device and server, while GPRS commands are sent. For this reason, active datalink timeout (global parameters in device configuration) is recommended to be set to 259200 (maximum value)."
|
||||
|
||||
> "AVL IDs that are higher than 255 will can be used only in the Codec16 protocol."
|
||||
|
||||
> "If command message IMEI is equal to actual device IMEI, received command will be executed and response will be sent with ACK (0x06) … If the command message IMEI doesn't match … response to the server will be sent with nACK (0x11)."
|
||||
|
||||
## Open questions / follow-ups
|
||||
|
||||
- Codec 8 Extended NX (variable-size IO) parser handling — what shape do those values take in our [[position-record]] `attributes` bag? Likely raw `Buffer`, length-aware. Needs explicit handling in the parser fixture suite ([[teltonika-ingestion-architecture]] §5.6).
|
||||
- Codec 16 Generation Type — should this be promoted into a typed field in [[position-record]] (since it's codec-defined, not model-defined)? Currently we only specify `priority`; Generation Type is similar in nature.
|
||||
- Should we plan to support the SMS-based protocols (Codec 4 / binary SMS) for fallback connectivity? Probably not for now — SMS is rare in the deployed fleet, and adding it means an SMS gateway integration well outside the TCP service.
|
||||
- Codec 15 is FMX6-only and out of scope. But if the platform ever onboards an FMX6 fleet, this becomes a Phase-3 line item.
|
||||
@@ -0,0 +1,63 @@
|
||||
---
|
||||
title: Teltonika Ingestion — Architecture Reference
|
||||
type: source
|
||||
created: 2026-04-30
|
||||
updated: 2026-04-30
|
||||
sources: [teltonika-ingestion-architecture]
|
||||
tags: [teltonika, ingestion, protocol, codec]
|
||||
source_path: raw/teltonika-ingestion-architecture.md
|
||||
source_kind: note
|
||||
---
|
||||
|
||||
# Teltonika Ingestion — Architecture Reference
|
||||
|
||||
## TL;DR
|
||||
|
||||
Design for the Teltonika protocol adapter inside the [[tcp-ingestion]] service. Goal: ingest telemetry from any Teltonika device — including unseen models — without parser changes, by leaning on the protocol's self-description (codec ID announces framing; IO bag carries opaque model-specific telemetry). Phase 1 implements data-sending codecs **8, 8E, 16**; Phase 2 will add command codecs **12, 13, 14**. Six design principles govern the adapter; the codec dispatch is a registry built so Phase 2 is additive, not a rewrite.
|
||||
|
||||
## Key claims
|
||||
|
||||
- **Project lives at `tcp-ingestion/`** — a single Node.js/TypeScript project containing the vendor-agnostic shell (`src/core/`) and per-vendor adapters (`src/adapters/`). Today the only adapter is Teltonika.
|
||||
- **Layout rules**: `core/` never imports from `adapters/`; adapters never import from each other; the Teltonika folder is self-contained so it can be lifted into its own service via `git mv` later.
|
||||
- **Phase 1 scope** = data-sending codecs 8, 8E, 16 (covers the deployed Teltonika telemetry fleet).
|
||||
- **Phase 2 scope** = GPRS command codecs 12, 13, 14 (server → device). Deferred because they are a **distinct feature** with different security implications, not an incremental codec. See [[phase-2-commands]].
|
||||
- **The parser is model-agnostic** — what matters is the codec ID byte, not the device model. Fixed AVL fields (timestamp, lat/lon/alt, angle, satellites, speed) are codec-defined; only the IO element bag varies, and it's passed through verbatim. New Teltonika models work on day one.
|
||||
- **`Position` shape** is the boundary contract — `device_id`, `timestamp`, `lat`, `lon`, `alt`, `angle`, `speed`, `satellites`, `priority`, `attributes` (raw IO map keyed by numeric ID as string). See [[position-record]].
|
||||
- **Six design principles** (priority order):
|
||||
1. Implement Codec 8, 8E, 16 — the closed set.
|
||||
2. Defer Codec 12, 13, 14.
|
||||
3. Pass the IO map through unchanged — naming/interpretation is a Processor concern.
|
||||
4. Log unknown codec IDs and drop the connection (loud failure > silent corruption).
|
||||
5. Validate CRC-16/IBM; NACK (no ACK) on mismatch — devices retransmit.
|
||||
6. Maintain a fixture suite of real packet captures + Teltonika-doc captures + synthetic edge cases.
|
||||
- **Codec dispatch is a flat registry** keyed on codec ID — not an inheritance hierarchy. Codec parsers are independent because their record shapes diverge. See [[codec-dispatch]].
|
||||
- **Phase 1 forward-compatibility seams** for Phase 2:
|
||||
- Codec dispatch is a registry, not a switch (§8.1).
|
||||
- Session owns the socket; handlers borrow write access via a `respond(bytes)` callback (§8.2).
|
||||
- Per-device state is local to the socket; no shared registry today. Phase 2 adds the connection registry alongside, not woven through (§8.3).
|
||||
- **Phase 2 outbound command flow**: `SPA → Directus → Redis Streams → Ingestion → device`. Directus enforces single auth surface; commands are persisted as rows in a `commands` collection before being routed; Redis is transport, not source of truth.
|
||||
- **Phase 2 connection registry**: Redis hash `connections:registry` mapping `imei → instance_id`. Per-instance heartbeat keys (`SET instance:heartbeat:{instance_id} EX 90`) plus a registry janitor handle crash recovery — Redis hashes don't support per-field TTL.
|
||||
- **Phase 2 command correlation**: Teltonika's command codecs carry no correlation ID, so the protocol assumes one outstanding command per connection. The Ingestion service enforces this via a per-socket write queue.
|
||||
|
||||
## TCP session lifecycle (Phase 1 happy path)
|
||||
|
||||
1. Device connects to the Teltonika port.
|
||||
2. IMEI handshake: device sends 2-byte length + ASCII IMEI; server responds `0x01` to accept.
|
||||
3. AVL data loop: read preamble (4×0x00) + length + payload + 4-byte CRC; validate CRC; dispatch on codec ID byte; emit `Position` records to Redis; ACK with 4-byte big-endian record count.
|
||||
4. Session ends on disconnect; no state preserved across sessions.
|
||||
|
||||
## Notable quotes
|
||||
|
||||
> "Ingest telemetry from any Teltonika device, including models we have never seen, without code changes to the parser."
|
||||
|
||||
> "Naming and interpreting IO elements is explicitly a Processor concern, driven by per-model configuration."
|
||||
|
||||
> "A loud failure (the device reconnects, fails again, shows up in logs) is strictly better than a quiet corruption."
|
||||
|
||||
> "A fixture suite is not optional infrastructure. It is the only place the parser's correctness is actually verified."
|
||||
|
||||
## Open questions / follow-ups
|
||||
|
||||
- Per-model IO dictionary: where does it live downstream — Directus collection, static config in the Processor, or both?
|
||||
- Phase 2 timing: no commitment given. Driver will be the first real need to issue commands (configuration, remote actions).
|
||||
- Pending-command sweeper cadence (30s) and command default TTL (5 min) — operational defaults, may want tuning once command volume is real.
|
||||
Reference in New Issue
Block a user