tcp-ingestion/.planning/ROADMAP.md

# tcp-ingestion — Roadmap

A Node.js TCP server that accepts persistent connections from GPS hardware, parses vendor binary protocols, and emits normalized `Position` records onto Redis Streams for the Processor to consume.

This file is the single navigation hub for all implementation planning. Each phase has its own folder with a README and granular task files. Update statuses here as work lands.

## Status legend

| Symbol | Meaning |
|--------|---------|
| ⬜ | Not started |
| 🟦 | Planned (designed, not coded) |
| 🟨 | In progress |
| 🟩 | Done |
| ⏸ | Paused / blocked |
| ❄️ | Frozen / future / optional |

## Architectural anchors

The service is specified by the wiki at `../docs/wiki/`. Implementing agents should read these pages before starting any task:

- **Architecture** — `docs/wiki/sources/gps-tracking-architecture.md`, `docs/wiki/concepts/plane-separation.md`, `docs/wiki/concepts/failure-domains.md`
- **Adapter design (this service)** — `docs/wiki/sources/teltonika-ingestion-architecture.md`, `docs/wiki/concepts/protocol-adapter.md`
- **Teltonika protocol** — `docs/wiki/entities/teltonika.md`, `docs/wiki/concepts/avl-data-format.md`, `docs/wiki/concepts/codec-dispatch.md`, `docs/wiki/concepts/position-record.md`, `docs/wiki/concepts/io-element-bag.md`
- **Phase 2 design** — `docs/wiki/concepts/phase-2-commands.md`
- **Canonical Teltonika spec** — `docs/wiki/sources/teltonika-data-sending-protocols.md`

## Non-negotiable design rules

These rules govern every task. Any deviation must be discussed and documented as a decision before code lands.

1. **Vendor-agnostic shell.** `src/core/` never imports from `src/adapters/`. Adapters never import from each other. Each adapter folder is self-contained.
2. **TCP handler never blocks on downstream work.** Redis pushes are queued/awaited but never inside a path that would back-pressure a device socket beyond the TCP buffer.
3. **IO element bag passes through unchanged.** No naming, no unit conversion, no model lookup in the parser. `attributes` is keyed by numeric IO ID as string.
4. **Loud failure on unknown codec.** Drop the connection, do not skip-ahead, do not partial-parse.
5. **NACK on CRC mismatch** = simply do not ACK; device retransmits.
6. **Codec dispatch is a flat registry**, not a switch or inheritance hierarchy. Phase 2 must be a pure addition.
7. **Fixture-based testing is mandatory** for every codec parser. Hex captures + expected `Position[]` pairs run on every CI build.

## Phases

### Phase 1 — Inbound telemetry (Codec 8, 8E, 16)

**Status:** 🟨 In progress (observability landed; hardening + device authority paused for pilot test)
**Outcome:** A production-ready Node.js TCP server ingesting Teltonika telemetry from any FMB/FMC/FMM/FMU device, publishing normalized `Position` records to Redis Streams, with full observability and CI/CD via Gitea.

[**See `phase-1-telemetry/README.md`**](./phase-1-telemetry/README.md)

| # | Task | Status | Landed in |
|---|------|--------|-----------|
| 1.1 | [Project scaffold](./phase-1-telemetry/01-project-scaffold.md) | 🟩 | `1e9219d` |
| 1.2 | [Core shell & framing types](./phase-1-telemetry/02-core-shell.md) | 🟩 | `1e9219d` |
| 1.3 | [Configuration & logging](./phase-1-telemetry/03-config-and-logging.md) | 🟩 | `1e9219d` |
| 1.4 | [Teltonika framing layer (envelope, CRC, handshake)](./phase-1-telemetry/04-teltonika-framing.md) | 🟩 | `1e9219d` |
| 1.5 | [Codec 8 parser](./phase-1-telemetry/05-codec-8.md) | 🟩 | `381287b` |
| 1.6 | [Codec 8 Extended parser (incl. NX)](./phase-1-telemetry/06-codec-8-extended.md) | 🟩 | `381287b` |
| 1.7 | [Codec 16 parser (incl. Generation Type)](./phase-1-telemetry/07-codec-16.md) | 🟩 | `381287b` |
| 1.8 | [Redis Streams publisher & main wiring](./phase-1-telemetry/08-redis-publisher.md) | 🟩 | `af06973` |
| 1.9 | [Fixture suite & testing strategy](./phase-1-telemetry/09-fixture-suite.md) | 🟩 | `381287b` |
| 1.10 | [Observability (Prometheus metrics)](./phase-1-telemetry/10-observability.md) | 🟩 | `26a1509` |
| 1.11 | [Dockerfile & Gitea workflow](./phase-1-telemetry/11-dockerfile-and-ci.md) | 🟩 | `88b742d` (slim pilot variant) |
| 1.12 | [Production hardening](./phase-1-telemetry/12-production-hardening.md) | ⏸ | *deferred — see below* |
| 1.13 | [Device authority (Redis allow-list refresher)](./phase-1-telemetry/13-device-authority.md) | ⏸ | *deferred — see below* |

#### Deferred (resume after the real-device pilot test)

These two tasks are paused so we can get the service onto real hardware as fast as possible. They are paused, not cancelled — each must be completed before the service is considered production-ready.

- **1.12 Production hardening.** Graceful shutdown is a stub today; uncaught-exception handlers are minimal. **Resume trigger:** before the pilot graduates to "always-on" or before any deployment that does rolling restarts. Acceptable for a manual pilot where we can stop/start the process by hand.
- **1.13 Device authority (Redis allow-list refresher).** Default `AllowAllAuthority` accepts every IMEI. **Resume trigger:** when Directus has a `devices` collection publishing the allow-list to Redis, or when the operational picture demands rejecting unknown IMEIs (`STRICT_DEVICE_AUTH=true`).

When resuming any of these, change the status from ⏸ back to ⬜ or 🟨 here and in the task file's status badge, and clear the deferral note in the task file.

### Phase 2 — Outbound commands (Codec 12, 14)

**Status:** ⬜ Not started (depends on Phase 1)
**Outcome:** Each Ingestion instance runs a parallel command consumer that reads from `commands:outbound:{instance_id}`, encodes Codec 12 or 14 frames, writes them to the appropriate device socket via a per-socket write queue, and publishes responses back to `commands:responses`. Includes the IMEI→instance connection registry and heartbeat.

[**See `phase-2-commands/README.md`**](./phase-2-commands/README.md)

| # | Task | Status |
|---|------|--------|
| 2.1 | [Connection registry & heartbeat](./phase-2-commands/01-connection-registry.md) | ⬜ |
| 2.2 | [Registry janitor (stale entry cleanup)](./phase-2-commands/02-registry-janitor.md) | ⬜ |
| 2.3 | [Per-socket write queue & outstanding-command tracker](./phase-2-commands/03-write-queue.md) | ⬜ |
| 2.4 | [Command consumer (stream reader)](./phase-2-commands/04-command-consumer.md) | ⬜ |
| 2.5 | [Codec 12 encoder + handler](./phase-2-commands/05-codec-12.md) | ⬜ |
| 2.6 | [Codec 14 encoder + ACK/nACK handler](./phase-2-commands/06-codec-14.md) | ⬜ |

### Phase 3 — Future / optional

**Status:** ❄️ Not committed
[**See `phase-3-future/README.md`**](./phase-3-future/README.md) for ideas on radar:

- Additional vendor adapters (Queclink, Concox).
- UDP transport for codecs 8/8E/16.
- Codec 15 (FMX6 RS232 modes) if such a fleet onboards.
- SMS-based protocols (Codec 4 24-position, binary SMS) if SMS fallback connectivity is needed.

## Operating model

- **Implementation agent contract.** Each task file is self-sufficient: goal, deliverables, specification, acceptance criteria. An agent should be able to complete one task without reading the whole wiki — but should skim the wiki references at the top of the task before starting.
- **Sequence within a phase.** Task numbering reflects intended order. Soft dependencies are explicit in each task's "Depends on" field. Tasks with no dependencies on each other can be done in parallel.
- **Status updates.** When a task is started, change its row in this ROADMAP to 🟨 and the task file's status badge accordingly. When done, 🟩 + a one-line note in the task file's "Done" section pointing at the merging commit/PR.
- **Drift control.** If implementation diverges from a task's spec, update the task file *before* the diverging code lands, with a note explaining why. Do not let plans rot — either fix the plan or fix the code.