tcp-ingestion/.planning/phase-1-telemetry/12-production-hardening.md

# Task 1.12 — Production hardening

**Phase:** 1 — Inbound telemetry
**Status:** ⏸ Paused — deferred until after the real-device pilot test. See ROADMAP.md "Deferred" section for resume triggers. `installGracefulShutdown` exists as a stub from task 1.8; this task fully implements signal handling, drain timeouts, unhandled-rejection handlers, and writes OPERATIONS.md. **Resume before any always-on deployment or rolling-restart workflow.**
**Depends on:** 1.8, 1.10, 1.11
**Wiki refs:** `docs/wiki/concepts/failure-domains.md`

## Goal

Make the service safe for unattended production operation: graceful shutdown, robust error handling, structured logging discipline, sane defaults for resource limits, and operational documentation.

## Deliverables

- `src/core/lifecycle.ts` — `installGracefulShutdown({ ... })` that wires SIGTERM/SIGINT/SIGHUP to a coordinated shutdown.
- `src/core/errors.ts` — typed error classes (`HandshakeError`, `FrameError`, `PublishOverflowError`, `RedisUnavailableError`).
- Updates to `src/main.ts` to install error handlers and shutdown.
- `OPERATIONS.md` (or section in `README.md`) covering: env var reference, signals, log fields, metric meanings, common alert rules, troubleshooting.
- (Optional) `docs/runbook.md` for on-call: "what to do when X alert fires."

## Specification

### Graceful shutdown

On SIGTERM (deployment rolling update) or SIGINT (Ctrl-C):

1. **Stop accepting new connections.** `server.close()` — existing sockets continue.
2. **Drain the publish queue.** Stop accepting new `publish()` calls; wait for the worker to flush queued records to Redis (with a timeout, e.g. 10s).
3. **Send a final goodbye on each open socket.** Optional: just let TCP FIN naturally; devices will reconnect to a new instance.
4. **Close Redis connection.**
5. **Exit cleanly with code 0.**

If shutdown takes longer than `SHUTDOWN_TIMEOUT_MS` (default 30s), log and exit with code 1 — the orchestrator will SIGKILL anyway, but exiting deliberately gives a cleaner signal.

```ts
export function installGracefulShutdown(handles: ShutdownHandles) {
  let shuttingDown = false;
  const shutdown = async (signal: string) => {
    if (shuttingDown) return;
    shuttingDown = true;
    handles.logger.info({ signal }, 'shutdown: starting');
    const deadline = setTimeout(() => {
      handles.logger.error({}, 'shutdown: timed out, forcing exit');
      process.exit(1);
    }, handles.timeoutMs ?? 30_000);
    try {
      await new Promise<void>((res) => handles.server.close(() => res()));
      await handles.publisher.drain(10_000);
      await handles.redis.quit();
      handles.metricsServer.close();
      clearTimeout(deadline);
      handles.logger.info({}, 'shutdown: clean exit');
      process.exit(0);
    } catch (err) {
      handles.logger.error({ err }, 'shutdown: error during drain');
      clearTimeout(deadline);
      process.exit(1);
    }
  };
  process.on('SIGTERM', () => shutdown('SIGTERM'));
  process.on('SIGINT', () => shutdown('SIGINT'));
}
```

### Unhandled promise / uncaught exception

```ts
process.on('unhandledRejection', (reason) => {
  logger.fatal({ reason }, 'unhandledRejection');
  process.exit(1);
});
process.on('uncaughtException', (err) => {
  logger.fatal({ err }, 'uncaughtException');
  process.exit(1);
});
```

Crashing the process on either is the right move — the orchestrator restarts, devices reconnect, no harm done. The wrong move is to log and continue; that hides real bugs.

ESLint's `no-floating-promises` (added in task 1.1) is the first line of defense; these handlers are the safety net.

### Per-socket error handling

In the session loop:

- Errors from `BufferedReader` / `frame.ts` / codec parsers: log at `warn` with `imei`, drop the socket.
- Errors from `ctx.publish` (specifically `PublishOverflowError`): skip the ACK, continue reading. Device retransmits.
- Errors from `ctx.publish` (other, unexpected): log at `error`, drop the socket. Open question: should we crash the process? Recommendation: drop the socket only; let the publisher's own logic decide whether the underlying issue (e.g. Redis hang) warrants process exit.

### Resource limits

- **Max concurrent connections per instance:** soft cap via gauge alert (`teltonika_connections_active > 5000`). No hard cap in code — let the OS-level fd limit be the real ceiling.
- **Per-connection memory:** the `BufferedReader` buffer is bounded by `MAX_AVL_PACKET_SIZE` (~1.3KB) per session. With 5,000 connections, ~6.5MB of buffer state — fine.
- **Node heap:** set via `NODE_OPTIONS=--max-old-space-size=512` in the Dockerfile or compose. 512MB is plenty for this workload.

### Logging discipline (audit pass)

Before declaring this task done, walk through every `logger.*` call site and confirm:

- `info`: lifecycle events (startup, shutdown, server bound).
- `warn`: recoverable per-frame issues (CRC fail, malformed handshake), per-connection drops.
- `error`: per-publish failures, unexpected per-session errors.
- `fatal`: process-killing conditions (Redis unreachable for >X seconds, `unhandledRejection`).
- `debug`: per-frame parse details, per-record publish details.
- No `console.log` anywhere in production paths. If there are any, replace.

### OPERATIONS.md outline

```
# tcp-ingestion — Operations

## Configuration
[table of env vars from task 1.3]

## Signals
| Signal | Effect |
|--------|--------|
| SIGTERM | Graceful shutdown (drain publish queue, close connections, exit 0) |
| SIGINT | Same as SIGTERM |

## Metrics
[table of metrics from task 1.10]

## Alerts (recommended)
- `teltonika_unknown_codec_total > 0` for 5 min: investigate codec coverage drift.
- `teltonika_publish_overflow_total > 0` for 1 min: Redis or downstream backed up.
- `rate(teltonika_frames_total{result="crc_fail"}[5m]) / rate(teltonika_frames_total[5m]) > 0.01`: high CRC error rate, suspect device firmware or line quality.
- `teltonika_connections_active{instance=...} == 0` for 10 min while peer instances have traffic: instance is silently broken; investigate.

## Troubleshooting
- "Devices not connecting" → check TCP_PORT firewall, /readyz response, Redis connectivity.
- "Records not appearing in Redis" → check publish queue depth metric, then Redis connectivity.
- "High CRC failures from one IMEI" → likely a firmware bug or bad cellular link; coordinate with device fleet ops.
```

## Acceptance criteria

- [ ] SIGTERM during steady-state traffic results in a clean exit with no data loss (verified by killing the process and confirming the publish queue drained, no `PublishOverflowError` in the last second of logs).
- [ ] SIGTERM under publish-queue-overflow conditions still exits within `SHUTDOWN_TIMEOUT_MS`.
- [ ] An `unhandledRejection` (intentionally injected via test) logs at fatal and exits non-zero.
- [ ] OPERATIONS.md is populated and accurate; an on-caller could read it cold and find the answer to "what does this metric mean."
- [ ] All log calls audited; no `console.log` in production paths.

## Risks / open questions

- The "drain publish queue with timeout" balance: too long blocks deployments; too short loses records on shutdown. Default 10s is a reasonable starting point; tune after real production data.
- Crashing on `unhandledRejection` is opinionated. Some teams prefer to log and continue. We choose crash because the alternative hides bugs and we have a fast restart path. Document the choice.

## Done

(Fill in once complete.)