Implement Phase 1 task 1.10 (Prometheus metrics + /healthz + /readyz)

Replaces the placeholder Metrics shim with a prom-client implementation
in src/observability/metrics.ts: all 10 Phase 1 metrics from the wiki
spec, plus nodejs_* defaults. Exposes /metrics, /healthz, /readyz over
node:http on METRICS_PORT (9090); /readyz returns 503 when Redis status
is not 'ready' or the TCP listener isn't bound.

The Metrics interface in src/core/types.ts is unchanged — adapter call
sites continue to use the same inc/observe shape. Only main.ts sees the
extended type that adds serializeMetrics().

Side effects:
- Dockerfile re-enables HEALTHCHECK pointing at /readyz, and EXPOSE 9090.
- frame-ingested log downgraded back to debug now that
  teltonika_records_published_total is scrapeable.
- 19 new unit tests covering exposition format, all metric types, and
  every HTTP endpoint path. Total now 98 passing.

Note: deploy/compose.yaml still does not expose 9090 — separate decision
about how Prometheus reaches the service (host port vs. internal scraper
on the same Docker network).
This commit is contained in:
2026-04-30 20:52:12 +02:00
parent ff9c8d67a4
commit d4a6d8f713
8 changed files with 720 additions and 27 deletions
+4 -5
View File
@@ -41,7 +41,7 @@ These rules govern every task. Any deviation must be discussed and documented as
### Phase 1 — Inbound telemetry (Codec 8, 8E, 16) ### Phase 1 — Inbound telemetry (Codec 8, 8E, 16)
**Status:** 🟨 In progress (core implementation done; observability + hardening + device authority paused for pilot test) **Status:** 🟨 In progress (observability landed; hardening + device authority paused for pilot test)
**Outcome:** A production-ready Node.js TCP server ingesting Teltonika telemetry from any FMB/FMC/FMM/FMU device, publishing normalized `Position` records to Redis Streams, with full observability and CI/CD via Gitea. **Outcome:** A production-ready Node.js TCP server ingesting Teltonika telemetry from any FMB/FMC/FMM/FMU device, publishing normalized `Position` records to Redis Streams, with full observability and CI/CD via Gitea.
[**See `phase-1-telemetry/README.md`**](./phase-1-telemetry/README.md) [**See `phase-1-telemetry/README.md`**](./phase-1-telemetry/README.md)
@@ -57,18 +57,17 @@ These rules govern every task. Any deviation must be discussed and documented as
| 1.7 | [Codec 16 parser (incl. Generation Type)](./phase-1-telemetry/07-codec-16.md) | 🟩 | `381287b` | | 1.7 | [Codec 16 parser (incl. Generation Type)](./phase-1-telemetry/07-codec-16.md) | 🟩 | `381287b` |
| 1.8 | [Redis Streams publisher & main wiring](./phase-1-telemetry/08-redis-publisher.md) | 🟩 | `af06973` | | 1.8 | [Redis Streams publisher & main wiring](./phase-1-telemetry/08-redis-publisher.md) | 🟩 | `af06973` |
| 1.9 | [Fixture suite & testing strategy](./phase-1-telemetry/09-fixture-suite.md) | 🟩 | `381287b` | | 1.9 | [Fixture suite & testing strategy](./phase-1-telemetry/09-fixture-suite.md) | 🟩 | `381287b` |
| 1.10 | [Observability (Prometheus metrics)](./phase-1-telemetry/10-observability.md) | | *deferred — see below* | | 1.10 | [Observability (Prometheus metrics)](./phase-1-telemetry/10-observability.md) | 🟩 | *(pending commit SHA)* |
| 1.11 | [Dockerfile & Gitea workflow](./phase-1-telemetry/11-dockerfile-and-ci.md) | 🟩 | `88b742d` (slim pilot variant) | | 1.11 | [Dockerfile & Gitea workflow](./phase-1-telemetry/11-dockerfile-and-ci.md) | 🟩 | `88b742d` (slim pilot variant) |
| 1.12 | [Production hardening](./phase-1-telemetry/12-production-hardening.md) | ⏸ | *deferred — see below* | | 1.12 | [Production hardening](./phase-1-telemetry/12-production-hardening.md) | ⏸ | *deferred — see below* |
| 1.13 | [Device authority (Redis allow-list refresher)](./phase-1-telemetry/13-device-authority.md) | ⏸ | *deferred — see below* | | 1.13 | [Device authority (Redis allow-list refresher)](./phase-1-telemetry/13-device-authority.md) | ⏸ | *deferred — see below* |
#### Deferred (resume after the real-device pilot test) #### Deferred (resume after the real-device pilot test)
These three tasks are paused so we can get the service onto real hardware as fast as possible. They are paused, not cancelled — each must be completed before the service is considered production-ready. These two tasks are paused so we can get the service onto real hardware as fast as possible. They are paused, not cancelled — each must be completed before the service is considered production-ready.
- **1.10 Observability (Prometheus metrics).** Tracking via the placeholder `Metrics` interface for now. **Resume trigger:** as soon as the pilot is generating real traffic and we want to measure it, *or* before any second instance is deployed (without metrics, "consumer lag" and "unknown codec" alerts cannot fire).
- **1.12 Production hardening.** Graceful shutdown is a stub today; uncaught-exception handlers are minimal. **Resume trigger:** before the pilot graduates to "always-on" or before any deployment that does rolling restarts. Acceptable for a manual pilot where we can stop/start the process by hand. - **1.12 Production hardening.** Graceful shutdown is a stub today; uncaught-exception handlers are minimal. **Resume trigger:** before the pilot graduates to "always-on" or before any deployment that does rolling restarts. Acceptable for a manual pilot where we can stop/start the process by hand.
- **1.13 Device authority (Redis allow-list refresher).** Default `AllowAllAuthority` accepts every IMEI; observability of `known | unknown` is moot until 1.10 lands. **Resume trigger:** when Directus has a `devices` collection publishing the allow-list to Redis, or when the operational picture demands rejecting unknown IMEIs (`STRICT_DEVICE_AUTH=true`). - **1.13 Device authority (Redis allow-list refresher).** Default `AllowAllAuthority` accepts every IMEI. **Resume trigger:** when Directus has a `devices` collection publishing the allow-list to Redis, or when the operational picture demands rejecting unknown IMEIs (`STRICT_DEVICE_AUTH=true`).
When resuming any of these, change the status from ⏸ back to ⬜ or 🟨 here and in the task file's status badge, and clear the deferral note in the task file. When resuming any of these, change the status from ⏸ back to ⬜ or 🟨 here and in the task file's status badge, and clear the deferral note in the task file.
@@ -1,7 +1,7 @@
# Task 1.10 — Observability (Prometheus metrics) # Task 1.10 — Observability (Prometheus metrics)
**Phase:** 1 — Inbound telemetry **Phase:** 1 — Inbound telemetry
**Status:** ⏸ Paused — deferred until after the real-device pilot test. See ROADMAP.md "Deferred" section for resume triggers. The placeholder `Metrics` interface in `src/core/types.ts` is what code currently uses; this task replaces it with `prom-client` and adds the `/metrics`, `/healthz`, `/readyz` HTTP endpoints. **Status:** 🟩
**Depends on:** 1.2, 1.3 **Depends on:** 1.2, 1.3
**Wiki refs:** `docs/wiki/sources/teltonika-ingestion-architecture.md` § 7. Observability, `docs/wiki/sources/gps-tracking-architecture.md` § 7.4 **Wiki refs:** `docs/wiki/sources/teltonika-ingestion-architecture.md` § 7. Observability, `docs/wiki/sources/gps-tracking-architecture.md` § 7.4
@@ -81,4 +81,4 @@ Use Node's `node:http` directly — no Express/Fastify dependency for two endpoi
## Done ## Done
(Fill in once complete.) Implemented `src/observability/metrics.ts` with `createMetrics()`, `startMetricsServer()`, and `ReadyzDeps`. Replaced the placeholder shim in `src/main.ts`, wired metrics server into boot and graceful shutdown, downgraded `frame ingested` log to debug, and re-enabled the Dockerfile `HEALTHCHECK`. Landed in: *(pending commit SHA)*
+3 -4
View File
@@ -24,9 +24,8 @@ COPY --from=build --chown=app:app /app/node_modules ./node_modules
COPY --from=build --chown=app:app /app/dist ./dist COPY --from=build --chown=app:app /app/dist ./dist
COPY --from=build --chown=app:app /app/package.json ./package.json COPY --from=build --chown=app:app /app/package.json ./package.json
USER app USER app
# Only the TCP port is exposed. METRICS_PORT (9090) is in the config schema but
# no HTTP server runs today — task 1.10 (observability) adds that server.
EXPOSE 5027 EXPOSE 5027
# HEALTHCHECK deferred — re-add `wget -qO- http://localhost:${METRICS_PORT}/readyz` EXPOSE 9090
# when task 1.10 (observability) ships and the HTTP server is running. HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD wget -qO- http://localhost:9090/readyz || exit 1
CMD ["node", "dist/main.js"] CMD ["node", "dist/main.js"]
-1
View File
@@ -77,7 +77,6 @@ The workflow uses `secrets.REGISTRY_USERNAME` and `secrets.REGISTRY_PASSWORD` fo
This service is running in pilot form. The following tasks are **paused** — they are not missing by accident, they are deferred by design to get onto real Teltonika hardware first: This service is running in pilot form. The following tasks are **paused** — they are not missing by accident, they are deferred by design to get onto real Teltonika hardware first:
- **Observability (task 1.10):** No `/metrics`, `/healthz`, or `/readyz` HTTP endpoints exist yet. `METRICS_PORT` is in the config schema but nothing listens on it. The Docker `HEALTHCHECK` is also absent for this reason.
- **Production hardening (task 1.12):** Graceful shutdown is a functional stub; uncaught-exception handling is minimal. - **Production hardening (task 1.12):** Graceful shutdown is a functional stub; uncaught-exception handling is minimal.
- **Device authority (task 1.13):** `AllowAllAuthority` is active — every IMEI is accepted. `STRICT_DEVICE_AUTH=true` is wired but the Redis allow-list refresher is not yet implemented. - **Device authority (task 1.13):** `AllowAllAuthority` is active — every IMEI is accepted. `STRICT_DEVICE_AUTH=true` is wired but the Redis allow-list refresher is not yet implemented.
+3 -3
View File
@@ -217,9 +217,9 @@ export function createTeltonikaAdapter(options: TeltonikaAdapterOptions): Adapte
result: 'ok', result: 'ok',
}); });
// Pilot-stage visibility into per-frame ingest. Downgrade to debug // teltonika_frames_total{result="ok"} and teltonika_records_published_total
// once task 1.10 wires up prom-client and frames_total is scrapeable. // now carry this signal in Prometheus; keep the log at debug to avoid noise.
sessionLogger.info( sessionLogger.debug(
{ codec: codecLabel, records: result.recordCount }, { codec: codecLabel, records: result.recordCount },
'frame ingested', 'frame ingested',
); );
+22 -12
View File
@@ -1,13 +1,14 @@
import type { Redis } from 'ioredis'; import type { Redis } from 'ioredis';
import type * as http from 'node:http';
import type * as net from 'node:net'; import type * as net from 'node:net';
import { loadConfig } from './config/load.js'; import { loadConfig } from './config/load.js';
import type { Config } from './config/load.js'; import type { Config } from './config/load.js';
import { createLogger } from './observability/logger.js'; import { createLogger } from './observability/logger.js';
import { createMetrics, startMetricsServer } from './observability/metrics.js';
import { createPublisher, connectRedis } from './core/publish.js'; import { createPublisher, connectRedis } from './core/publish.js';
import { startServer } from './core/server.js'; import { startServer } from './core/server.js';
import { createTeltonikaAdapter } from './adapters/teltonika/index.js'; import { createTeltonikaAdapter } from './adapters/teltonika/index.js';
import { AllowAllAuthority } from './adapters/teltonika/device-authority.js'; import { AllowAllAuthority } from './adapters/teltonika/device-authority.js';
import type { Metrics } from './core/types.js';
// ------------------------------------------------------------------------- // -------------------------------------------------------------------------
// Startup: validate config (fail fast on bad env), build logger, boot server // Startup: validate config (fail fast on bad env), build logger, boot server
@@ -31,12 +32,8 @@ const logger = createLogger({
logger.info('tcp-ingestion starting'); logger.info('tcp-ingestion starting');
// Placeholder metrics implementation — replaced in task 1.10. // Real prom-client metrics implementation (task 1.10).
// Using the Metrics interface from types.ts (no prom-client yet). const metrics = createMetrics();
const metrics: Metrics = {
inc: (name, labels) => logger.trace({ metric: name, labels }, 'metric inc'),
observe: (name, value, labels) => logger.trace({ metric: name, value, labels }, 'metric observe'),
};
// ------------------------------------------------------------------------- // -------------------------------------------------------------------------
// Wire up the pipeline // Wire up the pipeline
@@ -65,10 +62,18 @@ async function main(): Promise<void> {
metrics, metrics,
}); });
// 5. Install graceful shutdown (stub — full hardening in task 1.12) // 5. Start metrics HTTP server (task 1.10).
installGracefulShutdown({ server, redis, publisher, logger }); // readyzDeps use ioredis's synchronous `.status` field and net.Server's
// `.listening` boolean — no I/O, so these closures are always cheap.
const metricsServer = startMetricsServer(config.METRICS_PORT, metrics.serializeMetrics, {
isRedisReady: () => redis.status === 'ready',
isTcpListening: () => server.listening,
});
logger.info({ port: config.TELTONIKA_PORT }, 'tcp-ingestion ready'); // 6. Install graceful shutdown (stub — full hardening in task 1.12)
installGracefulShutdown({ server, metricsServer, redis, publisher, logger });
logger.info({ port: config.TELTONIKA_PORT, metricsPort: config.METRICS_PORT }, 'tcp-ingestion ready');
} }
// ------------------------------------------------------------------------- // -------------------------------------------------------------------------
@@ -77,13 +82,14 @@ async function main(): Promise<void> {
type ShutdownDeps = { type ShutdownDeps = {
readonly server: net.Server; readonly server: net.Server;
readonly metricsServer: http.Server;
readonly redis: Redis; readonly redis: Redis;
readonly publisher: { drain(timeoutMs: number): Promise<void> }; readonly publisher: { drain(timeoutMs: number): Promise<void> };
readonly logger: ReturnType<typeof createLogger>; readonly logger: ReturnType<typeof createLogger>;
}; };
function installGracefulShutdown(deps: ShutdownDeps): void { function installGracefulShutdown(deps: ShutdownDeps): void {
const { server, redis, publisher, logger: log } = deps; const { server, metricsServer, redis, publisher, logger: log } = deps;
let shuttingDown = false; let shuttingDown = false;
@@ -93,11 +99,15 @@ function installGracefulShutdown(deps: ShutdownDeps): void {
log.info({ signal }, 'shutdown signal received'); log.info({ signal }, 'shutdown signal received');
// Stop accepting new connections // Stop accepting new TCP connections
server.close(() => { server.close(() => {
log.info('TCP server closed'); log.info('TCP server closed');
}); });
// Close the metrics HTTP server before quitting Redis so /readyz reports
// not-ready during the drain window (task 1.12 will tighten this further).
metricsServer.close();
// Drain publisher queue then disconnect Redis // Drain publisher queue then disconnect Redis
publisher publisher
.drain(10_000) .drain(10_000)
+326
View File
@@ -0,0 +1,326 @@
import * as http from 'node:http';
import {
Registry,
Counter,
Gauge,
Histogram,
collectDefaultMetrics,
} from 'prom-client';
import type { Metrics } from '../core/types.js';
// ---------------------------------------------------------------------------
// Readiness probe dependencies — injected to keep this module free of
// adapters/ and core/ imports that would violate the layering rule.
// ---------------------------------------------------------------------------
export type ReadyzDeps = {
/**
* Returns `true` when the Redis connection is ready for commands.
* Typically: `() => redis.status === 'ready'`
*/
readonly isRedisReady: () => boolean;
/**
* Returns `true` when the TCP listener is bound.
* Typically: `() => tcpServer.listening`
*/
readonly isTcpListening: () => boolean;
};
// ---------------------------------------------------------------------------
// Internal metric registry type — one typed field per metric in the inventory.
// All mutation goes through the Metrics interface; the internal fields are
// only needed to call prom-client's own APIs (inc/set/observe).
// ---------------------------------------------------------------------------
type InternalRegistry = {
readonly registry: Registry;
readonly connectionsActive: Gauge;
readonly handshakeTotal: Counter;
readonly deviceAuthorityFailuresTotal: Counter;
readonly framesTotal: Counter;
readonly recordsPublishedTotal: Counter;
readonly parseDurationSeconds: Histogram;
readonly unknownCodecTotal: Counter;
readonly publishQueueDepth: Gauge;
readonly publishOverflowTotal: Counter;
readonly publishDurationSeconds: Histogram;
};
// ---------------------------------------------------------------------------
// createMetrics — builds the full prom-client registry and returns a Metrics
// wrapper that satisfies the existing call-site interface.
// ---------------------------------------------------------------------------
/**
* Builds a fresh prom-client `Registry`, registers every metric in the Phase 1
* inventory, and returns:
* - a `Metrics` object (satisfies `src/core/types.ts:Metrics`) for injection
* into adapters and the publisher
* - a `serializeMetrics()` function for exposition format
*
* `collectDefaultMetrics` is called once to enable Node.js process metrics
* (GC, event loop lag, heap stats, etc.) under the same registry.
*/
export function createMetrics(): Metrics & {
serializeMetrics: () => Promise<string>;
} {
const internal = buildInternalRegistry();
// Expose default Node.js process metrics (nodejs_*) on the same registry.
collectDefaultMetrics({ register: internal.registry });
const metricsImpl: Metrics & { serializeMetrics: () => Promise<string> } = {
inc(name: string, labels?: Record<string, string>): void {
dispatchInc(internal, name, labels);
},
observe(name: string, value: number, labels?: Record<string, string>): void {
dispatchObserve(internal, name, value, labels);
},
serializeMetrics(): Promise<string> {
return internal.registry.metrics();
},
};
return metricsImpl;
}
// ---------------------------------------------------------------------------
// serializeMetrics — standalone helper for use outside the Metrics wrapper.
// Exported so startMetricsServer can call it without holding a reference to
// the internal registry.
// ---------------------------------------------------------------------------
// Note: this is accessed via the metricsImpl.serializeMetrics closure above.
// A standalone export is not required; the server takes the bound method.
// ---------------------------------------------------------------------------
// startMetricsServer — minimal node:http server for /metrics, /healthz, /readyz
// ---------------------------------------------------------------------------
/**
* Starts the Prometheus metrics HTTP server on the given port.
*
* Endpoints:
* GET /metrics — Prometheus exposition format (text/plain; version=0.0.4)
* GET /healthz — 200 if the process is alive (liveness probe)
* GET /readyz — 200 if Redis is connected AND TCP server is listening;
* 503 otherwise (readiness probe)
*
* @param port Port to bind; 0 lets the OS pick (useful in tests).
* @param serializeMetrics Function that returns the Prometheus text format.
* @param readyzDeps Sync accessors for Redis and TCP readiness state.
*/
export function startMetricsServer(
port: number,
serializeMetrics: () => Promise<string>,
readyzDeps: ReadyzDeps,
): http.Server {
const server = http.createServer((req, res) => {
const url = req.url ?? '/';
const method = req.method ?? 'GET';
// Reject non-GET requests for all endpoints.
if (method !== 'GET') {
res.writeHead(405, { 'Content-Type': 'text/plain' });
res.end('Method Not Allowed');
return;
}
if (url === '/metrics') {
serializeMetrics()
.then((text) => {
res.writeHead(200, { 'Content-Type': 'text/plain; version=0.0.4; charset=utf-8' });
res.end(text);
})
.catch((err: unknown) => {
res.writeHead(500, { 'Content-Type': 'text/plain' });
res.end(`Internal Server Error: ${err instanceof Error ? err.message : String(err)}`);
});
return;
}
if (url === '/healthz') {
res.writeHead(200, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ status: 'ok' }));
return;
}
if (url === '/readyz') {
const redisOk = readyzDeps.isRedisReady();
const tcpOk = readyzDeps.isTcpListening();
if (redisOk && tcpOk) {
res.writeHead(200, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ status: 'ok' }));
} else {
res.writeHead(503, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ status: 'not ready', redis: redisOk, tcp: tcpOk }));
}
return;
}
res.writeHead(404, { 'Content-Type': 'text/plain' });
res.end('Not Found');
});
server.listen(port);
return server;
}
// ---------------------------------------------------------------------------
// Private: registry construction
// ---------------------------------------------------------------------------
function buildInternalRegistry(): InternalRegistry {
const registry = new Registry();
const connectionsActive = new Gauge({
name: 'teltonika_connections_active',
help: 'Currently open device sessions.',
registers: [registry],
});
const handshakeTotal = new Counter({
name: 'teltonika_handshake_total',
help: 'IMEI handshake outcomes.',
labelNames: ['result', 'known'],
registers: [registry],
});
const deviceAuthorityFailuresTotal = new Counter({
name: 'teltonika_device_authority_failures_total',
help: 'Times a DeviceAuthority.check call threw or timed out.',
registers: [registry],
});
const framesTotal = new Counter({
name: 'teltonika_frames_total',
help: 'Frame-level outcomes.',
labelNames: ['codec', 'result'],
registers: [registry],
});
const recordsPublishedTotal = new Counter({
name: 'teltonika_records_published_total',
help: 'AVL records emitted to Redis.',
labelNames: ['codec'],
registers: [registry],
});
const parseDurationSeconds = new Histogram({
name: 'teltonika_parse_duration_seconds',
help: 'Per-frame parse time.',
labelNames: ['codec'],
buckets: [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1],
registers: [registry],
});
const unknownCodecTotal = new Counter({
name: 'teltonika_unknown_codec_total',
help: 'Canary for codec coverage drift.',
labelNames: ['codec_id'],
registers: [registry],
});
const publishQueueDepth = new Gauge({
name: 'teltonika_publish_queue_depth',
help: 'Current bounded-queue depth.',
registers: [registry],
});
const publishOverflowTotal = new Counter({
name: 'teltonika_publish_overflow_total',
help: 'Records dropped because the queue was full.',
registers: [registry],
});
const publishDurationSeconds = new Histogram({
name: 'teltonika_publish_duration_seconds',
help: 'XADD latency.',
buckets: [0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0],
registers: [registry],
});
return {
registry,
connectionsActive,
handshakeTotal,
deviceAuthorityFailuresTotal,
framesTotal,
recordsPublishedTotal,
parseDurationSeconds,
unknownCodecTotal,
publishQueueDepth,
publishOverflowTotal,
publishDurationSeconds,
};
}
// ---------------------------------------------------------------------------
// Private: dispatch helpers — map string metric names to typed prom-client calls
// ---------------------------------------------------------------------------
function dispatchInc(
r: InternalRegistry,
name: string,
labels?: Record<string, string>,
): void {
switch (name) {
case 'teltonika_handshake_total':
r.handshakeTotal.inc(labels ?? {});
break;
case 'teltonika_device_authority_failures_total':
r.deviceAuthorityFailuresTotal.inc();
break;
case 'teltonika_frames_total':
r.framesTotal.inc(labels ?? {});
break;
case 'teltonika_records_published_total':
r.recordsPublishedTotal.inc(labels ?? {});
break;
case 'teltonika_unknown_codec_total':
r.unknownCodecTotal.inc(labels ?? {});
break;
case 'teltonika_publish_overflow_total':
r.publishOverflowTotal.inc(labels ?? {});
break;
case 'teltonika_connections_active':
// inc() on a gauge means +1 (connection opened)
r.connectionsActive.inc();
break;
default:
// Unknown metric name — silently ignore. This preserves the contract
// that the Metrics interface never throws, and avoids crashing the
// process when a call site references a metric not yet in the registry
// (e.g. during staged rollouts or future tasks).
break;
}
}
function dispatchObserve(
r: InternalRegistry,
name: string,
value: number,
labels?: Record<string, string>,
): void {
switch (name) {
case 'teltonika_parse_duration_seconds':
r.parseDurationSeconds.observe(labels ?? {}, value);
break;
case 'teltonika_publish_duration_seconds':
r.publishDurationSeconds.observe(value);
break;
case 'teltonika_publish_queue_depth':
r.publishQueueDepth.set(value);
break;
case 'teltonika_connections_active':
// observe() on a gauge means set(value) — caller controls the value
r.connectionsActive.set(value);
break;
default:
// Unknown metric name — silently ignore (see dispatchInc comment).
break;
}
}
+360
View File
@@ -0,0 +1,360 @@
/**
* Unit tests for src/observability/metrics.ts
*
* Covers:
* - serializeMetrics() returns valid Prometheus exposition format with the
* full metric inventory present (some at zero)
* - counter increments are reflected in the serialized output
* - teltonika_unknown_codec_total{codec_id="0xff"} appears after an unknown-codec event
* - startMetricsServer responds correctly to /metrics, /healthz, and /readyz
* - /readyz returns 503 when Redis is not ready or TCP is not listening
*/
import { describe, it, expect, afterEach } from 'vitest';
import * as http from 'node:http';
import { createMetrics, startMetricsServer } from '../src/observability/metrics.js';
import type { ReadyzDeps } from '../src/observability/metrics.js';
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
/**
* Makes an HTTP GET request and resolves with { statusCode, body }.
* Uses the raw node:http module to match the server's implementation.
*/
function get(port: number, path: string): Promise<{ statusCode: number; body: string }> {
return new Promise((resolve, reject) => {
const req = http.get(`http://localhost:${port}${path}`, (res) => {
let body = '';
res.on('data', (chunk: Buffer) => {
body += chunk.toString();
});
res.on('end', () => {
resolve({ statusCode: res.statusCode ?? 0, body });
});
});
req.on('error', reject);
});
}
/**
* Waits for a server to be listening, then runs `fn`, then closes the server.
* Returns after the server is fully closed.
*/
function withServer(
server: http.Server,
fn: (port: number) => Promise<void>,
): Promise<void> {
return new Promise((resolve, reject) => {
server.once('listening', () => {
const addr = server.address();
const port = typeof addr === 'object' && addr !== null ? addr.port : 0;
fn(port)
.then(() => {
server.close((err) => {
if (err) reject(err);
else resolve();
});
})
.catch((err: unknown) => {
server.close(() => reject(err));
});
});
server.once('error', reject);
});
}
// ---------------------------------------------------------------------------
// 1. serializeMetrics — Prometheus exposition format
// ---------------------------------------------------------------------------
describe('createMetrics — serializeMetrics', () => {
it('returns a non-empty string', async () => {
const metrics = createMetrics();
const text = await metrics.serializeMetrics();
expect(typeof text).toBe('string');
expect(text.length).toBeGreaterThan(0);
});
it('contains every teltonika_* metric in the Phase 1 inventory', async () => {
const metrics = createMetrics();
const text = await metrics.serializeMetrics();
const expectedMetrics = [
'teltonika_connections_active',
'teltonika_handshake_total',
'teltonika_device_authority_failures_total',
'teltonika_frames_total',
'teltonika_records_published_total',
'teltonika_parse_duration_seconds',
'teltonika_unknown_codec_total',
'teltonika_publish_queue_depth',
'teltonika_publish_overflow_total',
'teltonika_publish_duration_seconds',
];
for (const name of expectedMetrics) {
expect(text, `expected "${name}" in metrics output`).toContain(name);
}
});
it('contains nodejs_* default process metrics', async () => {
const metrics = createMetrics();
const text = await metrics.serializeMetrics();
// prom-client collectDefaultMetrics registers nodejs_version_info at minimum
expect(text).toContain('nodejs_');
});
it('exposition format contains HELP and TYPE lines', async () => {
const metrics = createMetrics();
const text = await metrics.serializeMetrics();
expect(text).toContain('# HELP teltonika_handshake_total');
expect(text).toContain('# TYPE teltonika_handshake_total counter');
expect(text).toContain('# HELP teltonika_frames_total');
expect(text).toContain('# TYPE teltonika_frames_total counter');
});
});
// ---------------------------------------------------------------------------
// 2. Counter increments
// ---------------------------------------------------------------------------
describe('createMetrics — counter increments', () => {
it('increments teltonika_frames_total and teltonika_records_published_total', async () => {
const metrics = createMetrics();
// Simulate a successful Codec 8E frame
metrics.inc('teltonika_frames_total', { codec: '8E', result: 'ok' });
metrics.inc('teltonika_records_published_total', { codec: '8E' });
const text = await metrics.serializeMetrics();
// The exposition format for a counter with labels looks like:
// teltonika_frames_total{codec="8E",result="ok"} 1
expect(text).toMatch(/teltonika_frames_total\{[^}]*codec="8E"[^}]*result="ok"[^}]*\}\s+1/);
expect(text).toMatch(/teltonika_records_published_total\{[^}]*codec="8E"[^}]*\}\s+1/);
});
it('accumulates multiple increments on the same label set', async () => {
const metrics = createMetrics();
metrics.inc('teltonika_frames_total', { codec: '8', result: 'ok' });
metrics.inc('teltonika_frames_total', { codec: '8', result: 'ok' });
metrics.inc('teltonika_frames_total', { codec: '8', result: 'ok' });
const text = await metrics.serializeMetrics();
expect(text).toMatch(/teltonika_frames_total\{[^}]*codec="8"[^}]*result="ok"[^}]*\}\s+3/);
});
it('tracks crc_fail result separately from ok', async () => {
const metrics = createMetrics();
metrics.inc('teltonika_frames_total', { codec: '8', result: 'ok' });
metrics.inc('teltonika_frames_total', { codec: '8', result: 'crc_fail' });
const text = await metrics.serializeMetrics();
expect(text).toMatch(/teltonika_frames_total\{[^}]*codec="8"[^}]*result="ok"[^}]*\}\s+1/);
expect(text).toMatch(/teltonika_frames_total\{[^}]*codec="8"[^}]*result="crc_fail"[^}]*\}\s+1/);
});
});
// ---------------------------------------------------------------------------
// 3. Unknown codec canary
// ---------------------------------------------------------------------------
describe('createMetrics — unknown codec canary', () => {
it('records teltonika_unknown_codec_total{codec_id="0xff"} after an unknown-codec event', async () => {
const metrics = createMetrics();
metrics.inc('teltonika_unknown_codec_total', { codec_id: '0xff' });
const text = await metrics.serializeMetrics();
expect(text).toMatch(/teltonika_unknown_codec_total\{[^}]*codec_id="0xff"[^}]*\}\s+1/);
});
it('distinguishes different codec_id values', async () => {
const metrics = createMetrics();
metrics.inc('teltonika_unknown_codec_total', { codec_id: '0x0f' });
metrics.inc('teltonika_unknown_codec_total', { codec_id: '0x0f' });
metrics.inc('teltonika_unknown_codec_total', { codec_id: '0x20' });
const text = await metrics.serializeMetrics();
expect(text).toMatch(/teltonika_unknown_codec_total\{[^}]*codec_id="0x0f"[^}]*\}\s+2/);
expect(text).toMatch(/teltonika_unknown_codec_total\{[^}]*codec_id="0x20"[^}]*\}\s+1/);
});
});
// ---------------------------------------------------------------------------
// 4. observe() — gauge and histogram
// ---------------------------------------------------------------------------
describe('createMetrics — observe', () => {
it('sets teltonika_publish_queue_depth gauge', async () => {
const metrics = createMetrics();
metrics.observe('teltonika_publish_queue_depth', 42);
const text = await metrics.serializeMetrics();
expect(text).toMatch(/teltonika_publish_queue_depth\s+42/);
});
it('records teltonika_parse_duration_seconds histogram observation', async () => {
const metrics = createMetrics();
metrics.observe('teltonika_parse_duration_seconds', 0.0003, { codec: '8E' });
const text = await metrics.serializeMetrics();
// The histogram sum should contain the observed value
expect(text).toMatch(/teltonika_parse_duration_seconds_sum\{[^}]*codec="8E"[^}]*\}\s+0\.0003/);
});
it('ignores unknown metric names without throwing', () => {
const metrics = createMetrics();
// Must not throw — the Metrics interface contract is never-throw
expect(() => metrics.inc('teltonika_nonexistent_metric')).not.toThrow();
expect(() => metrics.observe('teltonika_nonexistent_metric', 1.0)).not.toThrow();
});
});
// ---------------------------------------------------------------------------
// 5. startMetricsServer — HTTP endpoint behaviour
// ---------------------------------------------------------------------------
describe('startMetricsServer', () => {
// Track servers for cleanup in case of test failure
const openServers: http.Server[] = [];
afterEach(() => {
for (const s of openServers) {
if (s.listening) s.close();
}
openServers.length = 0;
});
it('GET /metrics returns 200 with Prometheus text', async () => {
const metrics = createMetrics();
const readyzDeps: ReadyzDeps = {
isRedisReady: () => true,
isTcpListening: () => true,
};
// port=0 → OS picks a free port
const server = startMetricsServer(0, metrics.serializeMetrics, readyzDeps);
openServers.push(server);
await withServer(server, async (port) => {
const { statusCode, body } = await get(port, '/metrics');
expect(statusCode).toBe(200);
expect(body).toContain('teltonika_');
expect(body).toContain('nodejs_');
});
});
it('GET /healthz returns 200 regardless of Redis/TCP state', async () => {
const metrics = createMetrics();
const readyzDeps: ReadyzDeps = {
isRedisReady: () => false, // deliberately unhealthy
isTcpListening: () => false,
};
const server = startMetricsServer(0, metrics.serializeMetrics, readyzDeps);
openServers.push(server);
await withServer(server, async (port) => {
const { statusCode, body } = await get(port, '/healthz');
expect(statusCode).toBe(200);
expect(JSON.parse(body)).toEqual({ status: 'ok' });
});
});
it('GET /readyz returns 200 when Redis is ready and TCP is listening', async () => {
const metrics = createMetrics();
const readyzDeps: ReadyzDeps = {
isRedisReady: () => true,
isTcpListening: () => true,
};
const server = startMetricsServer(0, metrics.serializeMetrics, readyzDeps);
openServers.push(server);
await withServer(server, async (port) => {
const { statusCode, body } = await get(port, '/readyz');
expect(statusCode).toBe(200);
expect(JSON.parse(body)).toEqual({ status: 'ok' });
});
});
it('GET /readyz returns 503 when Redis is not ready', async () => {
const metrics = createMetrics();
const readyzDeps: ReadyzDeps = {
isRedisReady: () => false,
isTcpListening: () => true,
};
const server = startMetricsServer(0, metrics.serializeMetrics, readyzDeps);
openServers.push(server);
await withServer(server, async (port) => {
const { statusCode, body } = await get(port, '/readyz');
expect(statusCode).toBe(503);
const parsed = JSON.parse(body) as { status: string; redis: boolean; tcp: boolean };
expect(parsed.status).toBe('not ready');
expect(parsed.redis).toBe(false);
expect(parsed.tcp).toBe(true);
});
});
it('GET /readyz returns 503 when TCP server is not listening', async () => {
const metrics = createMetrics();
const readyzDeps: ReadyzDeps = {
isRedisReady: () => true,
isTcpListening: () => false,
};
const server = startMetricsServer(0, metrics.serializeMetrics, readyzDeps);
openServers.push(server);
await withServer(server, async (port) => {
const { statusCode } = await get(port, '/readyz');
expect(statusCode).toBe(503);
});
});
it('GET /readyz returns 503 when both Redis and TCP are down', async () => {
const metrics = createMetrics();
const readyzDeps: ReadyzDeps = {
isRedisReady: () => false,
isTcpListening: () => false,
};
const server = startMetricsServer(0, metrics.serializeMetrics, readyzDeps);
openServers.push(server);
await withServer(server, async (port) => {
const { statusCode, body } = await get(port, '/readyz');
expect(statusCode).toBe(503);
const parsed = JSON.parse(body) as { status: string; redis: boolean; tcp: boolean };
expect(parsed.redis).toBe(false);
expect(parsed.tcp).toBe(false);
});
});
it('GET /unknown-path returns 404', async () => {
const metrics = createMetrics();
const readyzDeps: ReadyzDeps = {
isRedisReady: () => true,
isTcpListening: () => true,
};
const server = startMetricsServer(0, metrics.serializeMetrics, readyzDeps);
openServers.push(server);
await withServer(server, async (port) => {
const { statusCode } = await get(port, '/not-a-real-endpoint');
expect(statusCode).toBe(404);
});
});
});