Metrics: several families are registered but never incremented (export as 0)

[oreofeolurin]

Surfaced by #43, which wired up the Prometheus exporter. Now that /metrics is actually served, it's visible that several metric families are created and exported but never written to — they scrape as 0 forever, which reads as "this node is idle" rather than "this isn't instrumented".

The registry and the exporter are both fine. The gap is that the write paths never call the record* methods.

Current state

Counted callers outside the metrics modules themselves:

Family	Registered?	Recorded?	Effect
ServerMetrics (flo_commands_total, bytes, connections, uptime)	n/a	✅ recordCommand, recordBytesReceived/Sent, connectionOpened/Closed	works
ReplicationMetrics (flo_replication_*)	n/a	✅ wired in #16	works
ProcessingMetrics (flo_processing_*)	n/a	⚠️ partial — recordProcessed (2), recordCheckpoint (1), recordError (1); recordDropped, recordSubmitted, recordFailed have none	partly live
ShardMetrics (flo_shard_*)	sized by initShards (#43)	❌ shardMetrics() has 0 callers	all rows 0
StreamMetrics (flo_stream_*)	✅ registerStream ×8	❌ recordAppend / recordRead 0 callers	series appear, all values 0
QueueMetrics (flo_queue_*)	✅ registerQueue ×15	❌ recordEnqueue / recordDequeue 0 callers	queues appear, all values 0
WorkflowMetrics (flo_workflow_*)	n/a	❌ workflow/handler.zig never references metrics_registry at all	all 0
TieredLogMetrics (flo_tiered_log_*)	❌ registerTieredLog 0 callers	❌	family never appears
KVMetrics	⚠️ registerKVNamespace ×1	❌ recordGet / recordSet 0 callers	not emitted by exportPrometheus at all

Why it matters

A scrape today shows flo_stream_append_records_total{topic="orders"} 0 on a stream that is actively taking writes. That's worse than the family being absent: a dashboard or alert built on it silently reads "no traffic". It's the same shape as the fabricated ingest_rate: 0 / retention: "7d" values fixed in #29 — a number that looks real and isn't.

What to do

Call the existing record* methods from the write paths. The handlers already hold metrics_registry (and registerStream/registerQueue are already called there), so most of this is a one-line call next to the existing register:

• stream/handler.zig — recordAppend(records, bytes) on append, recordRead(...) on read/group-read; also the trim and consumer-group counters the family already defines.
• queue/handler.zig — recordEnqueue / recordDequeue / ack / nack / DLQ / lease counters.
• workflow/handler.zig — wire metrics_registry in (it has none today), then recordStarted / recordCompleted / recordFailed / recordSignalDelivered / recordStepExecuted.
• node/shard.zig — take registry.shardMetrics(self.id) once at wire-up and record commands / bytes / connections / reactor loops / inbox depth per shard. Note the shard already records the global server.* counters, so this is about attribution, not new data.
• processing/handler.zig — fill in recordSubmitted / recordFailed / recordDropped.
• Decide whether KVMetrics and TieredLogMetrics should be exported at all; if yes, add them to exportPrometheus and call registerTieredLog. If not, delete them rather than leaving dead structs.

Acceptance

Assert real values, not presence. E.g. append 3 records, scrape, assert flo_stream_append_records_total{...} 3 — a test that only checks the metric name appears would pass today and is exactly what let this hide.

Until this lands, docs/deployment/clustering.mdx carries a note that these families export as 0; remove it when they're populated.

[oreofeolurin]

All families are now instrumented. Landed across #64, #67 and #69.

Family	Status
flo_stream_*	append and read, counting batch contents rather than entries
flo_queue_*	enqueue, dequeue, empty-dequeue
flo_shard_*	shardMetrics() had zero callers; now resolved once at wire-up
flo_workflow_*	started, completed, failed, cancelled, timed out, signals, steps
flo_processing_*	submitted, cancelled, stopped
flo_kv_*	get, set, delete — and newly emitted, since exportPrometheus had no KV branch
flo_tiered_log_*	hot/warm/miss; registerTieredLog had zero callers, so the family never appeared

The acceptance criterion is met: every family has an e2e test asserting a value, parsed out of the exposition line rather than substring-matched, and each one was verified to fail with the instrumentation stashed. That check mattered — the issue's own warning about presence-only tests is what prompted it.

Two decisions worth recording:

KVMetrics and TieredLogMetrics were exported, not deleted. Per-namespace operation rates and hot/warm hit rates are what you would actually use to size a hot buffer or spot a workload change; deleting them would have removed the observability rather than the dead code.

flo_tiered_log_cold_hits_total is permanently 0 and is documented as such (floruntime/docs#11). The read path consults the hot ring and warm store only, never cold. A zero that is explained beats one that reads as a broken counter.

Three things found along the way, all now closed or stated:

• decrementKeyCount was an unguarded fetchSub, and key_count is not rebuilt on recovery — so deleting any key that survived a restart wrapped the gauge to u64 max. Fixed with a saturating CAS and a regression test.
• Wiring the workflow counters surfaced eight unit tests aborting on a Shard = undefined test helper whose metrics_registry was never assigned. Latent, not introduced.
• get_ops_total undercounts: the blocking-get and in-transaction read paths return before the record. bytes_stored only grows, since nothing subtracts on delete or overwrite. Both are noted on feat(metrics): export KV counters and populate the tiered-log family (#44) #69 as follow-ups rather than silently shipped.

The stale docs caveat this issue asked about is updated in floruntime/docs#11 — the "exports as 0" half is gone, the still-true half about unimplemented flo_cluster_* names stays.