Skip to content

fix(o11y): record output request + HEC ACK-poll latency in ms (PIPE-1404) - #292

Open
Dylan-M wants to merge 1 commit into
dylanmyers/pipe-1066-lifecycle-metricsfrom
dylanmyers/pipe-1404-output-latency-ms-buckets
Open

fix(o11y): record output request + HEC ACK-poll latency in ms (PIPE-1404)#292
Dylan-M wants to merge 1 commit into
dylanmyers/pipe-1066-lifecycle-metricsfrom
dylanmyers/pipe-1404-output-latency-ms-buckets

Conversation

@Dylan-M

@Dylan-M Dylan-M commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Proposed Change

blitz.output.request_latency and blitz.output.hec.ack_poll_latency were declared in seconds and recorded via .Seconds(). Both use OTel's default histogram buckets (0, 5, 10, 25, ... 10000), where the first boundary is 5. Output latencies almost always run well under 5 seconds, so every sample landed in the first bucket. The histograms carried no distribution signal.

Both now record in milliseconds, where the default buckets fit. A new output.DurationMillis helper converts a duration to fractional milliseconds. It stays float so a sub-millisecond sample is not truncated to zero. Truncating would also understate the histogram sum. The request-latency feeders (tcp, otlp_grpc, hec) and the HEC ACK-poll site record through it. Both metric.yaml units change from s to ms.

Stacked on #291.

How the reviewer can validate

Note: the manual validation procedure below is an AI-generated script. Review it before running.

Manual:

  1. make build
  2. Start a throwaway TCP sink: nc -lk 127.0.0.1 5140 >/dev/null &
  3. Run blitz to it with the self-metrics endpoint on:
    ./blitz --generator-type apache-common --generator-apache-common-rate 50ms \
      --output-type tcp --output-tcp-host 127.0.0.1 --output-tcp-port 5140 \
      --metrics-port 9100
    
  4. curl -s localhost:9100/metrics | grep request_latency
  5. Confirm the histogram renders in milliseconds: blitz_blitz_output_request_latency_milliseconds_* (previously ..._seconds_*). On a localhost sink, sub-5ms sends correctly sit in the first bucket. Any latency above 5ms now lands in a higher bucket, where in seconds it would have collapsed into the first.

The HEC ACK-poll histogram (blitz_blitz_output_hec_ack_poll_latency_milliseconds_*) validates the same way against a HEC endpoint with ack enabled.

Automated: go test ./output/.... TestDurationMillis covers the conversion including sub-millisecond preservation. Manual-reader tests assert the ms unit on both blitz.output.request_latency and blitz.output.hec.ack_poll_latency.

Checklist
  • Changes are tested
  • CI has passed

@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1404-output-latency-ms-buckets branch from b24bf34 to 792856e Compare August 14, 2026 19:22
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1066-lifecycle-metrics branch from be2b515 to c52f955 Compare August 14, 2026 19:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant