stats pipeline: structured request logs across services - #1463
Conversation
http_method and route are the bounded labels the instruments carry, so a dashboard-to-logs pivot is a copy-paste; the raw target rides in its own field, JSON-escaped — it is attacker-controlled, and an unescaped quote or control byte would let a crafted URI forge its own log entry. duration moves to microseconds to match the histogram. One file, four services: portrait, golf_hub, one_d4_v2, iili (#1459).
The rust services logged no requests at all — TraceLayer's events sit below the INFO default. One tracing event per request now carries the metrics vocabulary (http_method, route from the matched template or the shared sentinel, service_name) plus target, status, duration_us, trace_id and the raw x-forwarded-for, layered beside the metrics middleware so 429s appear too. init_logging installs the flattened-JSON subscriber in posterize, mithril and microgpt_serve, replacing bare fmt::init (#1459).
The C++ and Rust access lines must spell service_name, http_method, route, target, status, duration_us, trace_id and x_forwarded_for identically, or a cross-service log query silently misses a rail. Spelling pinned here, wire shape pinned per rail behaviorally — the same division the sentinel test uses. Java is deliberately absent: its services emit JSON app logs but no access line; caddy fronts them.
The oci image structure tests and the analysis-time rules_test guards were implicitly medium — each finishes in under a second. Set once in the generating macros, which covers all fourteen image tests and every rules_test fixture at their one declaration site.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
1d4-web | 42e28b9 | Commit Preview URL Branch Preview URL |
Sep 01 2026, 02:48 AM |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
iili | 42e28b9 | Commit Preview URL Branch Preview URL |
Sep 01 2026, 02:49 AM |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
ast-lol | 42e28b9 | Commit Preview URL Branch Preview URL |
Sep 01 2026, 02:50 AM |
middleware.h and the aura README still described the deleted key=value format; the service_name comment claimed a shared source the C++ rail does not have; the rust x_forwarded_for note pointed at a derived-client key this rail's governor does not use; two smaller references corrected.
The rust subscriber regressed RUST_LOG — the free fmt::init parses it into a Targets filter even without env-filter, so init_logging now does the same. aura's escaper replaces invalid UTF-8 with U+FFFD (a stray high byte is legal in a Beast request-target and would make the record unparseable) and caps the caller-controlled fields under absl's 15000- byte truncation. The discriminator key becomes event: docker's json-file envelope already owns 'log'. Access lines are now parsed by real JSON parsers in tests on both rails, service_name is asserted by value, the cross-rail pin matches emitted keys rather than identifiers, and the logback test pins the raw-template-plus-arguments message semantic the query stats will rely on.
The s3 client moves out of log_shipper into libs/s3lite and grows signed GETs and ListObjectsV2 (query-string canonicalization included, self-consistency-tested like the PUT path). The stats service lists the shipped objects, streams each once, and rolls lines up into bounded keys — vhost, status, nine-verb method, and a four-class user-agent vocabulary with AI scrapers split out — plus per-slug iili redirect counts. Rollup and processed marker commit in one transaction, so a crash re-processes and a duplicate no-ops. Served at /stats/v1/* via mucks; postgres via pgx, the store test gated on STATS_TEST_DB_URL.
stats_db_init provisions the role and database the way golf_hub's init does; the stats service joins the same profile as log_shipper — both halves need the S3 credentials in ~/.env, so a default up -d starts neither. Caddy routes GET api.muchq.com/stats/v1/* to it, pinned in the public-routes guard, GET-only on purpose.
COMPOSE_PROFILES=stats in ~/.env is what keeps the stats trio inside every unflagged deploy; a one-off --profile up starts them once and then silently stops being updated. Written down with the other ~/.env variables so a host rebuild finds it.
aaylward
left a comment
aaylward
left a comment
There was a problem hiding this comment.
Verdict
Comment (would Approve — 0 blocking, 1 non-blocking; cannot self-approve on GitHub)
This is a well-structured pipeline PR: structured access logs land consistently across C++/Rust/Java, the cross-rail vocabulary is contract-tested, and the stats aggregator is idempotent, bounded, and covered at unit, integration, and deploy-config levels. The review-panel hardening (JSON escaping, field caps, RUST_LOG restoration, real JSON parsers in tests) addresses the real risks. Safe to merge.
What Was Done Well
- Cross-rail contract discipline.
otel_contract/request_log_fields_test.gopins the shared field spelling without over-scoping (Java deliberately absent;response_bytescorrectly excluded from the cross-rail list). The division between "spelling pinned here, wire shape pinned per rail" is clean. - Security on attacker-controlled fields. Aura's
AppendJsonEscaped(control bytes, quotes, invalid UTF-8 → U+FFFD) plus 2KB/256B caps ontarget/x_forwarded_fordirectly address log-forgery risk. Rust mirrors the bounded-vocabulary pattern for metrics labels. - RUST_LOG regression caught and fixed.
init_loggingrestoringTargetsparsing is the right call — dropping it would have silently broken operator debugging. - Stats idempotency model.
ApplyRollupwrapping marker insert + upserts in one transaction, withON CONFLICT DO NOTHINGon the marker for concurrent/duplicate passes, is the correct crash-safe shape.store_test.goproves double-apply doesn't double-count. - s3lite growth is tested. GET/List pagination and query-string canonicalization have self-consistency signature tests matching the PUT path's bar.
- Deploy guards. Profile-gating the stats trio with
TestTheStatsPairIsProfileGatedTogetherand pinning the Caddy route inpublicRoutescloses the "half the pipeline starts and crash-loops" footgun.
Non-Blocking Issues
domains/platform/libs/s3lite/s3.go:22— the package comment still says "exactly one call: a header-signed PutObject"; worth updating now that GET/List exist, so the next reader doesn't assume the client is write-only.
Test Coverage
Adequate for merge. Highlights: aura and server_pal parse real JSON in behavioral tests; LogbackConfigTest pins the parameterized-message semantic the stats pipeline will query; stats has pure-unit tests for classify/aggregate/api/loop plus PG-gated store_test. Deploy config tests cover the new Caddy route and compose profile wiring.
Blast Radius
Wide. Every C++ aura service, every server_pal Rust binary, and shared Java logback output changes format. The stats service adds a new public read API (api.muchq.com/stats/v1/*) and a background S3→Postgres loop. s3lite moves to a shared lib consumed by log_shipper and stats. All intentional and mostly backward-compatible for operators (logs become more parseable, not less), but log pipelines and dashboards querying the old key=value or plain-text formats will need updating.
TCO
Moderate increase, well-contained. The stats loop is a single process with bounded aggregation keys and profile-gated deploy — no new always-on infrastructure on hosts that don't opt in. The main ongoing cost is maintaining the AI-scraper UA list (classify.go) and ensuring the cross-rail contract tests stay green when any rail touches access logging.
1 participant
Closes #1459. Closes #1460. One commit per piece: aura's access line becomes one JSON object per request in the metrics vocabulary, server_pal services get a request log and a JSON subscriber (they logged no requests at all before), the shared logback config moves to logback's own JsonEncoder, and otel_contract pins the field spelling across the C++ and Rust rails. On top of that, the stats service aggregates the shipped caddy logs into per-host/agent-class counts and iili slug popularity (s3 client moved to libs/s3lite and grew signed GET/List), deploy wiring puts the stats pair behind the same profile as the shipper, and muchq/muchq.github.io#288 adds the dashboard tab. Review-panel fixes are folded in as their own commits.
Tested with
bazel test //...and cargo test.