stats pipeline: named agents and scanner probe families - #1466
Merged
Conversation
The agent rollup keys each row by class plus a bounded name — the marker for AI scrapers and named bots, the UA's first product token (max 32 bytes) for the rest — and the probe rollup counts requests whose path matches a scanner family. Served at /stats/v1/agents and /stats/v1/probes. The store records a rollup version and re-aggregates from S3 when it changes, so the new tables fill from objects already processed. The store integration test was skipping in CI: nothing set STATS_TEST_DB_URL and the target did not inherit it.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
1d4-web | 6d273ec | Commit Preview URL Branch Preview URL |
Sep 01 2026, 10:22 PM |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
ast-lol | 6d273ec | Commit Preview URL Branch Preview URL |
Sep 01 2026, 10:23 PM |
The agent name lives on request_stats rather than in a second table that duplicated its dimensions; the store drops and recreates the aggregate tables on a version change, and the recorded version hashes the DDL so a column edit re-aggregates without a constant bump. Each object caps the anonymous agent tail at 500 names, and /stats/v1/agents takes a limit, busiest rows first. Classifier fixes: telegrambot before twitterbot, browser-shaped generic bots named by the marker they tripped, backup probes only at the root (Forgejo archives), and iili slugs only on the two routes that reach iili. AgentClassOf and the empty-UA branch were dead.
This was referenced Sep 1, 2026
Merged
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
iili | 6d273ec | Commit Preview URL Branch Preview URL |
Sep 01 2026, 10:22 PM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1365, the stats-table half of #1458.
Each
request_statsrow now carries a bounded agent name (the marker for AI scrapers and named bots, the UA's product token for the anonymous tail, capped at 500 distinct per object), and a newprobe_statscounts requests whose path matches one of eleven scanner families. Served atGET /stats/v1/agentsandGET /stats/v1/probes. The store records a rollup version (constant plus a DDL hash) and drops and recreates the aggregate tables when it changes, so the next pass recomputes from S3.Also fixes the store integration test silently skipping in CI since #1463 (no
STATS_TEST_DB_URL, noenv_inherit), and scopes iili slug counting to the two routes that actually reach iili.Review panel ran against the first push; the second commit is its findings. Not acted on:
servedovercounts on vhosts without a catch-all 404, which needs a Caddyfile restructure (follow-up issue). Dashboard side is muchq/muchq.github.io.Tested with
bazel test //domains/platform/apis/stats/...against a localpostgres:18; the rest of//...rides on CI.