stats pipeline: bot, attack, and AI-scraper traffic analysis

[aaylward]

depends on #1457. the caddy access log already contains the interesting negative space: 403s from the git.muchq.com bot blocks (#1447), 404 probes, and AI scraper user agents (GPTBot, ClaudeBot, meta-externalagent, Bytespider, PerplexityBot, ...).

questions to answer:

• who's crawling: UA and ip-range clusters, request rates, which vhosts and routes they hit
• what share of total traffic is bots vs humans
• ai scrapers specifically: which ones back off after a 403, what they fetch, volume over time
• attack shapes: common probe paths, geo distribution of junk traffic

output can be duckdb/spark queries or a notebook at first; promote anything worth watching continuously into the stats tables (#1460), and feed findings back into caddy block rules. ip clustering is also a decent graph problem if the memgraph experiment (#1461) happens.

[aaylward]

first slice is #1466 (tables + api) and muchq/muchq.github.io#289 (the /stats page with the per-host accordion). what it answers continuously:

• who's crawling, by name: request_stats.agent — marker for ai scrapers and named bots, product token for the anonymous tail (capped at 500 distinct per log object), per host per day, with the 403 count so "did it back off" is one query
• bots vs humans share: unchanged summary endpoint, now on the same page
• common probe paths: probe_stats with eleven scanner families, per host, with a served count — note the served column overcounts on api.muchq.com / gpt.muchq.com until caddy: catch-all 404 on api.muchq.com and gpt.muchq.com #1468

still ad hoc over the s3 partitions: ip-range clusters and geo (#1467). nothing fed back into caddy block rules yet; the busiest-agents table is where those candidates will show up.

[aaylward]

first finding fed back into caddy: #1473 refuses TLM-Audit-Scanner (the busiest bot on the new stats page, ~9k requests/day across five vhosts, credential-file probing per mitchellkrogza/apache-ultimate-bad-bot-blocker#235) by UA and by its two source addresses, via a refuse_scanners snippet every site imports. future blocks go in the same snippet.

[aaylward]

Named agents, probe families, per-host breakdown, geo panel, and the caddy block rules are all landed. The exploratory pass over the raw logs is #1482.