Skip to content

stats pipeline: geo breakdown of scraper, bot, and probe traffic #1467

Description

@aaylward

part of #1365, follow-on to #1458 / #1466. the stats page should show where scraper, bot, and probe traffic comes from — a country column on the per-host accordion and a top-level "junk traffic by country" table.

nothing in the repo maps an address to a country today. options:

  • ship a free ip-to-country db with the stats image (db-ip lite is cc-by; geolite2 needs a maxmind key) and resolve request.client_ip in the aggregator, adding a bounded country column to request_stats and probe_stats behind a RollupVersion bump
  • or resolve at query time in an ad hoc duckdb job over the s3 partitions and only promote the result if it earns a table

either way the key is a two-letter code, so it stays bounded. ip-range clusters (/24) are the same question at a finer grain and could ride along.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions