Skip to content

[finding] the generated shard-timings dataset has no scheduled refresh, and its documented refresh path is unreachable from an agent container #16222

Description

@claude

scripts/test-shard-timings.json is GENERATED and nothing refreshes it on a
schedule
. Its own provenance.refresh names two manual paths, and both are
weaker than they read:

  1. "Download all six test-core-run-summary-N-of-6 artifacts from any green
    queue build."
    — measured from an agent container while working CI: the shard-timings file is stale for the CLI package — 672s predicted vs 28m46s measured against a 30-minute timeout, so Test Core shard 1/6 is one slow run from being killed on any PR touching the CLI #16173:
    GET /repos/.../actions/artifacts/{id}/zip redirects to
    productionresultssa11.blob.core.windows.net, and the session egress policy
    answers 403 to CONNECT. The artifacts existed and were unexpired; the
    channel is what is missing. Job logs are reachable through the MCP
    GitHub tooling, which is how the numbers on CI: the shard-timings file is stale for the CLI package — 672s predicted vs 28m46s measured against a 30-minute timeout, so Test Core shard 1/6 is one slow run from being killed on any PR touching the CLI #16173 were obtained, but the
    summary JSON the generator consumes is not printed to a log — only its path
    on the runner is.
  2. "Or a local pnpm exec turbo run test --concurrency=4 --summarize." — a
    whole-workspace test run. The agent containers cap a foreground command at
    about ten minutes and the shared heavy-verify lock budget is 540s, and
    @objectstack/cli alone now measures 1231.52s. Not reachable either.

So the refresh is, in practice, maintainer-only and manual, for a file whose
entire design premise is that regenerating it is cheap. It was last measured
2026-08-24; thirteen days later one entry was wrong by 2.69x and a shard was
killed by the 30-minute wall.

Why this is separable from #16173

#16173 gets detection — a per-shard predicted-vs-measured drift check that
reds. Detection tells you the file has rotted; it does not un-rot it, and the
next person to see that red still has to reach a channel that is closed to
agent seats. This card is the prevention half of the question #16173 itself
poses: "Ask what makes it go stale and whether anything refreshes it on a
schedule."
#16173 does not answer it and is not expected to.

The shape that looks cheapest, offered as a lead and not a ruling

The download only exists because the regeneration happens somewhere the
summaries are not. Inside a merge_group job the summaries are already on
disk
.turbo/runs/ in the workspace, which is exactly what the existing
upload step publishes. A rollup leg (or a scheduled workflow) that collects the
six shards' summaries in-runner and runs the existing generator needs no
artifact download at all, and could open a PR with the regenerated dataset.

⚠️ Whoever takes this should expect the regenerated file to red
partition-test-shards.mjs --self-test's balance pins today — measured on
#16173, substituting the true CLI weight puts the heaviest single package at
1.54x the mean against a 1.3x bound. That is those pins working as designed, and
it is a decision sitting on #16173, not a defect in the refresh path. Sequencing
matters more than usual here.

Refs: #16173 (the stale-dataset bill, and where these numbers were measured).


Generated by Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions