ci: regenerate the shard-timings dataset on a timer, and assemble a sliced package per run

[claude]

Fixes #16464
Fixes #16473
Refs #16173
Refs #16222

Puts a clock on the Test Core balancing input, and fixes the generator defect that would have made the very first refresh untrustworthy for the one package it matters most for.

scripts/test-shard-timings.json is the per-package duration dataset the Test Core shard split is binned from. It is generated, but until now only when somebody remembered: it went 13 days without a refresh while ~700 test files were added, and the shard it mis-weighted was killed by the job wall twelve times in one day and ejected from the merge queue twice. The rot is one-directional and silent — suites only get slower, the table stays put, and a shard killed by the wall produces no measurement at all while the rollup reads green — which is why the fix is a timer rather than more discipline.

1. The prerequisite: a sliced package was bypassing the median rule (#16473)

buildDataset's slice ledger was keyed (package, slice count, slice index) with no notion of which run a summary came from, so every run collapsed into one entry and both merge rules broke at once, silently. Reproduced by driving the exported buildDataset directly, which is how flight C1 proved it:

                                       BEFORE   AFTER   correct
three runs measuring 400/600/1000s      1000      600     600   (the median)
unsliced control fed 100/300/500s         300      300     300   (control: the rule was alive)
run B 1/2 + B 2/2 + run C 1/2             800      600     600   (B's own set; 800 is a splice)
  and its skippedIncompleteSlices          []   named      named

The first row is last-wins: the package took whichever run was read last, no matter how many were fed. Every other package in the same refresh got its median, so the heaviest suite in the workspace — the one the slicing machinery exists for — was the single least robust reading in the file, with no line of output saying so. The second row is the control that proves the median rule was alive and the sliced path alone was bypassing it.

The third row is the sharper half: a set completed from slices of different runs was summed as one measurement, producing a duration no run observed, while skippedIncompleteSlices stayed empty — from that ledger's point of view the set was complete, it just was not from one run.

Both are this file's own signature hazard — a wrong number that reads exactly like a right one — surviving on the axis of which run.

The fix. The ledger is keyed by run first. Slices are summed within a run; the per-run sums are medianed across runs, exactly as an unsliced package's observations are; and a run that cannot assemble every slice contributes no sample and is named with its run.

The grouping key comes from the caller, because a run summary carries no run id to infer one from — the six artifacts of one CI run are named and fetched together, so the caller is the only actor that knows. --run opens a group; summaries before any --run share one implicit group, which is the ordinary single-run refresh. Feeding several runs without declaring them is refused by name, with --run given as the remedy, rather than resolved by taking the last value. That refusal is what makes the convenient default safe.

provenance.mergeRule said median across summaries, which was only ever true of unsliced packages; it now states the two composed rules, and provenance.runs records the declared ids.

Sixteen self-test cases across the two changes it drives (seven for the per-run rule, nine for the merge below), floor 34 to 50. Exercised end to end through the real CLI as well, on synthetic summaries: declared three runs give cli = 600 / control 300; the same six files with no --run exit 1 naming the slice and the flag; one run with no --run gives 600 with no ceremony.

2. The workflow

.github/workflows/shard-timings-refresh.yml — weekly (Monday 05:30Z) plus workflow_dispatch. It selects runs, downloads their test-core-run-summary-N-of-6 artifacts, re-runs the generator, and opens a PR only when the file changed by byte comparison.

Which path it took, and why. It downloads the artifacts, per ruling 2. The alternative — generating from the runner's own fresh summaries — is not available to a separate lane: a single job would have to run the entire workspace's suite unsharded to produce a complete summary set, which is the "special measurement run" this tooling exists to avoid. Downloading is safe here and unreachable elsewhere: GET /actions/artifacts/{id}/zip redirects to productionresultssa*.blob.core.windows.net, which every agent container's egress policy refuses (403 on CONNECT, reproduced four ways across two days and two containers — #16222, then again on #16173), while a GitHub-hosted runner reaches that host natively. Running here is therefore not a workaround for the denial; it removes the channel from the loop.

Choosing the run is the hard part, so it is a tested unit — scripts/ci/select-shard-timings-run.mjs, 30 self-test cases. "The newest run" is wrong three ways, all measured on this repo:

• Cancelled runs. cancel-in-progress censors most push runs on main — 36 of the last 60. Such a run still leaves artifacts, so "the artifacts exist" is not the test; all six Test Core (N/6) jobs must have concluded success.
• Cache replays. A run whose six shards all concluded success can still have replayed most of the workspace: shards were measured at 7s and 98s against a 672s prediction. The generator refuses cached tasks, so those packages simply vanish from the dataset — and an absent package reds nothing, it falls back to the test-file-count estimate. This is judged on what was measured: a package the committed dataset measured, that the workspace still contains, and that the refresh does not, is named and the run is rejected. No duration threshold is invented, and a package genuinely deleted from the monorepo does not block future refreshes because it is gone from turbo ls too.
• Expired artifacts. Retention is one day, so an empty candidate list is a normal quiet Monday — reported as a named refusal, never as a silent no-op.

Runs are accumulated rather than searched: each contributes its six summaries under its own --run group, and the loop stops at the first accumulation that covers the workspace. Section 4 is why that is not a nicety.

What it is not allowed to do. It never hand-edits the dataset; every byte comes out of the generator. It never touches timeout-minutes, the shard matrix, MAX_SHARD_OVER_MEAN, MAX_MEASURED_OVER_PREDICTED or FILE_SHARDED_PACKAGES. ci.yml is untouched by this PR.

The pins are run but never acted on. partition-test-shards.mjs --self-test grades the refreshed dataset, and it reds by design on a refresh that outgrows the acceptance bound, naming the remedy — raise the slice count, never the bound. Its pin 3c additionally demands a decision the day the CLI comes back under the bound on its own. Both are judgement, so the workflow reports the verdict verbatim into the PR body and opens the PR either way: withholding it would lose the measurement, which is the exact silent-rot failure this lane exists to end.

3. Two gate defects this diff was the first to trip

scripts/check-self-test-wired.mjs — a missing right boundary (declared as a file-surface supplement on #16464 before it was made). Its invocation extractor had a left boundary but none on its extension alternation, and js is a prefix of json, so the first workflow line naming a .json under scripts/ matched only as far as the .js:

scripts/test-shard-timings.json   ->  keyed as  scripts/test-shard-timings.js

That is precisely the phantom key the left boundary was added to abolish, arriving from the other end, and silent in both directions — the real file is audited by nothing and the phantom reconciles against no carrier. Ruling 2 requires the workflow to name that path, so no version of this deliverable avoids it. The remedy is the guard the same pattern already applies to --self-test two groups along. Proven red-then-green: with the boundary removed and the mutation confirmed on disk, the production run stays green at exit 0 while --self-test goes red naming the new pin — the shape that file's own header documents for this gate class — and restoring returns the byte-identical blob and a green self-test. Four cases, floor 4 to 8.

A scheduled-only gate family. dispatch-gates.mjs reds when a family is reachable only on a schedule, because such a family appears on no card's gate list and is graded by nobody until the next sweep. This lane was the first to arrive that way. The remedy is the posture half-state-patrol.yml and required-set-patrol.yml already keep: a pull_request trigger scoped to this lane's own files. On such a run everything executes for real — selection, download, regeneration, coverage check, byte comparison, partitioner verdict — and only the write is skipped, so a PR editing this workflow can never push a bot branch or open a second PR. Composing the PR body is therefore its own step, consumed either by the write or by a dry-run step that renders it to the run's step summary.

That trigger also supplies this PR's reverse verification: opening it runs the lane end to end against live artifacts, in dry-run mode, on a real runner.

4. What the live runs measured, and the merge rule they forced

The pull_request trigger ran the lane against live artifacts, and it measured a fact no ruling anticipated: not one retained green run measures the workspace, and neither does the union of all of them.

what each retained green push run on main MEASURED, of the 71 packages
the committed dataset holds:

  34070845319   52 / 71        34073696737   18 / 71
  34069255842   49 / 71        34072619046   13 / 71
  34077595330   22 / 71        34074971128    3 / 71
  34079753529    2 / 71

accumulating them, newest first:
  1 run -> 2      4 runs -> 28      6 runs -> 56
  2 runs -> 22    5 runs -> 35      7 runs -> 57   <- all of them, and it stops here

14 packages are turbo cache HITs in every retained run. That is the cache design rather than luck: the key is namespaced per shard and only main pushes write it, so a package whose inputs have not changed is a HIT — and the generator refuses hits rather than recording a replayed ~0.1s window as a suite's cost. The documented procedure ("download all six from any green queue build and re-run the generator") is optimistic in a way nobody had measured; that sentence is corrected in this PR.

So a refresh MERGES rather than replaces (--merge-into, ruled on the card). A package this pass did not measure keeps its previous weight — but only when a cache HIT witnesses it, and that witness is the whole argument: a HIT is not missing data, it is positive evidence that the package's inputs are unchanged since the run whose output was replayed, so its last measured weight still describes it. The file's invariant survives intact — every number in it is a real measurement of code as it stands, never an estimate.

The witness requirement is what stops that from becoming "keep whatever was there". A package absent for any other reason — never ran, suite failed, slices unassemblable — has no evidence behind it, is not carried, and the coverage check names it as a refusal. Both directions are pinned, and both were exercised against the real dataset: two runs measuring 55 packages carried the other 16 and passed; withholding two packages entirely (no MISS, no HIT) refused by name.

Carried packages are listed in a top-level carriedOver, beside skippedAsCached. That shape rather than a per-package measuredAt because the reader forces it: partition-test-shards.mjs reads packages as name → number (timings.packages[name] / sliceCount) and never opens provenance at all, so per-package dates would mean changing that shape and every consumer of it. A single provenance.measuredAt paired with the list carries the same information and needs no partitioner change.

Coverage is judged on measured ∪ carried, and the two ways a package can be missing are reported separately, because only one is a regression: a package that had a weight and has neither is a refusal by name; a workspace package that never had one is named as partitioner-estimated but is not a regression, since this refresh did not change its standing. Nothing is silently estimated in either case.

Verification

Everything below was run on head 94eeaa19b1, so the union is a reading about the tree this PR actually proposes rather than an earlier one.

• All 49 gates that node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack derives for this four-path change set were run and reconciled with --ran. The list was derived by the script from its own merge-base change set, not hand-assembled, and the derivation reported no stale tree.
• Whole-repo pnpm lint (eslint . --no-inline-config) through the shared verify lock, slot issue-16464: LINT-EXIT=0. Lock held 90s after an 86s wait — shared-box seconds, not idle-box.
• check-governed-merges --test on the final four-path file list: NOT governed — ordinary queue landing applies to a PR with exactly this file list.
• Every exit code was captured by redirecting first and reading the status before any pipe, so no verdict here comes from a pipeline's status rather than the gate's.
• Merge-tree proof against ci: the merge queue's Test Core runs the affected package set; selection moves into a script with a self-test #16471's head 05cb57b43c, twice, both clean and to the same tree — and that PR has since landed and merged into this branch cleanly.
• Live runs of the lane itself, through its own pull_request trigger, on real artifacts — the reverse verification. The first two reached the coverage check and refused, which is what measured section 4. The run on this head is green: it covered the workspace from a single run plus the carry (RUN_COUNT: 1, run 34087495912 at c383352cb7), composed a 70-line PR body and rendered it to the step summary with no branch pushed and no PR opened. Its PARTITIONER_EXIT: 1 is the designed behaviour from section 2 arriving live: the balancing pins red on an honest refresh, and the lane reports the verdict into the body rather than acting on it.

Two gate reds during development were fixed rather than worked around, and both are described above: check-step-collectors (a bare self-test sequence under bash -e) and check-self-test-wired --self-test plus check:pm-dispatch-gates (the two defects in section 3).

The dataset itself is not touched by this PR. scripts/test-shard-timings.json is written only by a workflow run, never by hand — which is the whole point.

Two notes for whoever lands this

workflow_dispatch on this branch (ruling 4's literal form) is not available to this seat: POST /actions/workflows/shard-timings-refresh.yml/dispatches answers 403 Resource not accessible by integration, so the session credential carries no Actions write scope. The pull_request trigger added in section 3 supplied the reverse verification instead, and did so better than a dispatch would have — it ran the lane end to end against live artifacts on a real runner, and it is what surfaced section 4's finding. Its result is on this PR's own checks.

The refresh PRs this lane opens carry only Refs, never a closing keyword. A weekly lane cannot know which cards a given run ought to retire, and a keyword baked into it would have every subsequent Monday try to re-retire cards already retired. Cards #16173 and #16222 therefore stay open when the first refresh PR merges — whoever lands that first refresh should retire them by hand, or add the keyword to that one PR.

⛔ This PR is left in draft deliberately. Do not flip it ready, enqueue it, or arm auto-merge from an agent seat.

---

Generated by Claude Code