[finding] measure-test-shard-timings: a file-sharded package bypasses the median merge rule — one run wins, and slices from different runs are summed as one measurement

[claude]

scripts/measure-test-shard-timings.mjs merges repeated measurements of a package by median across summaries — that is its stated rule, it is spelled in provenance.mergeRule, and it is why feeding several runs is supposed to be more trustworthy than feeding one. For a file-sharded package (today: @objectstack/cli, the heaviest suite in the workspace) the median rule does not apply. That package gets exactly one sample no matter how many runs are fed, and the sample is whichever run was read last.

Where

buildDataset in scripts/measure-test-shard-timings.mjs. Slices are held back in a ledger keyed name -> sliceCount -> sliceIndex -> seconds, and the innermost value is overwritten by each successive summary that carries that index. The completed set is then summed and pushed into the median pool as ONE sample. Nothing keys the ledger by the run a summary came from, and a run summary carries no run identifier for it to key on.

Measured, with a repro

Driving the exported buildDataset directly with three runs' worth of slices for one sliced package, plus an unsliced control fed the same three values:

run A slices 100 + 100 = 200s
run B slices 300 + 300 = 600s
run C slices 500 + 500 = 1000s

sliced package    = 1000     (median across the three runs would be 600)
unsliced control  = 300      (median across the same three values, correct)

The control proves the median rule is alive and that the sliced path alone bypasses it.

The sharper half: a cross-run splice, unreported

The ledger's indices are overwritten independently, so a set completed from different runs is summed as though it were one measurement:

run B slices 1/2 = 300, 2/2 = 300
run C slice  1/2 = 500          (run C's 2/2 not fed, or fed earlier)

recorded whole-package weight = 800s
skippedIncompleteSlices       = []      (empty — nothing warns)

800s is a duration no run observed. The file's own skippedIncompleteSlices guard does not fire, because from the ledger's point of view the set is complete — it just is not from one run.

Why this matters more than it looks

The script's header already names this exact hazard for the other direction ("a wrong number that reads exactly like a right one... n times too small") and refuses cache hits and incomplete slice sets to prevent it. The same hazard survives on the axis of which run, undefended, on the one package the slicing machinery exists for and the one whose mis-weighting killed a shard twelve times in a day.

It is also live for the current refresh procedure rather than hypothetical: the refresh ruling on #16173 directs feeding eighteen summaries from three fully green runs. Under that plan every ordinary package is a median of three observations and @objectstack/cli is a single observation from whichever run sorts last — the least robust reading for the package that most needs a robust one, with no line of output saying so.

Not a duplicate

Checked against the open cards on this dataset: #16173 (the stale dataset and its refresh), #16222 (the refresh path is unreachable from an agent container), #16464 (a scheduled refresh workflow — it feeds ONE run's six artifacts, so this defect is outside its completion scope), #16465 (the drift alarm). None of them touches the merge rule. Channel note: the repo-scoped REST search path answers 403 for this session ("sessions are bound to their configured repositories"), so the dedup sweep was one targeted MCP search_issues over this repo's open issues.

Shapes, not a ruling

• A. Key the ledger per input group. Treat each summary as belonging to a run and complete slice sets within a run, so a sliced package contributes one sample per run to the median pool. Needs a grouping key the summaries do not carry today; the caller knows it (the six files of one run are named and fetched together), so it could be passed in rather than inferred.
• B. Refuse the ambiguity. If more than one value is ever seen for the same package, slice count and slice index, fail loudly instead of taking the last. Cheapest, keeps the file honest, and turns a silent wrong number into a named refusal — at the cost of making multi-run refreshes of a sliced package impossible until A lands.
• C. Report it. Keep last-wins but name the sliced package in the generator's stderr summary and in the dataset, the way skippedIncompleteSlices already names its case. Weakest: it documents the defect rather than removing it.

Filed unassigned by the domain:devx execution seat while working #16173; #16173 is not addressed by this card and remains open.

---

Generated by Claude Code

[claude]

Claim: PM loop round 9 — flight C2a, maintainer-directed (「同意你的建议，你负责执行派发所有可行的优化」, 2026-09-07): the scheduled shard-timings refresh workflow, with #16473 folded in as its prerequisite — flight C1 (#16173) measured at 03:2xZ that the artifact host (productionresultssa*.blob.core.windows.net) is denied to every agent container by egress policy, so the documented refresh cannot run from a dev worktree; the refresh therefore runs IN-RUNNER, where .turbo/runs/*.json already sits, and this workflow's first workflow_dispatch is the refresh itself — the PR it opens is #16173's deliverable. Rider #16473: buildDataset in scripts/measure-test-shard-timings.mjs bypasses its median rule for a file-sharded package (last run wins; slices from different runs summed into a weight no run measured) — fixed first, with self-test cases, so the refresh's @objectstack/cli weight is a median like every other package's.
Session: session_019RfFHiRCSs3JXLK4cwcfox
Branch: claude/issue-16464-shard-timings-refresh-workflow
Worktree: objectstack-issue-16464
Domain: domain:devx (dispatched by the skills seat on the maintainer's direction)
File surface: .github/workflows/shard-timings-refresh.yml (new), scripts/measure-test-shard-timings.mjs (the slice ledger's merge rule + self-test), one helper under scripts/ci/ if the pin update needs one; scripts/test-shard-timings.json is written only by the workflow run, never by hand. ⛔ ci.yml untouched; no timeout-minutes, matrix or bound change.
Container & model: S, mode:subagent, opus — dispatch-gates.mjs --tier: no path-derived mandate.
Clause-②: no
Serial: no open PR touches the measure script or the JSON (C1 produced no PR); flights V and X hold ci.yml, which this card does not touch. Not governed ⇒ in-seat review, then the seat flips ready and arms SQUASH auto-merge; skip-changeset. The PR carries Fixes #16464 and Fixes #16473; #16173 and #16222 are Refs (they close when the workflow's first refresh PR merges).

---

Generated by Claude Code

[claude]

Implemented as the prerequisite rider of flight C2a, on branch claude/issue-16464-shard-timings-refresh-workflow. Draft PR: #16498 (it carries the closing keyword for this card). The full os-dev-report for both cards is #16464 (comment); this is the pointer, so the fix is discoverable from the card that describes it.

Shape taken: A, with B folded in as the guard that makes A's default safe. The card offered three. A alone (key the ledger per input group) leaves the ungrouped call resolving multiple runs by last-wins exactly as before; B alone (refuse any repeat) makes a multi-run refresh of a sliced package impossible, which the card names as its cost. Together they compose without either cost: slices are summed within a run, the per-run sums are medianed across runs, and a repeat within one declared group — which can only mean two runs sharing a group — is a named refusal with --run given as the remedy. C was not taken: it documents the defect rather than removing it, and the dataset it would annotate is machine-generated input to a scheduling decision, not something a human reads before each split.

The grouping key comes from the caller, as the card's shape A anticipated, because a run summary carries no run identifier to infer one from. --run opens a group; summaries before any --run share one implicit group, which is the ordinary single-run refresh the scheduled workflow performs.

Measured, red then green. The red leg reproduces flight C1's scratch driver of the exported buildDataset, and the numbers are the card's own:

                                       BEFORE   AFTER   correct
three runs measuring 400/600/1000s      1000      600     600
unsliced control (100/300/500)            300      300     300
run B 1/2 + B 2/2 + run C 1/2             800      600     600
  skippedIncompleteSlices                  []   named      named

The AFTER splice row reads cli in run C (missing 2/2) — the partial set is attributed to the run that could not assemble it, rather than silently completed from another run's slice.

Also corrected: provenance.mergeRule said median across summaries, which was only ever true of unsliced packages, since a sliced package's summaries are PARTS of one measurement rather than repeats of it. It now states both composed rules, and provenance.runs records the declared ids.

Seven self-test cases, battery floor 34 to 41, and the whole path exercised through the real CLI on synthetic summaries as well as through buildDataset directly.

Load-bearing for #16464 rather than incidental: the refresh workflow feeds one run today, but the moment anyone feeds more than one — which the #16173 refresh recipe explicitly directs — the heaviest package in the workspace would have taken a single observation while every other package took a median, with no line of output saying so.

---

Generated by Claude Code

[claude]

Pointer correction: the report link in my comment above now points at a superseded report. The current one is #16464 (comment) (status needs_decision).

⚠️ The change of status is about the WORKFLOW card only, and nothing about this card moved. This rider is fully repaired, self-tested and proven live, and PR #16498 still binds its closing keyword to this card alone. If anything, the live runs strengthened it: the workflow now has to feed SEVEN runs to the generator to cover the workspace, so the per-run assembly this card added is not a hygiene fix on a path nobody walks — it is the only reason that path is correct. Across those seven runs the generator raised no ambiguity refusal, which is the behaviour this card specified.

---

Generated by Claude Code

[claude]

Final pointer: the current report is #16464 (comment) (status done), superseding the two before it. PR #16498 binds this card's closing keyword and is green on head 94eeaa19b1.

Nothing about this card changed across the three reports — it was repaired, self-tested and proven live in the first round. What changed is how load-bearing it turned out to be: the refresh lane was ruled to merge rather than replace, so it now feeds the generator several runs' summaries under per-run --run groups as a matter of course, and the live green run did exactly that with no ambiguity refusal. The per-run assembly this card added is not hygiene on a path nobody walks; it is the reason that path is correct.

---

Generated by Claude Code