feat(tasks): discriminating screening roster — W4b/W3/W1b/W6 from verified commits - #14
Merged
Conversation
…screening roster Batch 3 hit a ceiling (27/27 controlled runs accepted, zero escalations), so the current three tasks cannot separate configurations on quality. This registers eight commit-mined candidates hard enough that acceptance is expected to vary. Sourcing per SPEC 5.1 and tasks/WORKLOAD-SELECTION.md 4: agent starts at N-1, the merged PR's own tests seal the gate. 2026-08-diversed-coding-tasks.md was used as class briefs only -- none of its fabricated SHAs or repos is carried forward. Verified via the GitHub REST API (urllib, User-Agent header; no clones, no model spend): 16/16 SHAs resolve -- 8 merge commits and their 8 parents. All eight PRs are merged squash merges, so parents[0] is unambiguously the agent start point. All eight repos are permissively licensed and all eight gate plans are hermetic (deps installed at image build; gate runs --network=none). Coverage: complex_bugfix x4 across three repos (sqlglot, anyio, zarr), migration x2 (sqlfluff, opentelemetry-python), test_generation x1 (zarr), W6 review-target proposal x1 (hono). C5 recommended as the SPEC 5.1 escalation probe. Nothing scaffolded, nothing pinned in the manifest, nothing runnable. Awaiting human selection. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 10-point validator and the deterministic gate hard-coded npm/jest/Prisma,
so no Python task could be validated at all. Split the toolchain out behind a
new `task.yaml stack:` field.
- stacks/{node,python,none}.sh + stacks/README.md — drivers own install,
deps probe, baseline, selected-test run, coverage and build; the task
supplies the commands via stack_cmds.
- A stack may declare a check not_applicable WITH A REASON. validate.sh
accounts for those separately and never counts them as passes (SPEC 2.8).
- check-public.sh: pr_review branch; stack-driven diff scope and tree status;
public_test_support (files the upstream PR adds alongside its new test
module, created empty in the subject tree).
- check-hidden.sh: test_generation/pr_review invoke a sealed hidden/check.sh,
hash all of hidden/, honour exit 0/1/2.
- gate/covpy_to_summary.py: coverage.py JSON -> the gate's coverage summary
shape, the Python counterpart of the existing jest reader.
- validate.sh: gate_type dispatch; stack + gate_type in the report. A
containerized run reports check 7 not_applicable with a reason rather than
failing, because .dockerignore deliberately keeps canonical/ out of the
image (subject isolation) — check 7 is host-verified.
- Dockerfile.python: clean-container validation env for stack: python
(uv 0.12.5 pinned, HYPOTHESIS_PROFILE=ci), sibling of the node Dockerfile.
Regression evidence, host 10-point validation on the refactored harness:
pilot 10/10, W1 10/10, W4 10/10 — all unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI's shellcheck and JSON/YAML steps used a bare find and would have linted sealed hidden/ gate scripts and .work/ run artifacts. Sealed material is human-held and must never be read by a gate. Prune set now matches tests/run-tests.sh prune_find. Latent today (a clean CI checkout has no sealed files, since they are gitignored) but live on a self-hosted or resumed runner. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d commits Batch 3 accepted 27/27 controlled runs with zero escalations. At that ceiling quality cannot separate configurations and cost-per-ACCEPTED-outcome collapses to raw cost. Four commit-mined tasks, all at contamination tier post_cutoff, all with hermetic gates, all pinned to SHAs verified through the GitHub API and re-verified against real clones (parents[0] of each merge == the recorded N-1). - W4b zarr-python#4227 complex_bugfix (ordering) 9 pass / 1 awaiting / 0 fail - W3 sqlfluff#7962 migration, ESCALATION PROBE 9 pass / 1 awaiting / 0 fail - W1b zarr-python#4054 test_generation 9 pass / 1 awaiting / 0 fail - W6 hono#5171 pr_review 4 pass / 2 awaiting / 4 n/a / 0 fail Every awaiting_human is a check waiting on a sealed artifact only the human may author; every not_applicable carries a reason. Nothing is reported as a pass that was not executed. W1b and W4b are SECOND TASKS of W1 and W4, not new workloads: they ship no workload.yaml, and the parent workload names them under second_task_for_class_claim. That is what unblocks a class-level claim under WORKLOAD-SELECTION 3 (extending SPEC 5.2) — both are post_cutoff and materially different from the famous originals in language, ecosystem and task shape. W3 is the designated escalation probe (SPEC 5.1); its task.yaml header records the obligation. The failure prediction is human-registered before any run. Notable pins: - sqlfluff's requirements_dev.txt is completely unpinned, so W3 ships a task-owned env/requirements.lock.txt (81 pins) — a pinned commit with an unpinned environment is not a pinned task. - W1b runs every command under HYPOTHESIS_PROFILE=ci (zarr's derandomize=True profile). A coverage threshold over randomised strategies is not a gate. Its 30% branch threshold sits inside a measured band: 26.19% for a shape-only smoke test, 34.13% for the canonical. - W6 withholds the upstream PR's own tests from the review artifact. No model API was called. No spend. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Add the four new task dirs; split TASK_DIRS into feasibility vs screening. The controlled P0/C2/P1 set is a FEASIBILITY invariant; screening tasks declare C1/C2/C3/C5 and run no companions. Both are now asserted. - Accept public_test_kind: pr_own_tests — the commit-mining default, where the public test is the upstream PR's own test file lifted verbatim, so the gate is sealed by construction (WORKLOAD-SELECTION 4). - Generalize the node-only .test.ts/.spec.ts assertions behind one _is_test_file() helper covering test_*.py and *_test.py (zarr uses the pytest default, sqlfluff sets python_files = "*_test.py"). - Handle gate_type: pr_review — no canonical_patch, no public_test. - Assert screening tasks declare task_phase: screening, and that gate_type is a known value. - hidden/ check moves from a blacklist of test suffixes to a whitelist: README-FOR-HUMAN.md is the only file any hidden/ may commit. That is what .gitignore already enforces, and unlike a suffix list it keeps covering sealed artifacts that are not test modules (check.sh, mutants/, defect-map.json, review-diff.patch). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Owner
Author
Full 8-candidate mining record (C1–C8), verbatimPasted here so the record survives terminal scrollback. Source of truth: Status: C4, C5, C7, C8 selected and scaffolded · C6 rejected · C1, C2, C3 Read the gate plan, difficulty rationale and setup estimate columns as what they Verification methodPython 16 of 16 SHAs resolved. All eight PRs merged. All eight merges are squash merges
Cross-cutting notes
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Feasibility batch 3 accepted 27/27 controlled runs with zero escalations. At that
ceiling, quality cannot separate configurations and cost-per-ACCEPTED-outcome degenerates
into raw cost. This PR adds four commit-mined screening tasks chosen to be hard enough
that acceptance is expected to vary, plus the harness work needed to run them.
No model API was called in this work. No spend; CP-SPEND not requested and not implied.
What lands
tasks/suite/W4b-zarr-consolidated-orderpost_cutofftasks/suite/W3-migrationpost_cutofftasks/suite/W1b-zarr-block-mask-propertiespost_cutofftasks/suite/W6-pr-reviewpost_cutoffEvery
awaiting_humanis check 6 and/or 7 waiting on a sealed artifact only you canauthor. Nothing is reported as a pass that was not executed, and no check is silently
skipped: W6's four
not_applicableresults each carry a reason (a review task installsnothing, compiles nothing, and runs no baseline suite).
W1bandW4bare second tasks of W1 and W4, not new workloads — they ship noworkload.yaml; the parent workload names them undersecond_task_for_class_claim.That is what unblocks a class-level claim for
test_generationandcomplex_bugfixunder
tasks/WORKLOAD-SELECTION.md§3 (extending SPEC §5.2): both arepost_cutoffand materially different from the
famousoriginals in language, ecosystem, and taskshape.
W3 is the designated escalation probe (SPEC §5.1). Its
task.yamlopens with acomment block recording the obligation; the failure prediction is yours to register in
manifest/preregistrations/before any run, and the result gets published either way.SHA re-verification at PR time
Re-ran the GitHub REST checks for all 8 candidates immediately before opening this PR:
merged == true,merge_commit_sha == N, N resolves, N−1 resolves, N has exactly oneparent, and
parents[0] == N−1.For the four scaffolded tasks this was additionally re-verified against real local
clones:
parents[0]of each merge equals the recorded N−1.Selection record
static
ruffcheck rather than the maintainers' own tests, and a competent economicaltier likely completes it.
later batch. C3 is worth keeping in particular: a genuine ordering/race bugfix with a
deterministic repro (
wait_all_tasks_blocked()+fail_after(3)), not a timing test.The complete 8-row table is reproduced verbatim as a PR comment below so the record
survives terminal scrollback, and lives permanently in
tasks/proposals/2026-08-commit-mined-candidates.md.Harness changes — and the scope deviation, stated plainly
Your first message set the bar "no changes outside
tasks/and the PR description".Your second required "run all ten-point checks that don't need my sealed input" — for a
Python migration task, a Python property-test task, and a TypeScript review task. Those
two cannot both hold: at the start of this work the validator hard-coded npm/jest/Prisma
primitives,
public_test_kindaccepted onlyrepro|feature, pins live inmanifest/delivery-manifest.yamlby design, andtests/test_tasks.pyenumerates taskdirs. I took the later, more specific instruction as governing and changed
harness/,manifest/,tests/and.github/as well. Every change is additive andbackward-compatible; the regression evidence is that all three pre-existing tasks still
validate at 10/10 on the refactored harness (below).
Flagging it rather than burying it — the harness commits are separable from the task
commits if you would rather split this.
What changed outside
tasks/:harness/task-tools/stacks/{node,python,none}.sh+stacks/README.md— toolchaindrivers behind a new
task.yaml stack:field. Install, dependency probe, baseline,selected-test run, coverage and build move out of the engine. A stack may declare a
check
not_applicablewith a reason; the validator reports those separately andnever counts them as passes.
harness/task-tools/gate/check-public.sh—pr_reviewbranch; stack-driven diff-scopeand tree status;
public_test_support(files the upstream PR adds alongside its newtest module, e.g. a package
__init__.py, created empty in the subject tree).harness/task-tools/gate/check-hidden.sh—test_generation/pr_reviewinvoke asealed
hidden/check.sh, hash all ofhidden/, and honour exit 0/1/2.harness/task-tools/gate/covpy_to_summary.py—coverage.pyJSON → the gate'scoverage summary shape (the Python counterpart of the existing jest reader).
harness/task-tools/validate.sh—gate_typedispatch,not_applicableaccounting,stack/gate_typein the report. Also: a containerized run now reports check 7 asnot_applicablewith a reason instead of failing, because.dockerignoredeliberately keeps
canonical/out of the image (subject isolation, FIX D) — thecontainer genuinely cannot perform that check, and check 7 is host-verified.
harness/task-tools/Dockerfile.python— clean-container validation env forstack: python(uv 0.12.5 pinned,HYPOTHESIS_PROFILE=ci), sibling of the existingnode
Dockerfile. Built and exercised — results below.manifest/delivery-manifest.yaml—w4b_task,w3_task(escalation_probe: true),w1b_task,w6_task. Each carriessealed_hidden_test: {status: awaiting_human, spec: <path>}— no invented sha256.tests/test_tasks.py— the four new dirs;TASK_DIRSsplit into feasibility vsscreening (the controlled
P0/C2/P1set is a feasibility invariant; screening tasksdeclare
C1/C2/C3/C5and run no companions — both now asserted);pr_own_testsaddedto the known
public_test_kinds; the node-only.test.ts/.spec.tsassertionsgeneralised behind one
_is_test_file()helper coveringtest_*.pyand*_test.py.The hidden-dir check moved from a blacklist of test suffixes to a whitelist —
README-FOR-HUMAN.mdis the only file anyhidden/may commit. That is what.gitignorealready enforces, and unlike a suffix list it keeps covering sealedartifacts that are not test modules (
check.sh,mutants/,defect-map.json,review-diff.patch)..github/workflows/ci.yml— CI'sshellcheckand JSON/YAML sweeps used a barefindand would have linted sealed
hidden/scripts and.work/artifacts. Now prunedto match
tests/run-tests.sh'sprune_find. Latent today (a clean CI checkout has nosealed files, since they are gitignored) but live on a self-hosted or resumed runner —
and a gate must never read sealed material.
Evidence
Quality gates
10-point validation, host
tasks/pilot-realworldtasks/suite/W1-test-generationtasks/suite/W4-complex-bugfixtasks/suite/W4b-zarr-consolidated-ordera6bc4a02e572be2b1856c36f1182bb8bdd46bb99tasks/suite/W3-migrationed87204769fb0601b6d776a3ca91a40939506c42tasks/suite/W1b-zarr-block-mask-propertiese4165e59c57f7ae2ede44680767f96c929339b9etasks/suite/W6-pr-reviewc98beb86cff2107a65c409694df654f08bce6449(columns: pass · awaiting_human · not_applicable · fail)
10-point validation, clean container —
Dockerfile.python, subject repo cloned freshat the pin inside the container, so check 9 reports
(container)rather than(host):W4ba6bc4a02…W3ed872047…W1be4165e59…W6c98beb86…The single
n/ain each is check 7, for the.dockerignorereason above. Everycontainer reset-tree hash is byte-identical to the host's — the pin reproduces from
scratch.
Measurements taken while scaffolding (all real; nothing estimated, nothing imputed):
1 failed, 2 passed in 0.50sat the pin →3 passed in 0.21swith the canonical applied.
ImportError: cannot import name 'ObjectReferenceLevel' from 'sqlfluff.core.dialects.common'(at the pin,common.pyis 24 lines holding onlyAliasInfo/ColumnAliasInfo, andtest/core/dialects/does not exist). With thecanonical applied: public test
9 passed in 9.30s; baseline part 1996 passed, 28 skipped in 39.29s(= the 987 that pass at the pin, plus the 9injected); baseline part 2
449 passed in 79.87s. Zero failures — behaviour parityconfirmed, which is the whole point of a migration gate.
HYPOTHESIS_PROFILE=ci, two consecutive runs, byte-identical:250/800 statements (31.94%), 86/252 branches (34.13%). A 10-line smoke test calling
all six interfaces once and asserting only shapes: 26.90% / 26.19%. The 30% branch
threshold sits inside that measured band with ~4 points of headroom each side — it is
measured, not chosen.
Notable scaffolding decisions
requirements_dev.txtis completely unpinned. A pinned commit with anunpinned environment is not a pinned task, so W3 ships a task-owned
env/requirements.lock.txt(81 pins fromuv pip freeze, Python 3.12.14).randomised strategies is not a gate. zarr registers a
ciprofile withderandomize=Trueintests/conftest.py; every W1b command selects it.-n 2because this box hasnproc == 2(-n 4measured2m39.782s vs 2m04.887s for
-n 2on the same tree).router.test.ts, +194) from the reviewartifact. The reviewer must judge the diff, not read the answer.
oracle-free suite can clear it. The sealed mutation runner is the authoritative signal
(SPEC §2.6), which is why W1b sits at 9/10 rather than claiming completeness.
What awaits you, per task
Each task's
hidden/README-FOR-HUMAN.mdis the full spec; this is the index. I have notread, written, or linted anything inside any
hidden/sealed set.W4b — zarr consolidated-metadata ordering
hidden/*_sealed_test.py. It must go beyond the public gate: the publictest covers three explicit key orders in-memory; the sealed test should cover a real
store and sibling groups differing only by case (upstream's
test_consolidated_metadata_case_differing_siblingsshape).awaiting_human → passand W4b is 10/10.Everything else is done: pins, reset, public gate from the PR's own tests, canonical
patch, container recipe, 9 of 10 checks.
W3 — sqlfluff segment-method migration · ESCALATION PROBE
Register the failure prediction in
manifest/preregistrations/before any run(SPEC §5.1 — prediction recorded first, result published either way).
task.yaml'sheader comment block lists the obligations verbatim.
Author⚠️ The glob is
hidden/*_sealed_test.py.*_sealed_test.py, nottest_*_sealed.py— sqlfluff setspython_files = "*_test.py", so atest_*.pymodule is silently uncollected and would pass vacuously. The README leads with this.
Five things it must add on top of the public gate:
DeprecationWarningwhilelinting with AL05/RF01/RF02/RF03/ST05/ST09/ST11 inside
warnings.catch_warnings(record=True). This is requirement 3 of the prompt and ithas no public-gate coverage: a submission that adds the free functions but
leaves the old call sites in place passes the public gate today;
ObjectReferenceLevelaccepts both the enum and a bare int;The README also carries an explicit do-not-assert list (internal helper names
_raw_refs/_level_to_int/deprecated_segment_method, the Protocol, exactwarning text, file layout) so the sealed test grades behaviour, not my scaffolding.
Verify both directions.
W1b — zarr block/mask property tests
hidden/check.sh,hidden/mutants/,hidden/VERSION. The README tables 8 candidate mutants (M1–M8) already anchored toreal line numbers —
BlockIndexer(l.1045),BlockIndex(l.1131),MaskIndexer(l.1299),
is_mask_selection(l.1159),BoolArrayDimIndexer(l.623),CoordinateIndexer(l.1169).threshold — otherwise the runner cannot distinguish "thorough" from "impossible".
check.shcontract (in the README): discover untrackedtests/test_*_properties.py; none → exit 1; apply/run/restore per mutant under atrap;HYPOTHESIS_PROFILE=ci; ~5 min timeout counts as not-caught; stderrCAUGHT/NOT-CAUGHT/TIMEOUT/CONTROL; exit 0 iff every non-control mutant iscaught; exit 2 reserved for a broken sealed set.
W6 — hono reg-exp-router PR review
Scaffolded except the defect map, as you asked.
hidden/README-FOR-HUMAN.md(227 lines)is written to be authorable in one sitting and spells out the seeding instructions
from the mining rationale:
hidden/review-diff.patch—review/base-diff.patch(the real 526-line upstreamdiff over
node.ts+88/−82,router.ts+73/−114,trie.ts+15/−6, withrouter.test.tsdeliberately withheld) with your k defects seeded in.hidden/defect-map.json— k = 6, one per class, each anchored to a real regionthe README names with its surrounding code:
trie.ts—this.paths[path] = [this.#index++, paramAssoc], the two reverse loopsnode.ts— theUnsupportedPathErrorthrow,regexpStr !== ONLY_WILDCARD_REG_EXP_STRcompareKey'sreturn b === TAIL_WILDCARD_REG_EXP_STR ? -1 : 1;#buildMatcherorderelse→ wildcard fallthroughif (pattern) {…} else {…}/i === len - 1ladderregExpMetaChars.has(k) ? `\\${k}` : kthis.#tries![method] = new Trie(),Object.create(null),clearWildcardRegExpCache()Schema:
{version, k, defects:[{id,class,file,line,note}], clean_regions:[{file,start,end,note}]},where
lineis 1-indexed in the new file.The ≥2 clean-region rule.
#buildMatcherand thechildStr === '' ? '' : …+#index = isStatic ? -1 : indexpruning are namedmandatory-unseeded. Zero-fabrication is only a meaningful half of the pass
condition if there are clean regions available to falsely accuse.
hidden/check.sh— matcher contract in the README: report at$SUBJECT_DIR/review-report.txt; missing → exit 1 (not 2); parse<path>:<line> — <desc>tolerating-/–/:; ±3 line window; collapse duplicates;stderr
DETECTED/MISSED/FABRICATED;exit 0 iff
detected >= k-1 && fabricated == 0; exit 2 only for a broken sealed set.Fixtures A/B/C for both-direction verification, plus the expected validator counts
once the map lands (W6 goes 4·2·4·0 → 6·0·4·0).
Not done / out of scope
pre-registered. Contamination tiers in
task.yamlare declared on a merge-date basisand must be confirmed against each subject's declared cutoff at pre-registration; the
register records a visibility-based fallback tier for each.
🤖 Generated with Claude Code