Wave-1 three-way matrix: 10 never-tuned corpora + DeepWiki/CodeWiki peer manifests - #1
Merged
Merged
Conversation
…odeWiki peer manifests Ten new corpora (resilience4j, ktor, ripgrep, sidekiq, laravel, Polly, Alamofire, argo-cd, fd, FluentValidation — 8 languages, none used to tune the pipeline; the last three ran as a sealed holdout) with full provenance and graph indexes. Nineteen hash-pinned peer manifests: DeepWiki with live-verified per-page URLs, CodeWiki with its displayed generation commit recorded per arm. New standalone fetch-codewiki pipeline (playwright, optional dependency) reproduces the CodeWiki extraction + prose-preserving citation normalization. Full-population results (same pinned judge, both sides, 0 judge failures): doc0 0.856 vs DeepWiki 0.588 vs CodeWiki 0.430 mean claim support; doc0 wins every per-repo head-to-head. Report includes the fairness protocol (peer-SHA pinning, extraction-artifact sensitivity runs) and an honesty section disclosing doc0's own defects found during the wave. Baseline pinning for the new corpora is deliberately deferred — v0.1-launch.baseline.json is mid-edit in separately staged work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 832ae418f4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… fetch pipeline - Peer manifests gain structured pinned_source_sha (set on all nine CodeWiki arms); the judged tier now checks each pinned peer out at its own commit instead of judging every peer against the corpus source SHA. ensureSourceRepo refactored to a repo/sha/dir-keyed ensureSourceAt both paths share. - peers:fetch handles codewiki manifests as verify-only (pages come from the fetch-codewiki playwright pipeline, not HTTP; unfetched arms skip with the exact command to run) instead of falling into the raw-HTML branch. - normalizeCitations is idempotent (already-normalized path:line labels are left untouched) and splitExtraction clears stale .md pages before writing, so rerunning the advertised CLIs cannot change prose, hashes, or leak removed sections into scoring. Both behaviors pinned in test/fetch-codewiki.test.ts. - Coverage detection in fetch.ts is structural (CodeWiki's not-found headline) rather than a bare "404" substring that wiki prose can trip. - README keeps only the latest full-population matrix; superseded n=40 snapshots remain in docs/benchmarks/. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e0f8db1cd9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…afe split, full-block dedup - peers:fetch exits 1 when codewiki pages haven't been browser-extracted yet, printing the exact fetch-codewiki command per manifest — a silent skip let the run report success while the judged runner rejected the same empty directory moments later. - splitExtraction tracks fenced-code state so a "## " shell/Ruby/Python comment inside a code block can't open a spurious section and split the fence across pages (pinned in test; the committed 2026-07-28 arms were audited — zero pages with unbalanced fences, so the scored corpus is unaffected). - extract.js deduplicates repeated DOM blocks by the full block, not a 200-char prefix, so distinct blocks sharing a long setup prefix are never dropped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
corpora/<repo>-2026-07/): resilience4j, ktor, ripgrep, sidekiq, laravel/framework, Polly, Alamofire, argo-cd, fd, FluentValidation — 8 languages, none ever used to tune the pipeline. argo-cd/fd/FluentValidation ran as a sealed holdout (selected in advance, generated last, zero pipeline changes mid-wave). Each corpus carries_meta.jsonprovenance (doc0 commit, source SHA, model stack) and a full_graph-index.json.corpora/peers/): DeepWiki arms for all ten repos (344 per-page URLs, live-verified) and CodeWiki arms for the eight it covers plus hono — sha256-pinned, content never redistributed, CodeWiki's displayed generation commit recorded per arm.fetch-codewiki/: standalone playwright pipeline (optional dependency, tsc-clean without it) reproducing the CodeWiki extraction, h2 splitting, and prose-preserving citation normalization.docs/benchmarks/2026-07-29-wave1-three-way-matrix.md) + raw per-arm results jsonl + README headline block.Headline (full population, one pinned judge, ~20k claim groups, 0 judge failures)
doc0 wins every per-repo head-to-head; the sealed holdout mean (0.886) came in above the main wave (0.843). The report documents the fairness protocol (CodeWiki judged at its own pinned commits — four rows commit-identical; extraction-artifact hypotheses tested with sensitivity runs before accepting peer numbers) and an honesty section disclosing doc0's own wave-discovered defects (ktor OpenAPI test-fixture pollution, JVM citation-resolution gap).
Deliberately out of scope
baselines/v0.1-launch.baseline.jsonentries for the new corpora — that file is mid-edit in separately staged local work; pin the deterministic metrics there with the usualWhy:line.Gates
npx tsc --noEmitclean · 95/95 tests green · eslint clean on new tooling · no internal paths in public-facing files🤖 Generated with Claude Code