feat(family-2): parser-safe adapter transforms + external-benchmarks table - #2
Merged
Conversation
Two measured loss modes in the pinned CodeWikiBench parser (probed directly
against 5e728fb4, 2026-08-31):
1. Preamble text under a heading with child headings has no dict key to
hang on and is dropped whole. Fix: hoistPreambles packages each preamble
as a synthetic first child heading ("Overview", with collision
fallbacks), the same key shape CodeWiki's own example output uses.
2. Within a section span, everything after the first list dies: a trailing
paragraph, a second list, and (the corpus staple) list items directly
above a Sources line. Fix: neutralizeNonFinalLists escapes the markers
of any list that is not the span's trailing list (1. -> 1\.), which
renders identically but parses as paragraphs, which always survive.
No words are added, removed, or reordered by either transform; this is
packaging for their parser, not editing.
Measured on the committed hono corpora (dry-run, all pages parsed both
before and after):
hono-2026-07-v2: retained-word fraction 0.7339 -> 0.9516
hono-2026-07: 0.73x -> 0.9508
Word-multiset diff shows the residual gap is ~2/3 markdown syntax tokens
(heading markers, blockquote prefixes, fence info-strings, alert tags), so
content-level retention is ~99%.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DfgE3Yucn5LUDZPpqJzhz2
…nch systems CodeWiki (Sonnet-4) is named as the benchmark authors' own generator, and the deepwiki-open and OpenDeepWiki rows join from the paper's Table 1. Notes the Family-2 comparability contract: their published per-repo rubrics + their exact judge panel, so doc0's row compares per-repo against every published row without re-running any peer. The 7-vs-21-repo scope discrepancy is cited as-published per docs/codewikibench-pinned.md (f). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DfgE3Yucn5LUDZPpqJzhz2
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e8b8569219
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…preambles The pinned evaluator parser (markdown_to_json 2.1.2) splits a section on headings at exactly one level below its own and discards every block before the first one it finds. A first child that skips a level (H1 straight to H3, the shape of corpora/redis-2026-07/cluster-routing.md) is never a key: the synthetic "## Overview" swallowed the H3, became childful, and the preamble was dropped again. Placing the synthetic heading at the child's level does not help either, since the parser then skips over both of them together (probed directly against the pinned parser). normalizeHeadingLevels re-levels every heading to exactly one deeper than its nearest shallower predecessor (stack walk, first heading keeps its level), and hoistPreambles runs it first so its one-level-deeper invariant holds. Heading text and order are untouched. On the redis page, parser retention goes from 0.834 to 0.997; the transform is a byte-identical no-op on the other 542 corpus pages. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HS8GHxjM3NfU2nPk1hwVJx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two content-preserving transforms in the CodeWikiBench adapter, closing the retained-word gap measured by the free dry-run, plus the external-benchmarks comparison table extended to all four published systems.
Measured loss modes in the pinned parser (probed directly against
5e728fb4with controlled markdown, 2026-08-31):markdown_to_json's tree and is dropped whole.hoistPreamblespackages each preamble as a synthetic first child heading ("Overview", with Introduction/Preamble/numbered fallbacks on sibling collision) — the same key shape CodeWiki's own example output uses.Sources:line.neutralizeNonFinalListsescapes the markers of any list that is not the span's trailing list (1.→1\.), which renders identically but parses as paragraphs, which always survive.Neither transform adds, removes, or reorders a word: packaging for their parser, not editing.
Results (dry-run, all pages parsed before and after)
A word-multiset diff of the residual shows ~2/3 is markdown syntax that can never reappear as words in the parsed JSON (
##,>, fence info-strings,[!NOTE]tags) — content-level retention is ~99%, the DESIGN target.Also
docs/comparison-systems.md's external-benchmarks table now lists all four published Table 1 systems — naming CodeWiki (Sonnet-4) as the benchmark authors' own generator — and states the Family-2 comparability contract (their published per-repo rubrics + their exact judge panel), so doc0's row will compare per-repo against every published row without re-running any peer.Tests
10 new cases pin both transforms (fence masking, collision fallbacks, tight lists, trailing-Sources shape, span independence). Full suite: 9 files / 116 tests green; eslint clean.
🤖 Generated with Claude Code
https://claude.ai/code/session_01DfgE3Yucn5LUDZPpqJzhz2
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Review round 1 (Codex)
fcd281c): the pinned parser splits each section on headings at exactly parent+1 and discards everything before the first match, so a first child that skips a level (H1 straight to H3,redis-2026-07/cluster-routing.md) made the synthetic## Overviewchildful and lost the preamble again. A synthetic heading at the child's own level is skipped over with it. NewnormalizeHeadingLevelspass (stack walk) runs first insidehoistPreambles. That page: 0.834 → 0.997 on the real parser; byte-identical no-op on the other 542 pages.