B-0024 steps 5–6: what the high labels are made of, and whether properties/args share a vocabulary - #135
Merged
Merged
Conversation
Step 4 measured which labels changed. This asks what the surviving ones rest on: `resolveToDir` walks any deeper mention back to its nearest real menu, so a section that never names a menu can still supply exact-match evidence for it — the strongest tier. Measured on v0.11.2-alpha.109 by extending the committed census: 19.7% of `high` (388 of 1,969) comes from collapsed evidence. Half of that is the collapse doing its job (the overlay calls the name settable at the menu for 196). The failure mode is a section documenting a command's *output* claiming to be about the command's *menu* — `/interface/wifi` + `ssid` returns the scan-result prose at `high` above the configuration prose at `medium`. Confirmable class: 24 rows. The cheap fix is measurably wrong: refusing to collapse read-only verbs would demote 147 rows, 54 of which the overlay says are settable at the menu anyway and 0 of which it says are read-only there. `cliref_fields.field_kind` is the signal; verb shape is not. Disposition: recorded, not built — 24 rows does not justify collapse provenance in menu-paths.ts plus a cliref dependency inside property-confidence.ts. Also marks briefing steps 2 and 5 done: #131 is narrowed to Option C in its issue body, and #58/#61 carry post-step-4 re-anchor comments re-running their original symptoms. No CHANGELOG entry: the census is a dev measurement tool, not a user-visible surface, and no grading behaviour changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe evaluation now measures whether high-confidence matches come from explicit menu names or collapsed command paths. It also checks CLI-Reference field kinds, reports read-only mismatches, and documents the resulting findings and deferred demotion approach. ChangesCommand prose evidence analysis
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
There was a problem hiding this comment.
Pull request overview
Extends the existing command↔prose join census (src/eval/command-prose-join.ts) with “Step 5” measurements to break down what evidence the shipped high property-confidence labels actually rest on (direct menu mentions vs resolveToDir-collapsed deeper paths), and records the new findings in the B-0024 briefing.
Changes:
- Add Step 5 counters and reporting to split surviving
highlabels by exact-match provenance (named menu vs collapsed deeper path) and corroborate subsets via the CLI-Reference overlay. - Update
briefings/B-0024-command-prose-join.mdwith Step 5 results, interpretation, and disposition notes; bumplast_revisited.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/eval/command-prose-join.ts | Adds Step 5 census logic to attribute high labels to direct vs collapsed evidence, plus overlay-based corroboration counts. |
| briefings/B-0024-command-prose-join.md | Documents Step 5 methodology/results and updates the recommended-next-steps narrative and revisit triggers. |
Steps 3-5 asked which menu a property row belongs to. This asks whether the two stores agree on what an attribute is called — a perfect menu join cannot align prose with schema if the name join fails underneath it. Measured on v0.11.2-alpha.109 by a new census, with the CLI-Reference overlay as referee (the only store distinguishing settable from read-only, and the only version-less one). The 1,271 property rows inspect has never heard of decompose into two mismatches, neither of which is about spelling: - 670 (52.7%) are read-only fields. inspect `arg` rows model settable input, so a documented output column can never appear there. - 280 (22.0%) are settable somewhere, just not on the device this tree was dumped from — CRS1xx/2xx switches alone account for 191, and /interface/ethernet/poe has 0 rows in `commands` at all. - 309 (24.3%) are unknown to the overlay too. Reverse direction, never counted before: of 2,720 distinct argument names, 1,104 (40.6%) have no prose row anywhere — #61's "known, undocumented" population, as a generous name-level upper bound. Dotted names are ambiguity, not absence: 244 dotted args, 0 documented under the full name, 150 under the bare leaf, and the model explains only 4 of the 1,271 unmatched rows. Consequences recorded in the briefing: the overlay is the vocabulary bridge because it is complementary exactly where inspect is blind; proximity answers "which menu" and does nothing for roughly half this gap; and the prior question to "what is the missing key" is "what is the missing row". No CHANGELOG entry: measurement only, no shipped behaviour changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mobileskyfi
changed the title
high labels are made ofhigh labels are made of, and whether properties/args share a vocabulary
Review fixes: - cspell: rename the census's `unk` CTE to `unmatched`; add `essid` and `wiliot` to project-words.txt. This is what failed CI. - Copilot: both censuses now probe for the cliref_* tables instead of unconditionally preparing against them. The overlay arrived in schema 11 and every other figure derives from properties/commands alone, so an older or overlay-less DB degrades to "no cross-check" with a printed note rather than crashing a census that is otherwise still valid. Docs, recording the layering model the maintainer proposed and this pass checked against the artifact: - DESIGN.md gains "What each store is authoritative for", because it governs which store should be the base for future alignment work. Three measurements behind it: the overlay has commands but not the CRUD verb layer (2 of 441 Command entries end in a CRUD verb); inspect is where the verbs live (print 540, get 539, set 428, add 301); and inspect's `arg` rows are a verb's *inputs*, so /print and /get expose CLI options, not read-only state — `running`, `rx-packet`, `tx-byte`, `last-seen` and `actual-mtu` appear 0 times as args anywhere. - B-0024 records the same claims as a checked table, including the one correction: print/get expose neither read-only nor writable fields. That strengthens the reading rather than weakening it — read-only fields are exclusively an overlay concern, with no inspect query that recovers them. - B-0024 step 8 re-orders the agenda around the store question, and demotes the step-7 extractor fix (it deepens reliance on the join we want to stop using). - BACKLOG gains a trigger for the overlay-as-base path vocabulary, which is a canonicalization question owned by centrs' explain work, not rosetta-local. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continues
briefings/B-0024-command-prose-join.mdafter step 4 (PR #134). Three parts: the outward-facing issue edits, and two measurements — the second of which reorders the briefing's agenda.Step 5 — provenance of the shipped
highlabelsresolveToDirwalks any deeper mention back to its nearest real menu (added so/certificate/import file-name=xcounts for/certificate), which means a section that never names a menu can still supply exact-match evidence for it.highHalf the collapsed set is the collapse working as designed (the overlay calls the name settable at the menu for 196 of 388). The failure mode is a section documenting a command's output claiming to be about its menu —
/interface/wifi+ssidreturns the scan-result prose athighabove the configuration prose atmedium, and the overlay confirms it independently (interface/wifi/scan · ssid=Read-only Argument). Confirmable class: 24 rows.The cheap fix is measurably wrong: refusing to collapse read-only verbs demotes 147 rows, of which the overlay says 54 are settable at the menu anyway and 0 are genuinely read-only there.
cliref_fields.field_kindis the signal; verb shape is not. Disposition: recorded, not built — 24 rows does not justify collapse provenance inmenu-paths.tsplus a cliref dependency inproperty-confidence.ts.Step 6 — do
propertiesandargseven share a vocabulary?New census (
src/eval/vocabulary-alignment.ts). Steps 3–5 all asked which menu a row belongs to; this asks whether the two stores agree on what an attribute is called, because a perfect menu join cannot align prose with schema if the name join fails underneath it.The 1,271 rows inspect has never heard of are mostly not a naming problem:
argrows model settable input; a documented output column (rx-packets,last-seen) can never appear there./interface/ethernet/poehas 0 rows incommands, so those property rows have no menu to join to at any granularity. This is arch as advisory + CLI-Reference overlay with quasi-provenance ('applies to stable X as of import') #25's arch-as-advisory half with a number on it.Reverse direction, never counted before: of 2,720 distinct argument names, 1,104 (40.6%) have no prose row anywhere — #61's "known, undocumented" population as a generous name-level upper bound.
Dotted names are ambiguity, not absence: 244 dotted args, 0 documented under the full name, 150 under the bare leaf — and the model explains only 4 of the 1,271 unmatched rows, because those leaves are usually also plain args elsewhere. So #61 BL-3's descriptions exist; the defect is one prose row standing in for several attributes.
What it changes
argrows to its 10,118 fields) but because it is complementary exactly where inspect is blind:field_kindand version-less/hardware-independent coverage.Issue edits (no code)
v0.11.2-alpha.109.Validation
bun test— 1,128 pass / 0 failbun run lint— cleanCHANGELOG.mdentry: measurement only, no shipped behaviour changed🤖 Generated with Claude Code