B-0024: measure the command↔prose join, then grade property lookups with it by mobileskyfi · Pull Request #134 · tikoci/rosetta

mobileskyfi · 2026-08-01T00:18:04Z

Two threads of work on B-0024: measure whether a property's own section can serve as the missing command↔property key (step 3), then use the answer to recalibrate routeros_lookup_property's confidence and ranking (step 4).

Step 3 — Option A is a ranking signal, not a key

Measured on v0.11.2-alpha.109 (the CI release artifact via make db-sync), census committed at src/eval/command-prose-join.ts — the db-census.ts precedent, not a CI gate.


Conditional precision (sections that name any path)	75.6%
Coverage — rows whose section names any path	68.1%
Property-owning sections naming no path	42.7%
Candidate paths per fragment, section vs page	1.0 vs 6.4

When a section names a path it is usually the right one; the dominant failure is silence, not misalignment — the opposite of the fuzzy page ranker's failure mode, which is why the two compose as tiers rather than one replacing the other.

Two sub-questions close outright:

Table granularity — dead end. Across 562 property-bearing tables: identical to its section's path set in 33 cases, narrower in 34, never wider (0) or disjoint (0), and 495 name none. A table-grained key can only lose information.
Proximity — needs new provenance, not a new query. No line/offset column exists on sections, page_tables, page_table_rows, or properties, only sort_order. This corrects the briefing, which claimed those tables retain line spans.

src/menu-paths.ts factors extraction out of link-commands.ts so census and linker cannot drift. That refactor is verified behaviour-preserving, not asserted: both versions run against separate DB copies, commands.page_id diffed across all 41,967 rows — identical, and the 1,575 linked rows match the figure in #131.

Step 4 — grade the row, not the query branch

high used to mean "the scoped branch found rows on the page commands.page_id links to" and low "it didn't". The label described which SQL ran. src/property-confidence.ts now scores each row against the requested menu:

high — the row's own section is about that menu (it names the menu, and of the menus it names that one owns most of its properties), and the command tree does not contradict it
medium — the section names a neighbouring menu; or names the requested one only in passing while documenting another; or only the page matches
low — nothing but the property name ties the row to the menu

Unscoped lookups stay medium throughout: with no menu to align to, the tier answers a different question.

The candidate set had to move too

Grading alone is not enough, and the first cut of this PR got that wrong — caught in review. While the scoped branch still selected rows, the fuzzy link kept its veto: /interface/bridge/host links to the IGMP-snooping page, which documents its own vid, so the lookup returned two medium rows from that page and never graded the high Static Entries section that names /interface/bridge/host outright. Same for /interface/wifi/provisioning + radio-mac.

Candidates now come from every page documenting the property, with page alignment per row. Two rules keep that from turning every lookup into a global dump:

an off-page row survives only if its tier is at least as good as the best the linked page offers;
within a tier, linked-page rows sort first.

That second rule fixes a regression the first cut introduced: tiers are menu-level, so name at /interface/ethernet is legitimately high on Ethernet, Bonding and PoE-Out, and with ties broken by page title explainCommand described name=ether2 as "Name of the bonding interface".

Measured

src/eval/command-prose-join.ts replays lookupProperty's candidate set and filter and grades every candidate with the shipped gradeRow/supportedPaths, so these figures regenerate rather than being asserted. Over the 14,832 (menu, property-name) pairs the command tree says are real, 115,926 row labels (absent = not returned at all):

Transition	Rows	Share
`low` → `low`	76,963	66.4%
`absent` → `absent`	33,521	28.9%
`high` → `medium`	2,295	2.0%
`high` → `high`	1,054	0.9%
`low` → `medium`	1,105	1.0%
`low` → `high`	835	0.7%
`absent` → `high`	80	0.1%
`absent` → `medium`	73	0.1%

31.5% of the labels that shipped as high survive; 1,940 rows escape a wrongly-low label; 153 rows previously suppressed outright are now returned, 80 of them high.

The support gate, and why the eval earned it

Whether a support ratio should gate high was step 4's open question. It must, and the retrieval eval proved it: without a gate, /ip/firewall/filter + action scored high on the bridge firewall section, which cites that menu in a single sentence while documenting /interface/bridge/filter. Naming is not aboutness.

The gate requires the requested menu to be among those accepting the most of the section's own property names, and distinguishes two zeroes: the command tree knowing none of the section's names is silence and cannot demote; knowing them and finding no named menu that takes any is evidence against every candidate. Of the alignments naming alone would accept, the gate keeps 84.4%; strict exclusivity (reject any section naming an unrelated menu) keeps 42.4% and discards correct alignments over incidental cross-references.

Effect on the motivating defects

Lookup	Result
`/interface/bridge/port` + `pvid` (#131)	both bridge-port sections `high`, unrelated Apps row `low` and last
`/ip/dhcp-server` + `address-pool` (#58)	DHCP server table `high`, DHCPv6 and the ambiguous `Properties` demoted
`/interface/bridge/vlan` + `vlan-ids` (#61)	`high` — despite `commands.page_id` still pointing at the wrong page
`/interface/bridge/host` + `vid`	the mislinked page no longer suppresses the section that names the menu

All of these rank correctly; none is fixed at the source. The bad commands.page_id join is still there — the grading stops it from mattering. Hence Part of, not Closes.

Eval fixture change — please read this one

src/eval/retrieval.ts's property assertion required a high row. Under the new contract that is stricter than intended: WireGuard's listen-port sits in a bare property table that never names /interface/wireguard, so the correct top row is honestly medium. It now requires the top-ranked row to be the expected page and not low, and still requires high where no page is expected.

That second clause matters: it keeps prop-firewall-filter-action (the known-failing BL-1 anchor) red. Without the support gate it had silently flipped green on the false high above — a regression disguised as an improvement, caught only because the anchor exists.

Not in scope

Two linker defects found while measuring — MENU_PATH_RE never matching bare top-level menus (~2.4pp coverage), and normalizeMenuPath fabricating pseudo-paths — are recorded with measured cost and not fixed. The top-level fix changes commands.page_id, so it needs a before/after link diff, not unit tests.
Fixing the underlying join. Proximity is the only live candidate and needs extraction-time provenance that does not exist yet.

Verification

make typecheck, make lint, bun test (1,107 pass / 0 fail) green; retrieval eval against the synced release DB shows no regression and the same three known failures as baseline.

Part of #131
Part of #58
Part of #61

Summary by CodeRabbit

New Features
- Added confidence scoring for property lookup results, with high, medium, and low tiers.
- Results are now ranked by confidence and better aligned with the requested command path.
- Lookup considers more candidate pages while preserving linked-page preferences.
- Added guidance for interpreting confidence and confirming low-confidence matches.
Bug Fixes
- Previously omitted property matches may now appear when their documentation alignment is strong.
Tests
- Expanded coverage for confidence scoring, path handling, ranking, and cross-page results.

… not a key Answers B-0024's gating question — is fragment-grained menu-path extraction precise and fine enough to be the missing command→property key? Measured on the CI artifact v0.11.2-alpha.109 (schema 11, source_commit 4cd7413): - 76.1% conditional precision, 6.5x narrower candidate set than page grain - but 42.7% of property-owning sections name no menu path, and 63.6% of property rows share a section with 10+ others So it is adopted as a ranking signal and rejected as a key. It discriminates correctly on both motivating families, including their negative cases: #131's pvid (exact on the two bridge-port sections, no signal on the Apps row) and #58's address-pool (exact on DHCP, no signal on the HotSpot mislink targets). Two sub-questions close outright. Table granularity is dead: no table names a path its section lacks and 495 of 562 name none, so a table-grained key can only lose information. Proximity — the one remaining candidate for a real key — needs new extraction-time provenance; no line/offset column exists on sections, page_tables, page_table_rows, or properties, correcting a claim in the briefing that they retain line spans. The measurement also surfaced two defects in the linker shipping today: bare top-level menus (/certificate, /queue) are invisible to MENU_PATH_RE, and normalizeMenuPath fabricates pseudo-paths from assignments. Both are recorded with their measured cost; neither is fixed here, since the first changes commands.page_id and needs a link diff rather than unit tests. Extraction moves to src/menu-paths.ts (pure, DB-free, anchor-tested) so the census and the linker measure the same rules instead of drifting copies. The link-commands.ts change is a behaviour-preserving refactor. Refs #131, #58, #61. Briefing records the step-4 confidence contract the bounds support. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… step 4) `routeros_lookup_property`'s confidence described which SQL ran: `high` meant "the scoped branch found rows on the page `commands.page_id` links to" and `low` meant "it didn't". So every property on a linked page was equally `high`, and a correct row reached by the global fallback was always `low`. Step 3 measured the alternative — menu paths named in the property's own section — at 76.1% conditional precision but silent for 42.7% of sections: a ranking signal, not a key. This uses it to grade rather than to select. The row set is unchanged; only the label and the order move. `src/property-confidence.ts` scores each row against the requested menu: high the row's own section names that menu, that menu is the one the section is about, and the command tree does not contradict it medium the section names a neighbouring menu, or names this one only as a cross-reference, or only the page aligns low nothing but the property name ties the row to the menu Over the 14,832 (menu, name) pairs the command tree says are real: 31.5% of today's `high` labels survive, 2,295 demote, and 1,940 rows escape a wrongly `low` one. `/interface/bridge/port` + `pvid` now ranks the two bridge-port sections above the unrelated Apps row (#131); `/ip/dhcp-server` + `address-pool` ranks the DHCP server table above the DHCPv6 one (#58). The support gate answers step 4's open question, and the retrieval eval is why it exists: without it, `/ip/firewall/filter` + `action` scored `high` on the *bridge* firewall section, which cites that menu in one sentence while documenting `/interface/bridge/filter`. Requiring the requested menu to accept the most of the section's own property names keeps 84.6% of `high` labels; the blunter "reject any section naming two menus" keeps 41.9% and discards correct alignments over incidental cross-references. The eval's property assertion moves with the contract rather than against it: `high` alone is no longer the bar, since a correct row whose section names no menu is honestly `medium` (WireGuard's bare property table). It now requires the top-ranked row to be the expected page and not `low`, and still requires `high` where no page is expected — which keeps the known-failing BL-1 anchor red, as it should be. Refs #131, #58, #61

coderabbitai · 2026-08-01T00:18:18Z

📝 Walkthrough

Walkthrough

Shared RouterOS menu-path extraction utilities normalize, validate, and resolve paths. Property rows receive high, medium, or low confidence from section alignment, supported paths, page link, and command acceptance evidence. Scoped lookups rank rows by confidence and linked-page preference. Unscoped lookups return medium-confidence rows. Evaluation tooling measures alignment across the corpus. Documentation and validation records cover the new contract.

Changes

Property confidence and menu-path alignment

Layer / File(s)	Summary
Shared menu-path extraction `src/menu-paths.ts`, `src/link-commands.ts`, `src/menu-paths.test.ts`	Adds shared path normalization, RouterOS validation, ancestor resolution, and de-duplicated extraction. `link-commands.ts` replaces local regex and hardcoded allowlists with shared utilities.
Property confidence grading `src/property-confidence.ts`, `src/property-confidence.test.ts`	Grades rows as `high`, `medium`, or `low` from section menu-path alignment, supported-path scoring, page link, and command-tree acceptance evidence. Pure grading functions and database-backed row analysis implement tiering.
Ranked property lookup integration `src/query.ts`, `src/query.test.ts`, `src/eval/retrieval.ts`	Lookups retrieve `section_id`, grade all matches against the command path, retain linked-page rows and off-page rows meeting linked-page tier, remove `section_id` from results, and sort by confidence tier.
Corpus evaluation and confidence validation `src/eval/command-prose-join.ts`	Census script measures extraction coverage, acceptance precision, oracle strength, support-ratio noise, and label transitions from pre-change confidence to implemented tiers.
Documentation and validation `src/mcp.ts`, `CHANGELOG.md`, `VALIDATION.md`, `project-words.txt`, `briefings/B-0024-command-prose-join.md`	Documents confidence tiers, validation coverage, measured outcomes, and supporting vocabulary. Briefing records completed validation and implementation status.

Estimated code review effort: 4 (Complex) | ~50 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant lookupProperty
  participant gradeRows
  participant Database
  participant acceptanceMap
  Client->>lookupProperty: request property with command_path
  lookupProperty->>Database: fetch all matching rows with section_id
  lookupProperty->>gradeRows: grade scoped rows against command_path
  gradeRows->>Database: read section paths and command acceptance
  gradeRows->>acceptanceMap: resolve property name to accepted menus
  gradeRows-->>lookupProperty: return confidence tiers
  lookupProperty-->>Client: return sorted rows by tier

Possibly related PRs

tikoci/rosetta#133 — Directly related to the B-0024 confidence and ranking work.
tikoci/rosetta#97 — Introduced section-based property attribution used by confidence grading.
tikoci/rosetta#62 — Related to command-to-documentation path alignment.

Suggested labels: area:docusaurus

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the PR's main changes: measuring the command-to-prose join and applying it to property lookup confidence grading.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch b0024-join-census-and-confidence

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Copilot

Pull request overview

Implements B-0024 steps 3–4 by (1) adding a corpus-wide census to measure whether section-level menu-path mentions can key the command↔prose join, and (2) using that result to recalibrate routeros_lookup_property confidence so it grades each returned row against the requested menu (and orders best evidence first), rather than reflecting which SQL branch ran.

Changes:

Add menu-paths.ts as the shared, DB-free menu-path extractor and refactor link-commands.ts to use it.
Introduce property-confidence.ts and wire it into lookupProperty() so confidence tiers are per-row and results are tier-ranked.
Add/adjust evaluation + validation artifacts (new census script, retrieval eval fixture changes, tests, VALIDATION.md, and CHANGELOG.md entry).

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
VALIDATION.md	Adds a blocking validation row for the new property-confidence contract and its tests.
src/query.ts	Wires per-row property confidence grading + tier-based ordering into `lookupProperty()`.
src/query.test.ts	Extends DB fixture and assertions to anchor the new confidence/ordering rules.
src/property-confidence.ts	Implements evidence gathering + tier rules (`high`/`medium`/`low`) for property rows.
src/property-confidence.test.ts	Pins the tier decision table and support-gate behavior as pure-function tests.
src/menu-paths.ts	Factors menu-path extraction into a shared module for linker + census + grading.
src/menu-paths.test.ts	Anchors extraction/normalization/resolve behavior with focused unit tests.
src/mcp.ts	Updates `routeros_lookup_property` tool description to describe new ordering/tier meaning.
src/link-commands.ts	Switches linker path extraction to use `extractMenuPaths()` from the new module.
src/eval/retrieval.ts	Updates property-surface eval assertions to match the stricter per-row tier semantics.
src/eval/command-prose-join.ts	Adds the step-3 join census script that prints provenance + measured outcomes.
project-words.txt	Adds new corpus/eval terminology to spellcheck allowlist.
fixtures/eval/queries.json	Updates notes to reflect new tier semantics (e.g., WireGuard now “medium” not “high”).
CHANGELOG.md	Adds an Unreleased “Changed” entry documenting user-visible confidence semantics change.
briefings/B-0024-command-prose-join.md	Promotes Option A from hypothesis to measured result and records conclusions + follow-ups.

coderabbitai

Actionable comments posted: 6

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@briefings/B-0024-command-prose-join.md`:
- Around line 524-526: Update the open-question entry for “What is high allowed
to mean?” in the Step 4 discussion to mark it resolved, removing the statement
that the support-ratio gate remains undecided. Align it with the established
requirement that the support-ratio filter must gate high and the shipped gate
noted later in the document.
- Around line 317-319: Correct the table-granularity claim in the prose to match
the census categories emitted by the command-prose-join analysis, treating the
count of 33 as the narrower category rather than stating that tables are never
wider than their sections. Also revise the repeated absolute claim “No table
names a path its section lacks” to use wording consistent with the actual
wider/disjoint counts, while preserving the surrounding Option A reasoning.

In `@src/eval/command-prose-join.ts`:
- Around line 63-77: Extract the shared arg-row-to-accepted-menus builder
currently duplicated in the local acceptsByName construction and
property-confidence.ts into the existing shared menu-paths utility or another
shared module, then consume it from both callers. Preserve the parent-menu
derivation via lastIndexOf("/"), and explicitly verify whether keys should
remain raw names here or consistently use name.toLowerCase() as acceptanceMap
does before choosing the shared contract.

In `@src/mcp.ts`:
- Around line 790-800: Update the confidence definitions in the command_path
guidance near routeros_command_tree so high confidence requires only
non-contradictory command evidence, not always an affirmative command-tree
match. Classify rows supported by documentation cross-reference alone as medium,
while preserving the existing parent/child and page-match medium cases and the
low tier for property-name-only candidates.

In `@src/property-confidence.ts`:
- Around line 98-114: Update supportedPaths to compute support even when paths
contains a single menu, preserving the existing best === 0 fallback for truly
unsupported sets. Ensure the caller’s confidence logic demotes the result to
medium when best is zero and the command tree recognizes at least one section
property name, rather than allowing that row to remain high solely because one
path was provided.
- Around line 139-152: Update gradeRows to lazily cache the directory paths in a
module-scoped cachedDirPaths/dirPathSet helper, rather than querying commands on
every lookup. Also hoist and reuse the sectionQuery, namesQuery, and
acceptanceQuery prepared statements at module scope so each gradeRows call only
binds and executes the existing queries.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 171c5718-99a2-46c2-9b2a-0678992af3eb

📥 Commits

Reviewing files that changed from the base of the PR and between 9557945 and 9e2ca01.

⛔ Files ignored due to path filters (1)

fixtures/eval/queries.json is excluded by !fixtures/**

📒 Files selected for processing (14)

CHANGELOG.md
VALIDATION.md
briefings/B-0024-command-prose-join.md
project-words.txt
src/eval/command-prose-join.ts
src/eval/retrieval.ts
src/link-commands.ts
src/mcp.ts
src/menu-paths.test.ts
src/menu-paths.ts
src/property-confidence.test.ts
src/property-confidence.ts
src/query.test.ts
src/query.ts

📜 Review details

⏰ Context from checks skipped due to timeout. (4)

GitHub Check: copilot-pull-request-reviewer
GitHub Check: Analyze (actions)
GitHub Check: Analyze (javascript-typescript)
GitHub Check: Analyze (javascript-typescript)

🧰 Additional context used

📓 Path-based instructions (9)

**/*.{ts,tsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Use Bun and TypeScript, including Bun-native runtime APIs such as bun:sqlite, Bun.serve, and bunx.

Rosetta uses Bun and TypeScript; prefer bun, bun test, and make verify where applicable, rather than Node/npm-oriented substitutes.

Files:

src/eval/command-prose-join.ts
src/eval/retrieval.ts
src/mcp.ts
src/menu-paths.test.ts
src/property-confidence.test.ts
src/menu-paths.ts
src/link-commands.ts
src/query.test.ts
src/query.ts
src/property-confidence.ts

src/**/*.{ts,tsx}

📄 CodeRabbit inference engine (AGENTS.md)

Rosetta is read-only documentation/schema context and must not connect to or modify a user's router.

Files:

src/eval/command-prose-join.ts
src/eval/retrieval.ts
src/mcp.ts
src/menu-paths.test.ts
src/property-confidence.test.ts
src/menu-paths.ts
src/link-commands.ts
src/query.test.ts
src/query.ts
src/property-confidence.ts

**/*.{ts,tsx,md}

📄 CodeRabbit inference engine (AGENTS.md)

Keep the attribution boundary visible when community RouterOS skills surface, because they are supplemental rather than official MikroTik documentation.

Files:

src/eval/command-prose-join.ts
src/eval/retrieval.ts
CHANGELOG.md
src/mcp.ts
src/menu-paths.test.ts
src/property-confidence.test.ts
VALIDATION.md
src/menu-paths.ts
src/link-commands.ts
src/query.test.ts
src/query.ts
src/property-confidence.ts
briefings/B-0024-command-prose-join.md

*

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Keep this Copilot instructions file short; put substantive rules in narrow instruction files under .github/instructions/*.instructions.md.

Files:

CHANGELOG.md
project-words.txt
VALIDATION.md

**/*.md

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.md: Keep each project documentation file limited to its canonical role; prefer the documented canonical home instead of creating a new top-level Markdown file.
Do not duplicate content from this routing index; move operational detail, schema blocks, or long rule lists to their canonical documentation or instruction files.
Apply the repository's Markdown linting and fenced-code conventions, while excluding LLM instruction files where the repository explicitly requires that exclusion.

Files:

CHANGELOG.md
VALIDATION.md
briefings/B-0024-command-prose-join.md

CHANGELOG.md

📄 CodeRabbit inference engine (CLAUDE.md)

Record user-visible shipped changes in CHANGELOG.md under [Unreleased] or release sections.

Files:

CHANGELOG.md

src/{query,mcp,browse}.ts

📄 CodeRabbit inference engine (AGENTS.md)

Route MCP, query, classifier, TUI, and canonicalizer behavior through shared core code, usually src/query.ts; keep src/mcp.ts and src/browse.ts as thin adapters.

Files:

src/mcp.ts
src/query.ts

VALIDATION.md

📄 CodeRabbit inference engine (CLAUDE.md)

Document load-bearing invariants and how CI proves them in VALIDATION.md.

Files:

VALIDATION.md

briefings/B-*.md

📄 CodeRabbit inference engine (CLAUDE.md)

Store grounded research and decision support in briefings/B-*.md.

Files:

briefings/B-0024-command-prose-join.md

🪛 LanguageTool

VALIDATION.md

[uncategorized] ~25-~25: The official name of this software platform is spelled with a capital “H”.
Context: ...Property confidence) via bun testin.github/workflows/test.yml| blocking |#131` |...

(GITHUB)

briefings/B-0024-command-prose-join.md

[style] ~208-~208: ‘none at all’ might be wordy. Consider a shorter alternative.
Context: ... sort_order, and properties carries none at all. Verified against v0.11.2-alpha.109; ...

(EN_WORDINESS_PREMIUM_NONE_AT_ALL)

[style] ~290-~290: Consider an alternative for the overused word “exactly”.
Context: ...xcellent or entirely spurious, which is exactly the shape a support-ratio filter can ...

(EXACTLY_PRECISELY)

[style] ~533-~533: ‘out of reach’ might be wordy. Consider a shorter alternative.
Context: ... - #61's prose-only properties remain out of reach. Section alignment needs a section; c...

(EN_WORDINESS_PREMIUM_OUT_OF_REACH)

🔇 Additional comments (18)

src/menu-paths.ts (2)

22-52: LGTM!

59-97: LGTM!

src/link-commands.ts (2)

22-26: LGTM!

Also applies to: 81-82

70-72: 🗄️ Data Integrity & Integration

No change needed. cmdPathToId is only populated before dirPathSet, and the remaining references are reads.

src/menu-paths.test.ts (1)

1-79: LGTM!

src/eval/command-prose-join.ts (1)

30-57: LGTM!

Also applies to: 79-338

src/property-confidence.ts (2)

68-89: LGTM!

154-202: LGTM!

Also applies to: 204-228

src/property-confidence.test.ts (1)

10-122: LGTM!

briefings/B-0024-command-prose-join.md (1)

157-157: LGTM!

Also applies to: 189-194, 206-211, 243-316, 320-337, 436-481
src/query.ts (3)
12-16: LGTM!

Also applies to: 1059-1059

1085-1145: LGTM!

1083-1083: 🎯 Functional Correctness

No duplicate type alias remains.

PropertyLookupRowUngraded appears only once, so there is no duplicate identifier issue to address.
			> Likely an incorrect or invalid review comment.
src/query.test.ts (1)

211-296: LGTM!

Also applies to: 1077-1080, 1092-1137, 1156-1159

src/eval/retrieval.ts (1)

344-358: LGTM!

CHANGELOG.md (1)

39-39: LGTM!

VALIDATION.md (1)

25-25: LGTM!

project-words.txt (1)

11-20: LGTM!

Also applies to: 40-40, 143-143, 235-235, 264-264, 372-373, 447-447

mobileskyfi

Codex Review

Independent review found two additional issues in the inline comments below.

One process loose end: this PR explicitly leaves the source linkage/property gaps open, so .github/instructions/issue-pr-linking.instructions.md calls for Part of #131, Part of #58, and Part of #61 in the PR body rather than Refs .... Please update that relationship before merge so the partial landing is represented consistently.

…gate The tier system could still be vetoed by the link it was built to survive. While the scoped branch selected the rows, a mislinked page that happened to document the same property name kept the right row out of the result entirely: `/interface/bridge/host` + `vid` returned two `medium` rows from the IGMP snooping page and never graded the `high` "Static Entries" section that names `/interface/bridge/host` outright. Same for `/interface/wifi/provisioning` + `radio-mac`. Caught in review, verified against the alpha.109 artifact. Candidates now come from every page documenting the property, with page alignment applied per row. Off-page rows must earn their place: once the linked page contributes anything, unaligned (`low`) rows elsewhere are dropped, so a correct link returns the tight result it always did. The support gate no longer exempts single-path sections, which was precisely backwards — a section naming one menu has no competitor to be measured against, so a lone passing mention was the least contradicted and most likely to be mistaken for authority. Scoring now separates the two zeroes: the command tree knowing none of the section's names is silence and cannot demote; knowing them and finding no named menu that takes any is evidence against every candidate. The step-4 figures are now produced by committed code rather than a scratch script. `eval/command-prose-join.ts` replays lookupProperty's candidate set and filter and grades with the shipped gradeRow/supportedPaths, so the briefing and CHANGELOG numbers regenerate after any rule change. Restated on that basis: 33.4% of `high` labels survive, 1,940 rows escape a wrong `low`, and 236 rows that were suppressed outright are now returned. Also from review: - `acceptanceMap` moves to property-confidence.ts and the census consumes it, so the census cannot describe rules the grader does not follow. Keys are lowercased in both; the census previously keyed raw names. - `dir` set and prepared statements cached lazily instead of per lookup. - MCP tool description covers the support gate and the cross-reference case. - TOP_LEVEL_MENUS doc no longer implies resolved extraction keeps roots with no `dir` row behind them. - Briefing: table-granularity counts stated per census category (identical 33, narrower 34, wider 0, disjoint 0); the `high` open question closed. Refs #131, #58, #61

…igures Widening the candidate set introduced a regression the reviewer caught: tiers are menu-level, so a ubiquitous name is legitimately `high` on several pages at once. `name` at /interface/ethernet grades `high` on Ethernet, Bonding and PoE-Out, ties were broken by page title, and explainCommand reported `/interface/ethernet set name=ether2` as "Name of the bonding interface" — a worse answer than before this branch, on a query the old scoped branch got right. Linked-page rows now win ties within a tier. The page link is weak evidence but it is still evidence, and it is the right thing to break a tie the section evidence cannot. Retention tightens with it: an off-page row is kept only if its tier is at least as good as the best the linked page offers, which subsumes the previous drop-`low` rule and collapses that lookup from 16 rows to 4. The suppressed-row fixes are unaffected — /interface/bridge/host + `vid` and /interface/wifi/provisioning + `radio-mac` still surface their `high` sections. Regression tests added for both halves: an explainCommand case where two pages tie on `high` and the linked one must win, and a fixture page that documents /interface/bridge so the tie exists at all. The lowercase fix to the census was incomplete — the commonality histogram still keyed raw names — so the step-3 figures were measured with a partly case-sensitive oracle. Corrected and regenerated throughout: scorable rows 3,223 -> 3,316 (72.3% of 4,587) conditional precision 76.1% -> 75.6% cascade 51.5/37.9/10.6 -> 50.1/37.2/12.7 zero-support pairs 26.6% -> 28.7% names at 26+ menus 21.0% -> 20.8% The support-gate comparison was also carrying scratch-script numbers, so the census now measures it directly: of the alignments naming alone would accept, the shipped gate keeps 87.2% and strict exclusivity 42.4% (was 84.6%/41.9%). The step-4 replay follows the new retention rule, so rescued rows read 148, not 236. CodeRabbit's recognized-zero-support case is now covered explicitly: a section whose names the command tree knows, where no named menu accepts any of them, must keep nothing — distinct from silence, which keeps everything. Part of #131, #58, #61

mobileskyfi · 2026-08-01T02:45:38Z

@coderabbitai review

coderabbitai · 2026-08-01T02:45:43Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

coderabbitai

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

briefings/B-0024-command-prose-join.md (1)
410-414: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

State that acceptance cannot promote a row by itself.

“Nothing reaches high on acceptance” and “never for a name accepted at 26+ menus” imply that name can never be high. Lines 368-370 show that name can be high when the section directly aligns with a menu. Qualify both statements with “from acceptance alone” or “from field existence alone.”
Proposed wording
-Ubiquitous names cannot reach `high` on acceptance.
+Ubiquitous names cannot reach `high` from acceptance alone.

-never for a name accepted at 26+ menus
+never from field existence alone, including names accepted at 26+ menus
Also applies to: 560-562
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@briefings/B-0024-command-prose-join.md` around lines 410 - 414, Qualify the
statements in the “Field existence” and “Ubiquitous names” bullets, including
the corresponding passage around the later repeated wording, to say that
acceptance or field existence alone cannot promote a row to high. Preserve the
distinction that direct section alignment with a menu can still make name high.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@briefings/B-0024-command-prose-join.md`:
- Around line 264-265: Reconcile every published census percentage and
explicitly state its denominator: in briefings/B-0024-command-prose-join.md
lines 264-265, change the unambiguous count to 1,446 / 3,316 = 43.6% (or
otherwise name its denominator); on lines 287-291 and 493-495, recalculate the
spurious-path percentage from 128 / 433 = 29.6%; on lines 569-572, change the
value to 1,271 / 4,587 = 27.7%; and in src/mcp.ts lines 799-801, identify that
200 / 468 is the property-owning-section denominator.

In `@src/eval/command-prose-join.ts`:
- Around line 404-419: Update the census grading loop around gradeRow and its
acceptsByName lookup to build acceptance data separately for each candidate
section_id, using that section’s own property names as gradeRows does via
acceptance.all(id). Pass the section-scoped acceptance map into acceptsName and
ensure gateNone, gateSupport, and gateExclusive use the same scoped data,
replacing the corpus-wide acceptsByName behavior while preserving existing
candidate and path handling.
- Around line 380-385: Update the linkedPageOf construction in
command-prose-join.ts to resolve each menu path using the same per-path DISTINCT
page_id lookup as lookupProperty, rather than relying on unordered table-scan
results. Query non-null page IDs for each path and use that result consistently
when populating linkedPageOf, preserving the existing downstream
linkContributes, modeled before-label, and oldHigh/oldLow behavior.

---

Outside diff comments:
In `@briefings/B-0024-command-prose-join.md`:
- Around line 410-414: Qualify the statements in the “Field existence” and
“Ubiquitous names” bullets, including the corresponding passage around the later
repeated wording, to say that acceptance or field existence alone cannot promote
a row to high. Preserve the distinction that direct section alignment with a
menu can still make name high.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 23e59177-e45e-4ec2-99ec-b3387bfe7b94

📥 Commits

Reviewing files that changed from the base of the PR and between 9e2ca01 and a1205cd.

📒 Files selected for processing (9)

CHANGELOG.md
briefings/B-0024-command-prose-join.md
src/eval/command-prose-join.ts
src/mcp.ts
src/menu-paths.ts
src/property-confidence.test.ts
src/property-confidence.ts
src/query.test.ts
src/query.ts

📜 Review details

🧰 Additional context used

📓 Path-based instructions (8)

**/*.{ts,tsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Use Bun and TypeScript, including Bun-native runtime APIs such as bun:sqlite, Bun.serve, and bunx.

Rosetta uses Bun and TypeScript; prefer bun, bun test, and make verify where applicable, rather than Node/npm-oriented substitutes.

Files:

src/property-confidence.test.ts
src/mcp.ts
src/query.test.ts
src/menu-paths.ts
src/eval/command-prose-join.ts
src/property-confidence.ts
src/query.ts

src/**/*.{ts,tsx}

📄 CodeRabbit inference engine (AGENTS.md)

Rosetta is read-only documentation/schema context and must not connect to or modify a user's router.

Files:

src/property-confidence.test.ts
src/mcp.ts
src/query.test.ts
src/menu-paths.ts
src/eval/command-prose-join.ts
src/property-confidence.ts
src/query.ts

**/*.{ts,tsx,md}

📄 CodeRabbit inference engine (AGENTS.md)

Keep the attribution boundary visible when community RouterOS skills surface, because they are supplemental rather than official MikroTik documentation.

Files:

src/property-confidence.test.ts
CHANGELOG.md
src/mcp.ts
src/query.test.ts
src/menu-paths.ts
briefings/B-0024-command-prose-join.md
src/eval/command-prose-join.ts
src/property-confidence.ts
src/query.ts

*

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Keep this Copilot instructions file short; put substantive rules in narrow instruction files under .github/instructions/*.instructions.md.

Files:

CHANGELOG.md

**/*.md

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.md: Keep each project documentation file limited to its canonical role; prefer the documented canonical home instead of creating a new top-level Markdown file.
Do not duplicate content from this routing index; move operational detail, schema blocks, or long rule lists to their canonical documentation or instruction files.
Apply the repository's Markdown linting and fenced-code conventions, while excluding LLM instruction files where the repository explicitly requires that exclusion.

Files:

CHANGELOG.md
briefings/B-0024-command-prose-join.md

CHANGELOG.md

📄 CodeRabbit inference engine (CLAUDE.md)

Record user-visible shipped changes in CHANGELOG.md under [Unreleased] or release sections.

Files:

CHANGELOG.md

src/{query,mcp,browse}.ts

📄 CodeRabbit inference engine (AGENTS.md)

Route MCP, query, classifier, TUI, and canonicalizer behavior through shared core code, usually src/query.ts; keep src/mcp.ts and src/browse.ts as thin adapters.

Files:

src/mcp.ts
src/query.ts

briefings/B-*.md

📄 CodeRabbit inference engine (CLAUDE.md)

Store grounded research and decision support in briefings/B-*.md.

Files:

briefings/B-0024-command-prose-join.md

🧠 Learnings (1)

📚 Learning: 2026-08-01T01:16:11.873Z

Learnt from: mobileskyfi
Repo: tikoci/rosetta PR: 134
File: src/mcp.ts:790-802
Timestamp: 2026-08-01T01:16:11.873Z
Learning: In src/property-confidence.ts and src/mcp.ts, treat documentation sections that cite a requested RouterOS menu but describe a different menu as cross-reference-only evidence and assign them medium, not high, confidence. The support gate must use the properties associated with the section's cited menus, preventing incidental mentions—such as /ip/firewall/filter in /interface/bridge/filter documentation—from qualifying for high confidence.

Applied to files:

src/mcp.ts
src/property-confidence.ts

🪛 LanguageTool

briefings/B-0024-command-prose-join.md

[style] ~290-~290: Consider an alternative for the overused word “exactly”.
Context: ...xcellent or entirely spurious, which is exactly the shape a support-ratio filter can ...

(EXACTLY_PRECISELY)

[style] ~354-~354: ‘vid’ is informal. Consider replacing it.
Context: ...page, which happens to document its own vid, so the lookup returned two medium r...

(VID)

[style] ~374-~374: Consider an alternative for the overused word “exactly”.
Context: ...ce, but it is still evidence, and it is exactly the right thing to break a tie the sect...

(EXACTLY_PRECISELY)

[style] ~427-~427: Consider simply using “of” instead.
Context: ...: when the command tree has never heard of any of the section's property names it cannot ...

(OF_ANY_OF)

🔇 Additional comments (10)

src/mcp.ts (1)

792-798: LGTM!

CHANGELOG.md (1)

39-40: LGTM!

briefings/B-0024-command-prose-join.md (1)

157-157: LGTM!

Also applies to: 189-194, 206-211, 243-263, 266-286, 292-338, 340-409, 415-441, 486-493, 496-516, 556-559, 563-568, 573-574

src/eval/command-prose-join.ts (2)

32-39: LGTM!

Also applies to: 68-75, 230-231

403-403: 📐 Maintainability & Code Quality

No shared related helper to move.

src/query.ts only binds related as a SearchAllRelated object; it does not define the string ancestor/descendant related helper.

src/menu-paths.ts (1)

33-38: LGTM!

src/property-confidence.ts (1)

11-13: LGTM!

Also applies to: 31-33, 92-119, 121-163, 168-201, 203-246, 248-267

src/property-confidence.test.ts (1)

104-120: LGTM!

src/query.ts (1)

1082-1124: LGTM!

Also applies to: 1126-1160

src/query.test.ts (1)

211-323: LGTM!

Also applies to: 1119-1196

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

…nator `gradeRows` scores `SELECT DISTINCT name` for a section's properties; the census passed the raw list. A duplicated name is then weighted twice, which can flip which menu wins support — so the census was not measuring the shipped rule. It was also over-reporting the result: high labels surviving 1,119 (33.4%) -> 1,054 (31.5%) demoted to medium 2,230 -> 2,295 support-gate retention 87.2% -> 84.4% rescued rows 148 -> 153 Published figures corrected accordingly, in the briefing, the CHANGELOG and the property-confidence.ts header. Separately, several published percentages had been recomputed from a stale denominator or shipped without one. The census now prints the denominator at source — the M4 pair population, the unambiguous-alignment base, and the unscorable-row count as a fraction — rather than leaving it to be inferred, and the derived figures are regenerated: unambiguous alignments 1,446 (44.9%) -> 1,446 / 3,316 (43.6%) (section, path) pairs 433 -> 446 90-100% support 42.5% -> 41.3% names at one menu 36.9% -> 37.0% unscorable rows 1,271 (29.7%) -> 1,271 / 4,587 (27.7%) The MCP tool description now names its denominator too: 200 of the 468 documentation sections that carry properties, not 42.7% of all sections. Not changed: `linkedPageOf` resolving a menu's page without ORDER BY. Flagged as depending on table-scan order, but `commands.path` is `TEXT NOT NULL UNIQUE`, so at most one row exists per path and both the census and `lookupProperty` are deterministic. Verified on the artifact: 0 paths carry more than one page_id. Part of #131, #58, #61

mobileskyfi and others added 2 commits July 31, 2026 14:56

Copilot AI review requested due to automatic review settings August 1, 2026 00:18

Copilot started reviewing on behalf of mobileskyfi August 1, 2026 00:18 View session

Copilot AI reviewed Aug 1, 2026

View reviewed changes

Comment thread src/mcp.ts Outdated

Comment thread src/menu-paths.ts

Comment thread src/property-confidence.ts Outdated

coderabbitai Bot reviewed Aug 1, 2026

View reviewed changes

mobileskyfi commented Aug 1, 2026

View reviewed changes

Comment thread src/query.ts Outdated

Comment thread briefings/B-0024-command-prose-join.md Outdated

mobileskyfi added 2 commits July 31, 2026 18:15

coderabbitai Bot reviewed Aug 1, 2026

View reviewed changes

Comment thread briefings/B-0024-command-prose-join.md Outdated

Comment thread src/eval/command-prose-join.ts

Comment thread src/eval/command-prose-join.ts

mobileskyfi requested a review from Copilot August 1, 2026 03:09

Copilot started reviewing on behalf of mobileskyfi August 1, 2026 03:10 View session

Copilot AI reviewed Aug 1, 2026

View reviewed changes

mobileskyfi merged commit 377bcd8 into main Aug 1, 2026
11 of 14 checks passed

mobileskyfi deleted the b0024-join-census-and-confidence branch August 1, 2026 05:05

Uh oh!

Conversation

mobileskyfi commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Step 3 — Option A is a ranking signal, not a key

Step 4 — grade the row, not the query branch

The candidate set had to move too

Measured

The support gate, and why the eval earned it

Effect on the motivating defects

Eval fixture change — please read this one

Not in scope

Verification

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mobileskyfi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Codex Review

Uh oh!

Uh oh!

Uh oh!

mobileskyfi commented Aug 1, 2026

Uh oh!

coderabbitai Bot commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mobileskyfi commented Aug 1, 2026 •

edited

Loading

coderabbitai Bot commented Aug 1, 2026 •

edited

Loading

mobileskyfi left a comment •

edited

Loading

coderabbitai Bot commented Aug 1, 2026 •

edited

Loading