fix(website): docs search indexes shell comments as headings

[vivek7405]

Problem

The docs search index treats shell comments inside code samples as page headings, and scores them as such.

website/app/api/search/route.ts:35-39 builds each entry's headings by filtering the page's generated markdown for lines starting with #:

headings: page.markdown
  .split('\n')
  .filter((line) => line.startsWith('#'))
  .map((line) => line.replace(/^#+\s*/, '').trim()),

There is no fence tracking, so a line inside a fenced code block counts. Docs samples routinely open a line with # , because that is a shell comment. /docs/getting-started alone contributes about eight, including # scaffold a new app (no global install needed), # full-stack app (pages + API + components + Drizzle/SQLite + gallery), and # auto-detected: scaffolding through bun implies --runtime bun. Others live in website/app/docs/backend-only/page.ts and website/app/docs/deployment/page.ts.

Two consequences. A heading match scores 5 per term in GET (route.ts:53-56), well above a body match, so a query matching words in a shell comment ranks that page as though the comment were a section title. And whatever consumes headings for display shows text that is not a heading on the page at all.

This is long-standing and independent of the <code-block> rename; the markdown has carried fenced samples since llms.txt landed (#261).

Implementation plan

Decision. Add fence tracking to the heading extraction, and KEEP fenced code in the body match corpus at its existing weight. Both rulings are settled below; nothing here is left to the implementer's judgement.

Measured against the live corpus (43 doc pages, read through getDocPages()): 736 lines start with #, but only 683 sit outside a fence. So 53 phantom headings are indexed today, spread over 9 of the 43 pages, 8 of them on /docs/getting-started. Separately, 4605 distinct tokens appear ONLY inside fenced samples (/docs/components contributes 355 of them, /docs/websockets 307).

Ruling 1: fence-tracked headings, extracted as a pure helper. The filter moves out of the route into website/lib/utils/doc-headings.ts as extractHeadings(markdown), beside faq.ts and frontmatter.ts. website/AGENTS.md keeps app/ routing-only and lib/utils/ for pure compute, and a helper is directly unit-testable where a closure inside buildIndex() is only reachable over HTTP. The fence detector is line.trimStart().startsWith('```'), matching the one bodyToMarkdown already runs in its normalisation pass (website/lib/docs-llms.server.ts:206), so the two passes cannot disagree about where a fence begins. The toggle cannot desync: fences are emitted in pairs at docs-llms.server.ts:195-197, and a doc page's html template cannot contain a literal backtick (root AGENTS.md invariant 9), so no sample can open a stray fence.

Ruling 2: fenced code stays in text (route.ts:40), at the same weight, and the snippet keeps drawing from the same string. The scoring in GET (route.ts:55-62) is, per term: title contains it, +10. Each heading containing it, +5, accumulating over headings. Body text contains it, +1, boolean rather than a frequency count. A code-only match therefore already sits at the floor and cannot outrank a title or a heading hit. The bug was never that code is searchable, it was that a shell comment got promoted into the 5-point heading tier and could repeat, which is how /docs/backend-only reaches 16 for localhost today off nothing but # → http://localhost:8080 lines. Fixing the heading filter removes the promotion and the rest of the scoring is already correct. Dropping code from text would turn a ranking bug into a zero-results bug for 4605 tokens that exist nowhere else on their page, and readers search a docs site for exactly those: a flag, an env var, a header name, an import specifier.

No third weight tier for code. It would need a second lowercased corpus per entry, and it could only change an outcome when a page matches inside a sample but not in prose, which is precisely the case the flat +1 body tier already scores at the floor. Precedent for the two-tier split: Algolia DocSearch, which the Tailwind CSS docs consume, carries hierarchy.lvl0 through lvl6 for the heading path plus one flat content field for everything else, with no separate code tier (/home/vivek/Documents/Projects/frameworks/tailwindcss.com/src/components/search.tsx:124-136; the crawler's selector config lives on Algolia's side, so the repo shows only the consumption shape).

Rejected:

• Drop fenced code from the text corpus. 4605 code-only tokens across the corpus become unsearchable, which is a worse failure than a mis-ranked hit.
• Give fenced code its own lower weight tier. Costs a second corpus per entry and cannot produce an ordering the flat +1 body tier does not already produce.
• Strip in-fence lines before indexing and keep one corpus. Same loss as above, and it also empties the snippet for a code-only hit.
• Change what bodyToMarkdown emits. Its output is also /llms.txt and /llms-full.txt, where the fenced samples are the point. The index is the consumer that must get smarter.
• Lower or de-duplicate the accumulating +5 per heading. Out of scope and not wrong: a term recurring across several REAL section headings is genuine evidence of relevance, unlike a repeated shell comment.
• Keep the loop inline in buildIndex(). Testable only through the HTTP endpoint, which does not expose headings at all.
• Detect a fence with the bare line.startsWith('```') the sketch above uses. It disagrees with bodyToMarkdown, which tolerates leading whitespace on a fence line.

Steps

1. Land fix(website): a stray decoded angle bracket eats code samples from llms.txt #1261 first, then write this issue's expectations against the corpus it produces. See the coupling note under Implementation notes.
2. Add website/lib/utils/doc-headings.ts, exporting extractHeadings(markdown: string): string[]. Pure, no imports. Walk the lines, toggle inFence on line.trimStart().startsWith('```') and skip that line, and outside a fence push line.replace(/^#+\s*/, '').trim() for a line starting with #. Skip an empty remainder, since an empty heading can never match a term.
3. In website/app/api/search/route.ts, import extractHeadings and replace the headings filter chain (currently L36-L39) with headings: extractHeadings(page.markdown),. Update the comment above it (L34-L35) to state that fenced lines are skipped, so a shell comment is not a heading.
4. Leave text (L40), the scoring loop (L55-L62), and the snippet slice (L63-L73) exactly as they are. Record ruling 2 in the module's header comment (L1-L14): code samples are searchable, at body weight, deliberately.
5. Run webjs check in website/ and the website test suite.

Tests

All in website/test/ssr/docs-search.test.ts.

• Unit, the helper against a fixture. extractHeadings on a hand-written markdown string that carries a ## heading, a ### heading, a fenced block containing # not a heading, and an indented fence, and deepEqual against the two real headings only.
• Unit, the helper against the real corpus. getDocPages(), take /docs/getting-started, and assert the extracted headings include Quick Start and Using the scaffold (a real heading at both levels survives), exclude scaffold a new app (no global install needed) (the shell comment authored at website/app/docs/getting-started/page.ts:17), and number at least 10. The length floor is what stops an empty list passing the exclusion, the trap flagged below.
• Endpoint, the ranking the fix exists for. search('localhost') and assert every hit scores exactly 1. The term appears in the corpus only inside # → http://localhost:8080 shell comments, so after the fix every page matching it sits at the body floor. Today /docs/backend-only scores 16 and /docs/getting-started 11. Note in the test that a future REAL heading containing localhost would legitimately change this number.
• Counterfactual. Restoring the bare .filter((line) => line.startsWith('#')) reds all three: the fixture test gains not a heading, the corpus test gains the shell comment, and the endpoint test sees 16 where it wants 1.

Doc surfaces

• None. This is website-internal ranking behaviour with no public or agent-facing surface: no export, no CLI flag, no webjs config key, no convention. The framework docs, the skill at .agents/skills/webjs/, the scaffold templates, the MCP, and the changelog are all N/A. The decision itself is recorded in the route module's header comment (step 4), which is where the next reader of the index will look.

Implementation notes (for the implementing agent)

Where to edit

• website/app/api/search/route.ts, buildIndex() at L28-L43. The heading filter is L36-L39; the scoring that makes it matter is in GET at L45-L81, specifically L55-L62.
• website/lib/docs-llms.server.ts is where the markdown comes from. bodyToMarkdown (L150, module-local, NOT exported) restores the fences at L195-L197 and normalises around them at L202-L211; that is the format to track.

Coupling with #1261

• fix(website): a stray decoded angle bracket eats code samples from llms.txt #1261 fixes a corpus bug in website/lib/docs-llms.server.ts and this index consumes that corpus, so land fix(website): a stray decoded angle bracket eats code samples from llms.txt #1261 FIRST and write these test expectations against the post-fix(website): a stray decoded angle bracket eats code samples from llms.txt #1261 markdown. The two are file-independent, so they will not conflict in the diff, but /docs/metadata-routes currently reaches the corpus missing 5 of its 9 samples, and restoring them changes both the heading lists and the corpus-wide counts quoted above.
• The coupling is expectation-level only, not a mechanism risk. fix(website): a stray decoded angle bracket eats code samples from llms.txt #1261's stray-angle-bracket strip deletes whole CODE<n> sentinels BEFORE the fences are generated from the survivors (docs-llms.server.ts:195-197), so it drops complete blocks and can never leave a half-open fence for this fix's toggle to trip over.

Landmines / gotchas

• The index is built once and cached in module scope (let index at L26, if (index) return index at L29). A dev server holds a stale index across edits, so restart rather than concluding a fix did not work.
• Do NOT fix this by changing what bodyToMarkdown emits. Its output is also /llms.txt and /llms-full.txt, where fenced samples are the point. The index is the consumer that needs to be smarter.
• /docs/metadata-routes currently loses 5 of its 9 samples from that markdown for an unrelated reason, tracked as fix(website): a stray decoded angle bracket eats code samples from llms.txt #1261. Do not be confused by it while testing, and do not fix it here.
• Headings are also emitted from real ## / ### markup, so the fix must keep those. A test asserting only the absence of shell comments would pass on an empty list, which is why the corpus test carries a length floor.
• One correction to the Problem statement above: headings is an index-internal field and never reaches the client. GET returns { path, title, score, snippet } (route.ts:74) and website/components/doc-search.ts renders only the title and the snippet, so the live damage is the scoring, not a display of fake headings. That is worth knowing before hunting for a rendering bug that does not exist.

Invariants to respect

• Root AGENTS.md invariant 11 for any prose added.
• Website-only change; no framework surface.

Acceptance criteria

• A line-leading # inside a fenced code sample is not indexed as a heading
• Real ## / ### headings are still indexed
• A counterfactual proves the new assertions fire when the fence tracking is removed
• Fenced code REMAINS in the text match corpus at its existing weight, and the decision is recorded in the route module's header comment
• extractHeadings lives in website/lib/utils/doc-headings.ts and is unit-tested directly, not only through the endpoint
• The fence detector tolerates leading whitespace, matching bodyToMarkdown
• A localhost query returns every hit at score 1, where /docs/backend-only scores 16 today
• website/test/ssr/docs-search.test.ts covers the fixture, the real corpus, and the endpoint ranking
• fix(website): a stray decoded angle bracket eats code samples from llms.txt #1261 landed first and the expectations are written against its corpus