fix(website): a stray decoded angle bracket eats code samples from llms.txt

[vivek7405]

Problem

/docs/metadata-routes reaches the llms corpus with 4 of its 9 code samples, and loses the paragraphs between them too. This affects /llms-full.txt, /docs/metadata-routes/llms.txt, and the docs search index, all of which are built from the same markdown.

It is not a truncation of the page tail, which is what it looks like at first. The mechanism is a decode-then-strip ordering bug in website/lib/docs-llms.server.ts:

1. oneLine() (L121) strips tags first and THEN decodes entities, so < in a paragraph becomes a bare < in the rewritten output.
2. website/app/docs/metadata-routes/page.ts:52 contains exactly that, a paragraph teaching XML escaping: a value with <code>&amp;</code> or <code>&lt;</code>.
3. bodyToMarkdown then runs its generic tag strip, .replace(/<[^>]+>/g, ' ') at L223, over a body that now contains that stray <. The regex matches from it to the next > anywhere later in the document, deleting everything in between, including the CODE<n> sentinels standing in for the extracted code blocks.

Confirmed by removing the angle-bracket decode from oneLine(), which restores all 9 samples and takes site-wide authored-vs-fenced mismatches from 1 to 0. That is a diagnostic, not the fix: the decode is what makes prose about markup readable in the corpus.

Any docs page whose prose teaches an escaped < is exposed to this. /docs/metadata-routes is currently the only one that trips it.

Implementation plan

Decision: strip tags at every stage, decode entities exactly once at the end. Concretely, delete the entity-decode chain from oneLine() (L124-L130), leaving it a tag strip plus a whitespace collapse, and let the single decodeEntities(body) at L227 do all prose decoding. That call already sits after the generic strip at L223, so the pipeline becomes correct by removal rather than by reordering, and no stage ever sees text that an earlier stage decoded. Code samples are unaffected: they are captured and decoded on their own path (L205) and restored at L230, after the body decode.

The same inversion exists on the description path in extractPage, where L290 and L294 both call oneLine(decodeEntities(...)), decoding before the strip. A metadata description or first paragraph teaching <code> loses the decoded tags there today. Fix it under the same rule: hoist the normalization into one exported helper plainText(s) that runs oneLine first and decodeEntities after, and call it from both sites.

Why this variant of "do not let decoded text reach a later tag strip" and not the other one the earlier draft floated: running the generic strip BEFORE the per-block rewrites is not implementable. The generic strip deletes every tag, including the <h2> / <p> / <li> boundaries that the rewrites at L211-L219 match on, so a strip-first body arrives at those rewrites as one undifferentiated blob with no headings and no list items. The ordering constraint therefore has exactly one solution: strip first at every stage, decode last, once.

A side effect to expect and accept: prose stops being double-decoded. Today a paragraph passes oneLine's decode and then the L227 decode, so authored <div> arrives in the corpus as <div>. After the fix it arrives as <div>, which is what the page author wrote and what an agent reading the corpus should see.

Rejected:

• Generic strip before the per-block rewrites: destroys the tag boundaries those rewrites match on, so headings and list items are lost.
• Make the sentinels unmatchable by the generic strip: bounds the damage to prose instead of removing the cause, so a paragraph is still eaten silently.
• Escape or re-protect oneLine()'s decoded output before it rejoins the body: adds an encode and decode round trip to hide an ordering bug that a deletion fixes outright.
• Rewrite the prose at website/app/docs/metadata-routes/page.ts:52: the documentation is correct, and the next page teaching escaping trips the same bug.

Steps

1. website/lib/docs-llms.server.ts, oneLine() (L121-L133): delete the seven entity replacements at L124-L130. Keep the tag strip (L123), the whitespace collapse (L131), and the trim (L132). Update its doc comment to state the rule, that it deliberately does not decode, because a decoded < re-entering a later tag strip is what eats the rest of the document.
2. Same file, add export function plainText(s: string): string next to oneLine, defined as decodeEntities(oneLine(s)) followed by a whitespace collapse and trim (the collapse is re-run because   and … only become whitespace-relevant after decoding). Document that it is exported for the same reason bodyToMarkdown is, so a unit test can drive it on a fixture.
3. Same file, extractPage() (L272): replace oneLine(decodeEntities(unescapeJs(...))) at L290 with plainText(unescapeJs(...)), and oneLine(decodeEntities(pMatch[1])) at L294 with plainText(pMatch[1]).
4. Same file, bodyToMarkdown(): no pipeline change. Delete the comment block at L193-L202, which describes this loss as live and unfixed, and put in its place a short note stating the invariant the file now holds (tags are stripped at every stage, entities are decoded once at L227 for prose and once at L205 for a captured sample).
5. website/test/ssr/docs-llms.test.ts: delete the KNOWN_TRUNCATED map (L79), the test the truncation exemption still describes reality (L81-L92), the comment block at L58-L78 that explains the exemption, and the KNOWN_TRUNCATED.has(page.path) skip at L104 inside every sample a page authors reaches the corpus, so that walk covers every page with no exemption.
6. Same test file, add the fixture tests below to the fixture section at the bottom.
7. website/AGENTS.md: add one bullet recording the ordering invariant so a future edit does not reintroduce it (see Doc surfaces).

Tests

• Unit, website/test/ssr/docs-llms.test.ts (fixture section at the bottom, driving the exported functions directly rather than planting content in a real docs page):

  • A paragraph teaching an escaped angle bracket followed by a code block keeps both. Drive bodyToMarkdown on a fixture body holding <p>a value with &lt;code&gt; here</p> followed by a <code-block> sample, and assert the output contains the decoded prose a value with <code> here AND the fenced sample. Today the stray < eats from mid-paragraph through the sentinel, so the fence is gone.
  • Prose is decoded exactly once. Drive bodyToMarkdown on a <p> containing <div> and assert it yields <div>, not <div>. This pins the double-decode that removing oneLine's decode chain also fixes, so a later reviewer does not "restore" it.
  • The description path strips before it decodes. Assert plainText('a value with <code> here') returns a value with <code> here. Today the decoded tag is deleted out of it.
  • Delete the KNOWN_TRUNCATED entry and the map with it, per step 5. That entry asserts the broken counts in both directions, so a correct fix reds it by design, and deleting it is the intended response rather than a weakening of the pin.

• Corpus diff, before and after, across all 43 pages. The change touches every page's output, so a single-page check proves nothing. Capture the baseline BEFORE making any edit:

  cd website
  node --input-type=module -e "const m = await import('./lib/docs-llms.server.ts'); process.stdout.write(await m.renderLlmsFull());" > /tmp/llms-before.txt
  # apply the fix, then
  node --input-type=module -e "const m = await import('./lib/docs-llms.server.ts'); process.stdout.write(await m.renderLlmsFull());" > /tmp/llms-after.txt
  diff -u /tmp/llms-before.txt /tmp/llms-after.txt | less

  Node 24 strips the types natively and #lib/env.ts resolves through website/package.json, so this needs no server and no build. What to look for:

  • /docs/metadata-routes gains 5 fenced samples (10 fence lines) plus the paragraphs that were swallowed with them. The corpus-wide count of fence lines should rise by exactly 10, and no page should lose one, which a grep -c for the fence marker over each file settles in one command.
  • Every other difference must be a prose line that previously double-decoded an entity and now decodes once. A diff line that is neither that shape nor on /docs/metadata-routes is unexplained and blocks the change.
  • grep -n '<\|>\|&\|"' /tmp/llms-after.txt: every remaining entity must trace back to source that double-escaped it deliberately. Ordinary prose about markup has to read as <div>, not <div>, or the fix has traded one defect for another.

• Counterfactual. Reverting the source fix alone (restoring the decode chain in oneLine, and the decode-before-strip order on the description path) while keeping the tests must red: both bodyToMarkdown fixtures, the plainText fixture, and the now-unexempted corpus walk every sample a page authors reaches the corpus, which reports /docs/metadata-routes: 9 authored, 4 fenced.

Doc surfaces

• website/AGENTS.md: add one bullet near the lib/ inventory (L106 names docs-llms.server.ts) stating the invariant, that the llms extractor strips tags at every stage and decodes entities exactly once at the end, because decoded text re-entering a tag strip deletes everything up to the next >. That is the only doc surface. The module is website-internal server code with no public export, no CLI flag, and no webjs config key, so the framework docs, the docs site, the skill, the scaffold, the MCP, and the changelog are all N/A.

Ordering against #1262

#1261 lands FIRST. It changes the corpus markdown, and #1262 (docs search indexes shell comments as headings) edits website/app/api/search/route.ts, which builds its index from getDocPages() in this same module. The two touch different files and will not conflict textually, but #1262's expectations are stated against corpus content that #1261 moves, and /docs/metadata-routes gains five samples plus paragraphs that #1262's heading extraction then has to walk. Landing #1261 first means #1262 is written and verified against the fixed corpus once. #1262 owns refreshing any shared expectation that shifts, because it is the consumer and the later merge.

Implementation notes (for the implementing agent)

Where to edit (anchors verified against origin/main after #1249, which moved this file by about four lines)

• website/lib/docs-llms.server.ts. oneLine() is L121-L133 (tag strip L123, entity decodes L124-L130, whitespace collapse L131, trim L132); bodyToMarkdown() is L154 (exported); the code-sample capture is L203-L207 (the captured text is decoded at L205); the block-level rewrites that call oneLine are L209-L221 (h1 through h4 at L211-L214, <li> at L216, <p> at L218, <blockquote> at L219); the generic strip is L223; the template-hole strip is L225; the single body decode is L227; the sentinel restore is L230-L232; decodeEntities() is L257; extractPage() is L272, with the two description calls at L290 and L294.
• website/app/docs/metadata-routes/page.ts:52 is the paragraph that triggers it. Do NOT fix by editing that prose: it is correct documentation, and the next page to teach escaping would hit the same bug.

Landmines / gotchas

• bodyToMarkdown is exported specifically so it can be unit-tested on fixtures (done in fix(website): one code-block element owns the grammar and focus stop #1249). Use that rather than planting content in a real docs page; there are existing fixture tests at the bottom of website/test/ssr/docs-llms.test.ts to copy the shape from. plainText is exported for the same reason and should carry the same rationale comment.
• website/test/ssr/docs-llms.test.ts pins this page's current broken counts in KNOWN_TRUNCATED ({ authored: 9, fenced: 4 }) at L79. The test the truncation exemption still describes reality (L81) asserts them in BOTH directions, so a correct fix FAILS it. That failure is the intended signal: delete the KNOWN_TRUNCATED entry and the map with it once the page reaches 9 of 9, and drop the KNOWN_TRUNCATED.has(page.path) skip at L104.
• The sentinel is written as the escape \uE000, not a literal byte, so the file stays diffable. Keep it that way.
• Prose readability in the corpus is the reason the decode exists at all. A fix that leaves <div> literal in /llms-full.txt trades one defect for another.
• The docs search index (website/app/api/search/route.ts) imports getDocPages from this module, so its output changes too. See the ordering note above.
• The comment block at L193-L202 inside bodyToMarkdown describes this loss as live and not fixed, and the comment at L58-L78 in the test file explains the exemption. Both become false with this change and must be rewritten rather than left standing.

Invariants to respect

• Root AGENTS.md invariant 11 for any prose added (no em-dashes, brand casing).
• website/AGENTS.md: one tokenizer, and code samples under /docs are <code-block>.

Acceptance criteria

• /docs/metadata-routes reaches the corpus with all 9 samples
• The paragraphs between them are present too, not just the code blocks
• oneLine() no longer decodes entities, and prose is decoded exactly once, at decodeEntities(body)
• The description path (extractPage L290 and L294) strips before it decodes, through the shared plainText helper
• A fixture test drives bodyToMarkdown on a paragraph containing an escaped < followed by a code block, and reds without the fix
• A fixture test pins single decoding: authored <div> reaches the corpus as <div>
• A fixture test drives plainText on text teaching <code> and reds without the fix
• The KNOWN_TRUNCATED entry in website/test/ssr/docs-llms.test.ts is DELETED, and per-page parity covers every page with no exemption
• Prose about markup still reads correctly in the corpus (<div> renders as <div>, not left escaped)
• The full-corpus output is diffed before and after across all 43 pages, and every difference is accounted for
• The stale comments at docs-llms.server.ts L193-L202 and at the test file's L58-L78 are gone
• website/AGENTS.md records the strip-then-decode invariant