Inline parse cost linear in spans per region, not quadratic (#109)

[wildthink]

Fixes #109, kept out of the directives PRs as you asked. Independent of #120 — it's the pass underneath, and touches different functions in InlineParser.

The two scans

Both come from the same place, which is why one invariant removes both.

Claimed-range membership was a full array scan. scanEscapes and collectDelimiterRuns asked it once per character, scanLinkFamily once per candidate. So the cost of every pass after the first scaled with how much the earlier passes had claimed — worst for code spans, which claim first and are consulted by all three.

buildTree decided containment pairwise. isChild looped over every span in the region, called once per span via inRegion.filter { !isChild($0) }, plus a second inRegion.filter per emphasis to gather its children.

Both are avoidable for the same reason: the passes walk the string left to right and never look back, and claimed ranges are non-overlapping by construction. So a cursor over the sorted ranges answers membership in amortised constant time — the answer for index i only ever involves the first range ending after i. Containment falls out of the same invariant: sorting by start ascending / length descending puts every span immediately after the one that contains it, so buildTree becomes a single ordered walk with the cursor threaded through the recursion.

That makes the non-overlap invariant load-bearing for cost, not just for correctness, so I noted it in the file header — a pass that claimed a partially overlapping span would now break the walk, not just the tree.

ClaimedIndex sorts in its own initialiser rather than documenting an ordering precondition, so no call site can get it wrong. Three sorts per parse, and they don't show up.

Numbers

ms per DocumentAST.parse of one paragraph with n spans (M-series, debug):

n	code before	code after	links before	links after	emphasis before	emphasis after
40	1.23	0.11	0.59	0.27	0.78	0.17
80	4.01	0.20	1.54	0.59	2.30	0.34
120	8.51	0.29	2.96	0.78	4.26	0.51
240	32.92	0.58	9.31	1.56	16.07	1.02

6x the spans cost ~30x the parse before and ~6x now. At n=240 code spans that's 57x less work — the case where three passes were each rescanning 240 claimed ranges per character.

Ordinary documents won't notice; nothing here changes the constant at low density. What it buys is that a paragraph with a few hundred inline spans stops blowing the frame budget on its own.

That it changes nothing

The risk in this change is behavioural, not performance, so that's what I tested hardest.

InlineSpanDensityTests.corpusFingerprint folds the parsed tree of 4000 pseudo-random inputs into a single value. The corpus is built from bare and paired delimiters, escapes, and the openers of every claimed-span construct, so it's dense in half-formed, overlapping and nested spans rather than in valid markdown — the shapes I wouldn't have thought to write by hand. Deterministic LCG so both sides see identical input, hand-rolled FNV because Hasher is per-process seeded.

The baseline b4b562f2c6be080b is recorded on the pre-rewrite parser at eaed9dd — same idea as your GoldenCorpusTests. It passes on both parsers, which is the point; it's there to fail if the walk ever diverges.

The five scaling assertions are the regression detectors, and they fail on the old parser — 14.6x (links), 15.8x (highlight), 20.1x (emphasis), 25.3x (mixed), 31.0x (code) against a 12x bound. Bound is 2x linear, measured is ~5.3-6x, and it's a minimum of 7 runs rather than a mean, since scheduler noise only ever adds time.

306 tests green, demo builds.

Two deletions worth flagging

Span.containerContent and equalRange are both gone — the ordered walk derives the emphasis content range inline and consumes the span itself before recursing, so neither had a caller left.

equalRange was guarding a case that can't arise: two spans with identical ranges. Under the old code both would be excluded from top and dropped; under the new one the second is skipped as nested in the first. Different handling of an impossible input, and I'd rather say so than have you find it.

The new walk also skips anything nested inside a non-container span. Every claimed span but emphasis is opaque today so nothing ever is, but the old code would have emitted such a span after its parent with the cursor already past it — the skip keeps the walk well-formed instead.

[luca-chen198]

Reviewed, and the change is right — the cursor over the sorted ranges and the ordered walk are both the obvious thing in hindsight, which is the good kind of fix. Sorry for the slow reply.

main moved under you while this sat: #118 landed and relaxed the exact invariant this rests on. A Markdown link may now contain a claimed code span inside its label, so claimed ranges are no longer strictly disjoint — they're disjoint or properly nested. The two conflict hunks are precisely that collision (Span.containerContent, and the overlap check in scanLinkFamily), and overlaps() -> Bool can't express #118's rule, so resolving in favour of this branch would have silently reverted it with nothing going red.

Rather than bounce it back, I've rebased it for you and force-pushed — maintainer_can_modify was on. Shout if you'd rather redo it yourself and I'll drop mine.

What the rebase changed

• ClaimedIndex gains overlapping, which enumerates every claimed range meeting a candidate rather than answering yes/no. Only the link case needs it; everything else still short-circuits on the first overlap. It peeks forward from the cursor instead of advancing it, so the left-to-right walk is untouched and the cost property holds.
• contains needed nothing. A nested range sorts after its container, which already covers it — that fell out of your design rather than needing a patch, which is a decent sign the shape was right.
• The corpus baseline was re-recorded on the pre-rewrite parser at the new merge base (1a2bd74), since b4b562f2c6be080b predates Allow inline code spans in Markdown link labels #118. On current main that corpus fingerprints to b74649ffbbbe237a, and the rewrite reproduces it exactly — so it's still a pure performance change, now on top of Allow inline code spans in Markdown link labels #118 as well. I added a note that it must only ever be re-recorded on a parser predating the rewrite, otherwise it just ratifies whatever the rewrite does.
• The file header was saying both things at once after the merge — your added paragraph asserts non-overlap, Allow inline code spans in Markdown link labels #118's edit three lines above documents the exception. Restated as "disjoint OR properly nested; a link label may hold one, nothing else may", with partial overlap still called out as the thing that would break both the cursor and the sort.

One substantive change: the bound moved from 12x to 8x

Your scaling assertions are the right idea, but I measured them on the old parser and two of the five don't detect anything. On this machine, pre-rewrite:

	before	after
code	27.8x, 27.0x	5.6x, 5.3x
mixed	23.0x, 23.3x	4.0x, 4.7x
emphasis	16.6x, 16.6x	5.1x, 5.2x
links	11.5x, 12.1x	3.6x, 4.2x
highlight	11.3x, 11.8x	3.2x, 2.8x

highlight passes the 12x bound on the old parser outright, and links straddles it. Your PR quotes 14.6x and 15.8x for those two, so this looks machine-dependent rather than wrong — which is itself the argument for more margin. Worst case after the rewrite is 5.6x, best case before it is 11.3x; 8 is the geometric midpoint of that gap, ~1.4x either side. Happy to hear an argument for a different number.

Verification

312 tests green over repeated runs, #118's link-label tests included. The corpus fingerprint matching main byte-for-byte is what actually convinced me this is behaviour-neutral — a test list can miss a shape, 4000 adversarial inputs folding to the same value can't.

Thanks for splitting this out of the directives work rather than folding it in. It's much easier to be confident about in isolation, which is the whole point.

[wildthink]

Heads-up on an overlap with #118, which I found after opening this.

#118 (inline code inside link labels) touches the two exact things this PR rewrites — Span.containerContent and scanLinkFamily's overlap check — so the patches don't apply to each other. Whichever lands second needs a rebase, and I'm happy for that to be mine.

There's a substantive interaction underneath the textual one, in this PR's favour, and I checked it rather than assuming:

#118 introduces nesting into claimed — a link that keeps a code span inside its label — and ClaimedIndex was written against the non-overlap invariant. Nesting is safe: sorted by location, the enclosing range comes first and ends last, so it dominates every query inside it. What would break the cursor is partial overlap, which #118 still rejects. My doc comment says "non-overlapping" where it should say "never partially overlapping" — I'll tighten that wording regardless of ordering, since it's the invariant that's actually load-bearing.

#118 gets smaller on top of this PR. It extends containerContent to make links containers so label-nested spans are suppressed at top level; the ordered walk here already skips anything nested inside the span it just consumed, so that change isn't needed. I ported #118's rule onto the cursor locally — about 10 lines — and all five of its parser tests plus its styler test pass with containerContent deleted.

And the composition is equivalent to #118 alone. The 4000-input corpus fingerprint is b74649ffbbbe237a both for #140+#118 and for #118 on main — identical trees on every input, not just on its own tests.

One ordering argument worth having explicitly: #118's new check replaces a short-circuiting contains with claimed.filter { … }, which allocates per candidate at every position. On main that costs links 0.59ms → 1.05ms at 40 spans and 9.31ms → 12.77ms at 240. On top of this PR they stay linear. So landing this first means #118's rebase deletes code instead of adding it, and #118 doesn't land a slowdown on the path #109 is about.

Also worth stating: when #118 lands, corpusFingerprint's baseline has to be re-recorded to b74649ffbbbe237a. That's the test working as intended — it's a change-detector, and #118 is a deliberate parse change, so it should fail and be re-recorded rather than loosened.

#120 merges cleanly with this one — the directive hook is inside matchClaimedSpan, which this PR doesn't touch.

I've left #118 a note with the same findings and offered them the port.

[wildthink]

Thanks for rebasing this rather than sending it back, and for the CI catch.

The red was mine and your diagnosis is exact. "Minimum of several runs, noise only adds time" is sound for an absolute measurement and simply doesn't transfer to a ratio — under load there's no quiet run to floor against, and the smaller number inflates proportionally more. That the pre-rewrite floor sits below the post-rewrite CI reading settles it: no bound separates the two parsers, so there was no number for me to argue for.

Ignore my comment above — it crossed with your merge, and by then you'd already found all of it independently, including that contains needed no change and that the baseline had to be re-recorded on a pre-rewrite parser. My mistake was checking PR state at the start of a session and posting hours later without re-reading it.

Your overlapping is the right shape — peeking from the cursor rather than advancing keeps both the walk and the cost property intact, and confining it to the link case leaves everything else short-circuiting.

Rather than argue for a different bound I've opened #146, which asserts on counted work instead of elapsed time: 6.0x for 6x the spans against 33.9x with the old pairwise containment restored, both exact and machine-independent. Those go back on CI; your timed ones stay opt-in for absolute numbers.