feat: Go elides — step 3 of 3, four languages reduce instead of three - #34
Merged
Conversation
Step 3 of three, and the one that changes output. 56 fixed the order and
measured the precondition, 59 gave the drift gate Go symbols, 60 gave Go a
validator. This adds the scanner.
scanGoBraceSpans a Go brace scanner. Separate from scanBraceSpans for
60's reason: Go's raw string spans lines, takes no
escapes, and holds braces and backslashes, so a TS
scanner reads `C:\path\` as an unterminated template
literal and every brace after it as string content. No
regex-literal state, because Go has none.
GO_FUNCTION_HEADER /^func\b/ — a keyword test where TypeScript needs a
shape test, which is most of why Go went first.
FUNCTION_HEADER matches anything ending in `)`, so
CONTROL_FLOW_HEADER has to subtract if/for/while back
out; `^func` needs no subtraction and excludes struct
decls, composite literals, type switches and every
closure form by construction.
splitGoStatements Go ends statements at a newline, not at a `;`.
Semicolon insertion means gofmt-ed source has almost
none written down, so splitTypeScriptStatements finds
only the `}` boundaries and calls most bodies
indivisible — 50's overshoot, one language along.
Measured, frozen 80-file Go corpus at target 0.3:
application Go (cli/cli, gin, cobra) 40 files 32 reduce 27.46%
stdlib (golang/go src/) 40 files 25 reduce 19.42%
_test.go, across both 40 files 32 reduce 26.88%
source, across both 40 files 25 reduce 14.42%
56 projected 23-28%; application Go landed at the top of it and above this
repo's TypeScript at 21.22%. 56's "test files are the larger prize" is
confirmed end-to-end: nearly double the saving and half the fallbacks,
because Go's table-driven tests put large literal slices inside bodies.
The 287-file main corpus is 574/574 byte-identical, 0 rows differing across
17 fields — every Go path is gated on the language, so TypeScript and Python
do not move.
THE ORDERING PAID FOR ITSELF TWICE. Neutering 59's symbol patterns and
re-running step 3 reproduces 56's hazard on real input rather than in
simulation:
scanner-first (no 59) shipped
file-route fallbacks 43/80 20/80
rows where drift measured anything 55/80 80/80
median symbolsBefore 2 8
application Go aggregate 14.45% 27.46%
32 files elide at S_k = 0.0000 with astMeasured: true, both gates passing and
no fallback, on 1-5 symbols that are all type: and import:. accessibility.go
loses 78.4% of its tokens that way. With 59 in place the same 32 still pass —
correctly, since signature-preserving elision loses no symbols — but on a
median of 8 real symbols rather than on an import.
And 59 turned out to be a precondition for reduction, not a tax on it:
without Go symbols many files have no symbols at all, so 33's measurement
gate refuses them outright. The safety step and the reduction step were the
same step.
Go's fallback rate, which 56 could not measure: 20 of 80 (25%), of which 18
are CONSTRAINT_DIRECTIVE_LOST and 2 are drift. 56 expected Go's lower comment
density to make that gate fire LESS. It does not — it dominates exactly as it
does for TypeScript, at the same rate (24%, 15 of 62). Density was the wrong
variable; Go's defensive comment style is 52's still-open descriptive-present-
tense axis verbatim ("// This never happens in practice", "// Should never
happen, but we"). The largest remaining gain on Go is that gate, not the
scanner.
Ten existing tests asserted Go was not elidable and now fail on purpose;
they are updated in the same commit, as this repo's contract requires, with
the prior assertion recorded in each. language-support.test.ts moves its
"incidental symbol is not support" example from Go to Rust, which is in the
position Go used to occupy. The "elision reduces TypeScript/JavaScript and
Python only" explanation is corrected wherever it is emitted.
test/unit/go-region-elision.test.ts: 13 known-answer cases, 9 of which fail
against the unfixed engine. The 4 that pass both ways assert an absence and
are trivially true when the scanner selects nothing, so they are controls on
over-selection rather than evidence.
DECISIONS 61.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 3 of three, and the one that changes output. §56 fixed the order and measured the precondition; #33 shipped §59 (the drift gate can see Go symbols) and §60 (Go has a validator). This adds the scanner, and Go reduces.
Closes the sole open roadmap item — §7.3 of the status doc. Elision now reduces 4 of 17 probed languages instead of 3.
What it adds
scanGoBraceSpans— a Go brace scanner, separate fromscanBraceSpansfor exactly §60's reason. Go's`raw string spans lines, takes no escapes, and routinely holds{,}and\, so the TypeScript scanner reads`C:\path\`as an unterminated template literal and every brace after it as string content. A region boundary computed from that is not a function body. No regex-literal state, because Go has none.GO_FUNCTION_HEADER = /^func\b/— a keyword test where TypeScript needs a shape test, which is most of why Go went first.FUNCTION_HEADERmatches anything ending in), soCONTROL_FLOW_HEADERhas to subtractif/for/whileback out.^funcneeds no subtraction:type Point struct {,Config{,switch v := x.(type) {and every closure form (go func() {,defer func() {,handler := func() {) are excluded by construction.splitGoStatements— Go ends statements at a newline, not at a;. Semicolon insertion means gofmt-ed source has almost none written down, sosplitTypeScriptStatementsfinds only the}boundaries and calls most bodies indivisible — §50's overshoot, one language along. The boundary is a newline at depth 0, so a multi-line call, a composite literal and a nested block are each one span.'go'inREGION_ELISION_LANGUAGES, plus a Go arm onisSubstantiveRegion(stripGo).Measured
Frozen 80-file Go corpus, target 0.3, engine varied and nothing else:
cli/cli,gin,cobra)golang/gosrc/)_test.go, across both§56 projected 23–28% from the ceiling, borrowing TypeScript's conversion factor. Application Go landed at the top of that range and above this repo's TypeScript at 21.22% on the same engine — the newest language is the strongest. §56's "test files are the larger prize" is confirmed end-to-end: nearly double the saving and half the fallbacks, because Go's table-driven tests put large literal slices inside function bodies.
Adherence over the 57 reducing files: median 35.8%, 20 in the 25–35% band, 17 in 35–50%, 14 above 50% — the profile TypeScript has after §50.
The 287-file main corpus is 574/574 byte-identical, 0 rows differing across 17 fields. Every Go path is gated on
language === 'go', so TypeScript and Python do not move.The ordering paid for itself twice
Neutering §59's symbol patterns and re-running step 3 reproduces the configuration §56 warned about — the region scanner without the drift gate that can witness it — on real input rather than in simulation:
symbolsBefore32 files elide with
S_k = 0.0000,astMeasured: true, both gates passing and no fallback, on 1–5 symbols that are alltype:andimport:.accessibility.goloses 78.4% of its tokens that way. With §59 in place the same 32 still pass — correctly, since signature-preserving elision genuinely loses no symbols — but the verdict now rests on a median of 8 real symbols instead of on an import.And §59 is a precondition for reduction, not a tax on it. Without Go symbols many files have no symbols at all, so §33's measurement gate refuses them outright: fallbacks more than double and application Go reads 14.45%. The safety step and the reduction step turned out to be the same step.
An expectation of §56's that measurement contradicts
§56 could not measure Go's fallback rate and said so. Now: 20 of 80 (25%) — 18
CONSTRAINT_DIRECTIVE_LOSTand 2SEMANTIC_DRIFT_EXCEEDED.§56 expected Go's lower comment density to make the constraint gate fire less. It does not. The gate dominates Go's fallbacks exactly as it dominates TypeScript's, at the same rate within noise — 25% against 24% (15 of 62). Density was the wrong variable: what trips it is Go's defensive comment style, and it is §7.5's still-open axis verbatim —
// This never happens in practice,// Should never happen, but we,// The user data should always,// Must read more data.Every one is descriptive present tense, which §52 deliberately did not attempt.So the largest remaining gain on Go is not in the scanner. It is in that gate.
Tests
test/unit/go-region-elision.test.ts— 13 known-answer cases, the set thewiden-languageskill requires before a scanner's output is used to justify a number: the raw-string form, both comment forms (including a cgo C preamble), a declaration with no body, a nested closure counted once, and braces inside string and rune literals. 9 of 13 fail against the unfixed engine. The 4 that pass both ways assert an absence and are trivially true when the scanner selects nothing, so they are controls on over-selection rather than evidence.Ten existing tests asserted Go was not elidable and now fail on purpose. They are updated in the same commit, as this repo's contract requires, with the prior assertion recorded in each.
language-support.test.tsmoves its "an incidental symbol is not support" example from Go to Rust, which is in the position Go used to occupy — astructyieldstype:Pointthrough the TypeScript class regex and there is no Rust scanner, so the reasoning is unchanged even though the example had to move.The "elision reduces TypeScript/JavaScript and Python only" explanation is corrected wherever it is emitted, and Go now reports as supported on
trace.languageSupport.Suite 737 passing, typecheck, lint and build clean.
What this does not establish
scanBraceSpanstakes the header from the line carrying the{, so a gofmt-wrapped signature presents as) error. Under-selection costs reduction, never content, which is why it ships characterized rather than fixed.token-hashing-reversibility.test.ts; nothing here tested a Go elision round-tripping through a real MCP session.int foo(...)cannot be told from a prototype, wherefuncis unambiguous.DECISIONS §61.
🤖 Generated with Claude Code