Mutation testing on the generator

[iderex]

Depends on the golden files issue and the parity ledger issue.

Coverage says a line ran. It does not say the suite would have noticed if the
line were wrong. Mutation testing asks the second question by changing the
implementation and checking that something goes red.

The target gate uses a mutation tool that exists only for .NET, so the tool here
is a different one and the ledger records that as a deviation with this reason.
What does not change is the obligation: the suite has to fail when the
implementation is wrong.

Scope it to the code where a silent wrong answer matters, which is the parser,
the state derivation and the escaping path, rather than the whole tree. A score
over template plumbing is a number that moves without meaning anything.

The score is recorded with the run rather than turned straight into a threshold.
A threshold set before there is a baseline is a number somebody invented, and
the first thing it does is get lowered.

If no maintained tool clears the bar for this toolchain, this issue does not
close by declaring the obligation met. It closes by recording which tools were
tried, what each one did, and what would retire the gap. A debt written down is
a different thing from a gap nobody mentioned.

Done when

Either a named tool runs over the three scoped packages in the gate and the
score is recorded with the run, or the issue carries the tools that were tried,
what each one failed at, and the condition that would retire the debt.

[iderex]

I tried the candidate tools rather than assuming what is available, so the
alternative this issue offers is settled by a run instead of by a guess. Every
run below is against 9985046, in a clone outside the working tree, because two
of the three candidates rewrite source files where they find them.

One tool clears the bar.

go install github.com/go-gremlins/gremlins/cmd/gremlins@v0.6.0
gremlins --version
gremlins version dev windows/amd64

The binary reports dev rather than the tag it was built from, which matters for
anything that later has to pin it.

Two of the three scoped areas are in the tree today, and the markup checker is
the third place where a silent wrong answer reaches a page:

for p in roster site markup; do
  gremlins unleash --timeout-coefficient=20 --workers=2 ./internal/$p
done
### ./internal/roster
Killed: 20, Lived: 2, Not covered: 4
Timed out: 0, Not viable: 0, Skipped: 0
Test efficacy: 90.91%
Mutator coverage: 84.62%
### ./internal/site
Killed: 45, Lived: 2, Not covered: 8
Timed out: 0, Not viable: 0, Skipped: 0
Test efficacy: 95.74%
Mutator coverage: 85.45%
### ./internal/markup
Killed: 60, Lived: 30, Not covered: 3
Timed out: 11, Not viable: 0, Skipped: 0
Test efficacy: 66.67%
Mutator coverage: 96.77%

Run 2026-08-12.

The coefficient is part of the result rather than a detail of how it was invoked.
At the default the same package reports mostly timeouts, and the score is then
computed over the few that finished:

gremlins unleash ./internal/roster
Killed: 3, Lived: 0, Not covered: 4
Timed out: 19, Not viable: 0, Skipped: 0
Test efficacy: 100.00%
Mutator coverage: 42.86%

Run 2026-08-12. A hundred per cent over three of twenty-six mutants is the number
this issue warns about, and here it arrives out of a default rather than from
anybody choosing it. Whatever run this becomes, the coefficient belongs recorded
beside the score.

The two mutants that survive in the parser are both the row number a refusal
names:

roster.go:104:35   where = fmt.Sprintf("row %d", i+1)
roster.go:146:20   seen[e.ID] = i + 1

Replacing either i + 1 with i - 1 leaves the suite green. The package says of
itself that every refusal names the row it is about, so the row number is a
property it claims and nothing asserts it.

The other two candidates, and what each one did.

go install github.com/avito-tech/go-mutesting/cmd/go-mutesting@v2.3.1
go: github.com/avito-tech/go-mutesting/cmd/go-mutesting@v2.3.1: github.com/avito-tech/go-mutesting@v2.3.1: invalid version: module contains a go.mod file, so module path must match major version ("github.com/avito-tech/go-mutesting/v2")
go install github.com/avito-tech/go-mutesting/v2/cmd/go-mutesting@v2.3.1
go: github.com/avito-tech/go-mutesting/v2/cmd/go-mutesting@v2.3.1: github.com/avito-tech/go-mutesting@v2.3.1: invalid version: module contains a go.mod file, so module path must match major version ("github.com/avito-tech/go-mutesting/v2")

Run 2026-08-12. The tag says v2 and the module path inside the file does not, so
both spellings are refused and there is no third one to try.

go install github.com/zimmski/go-mutesting/cmd/go-mutesting@latest
.../zimmski/osutil@v0.0.0-20190128123334-0d0b3ca231ac/limits.go:15:18: undefined: syscall.Setrlimit
.../zimmski/osutil@v0.0.0-20190128123334-0d0b3ca231ac/capture.go:85:31: undefined: syscall.Dup

Run 2026-08-12, with the local module cache prefix elided from each path. It does
not build on windows/amd64, because a dependency calls syscalls that exist only
on POSIX. That is a failure on the platform it was run on and not a statement
about the tool anywhere else, which was not measured.

So the second half of the done-when does not describe the state this repository
is in. A maintained tool runs here and produces numbers, and recording a debt
instead would be recording something that is not true.

What the first half waits on is two things rather than one, and neither of them
is a change to the packages measured above.

The scoped set is three areas and the tree holds two of them. The parser and the
escaping path are here. The state derivation is the code that computes whether a
plugin ships, and there is none in the tree today.

And in the gate is where the run does not fit as the gate is built. Every leg
is decided by the toolchain alone, deliberately, and the one leg that could have
shelled out does not:

grep -n 'go/format' internal/gate/gate.go
13:	"go/format"

Run 2026-08-12. That leg reads the standard library rather than a gofmt binary so
that the gate depends on nothing beside the toolchain, and the two verbs that
read something outside this tree are kept off the leg list rather than added to
it. A mutation leg would be the first leg needing a program the toolchain does
not carry. Where the run lives instead is open, and it changes what the first
half of the done-when is asking for.

[iderex]

The two mutants this issue recorded as the only survivors in the parser are
dead, on c31d98a. Both were the row number a refusal names, and the repair is
in the suite rather than in the parser: what the fixture table compared was a
fragment of the reason, so a reason opening at the wrong row still carried the
identifier the table looked for and passed.

gremlins unleash --timeout-coefficient=20 --workers=2 ./internal/roster
Killed: 22, Lived: 0, Not covered: 4
Test efficacy: 100.00%
Mutator coverage: 84.62%

Run 2026-08-17 against c31d98a, in a clone outside the working tree, at the
coefficient this issue already argues belongs beside any score. The same run at
72a768a reports Killed: 20, Lived: 2 and 90.91%, which is the number
recorded here in August. Mutator coverage does not move: the four uncovered
mutants are the ones nothing reaches, and these two were the ones nothing
asserted.

Getting there needed two fixtures the table did not have, a row whose
identifier is present and empty and a row whose repository is present and
empty. The first is the only route to the form the parser uses for a row
carrying no identifier, so the number in that form could not have been asserted
without it. Statement coverage over the package reads 100% either way, because
the fuzz corpus reaches both lines and asserts only that a refusal came back,
which is the difference this issue is about stated on one package.

The four proofs are in #171 with the edit that produced each red one.

What the done-when asks for has not moved, and what it waits on is what the
comment above already names rather than anything new. The number this issue
holds is now 100.00% over ./internal/roster at the coefficient quoted, and it
is a score somebody took rather than one anything re-runs.

[iderex]

The third scoped area exists now, and it is the half of the first done-when clause that had nothing behind it

The comment above says the scoped set is three areas and the tree holds two of
them: the parser and the escaping path are here, and the state derivation, the
code that computes whether a plugin ships, is not. That has moved. It landed
under #22 as internal/releases, which reads what each repository has published
and applies the rule decisions/0009-what-counts-as-shipping.md fixes:

git show origin/main:internal/releases/releases.go | grep -n 'func (r Repository) Ships'
64:func (r Repository) Ships() bool { return r.Finished > 0 }

Run 2026-08-25 against 440c545.

The score over the two areas I could re-take tonight

Both runs are in a clone outside the working tree, because the tool rewrites
source files where it finds them, and at the coefficient this issue already
argues belongs beside any score:

go install github.com/go-gremlins/gremlins/cmd/gremlins@v0.6.0
gremlins --version
gremlins version dev windows/amd64

for p in roster releases; do gremlins unleash --timeout-coefficient=20 --workers=2 ./internal/$p; done
### ./internal/roster
Killed: 23, Lived: 0, Not covered: 8
Timed out: 0, Not viable: 0, Skipped: 0
Test efficacy: 100.00%
Mutator coverage: 74.19%
### ./internal/releases
Killed: 20, Lived: 0, Not covered: 14
Timed out: 0, Not viable: 0, Skipped: 0
Test efficacy: 100.00%
Mutator coverage: 58.82%

All run 2026-08-25 against 440c545, in a clone at that commit. The binary
still reports dev rather than the tag it was built from, which is the same
thing this issue already recorded and still matters for anything that later has
to pin it.

The parser's efficacy is unchanged at 100.00% and its mutant population has
grown: 31 mutants against the 26 recorded in August, because the identifier rule
that landed with #28 added mutable expressions. Its mutator coverage fell from
84.62% to 74.19% for the same reason - the four uncovered mutants recorded
before are now eight, and what nothing reaches has grown faster than what does.

The state derivation reads 100.00% over 20 mutants with 14 uncovered, which is
the lowest mutator coverage of the two and the number to read rather than the
efficacy beside it. A hundred per cent over the mutants a suite reaches says
nothing about the ones it does not.

Two readings I did not take, and why they are absent rather than pending

./internal/markup and ./internal/site were started and had not finished
after twenty minutes, so no number for either is in this comment. The markup
package is the one this issue already records as producing eleven timed-out
mutants at this coefficient, and each timed-out mutant costs the whole timeout;
that cost has grown with the package. Reporting a score for either from the
August run would be reporting a measurement of a tree that no longer exists,
and reporting one from a shorter coefficient would be the failure this issue
opens by naming.

So the scoped set is complete in the tree and the score over it is not.

What the done-when still waits on, and it is one thing rather than two

The comment above names two things in front of the first half. One of them is
answered above. The other is unchanged and is not a measurement:

go run . ci | head -1
gate: 7 legs, in order: format, vet, test, build, links, sitemap, invariants
grep -n 'go/format' internal/gate/gate.go
15:	"go/format"

Both run 2026-08-25 against 440c545. Every leg is still decided by the
toolchain alone, and the one leg that could have shelled out reads the standard
library instead so that the gate depends on nothing beside the toolchain. A
mutation leg would be the first leg needing a program the toolchain does not
carry, and the two verbs that read something outside this tree are kept off the
leg list rather than added to it. Where the run lives instead is the question,
and it is a call rather than something a reading settles, so I have recorded it
here rather than taking it from inside the work it would judge.

[iderex]

The third scoped area has a score now, and it is the one this issue records as absent

The comment of 2026-08-25 leaves ./internal/markup unmeasured because the run
had not finished after twenty minutes. It finishes. Same tool, same coefficient,
same kind of clone outside the working tree, at ad8b25c:

gremlins --version
gremlins version dev windows/amd64
gremlins unleash --timeout-coefficient=20 --workers=2 ./internal/markup
Mutation testing completed in 6 minutes 46 seconds
Killed: 60, Lived: 30, Not covered: 3
Timed out: 11, Not viable: 0, Skipped: 0
Test efficacy: 66.67%
Mutator coverage: 96.77%

Run 2026-08-26. Every figure reproduces the August reading of that package
exactly, including the eleven timed-out mutants, so the escaping path is where it
was: the lowest efficacy of the three areas and the only one with survivors in
double figures. What that costs to take is the seven minutes above rather than
the twenty the last attempt spent, and the difference is that ./internal/site
is not in the same loop; that package is outside the three this issue scopes.

So the scoped set is complete in the tree and now has a number against each of
its three areas at one commit.

Re-taking the other two found a regression rather than confirming them

./internal/roster is unchanged at 100.00% over 23 killed with 8 uncovered.
./internal/releases moved, and it moved the wrong way:

git checkout 440c545
gremlins unleash --timeout-coefficient=20 --workers=2 ./internal/releases
Killed: 20, Lived: 0, Not covered: 14
Test efficacy: 100.00%
Mutator coverage: 58.82%

git checkout ad8b25c
gremlins unleash --timeout-coefficient=20 --workers=2 ./internal/releases
Killed: 37, Lived: 5, Not covered: 30
Test efficacy: 88.10%
Mutator coverage: 58.33%

git log --oneline 440c545..ad8b25c -- internal/releases/
b3768b5 Take the server generation from what each release published [#91]

All three run 2026-08-26. Both readings are mine at those two commits rather than
the August number quoted back, which is why the comparison is a measurement and
not a difference between a run and a paste.

Four of the five survivors were the four field comparisons in same, the
function that decides whether the refresh reports a repository as MOVED or as
unchanged. Its own comment says it exists to keep that comparison honest, and
nothing asserted it: the only case reaching it moved two fields at once, so a
comparison that had stopped reading one field still answered MOVED there.

The repair, and the score after it

In the suite rather than in the package, which is the shape the parser's two
survivors were repaired in under #171. One case compares an entry against itself
and against five neighbours differing in exactly one thing; negating any of the
four comparisons makes an entry differ from itself, and each mutation was applied
by hand and produced a red run naming the field it removed. The four reds are in
#204 with the edit that produced each one.

gremlins unleash --timeout-coefficient=20 --workers=2 ./internal/releases
Killed: 42, Lived: 1, Not covered: 29
Test efficacy: 97.67%
Mutator coverage: 59.72%

Run 2026-08-26 in the same clone at the same commit, with the case in. Landed on
9a2bf51.

The survivor that is left is not a debt

CONDITIONALS_BOUNDARY at releases.go:139:17 is the boundary on the count of
finished releases in the refusal about a finished release with no server
generation. Three guards above it leave every entry reaching that line with two
counts that are not negative and no more releases stating no generation than were
published, so the only inputs the boundary changes the verdict for are refused
two guards earlier. It is a reading of those four lines and not a measurement;
the suite staying green under the mutation is consistent with it and proves
nothing alone. Chasing it would mean editing a refusal to suit the tool.

What this does and does not move

It fills the measurement half of the first done-when clause: a named tool has now
run over all three scoped areas at one commit and each score is recorded with the
run and with the coefficient. It moves nothing about where that run lives, which
is the other half and is unchanged by anything above.