Skip to content

Assert the order the page reader hands its problems back in - #209

Merged
iderex merged 1 commit into
mainfrom
tests/the-order-the-page-reader-hands-its-problems-back-in
Aug 27, 2026
Merged

Assert the order the page reader hands its problems back in#209
iderex merged 1 commit into
mainfrom
tests/the-order-the-page-reader-hands-its-problems-back-in

Conversation

@iderex

@iderex iderex commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

What was wrong

Read sorts the problems it found by the line they are on, and nothing asserted
the sort. Both mutants the tool reports at that comparison were applied by hand
at 0112a59, in the working tree with the tree restored between, and both left
the suite green:

internal/markup/markup.go:227  Line < Line  ->  Line >= Line   SUITE GREEN
internal/markup/markup.go:227  Line < Line  ->  Line <= Line   SUITE GREEN

Run 2026-08-27. Those two are the whole of what the tool reports as NOT COVERED
in this package, and unlike the four in ./internal/roster that the same reading
found dead, these are a gap in the suite rather than a position the tool cannot
see.

The order is not incidental to what this package is for. The list is what
somebody repairs a page from, and the reversed comparison hands them the bottom
of the file first. It is also not something the code gets right by accident: an
element still open when the input ends is only known to be open once the input
has ended, so its problem is appended after every problem found further down the
page, and the sort is the only thing that puts line 2 back in front of line 10.

What this does

Two cases, one per direction of the same comparison.

The first reads a document ending with three elements still open and a skipped
heading level below them, and asserts the four lines come back ascending. The
document is not the wrapper the rest of the file uses, because that wrapper
closes everything it opens and the out-of-order append cannot arise inside it.

The second puts three problems on one line and asserts they stay in the order the
reader found them. That is the half the first case cannot see: every line in it is
different, and a comparison that has stopped being strict sorts distinct keys
correctly and swaps equal ones. Three kinds on one line separate the two.

Closes

Nothing. It belongs to #44 and does not close it: what that issue waits on is
where a mutation run lives, which is a call rather than a measurement and is not
touched here.

What was run

The gate, at the commit being pushed:

go run . ci
gate: 7 legs, in order: format, vet, test, build, links, sitemap, invariants
  needs-network was not asked for. Asking costs a request to the public name from whatever machine runs it, and a verdict that moves when somebody else's service does rather than when this tree changes. Ask with: go run ./harness/needs-network
  format: ok, 74 file(s)
  vet: ok
  test: ok, 37 test file(s)
  build: ok, 22 file(s)
  links: every reference that stays inside this site resolves to a file the build wrote
  sitemap: every page the build wrote is listed once, and every entry has a page behind it
  invariants: ok, 39 rule(s) decided, 1 owed and not decided
7 of 7 legs ran. None was skipped.

Three mutations of that comparison applied by hand with these cases in, one at a
time with the tree restored between. The third is not one the tool reports and is
here because it is the mistake somebody actually makes at a comparator:

Line < Line -> Line >= Line
--- FAIL: TestTheProblemsComeBackInTheOrderThePageReadsIn
    problem 1 is on line 10, and the page reads them in the order [2 7 8 10]
--- FAIL: TestProblemsOnOneLineStayInTheOrderTheyWereFound
    problem 1 is identity, and the reader finds them in the order [heading alt identity]

Line < Line -> Line <= Line
--- FAIL: TestProblemsOnOneLineStayInTheOrderTheyWereFound
    problem 1 is identity, and the reader finds them in the order [heading alt identity]

Line < Line -> Line > Line
--- FAIL: TestTheProblemsComeBackInTheOrderThePageReadsIn
    problem 1 is on line 10, and the page reads them in the order [2 7 8 10]

All run 2026-08-27. Each is red for the property it removed, and the middle one
is red only in the second case, which is why both cases are here.

The tool, in a clone outside the working tree at the two commits, at the
coefficient #44 argues belongs beside any score:

gremlins --version
gremlins version dev windows/amd64
gremlins unleash --timeout-coefficient=20 --workers=2 ./internal/markup

at 0112a59   Killed: 73, Lived: 14, Not covered: 2, Timed out: 11, Not viable: 4
             Test efficacy: 83.91%   Mutator coverage: 97.75%
at 1ac78a2   Killed: 79, Lived: 14, Not covered: 0, Timed out: 11, Not viable: 0
             Test efficacy: 84.95%   Mutator coverage: 100.00%

Both run 2026-08-27. Read the killed counts with the run this package is already
known to give rather than as a difference these two cases produced. Six more
mutants are killed and only two of them are the ones above: the other four are
the four the earlier run reported as NOT VIABLE and this one did not report at
all. #44 already records that this package does not reproduce run to run, with a
mutant killed on one of three runs and timed out on the other two, and this pair
is another instance of it rather than a new fact. What does reproduce across both
runs is the fourteen survivors, which are unmoved and are the ones #44 reasons
about site by site.

No test was skipped. Nothing here needs a display, elevation, a socket or a
network.

The means

Go test cases in the package they judge, which is what every other suite in this
tree is made of. It adds no language, no runtime and no dependency: go test
already runs it as the third leg of the gate, and the tool the figures above come
from is run by hand outside the gate and is not made a dependency of it here.

Each case is shown to bite by removing the thing it exists for and watching the
run go red naming that thing, which is the three readings above, and every claim
in this body carries the command that produced it.

Who read it

Nobody else. This change carries no second reader, and the evidence above stands
in place of one: the two mutations that found the gap, the three replayed against
the cases, and the two tool runs at the two commits. The ruleset on this branch
requires no approving review, so the merge is not evidence of one either.

Read sorts the problems it found by the line they are on, and nothing
asserted the sort. Both mutants the tool reports at that comparison were
applied by hand at 0112a59 and left the suite green:

    internal/markup/markup.go:227  Line < Line  ->  Line >= Line
    internal/markup/markup.go:227  Line < Line  ->  Line <= Line

The order is not incidental to what the checker is for. The list is what
somebody repairs a page from, and reversing the comparison hands them the
bottom of the file first. It cannot be got right by accident either: an
element still open when the input ends is only known to be open once the
input has ended, so its problem is appended after every problem found
further down the page, and the sort is the only thing that puts line 2 back
in front of line 10.

Two cases, one per direction of the same comparison.

The first reads a document that ends with three elements still open and a
skipped heading level below them, and asserts the four lines come back
ascending. The document is not the wrapper the rest of the file uses,
because that wrapper closes everything it opens and the out-of-order append
cannot arise in it.

The second puts three problems on one line and asserts they stay in the
order the reader found them. That is the half the first case cannot see:
every line in it is different, and a comparison that is no longer strict
sorts distinct keys correctly and swaps equal ones. Three kinds on one line
separate the two.

Each of the two mutations above was applied again with these cases in, and a
third, and all three produce a red run naming the property they removed.

Signed-off-by: Nils Lehnen <30603423+iderex@users.noreply.github.com>
@iderex
iderex merged commit 21bc843 into main Aug 27, 2026
17 checks passed
@iderex
iderex deleted the tests/the-order-the-page-reader-hands-its-problems-back-in branch August 27, 2026 06:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant