evolith validate with no flag checks 41 of 159 rules and never says so.
Measured today against @beyondnet/evolith-cli@1.3.2 from the public registry, on
a satellite created by evolith init and nothing else — same repository, same
version, same corpus, back to back:
|
validate --engine opa |
validate (default) |
| Rules checked |
133 |
41 |
| Rules skipped |
26 |
118 |
| Issue rows |
72 |
114 |
| Blocking |
37 |
77 |
| Exit code |
2 |
2 |
Full capture of both runs: docs/evidence/first-run-capture.md
This is not a request to make the engines agree
.harness/scripts/ci/68-validate-engine-verdict-parity.mjs states the position in
its own header: the two engines are allowed to disagree about coverage, and are
only held to agreement about facts. That position is deliberate and this issue does
not argue with it.
What is actually wrong
The command a user runs by default is the one with the lower coverage, and nothing
in the output says so. **Rules Skipped:** 118 is printed, but a first-time user has
no way to know that 118 is a property of the engine they did not choose rather than
a property of their repository. Every one of those skips is reported honestly and
the total is published — the project's whole thesis holds — yet the reader still
draws the wrong conclusion, because the denominator that moved is the engine's, not
theirs.
That is the same class of defect the README opened with for months: two commands
with different denominators, and no line explaining the difference. The front page
half of it is fixed — it now uses --engine opa consistently and discloses the gap.
The tool half is not.
Suggested fix
Whichever of these is cheapest, in preference order:
- The native evaluator names itself in the report, with its coverage relative to the
selected corpus — one line, e.g. Engine: native (41/159 decidable; --engine opa decides 133).
- Failing that, a note in the summary whenever
skipped exceeds checked, pointing
at --engine opa.
- Failing that, make
--engine opa the default and keep the native evaluator behind
the flag.
Done when
A user running evolith validate with no flag can tell from the output alone that
the engine, not their repository, is responsible for most of the skips.
evolith validatewith no flag checks 41 of 159 rules and never says so.Measured today against
@beyondnet/evolith-cli@1.3.2from the public registry, ona satellite created by
evolith initand nothing else — same repository, sameversion, same corpus, back to back:
validate --engine opavalidate(default)Full capture of both runs:
docs/evidence/first-run-capture.mdThis is not a request to make the engines agree
.harness/scripts/ci/68-validate-engine-verdict-parity.mjsstates the position inits own header: the two engines are allowed to disagree about coverage, and are
only held to agreement about facts. That position is deliberate and this issue does
not argue with it.
What is actually wrong
The command a user runs by default is the one with the lower coverage, and nothing
in the output says so.
**Rules Skipped:** 118is printed, but a first-time user hasno way to know that 118 is a property of the engine they did not choose rather than
a property of their repository. Every one of those skips is reported honestly and
the total is published — the project's whole thesis holds — yet the reader still
draws the wrong conclusion, because the denominator that moved is the engine's, not
theirs.
That is the same class of defect the README opened with for months: two commands
with different denominators, and no line explaining the difference. The front page
half of it is fixed — it now uses
--engine opaconsistently and discloses the gap.The tool half is not.
Suggested fix
Whichever of these is cheapest, in preference order:
selected corpus — one line, e.g.
Engine: native (41/159 decidable; --engine opa decides 133).skippedexceedschecked, pointingat
--engine opa.--engine opathe default and keep the native evaluator behindthe flag.
Done when
A user running
evolith validatewith no flag can tell from the output alone thatthe engine, not their repository, is responsible for most of the skips.