Skip to content

The default engine checks 41 of 159 rules and the output never says which engine you got #628

Description

@beyondnetPeru

evolith validate with no flag checks 41 of 159 rules and never says so.

Measured today against @beyondnet/evolith-cli@1.3.2 from the public registry, on
a satellite created by evolith init and nothing else — same repository, same
version, same corpus, back to back:

validate --engine opa validate (default)
Rules checked 133 41
Rules skipped 26 118
Issue rows 72 114
Blocking 37 77
Exit code 2 2

Full capture of both runs: docs/evidence/first-run-capture.md

This is not a request to make the engines agree

.harness/scripts/ci/68-validate-engine-verdict-parity.mjs states the position in
its own header: the two engines are allowed to disagree about coverage, and are
only held to agreement about facts. That position is deliberate and this issue does
not argue with it.

What is actually wrong

The command a user runs by default is the one with the lower coverage, and nothing
in the output says so. **Rules Skipped:** 118 is printed, but a first-time user has
no way to know that 118 is a property of the engine they did not choose rather than
a property of their repository. Every one of those skips is reported honestly and
the total is published — the project's whole thesis holds — yet the reader still
draws the wrong conclusion, because the denominator that moved is the engine's, not
theirs.

That is the same class of defect the README opened with for months: two commands
with different denominators, and no line explaining the difference. The front page
half of it is fixed — it now uses --engine opa consistently and discloses the gap.
The tool half is not.

Suggested fix

Whichever of these is cheapest, in preference order:

  1. The native evaluator names itself in the report, with its coverage relative to the
    selected corpus — one line, e.g. Engine: native (41/159 decidable; --engine opa decides 133).
  2. Failing that, a note in the summary whenever skipped exceeds checked, pointing
    at --engine opa.
  3. Failing that, make --engine opa the default and keep the native evaluator behind
    the flag.

Done when

A user running evolith validate with no flag can tell from the output alone that
the engine, not their repository, is responsible for most of the skips.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions