Skip to content

Run mutation testing on the core modules, not by hand #163

Description

@yuanhao

Fourth of the trustworthiness items.

Why

Hand-run mutation testing found a decorative test nearly every time it was tried during 0.18 work. Tests that could not fail for the reason they stated:

  • EVENT_VARIANT_COUNT — compared two hand-written lists; a new variant appeared in neither (EVENT_VARIANT_COUNT does not guard what its failure message claims #137)
  • unpriced_presets_report_unknown_rather_than_free — survived flipping || to &&
  • file_backend_evicts_oldest_to_stay_under_its_cap — keys sorted in write order, so the tiebreak agreed with mtime and it passed either way
  • A shared-state leak test that survived the function under test being renamed
  • assert_eq!(seen.len(), samples.len()) — tautological, in the PR that fixed decorative assertions
  • A claude_sonnet_5 price assertion pinning one field of four, which is how a 1.5x error survived 18 releases
  • Two assertions using context::message_text, which returns empty for anything but a ToolResult — so one passed having inspected nothing

Seven, found by hand, in one release cycle. The pattern is not going to stop on its own.

Proposal

cargo-mutants over the modules where a silent failure costs the most:

  • src/context.rs — compaction, truncation, execution limits
  • src/agent_loop.rs — the loop itself
  • src/llm_compaction.rs — the state machine
  • src/provider/model.rs — pricing

Not per-PR: it is slow, and the signal needs a human to triage. A scheduled run (weekly, or pre-release) with the surviving mutants reported is the right cadence.

Acceptance

  • cargo-mutants runs against the listed modules and the output is readable.
  • A baseline is recorded — the point is the trend, not a one-off number.
  • Known-acceptable survivors are documented so triage is about what changed, not re-litigating the same ones.

Caveat worth stating

Mutation score is a means, not a target. A test suite optimised for killing mutants can still be one that nobody can read. The value here is specifically finding assertions that cannot fail, which is a narrower and more useful signal than coverage.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions