Skip to content

Qualify changed-file mutation evidence with StrykerJS #196

Description

@sarthakagrawal927

Why

A StrykerJS 10.0.0 trial against scripts/qualify-codex-accounting-oracle.mjs exposed a large gap between passing tests and behavioral strength:

  • 218 mutants generated.
  • 88 killed and 130 survived.
  • Mutation score: 40.37%.
  • One command-runner process, coverage analysis off, 14 seconds wall time.

The trial also found integration costs before mutation began: isolated pnpm dlx resolution needed TypeScript to be co-installed, and the sandbox copier failed on repository-local socket/symlink trees until explicit ignore patterns were added.

Requirement

Mutation evidence SHALL remain a bounded verification oracle, not a universal full-suite gate. It SHALL target changed deterministic TypeScript/JavaScript verification logic, retain tool/version/config attribution, and fail closed when the dry run, sandbox copy, or report production fails.

Design

  • Start with one high-value verification module and its focused tests.
  • Improve tests against survived correctness-boundary mutants before setting a threshold.
  • Prefer the Node TAP runner only if its coverage acceleration materially beats the command runner; otherwise avoid an extra plugin.
  • Keep mutation output in an attributable JSON receipt and report killed, survived, timeout, error, and no-coverage separately.
  • Do not treat mutation score as the final CodeVetter verdict.

Tasks

  • Classify the 130 survivors into meaningful gaps, equivalent mutants, CLI-only paths, and message-only changes.
  • Add focused tests for negative numbers, invalid costs, provider selection, missing daily buckets, CLI arguments, process failures, and exit behavior.
  • Re-run the same 218-mutant scope and record score/time deltas.
  • Compare command runner with @stryker-mutator/tap-runner before adding a plugin.
  • Decide whether the dependency and transitive package cost justifies a maintained integration.
  • Add a bounded changed-file or weekly mutation command only after the score is meaningful and stable.
  • Define a baseline/ratchet policy that cannot be gamed by mutator exclusions.

Evidence

Discovered during #193. The reproducible trial config and JSON report are ignored local scratch under artifacts/tooling/trials/stryker/; no repository dependency was added.

Metadata

Metadata

Labels

openspecOpenSpec feature change

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions