Skip to content

test(e2e): assert the generate matrix covers every declared tool target#2116

Open
saitota wants to merge 1 commit into
dyoshikawa:mainfrom
saitota:test-e2e-matrix-covers-registry-targets
Open

test(e2e): assert the generate matrix covers every declared tool target#2116
saitota wants to merge 1 commit into
dyoshikawa:mainfrom
saitota:test-e2e-matrix-covers-registry-targets

Conversation

@saitota

@saitota saitota commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Background

Which tools a feature supports, and where each writes output, lives in two places — the implementation (getToolTargets()) and the hardcoded it.each arrays in src/e2e/*.spec.ts — with nothing connecting them. CLAUDE.md having to tell a human "preserve the Tool × Feature matrix tests" is the evidence it was never mechanized. So adding a tool (32 in H1 2026) means reconstructing from memory which specs need new rows; a miss leaves CI green while coverage silently drifts.

Changes

  • Add assertGenerateMatrixCoversTargets (e2e-helper): derives the tool set from getToolTargets() and asserts declared targets partition exactly into the matrix's tested tools plus an explicit, reasoned untested list — a declared-but-absent tool or a stray entry now fails CI.
  • Wire it into all 8 features (project + global where supported). -legacy aliases are excluded, as the supported-tools tables already do.

Same "derive from implementation, fail on drift" idiom as the TOOL_DISPLAY check and gitignore derivation. Per-tool expected paths stay hand-written; only the enumeration is derived.

Test Plan

  • pnpm cicheck green (6740 tests).
  • Verified drift detection: removing a tool from a matrix array fails with "declared by the processor but missing from the e2e matrix".

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Unfortunately, this PR has 1354 added lines, which exceeds the limit of 1000 lines for external contributors.

Please split your changes into smaller PRs. See CONTRIBUTING.md for details.

@saitota saitota force-pushed the test-e2e-matrix-covers-registry-targets branch 2 times, most recently from aedc93f to e346d24 Compare July 3, 2026 10:40
@saitota

saitota commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Trimmed comments: now +1,318 / -884.

@saitota saitota force-pushed the test-e2e-matrix-covers-registry-targets branch from e346d24 to 8067396 Compare July 3, 2026 14:25
@saitota saitota force-pushed the test-e2e-matrix-covers-registry-targets branch from 8067396 to 469ba7e Compare July 3, 2026 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant