Skip to content

feat: add replay test reporters#936

Merged
thymikee merged 6 commits into
mainfrom
codex/extensible-test-reporters
Jun 29, 2026
Merged

feat: add replay test reporters#936
thymikee merged 6 commits into
mainfrom
codex/extensible-test-reporters

Conversation

@thymikee

@thymikee thymikee commented Jun 29, 2026

Copy link
Copy Markdown
Member

Summary

Add an extensible replay test reporter layer for the test command.

Details:

  • Extract default console and JUnit output behind ReplayTestReporter modules.
  • Add repeatable --reporter, with built-in default and junit:<path> reporters.
  • Support CLI-only custom reporters loaded from file paths: --reporter ./scripts/reporter.mjs.
  • Keep the custom reporter contract intentionally small: no CLI reporter options or tuple syntax; custom reporter code owns its own configuration.
  • Export reporter types from the package root for TypeScript custom reporters and document that runtime reporters should be .mjs/.js or precompiled from TS.
  • Keep --report-junit <path> as a compatibility alias while preventing reporter flags from reaching daemon requests.
  • Document built-in and custom reporter usage in website/docs/docs/replay-e2e.md.

Touched-file count: 20. Scope stayed within replay test CLI/reporting, public reporter types, docs, and related tests.

Validation

Verified with focused CLI parser/reporting tests, TypeScript, formatting, integration-progress classification, and fallow:

  • ./node_modules/.bin/vitest run src/__tests__/cli-test-reporters-spec.test.ts src/__tests__/cli-network.test.ts src/utils/__tests__/args.test.ts src/commands/replay/index.test.ts
  • ./node_modules/.bin/tsc -p tsconfig.json
  • node --experimental-strip-types scripts/integration-progress.ts --check
  • ./node_modules/.bin/fallow audit --base origin/main
  • node ./node_modules/oxfmt/bin/oxfmt --write ...

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 1.4 MB 1.4 MB +2.9 kB
JS gzip 450.7 kB 451.6 kB +955 B
npm tarball 553.6 kB 555.1 kB +1.4 kB
npm unpacked 2.0 MB 2.0 MB +5.7 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 27.5 ms 27.6 ms +0.1 ms
CLI --help 48.1 ms 48.6 ms +0.5 ms

Top changed chunks:

Chunk Raw diff Gzip diff
dist/src/2948.js +232 B +63 B

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-06-29 18:37 UTC

@thymikee

Copy link
Copy Markdown
Member Author

Current head 7acf07a is blocked by CI. Fallow fails on src/cli-test-reporters/custom.ts:100 validateCustomReplayTestReporter (11 cyclomatic / 10 cognitive / CRAP 37.1); split the validation branches or extract helpers. Integration progress check also fails because the new public CLI flag reporter is unclassified: add reporter to the progress-script coverage classification (likely parser/client-only command flags or replay reporting owner) so pnpm test:integration:progress:check passes.

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed current head f8bfb9b after the CI fixes. I did not find actionable blockers: reporter handling is contained to CLI presentation, reportJunit/reporter are stripped before daemon requests, the default/JUnit output behavior is preserved behind reporters, custom reporter loading validates the expected hooks, docs cover the explicit reporter list semantics, and all 22 checks are green. Marking ready-for-human.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Jun 29, 2026
@thymikee

Copy link
Copy Markdown
Member Author

Re-reviewed the post-ready delta on current head d2c6718 (refactor: modularize replay test reporters, refactor: trim replay reporter parsing, refactor: simplify custom replay reporters). I do not see a new actionable blocker: reporter spec parsing is centralized, built-ins/custom paths are separated cleanly, custom factories still receive load context, built-in reporters use the reporter IO context, and help/docs/tests now consistently describe the supported default, junit:<path>, or custom-path contract without JSON reporter options. The existing ready-for-human label still looks valid subject to the remaining pending smoke shard finishing green.

@thymikee thymikee merged commit 8cc6bb8 into main Jun 29, 2026
22 of 23 checks passed
@thymikee thymikee deleted the codex/extensible-test-reporters branch June 29, 2026 18:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant