Skip to content

feat: Post-Generation Verification — auto-run and heal after test generation (QUA-647)#108

Merged
Desperado merged 2 commits into
mainfrom
strazhnyk/qua-647-test-generation-heal-pass
May 24, 2026
Merged

feat: Post-Generation Verification — auto-run and heal after test generation (QUA-647)#108
Desperado merged 2 commits into
mainfrom
strazhnyk/qua-647-test-generation-heal-pass

Conversation

@Desperado
Copy link
Copy Markdown
Contributor

Summary

After generate_test_code returns a script_id, the agent previously reported success without running the test. In dogfooding, 4 of 26 generated assertions had signature mismatches or wrong regex behavior that a single run would have caught immediately.

Adds a Post-Generation Verification section to both system prompts (professional and cat mode), instructing the agent to treat generation as a three-step pipeline:

  1. Run — execute immediately after generation (run_test for Playwright/Cypress, run_native_test for pytest/rust/go)
  2. Heal — if it fails, apply the existing Heal loop (up to 3 fix cycles, already in the prompt)
  3. Report✅ generated and verified or ⚠ generated — N issue(s) remain after 3 fix attempts

The new section sits directly above the existing "Test Healing" section so the two read as one coherent generate → verify → heal pipeline.

Closes QUA-647

Test plan

  • Ask the agent to generate a test — it should immediately run it after generate_test_code returns
  • If the generated test fails, it should enter the heal loop and attempt to fix (up to 3 cycles)
  • Agent should never report a script_id as "done" without first running it

🤖 Generated with Claude Code

After generate_test_code returns a script_id, the agent previously
reported success without running the test. In dogfooding, 4 of 26
generated assertions had signature mismatches that a single run
would have caught immediately.

Adds a "Post-Generation Verification" section to both system prompts
(professional and cat mode) instructing the agent to:
- Run the test immediately after generation (run_test / run_native_test)
- Apply the existing Heal loop if it fails (up to 3 fix cycles)
- Report "✅ generated and verified" or "⚠ N issues remain" — never
  hand back a script_id as finished without confirming it executes

Sits directly above the existing "Test Healing" section so the two
workflows read as a single generate → verify → heal pipeline.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@qualitymaxapp
Copy link
Copy Markdown

qualitymaxapp Bot commented May 24, 2026

✅ QualityMax Pipeline

Gate Result
🔍 AI Review ✅ Clean
🧪 Repo Tests ✅ 298/298 passed (go)
🤖 AI Tests ✅ 28/32 passed

Powered by QualityMax — AI-Powered Test Automation

Three fixes from code review:
- Correct pytest tool: run_local_test (local) not run_native_test (cloud API)
- Resolve confirmation conflict: exempt the post-gen run from Rule 2's
  "confirm before running" since the user already confirmed generation
- Add enhance_test_case to the trigger list alongside generate_test_code
  and generate_gap_tests

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Desperado Desperado merged commit eda992f into main May 24, 2026
6 checks passed
@Desperado Desperado deleted the strazhnyk/qua-647-test-generation-heal-pass branch May 24, 2026 11:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant