feat: Post-Generation Verification — auto-run and heal after test generation (QUA-647)#108
Merged
Merged
Conversation
After generate_test_code returns a script_id, the agent previously reported success without running the test. In dogfooding, 4 of 26 generated assertions had signature mismatches that a single run would have caught immediately. Adds a "Post-Generation Verification" section to both system prompts (professional and cat mode) instructing the agent to: - Run the test immediately after generation (run_test / run_native_test) - Apply the existing Heal loop if it fails (up to 3 fix cycles) - Report "✅ generated and verified" or "⚠ N issues remain" — never hand back a script_id as finished without confirming it executes Sits directly above the existing "Test Healing" section so the two workflows read as a single generate → verify → heal pipeline. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
✅ QualityMax Pipeline
Powered by QualityMax — AI-Powered Test Automation |
Three fixes from code review: - Correct pytest tool: run_local_test (local) not run_native_test (cloud API) - Resolve confirmation conflict: exempt the post-gen run from Rule 2's "confirm before running" since the user already confirmed generation - Add enhance_test_case to the trigger list alongside generate_test_code and generate_gap_tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
After
generate_test_codereturns a script_id, the agent previously reported success without running the test. In dogfooding, 4 of 26 generated assertions had signature mismatches or wrong regex behavior that a single run would have caught immediately.Adds a Post-Generation Verification section to both system prompts (professional and cat mode), instructing the agent to treat generation as a three-step pipeline:
run_testfor Playwright/Cypress,run_native_testfor pytest/rust/go)✅ generated and verifiedor⚠ generated — N issue(s) remain after 3 fix attemptsThe new section sits directly above the existing "Test Healing" section so the two read as one coherent generate → verify → heal pipeline.
Closes QUA-647
Test plan
generate_test_codereturns🤖 Generated with Claude Code