feat(#677): add structured needs_input signal for code agent - #1058
feat(#677): add structured needs_input signal for code agent#1058fullsend-ai-coder[bot] wants to merge 8 commits into
Conversation
When the code agent cannot proceed without human intervention (broken tooling, missing scan-secrets, uninterpretable issue), it now emits a structured needs_input signal in agent-result.json instead of silently no-oping. Changes: - Schema: add needs_input (boolean) and needs_input_reason (string) properties to code-result.schema.json - Skill: update SKILL.md steps 8/9a/9c to reference the new "Emitting needs_input" procedure section - Post-script: handle needs_input before the feature branch check — apply fs-code-needs-input label, post explanatory comment, exit 0 - Tests: add detection logic tests, comment body tests, and bundled script grep checks to post-code-test.sh - Bundle: rebuild post-code.sh from post-code.src.sh The fs-code-needs-input label is created on first use (yellow, like requires-manual-review) and reused for all causes — the comment body carries the specifics. Closes #677
|
🤖 Finished Review · ✅ Success · Started 8:43 PM UTC · Completed 9:02 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $9.02 |
|
Risk Assessment: moderate (2/5) DetailsRe-review anchoring applied: all Tier 1 signals are identical to the prior assessment (PROTECTED_PATH_COUNT=6, TEST_FILE_RATIO=0.17, bot author, no security exposure), and Tier 2 high-churn and multi-author signals remain repository-wide characteristics already noted in the prior rationale; Tier 3 issue alignment is unchanged and strong, keeping the composite at 2 (moderate). Previous runRisk Assessment: moderate (2/5) DetailsModerate risk: Tier 1 signals unchanged from prior assessment — bot-authored additive feature (new schema fields, post-scripts, eval case) with clear issue alignment and no security exposure, offset by 6 protected-path touches and a test-file ratio of 0.17; Tier 2 flags persistent high churn on hot files and 13 distinct authors with fix/revert touches in 90 days, but these are repository-wide characteristics already present in the prior assessment, and the low-risk Tier 3 issue alignment keeps the composite at 2. Previous run (2)Risk Assessment: moderate (2/5) DetailsModerate risk: signals are unchanged from prior assessment — bot-authored additive feature (new schema fields, post-scripts, eval case) with clear issue alignment and no security exposure, offset by 6 protected-path touches, meaningful repository churn (71 commits/30d, 13 authors/90d), and a test-file ratio below the preferred 0.30 threshold. Previous run (3)Risk Assessment: moderate (2/5) DetailsModerate risk: signals are unchanged from prior assessment - bot-authored additive feature (new schema fields, post-scripts, eval case) with clear issue alignment and no security exposure, offset by 6 protected-path touches, meaningful repository churn (71 commits/30d, 13 authors/90d), and a test-file ratio below the preferred 0.30 threshold. Previous run (4)Risk Assessment: moderate (2/5) DetailsModerate risk: additive, bot-authored feature (new schema fields, post-scripts, eval case) with clear issue alignment and no security exposure, offset by 6 protected-path touches, meaningful repository churn (71 commits/30d, 13 authors/90d), and a test-file ratio below the preferred 0.30 threshold. Previous run (5)Risk Assessment: moderate (2/5) DetailsModerate risk: well-scoped additive feature (new schema fields, post-script handling, skill updates, eval case) with bot authorship and clear issue alignment, offset by protected path modifications in scripts/ and skills/. Previous run (6)Risk Assessment: moderate (2/5) DetailsModerate risk: well-scoped additive feature (new schema fields, post-script handling, skill updates) with bot authorship and clear issue alignment, offset by protected path modifications in scripts/ and skills/. Previous run (7)Risk Assessment: moderate (2/5) DetailsModerate risk: a well-scoped additive feature (new schema, scripts, and skill updates) with bot authorship and clear issue alignment, offset by high churn and multi-author contention on the heavily-modified post-code scripts. |
ReviewFindingsMedium
Low
Previous runReviewFindingsMedium
Low
Next steps:
Previous run (2)ReviewFindingsMedium
Low
Next steps:
Previous run (3)ReviewFindingsMedium
Low
Next steps:
Previous run (4)ReviewFindingsMedium
Low
Next steps:
Previous run (5)ReviewFindingsMedium
Low
Previous run (6)ReviewFindingsMedium
Low
Previous run (7)ReviewFindingsHigh
Medium
Low
Labels: PR modifies code agent post-script and skill behavior Next steps:
|
|
🤖 Finished Fix · ✅ Success · Started 9:04 PM UTC · Completed 9:12 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $1.97 |
- Update SKILL.md schema compliance note to include needs_input and needs_input_reason in the allowed fields list (stale-doc, high) - Add JSON Schema if/then conditional requiring needs_input_reason when needs_input is true, matching the triage schema pattern (api-contract, medium) - Update agents/code.md structured output section to mention needs_input and needs_input_reason fields (stale-doc, medium) - Add third exit state (needs_input) to agents/code.md failure handling section (stale-doc, low) - Add fs-code-needs-input label to docs/code.md control labels table (stale-doc, medium) - Add schema validation test cases for needs_input fields in validate-code-output-test.sh (test-coverage, low) Addresses review feedback on #1058
🔧 Fix agent — iteration 1 (bot-triggered)Addressed 6 of 10 review findings with code changes. Fixed the high-severity stale allowlist in SKILL.md, added if/then schema enforcement for needs_input_reason, updated agents/code.md with new fields and exit state, added fs-code-needs-input to docs/code.md control labels table, and added 5 schema validation test cases. Disagreed with 4 findings: protected-path (informational), section-numbering (intentional), test-adequacy (environment constraint), label-naming (intentional namespacing). Fixed (6):
Disagreed (4):
Tests: passedNext steps:
|
|
🤖 Finished Review · ✅ Success · Started 9:14 PM UTC · Completed 9:35 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $8.87 |
Superseded by updated review
|
/fs-fix add an functional-test eval case for this in |
|
🤖 Finished Fix · ✅ Success · Started 9:12 PM UTC · Completed 9:20 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $2.64 |
Add eval case 002-needs-input-uninterpretable that exercises the needs_input signal end-to-end: an issue with redacted context and no actionable scope should cause the code agent to emit needs_input: true, triggering the post-script to apply the fs-code-needs-input label and post a blocker comment instead of creating a PR. Update eval judges (pr_created, expected_files) to handle the expect_needs_input annotation, and add a needs_input_signal judge that verifies the label and comment are present. Addresses #1058
🔧 Fix agent — iteration 2 (human-triggered)Added eval/code/cases/002-needs-input-uninterpretable with judges to test the needs_input signal end-to-end: uninterpretable issue → agent emits needs_input → post-script applies label and posts comment → no PR created Fixed (3):
Tests: passed Decision points
Next steps:
|
|
🤖 Finished Review · ✅ Success · Started 9:22 PM UTC · Completed 9:42 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $5.79 |
The 800/700 pair was derived for two cases, but fullsend-ai#1058 adds a third to the same parallelism: 1 leg, and whichever of the two PRs merges second would land a budget that no longer holds: 3 x (120 + 800 + 180 + 30) = 3390s against a 2700s cap. 420/320 gives 3 x 750 = 2250s and keeps a 450s setup allowance — the same margin the two-case derivation reserved, and enough for podman's own ~190s plus the rest. Two cases at this value are 1500s, so merging before fullsend-ai#1058 is safe either way. 500 (the value whose arithmetic reaches 2490s) leaves 210s, which is inside podman alone. The window is a job-budget quotient, not a measured fit: 003 has never run, so if its first real run needs more than 320s the code cases need their own matrix legs rather than a larger share of this one. Said the same thing plainly in 003's budget comment, which called the ceiling "unmeasured" without saying the harness has never executed the case — every Functional Tests run on this branch was gated off as unauthorized. after_each's derivation picks up run_go_checks' 5s kill grace: ~143s worst case, not ~135s. Signed-off-by: guy oron <goron@redhat.com>
|
🤖 Finished Review · ✅ Success · Started 5:01 PM UTC · Completed 5:22 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $6.63 |
Keep the needs_input contract in the code agent definition while removing duplicated procedure text from the implementation skill. This returns the skill to the existing Skillsaw context-budget ceiling without changing the structured-output behavior.
|
🤖 Finished Fix · ❌ Failure (post-script /home/runner/work/agents/agents/.fullsend/.fullsend-cache/resources/sha256/f0b37e364cfe852baffb406950fba12d481781d1ec6d4ec0d4d22f581fa19dd8/scripts/post-fix.sh failed: exit status 1) · Started 5:24 PM UTC · Completed 5:31 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $1.89 |
|
🤖 Review · ❌ Terminated · Started 5:25 PM UTC · Ended 5:46 PM UTC Commit: |
|
The fix agent completed, but the post-fix script failed before finishing. Workflow run: https://github.com/fullsend-ai/agents/actions/runs/34256691147 Details: |
|
🤖 Finished Review · ✅ Success · Started 5:25 PM UTC · Completed 5:46 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $5.44 |
|
🤖 Finished Fix · ✅ Success · Started 5:47 PM UTC · Completed 5:55 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $1.96 |
- Add EXTERNAL_WORK_ITEM guard to the needs_input handler in both post-code.src.sh and post-code.sh, matching the pattern used by post_noop_comment (logic-error, medium) - Use **STOP** (bold uppercase) instead of lowercase 'stop' in the two new needs_input halt instructions in SKILL.md, matching the established convention at line 45 (emphasis-convention, low) - Clarify 'No code changes are expected' in agents/code.md to state this is a convention the post-script does not enforce (intent-alignment, low) Note: pre-commit hooks could not complete (infrastructure failure — network access to remote hook repositories was blocked). Shell files were verified with basic checks and all post-code and validate-code-output tests pass. Note: harness-jira-test.sh failures are pre-existing (yq not installed in this environment) and unrelated to these changes. Addresses #1058
🔧 Fix agent — iteration 3 (bot-triggered)Fixed three review findings: added EXTERNAL_WORK_ITEM guard to prevent silent needs_input signal loss with external work items, normalized halt-instruction emphasis to STOP convention, and clarified needs_input documentation as convention rather than enforced invariant. Six informational findings disagreed with as out-of-scope or by-design. Fixed (3):
Disagreed (6):
Tests: passed Decision points
Next steps:
|
|
🤖 Finished Review · ✅ Success · Started 5:57 PM UTC · Completed 6:18 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $7.51 |
|
🤖 Finished Fix · ✅ Success · Started 6:20 PM UTC · Completed 6:28 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $2.47 |
Address review feedback: (1) restructure schema allOf so target_branch is required only when needs_input is not true, preventing the validation loop from swallowing the needs_input signal; (2) align schema description with the advisory language already used in agents/code.md; (3) rename lowercase variables to UPPER_SNAKE_CASE in the needs_input handler to match the file's existing convention. Addresses #1058
🔧 Fix agent — iteration 4 (bot-triggered)Addressed three actionable findings: (1) restructured schema to make target_branch conditional on needs_input, (2) aligned schema description with advisory language, (3) renamed variables to UPPER_SNAKE_CASE. Six informational findings recorded as disagreements with rationale. Strategy escalation: prior iterations (1-3) addressed doc/skill files but not the schema constraint or variable naming directly — this iteration fixes the root causes in the schema and scripts. Fixed (3):
Disagreed (6):
Tests: passed
Decision points
Next steps:
|
|
🤖 Finished Review · ✅ Success · Started 6:30 PM UTC · Completed 6:49 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $5.12 |
|
🤖 Finished Fix · ✅ Success · Started 6:51 PM UTC · Completed 6:59 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $1.51 |
Rewrite structured-output documentation in SKILL.md and agents/code.md to accurately reflect the conditional validation: target_branch is required for normal runs, needs_input_reason is required when needs_input is true. Add a comment to the schema allOf/if explaining the vacuous-truth guard. Update target_branch description field to note conditional requirement. Rename misleading test name schema-valid-needs-input-reason-only to -without-flag. Note: pre-commit hooks were not run. pre-commit could not complete (infrastructure failure). Hooks were run directly where possible (trailing whitespace, end-of-file, JSON validation). Addresses #1058
🔧 Fix agent — iteration 5 (bot-triggered)Addressed all actionable findings: rewrote docs in SKILL.md and agents/code.md to accurately describe conditional schema requirements, added $comment to schema explaining vacuous truth guard, documented conditional requirement in target_branch description, renamed misleading test. Recorded reasoned disagreements for 6 informational/pre-existing/deliberate-design findings. Fixed (4):
Disagreed (6):
Tests: passed
Decision points
Next steps:
|
|
🤖 Finished Review · ✅ Success · Started 7:01 PM UTC · Completed 7:22 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $5.76 |
Superseded by updated review
Summary
Adds a structured
needs_inputsignal so the code agent can explicitly say "I cannot proceed — a human needs to act" instead of silently no-oping.Problem: When the code agent encounters broken tooling, a missing
scan-secretsbinary, or a genuinely uninterpretable issue, it currently produces no informative signal — the post-script falls through to a generic "No changed files" no-op comment with no explanation.Solution: Three coordinated changes give the agent a structured way to push back:
schemas/code-result.schema.json): New optionalneeds_input(boolean) andneeds_input_reason(string) propertiesskills/code-implementation/SKILL.md): Steps 8 (uninterpretable issue), 9a (missing scan-secrets), and 9c (broken build tooling) now reference a new "Emitting needs_input" procedure sectionscripts/post-code.src.sh): Checks forneeds_inputbefore the feature branch check — appliesfs-code-needs-inputlabel to the issue, posts an explanatory comment with the agent's reason, and exits cleanly (exit 0, no PR)A single label (
fs-code-needs-input) is reused for all causes — the comment body carries the specifics.Testing
make check-bundleCloses #677
Post-script verification
agent/677-code-agent-needs-input)2fbb7da338097ad4c3d659820444c7523a0f1e63..HEAD)