Review correctness sub-agent should flag missing validation coverage for new config file types

[fullsend-ai-retro]

What happened

On PR #722, the author introduced 6 new YAML measurement manifests under eval/measurements/ -- a new config file category with no pre-existing CI or pre-commit validation. The human reviewer flagged this on the first review sweep (Aug 10), noting that: (1) the repo has lint-agent-docs for harness/*.yaml and eval/lint-cases.sh for eval cases, but neither covers eval/measurements/*.yaml, (2) no CI workflow step validates these files, and (3) the consumer code in fullsend's ScoreTrace silently skips unknown scorer values via a default: switch case, meaning a typo would silently produce zero measurements in production. The review agent approved 3 times (Aug 10--13) without flagging this gap. The human's finding was the catalyst for the entire eval/lint-measurements.sh script (395 lines of validation logic plus 301 lines of tests).

What could go better

The review agent's correctness sub-agent should detect when a PR introduces new config file types that fall outside all existing validation infrastructure. The first two checks are mechanically detectable: (1) identify new files whose paths don't match any files: pattern in .pre-commit-config.yaml hooks, and (2) check whether any CI workflow step explicitly validates files at that path. A third check (whether the consumer code uses silent-skip semantics for unknown values) requires cross-repo reasoning and is harder. Confidence is high that checks 1 and 2 are feasible and would have caught this -- the correctness sub-agent already has access to .pre-commit-config.yaml and .github/workflows/ in the repo. This is distinct from #394 (CI workflow file correctness), #397 (test coverage estimation for code), and #562 (config-loader defensive validation).

Proposed change

Add guidance to the correctness sub-agent definition in skills/pr-review/sub-agents/ instructing it to: when a PR introduces files under a new directory path or file pattern that doesn't match any existing pre-commit hook's files: regex or CI workflow step's path filter, flag a LOW finding noting the validation gap. Escalate to MEDIUM when the consumer code (if identifiable from the PR body or README) uses silent-skip or fallback semantics for unknown values, since typos would silently no-op in production. The check should cross-reference .pre-commit-config.yaml hooks and .github/workflows/ steps in the repo to determine existing validation coverage.

Validation criteria

On the next PR in this repository that adds a new config file category without corresponding lint or CI validation, the review agent should produce at least a LOW finding noting the missing validation coverage. Measurable over the next 3 PRs that introduce new config file types.

---

Generated by retro agent from #722

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 4:32 PM UTC · Completed 4:35 PM UTC

Commit: 0a5599e · View workflow run →

[fullsend-ai-triage]

Triage summary

Severity: Medium · Category: Feature

Related issues

The issue correctly identifies itself as distinct from:

• Add CI workflow correctness checks to review agent sub-agents #394 (CI workflow correctness checks — about validating the workflows themselves)
• Correctness sub-agent should estimate aggregate coverage for new-code PRs and escalate when below merge-gate threshold #397 (test coverage estimation for code — about code coverage, not config validation)
• Correctness sub-agent should check config-loader defensive validation patterns #562 (config-loader defensive validation — about code-level validation patterns, not pre-commit/CI coverage)

Implementation notes

The target file is skills/pr-review/sub-agents/correctness.md. A new section should be added after the existing "Consumer completeness" or "Removal / rename staleness" guidance. The two mechanically-detectable checks (pre-commit hook files: regex matching and CI workflow path filter matching) are well-scoped and feasible given the sub-agent already has Read, Grep, and Glob tools available.

Proposed test case

Eval case: new-config-file-type-validation-gap

Setup:
  - PR diff introduces files under a novel path (e.g., config/scoring/*.yaml)
  - .pre-commit-config.yaml contains hooks whose files: patterns do NOT match that path
  - No .github/workflows/ step filters on that path

Expected:
  - Correctness sub-agent emits >= 1 LOW finding about missing validation coverage

Variant (escalation to MEDIUM):
  - PR body or referenced consumer code contains a switch/case with default:/fallback
    that silently skips unknown values
  - Expected: finding severity escalated to MEDIUM

---

Labels: This is a feature request for the review agent's correctness sub-agent. Applying 'review-agent' for component ownership and 'feature' for request type.

---

Next steps:

• /fs-code — agent creates a PR to implement this issue
• /fs-code <your instruction> — agent implements with your specific guidance

[fullsend-ai-prioritize]

RICE Priority Score: 1.2

Score breakdown

Dimension	Score	Reasoning
Reach	1	This improves the review agent's correctness sub-agent, which is used across all PRs in the repo. However, the specific scenario — a PR introducing an entirely new config file category without validation coverage — is relatively infrequent, affecting a moderate number of users over time rather than all users on every PR.
Impact	1.5	When this gap is hit, the consequences are significant: the consumer code uses silent-skip semantics (default switch case), meaning typos in new config files silently produce zero measurements in production with no error. The review agent missed this 3 times while a human caught it on the first sweep, indicating a real correctness gap in the agent's review capabilities.
Confidence	0.8	High confidence. The issue is exceptionally well-described with a concrete PR reference (#722), clear proposed implementation path (add guidance to skills/pr-review/sub-agents/correctness.md), two mechanically-detectable checks that are feasible with existing tools, and explicit validation criteria. The triage agent confirmed feasibility and provided a proposed eval case.
Effort	1	Medium effort. The core change is adding a new check section to the correctness sub-agent definition, cross-referencing .pre-commit-config.yaml hooks and CI workflow path filters. The sub-agent already has Read, Grep, and Glob tools available. An eval case with setup and expected findings also needs to be created. The scope is well-contained to one sub-agent's guidance file plus eval infrastructure.

Formula: (1 x 1.5 x 0.8) / 1 = 1.2