You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The CodeRabbit custom check Require AI Unit Tests warns on every pull request that lacks AI-generated unit tests and the copilot-rabbit label, including diffs that contain no code.
On #1410 — a one-file change to MERGE_POLICY.md — it asked for unit tests for a markdown document. There is no function, branch, or return value to assert against, so the only way to clear it is a vacuous test. That is work directly against #1122 ("de-vacuify its security tests") and #1236 ("make OpenAI DNS-validation tests non-vacuous"), both open specifically to delete tests of that kind.
Why this is worth fixing rather than ignoring
This is the same failure mode MERGE_POLICY.md was written to name — a gate naming an action the author cannot take — reproduced inside the repository's own custom checks. It is the third instance found on that file's lineage:
Require AI Unit Tests on a markdown-only diff — this issue.
Gate 2 already carries the remedy pattern and the rule that catches it:
Before adding any check to branch protection, confirm the exact string appears in GitHub's required-checks picker, and confirm it reports on a documentation-only pull request. If it does not report, it belongs in the conditional list.
Severity: low — it is advisory today
Confirmed on #1410: the active CodeRabbit configuration sets both Require AI Unit Tests and Enforce Copilot Verification to mode: warning, so neither blocks a merge. #1408 merged with documentation + ci-cd labels, no copilot-rabbit, and zero reviews.
So this is noise reduction and signal hygiene, not an unblocking task. It matters because a check that warns on every docs PR trains people to ignore it, which is how a real finding gets missed later — the same dynamic that made agent-completion/truth-gate worthless by being red on ~100% of pull requests.
Proposed change
Scope the check to diffs that contain code — src/**, apps/**, packages/**, mcp-servers/**, scripts/** — and have it report not-applicable for diffs limited to *.md, docs/**, and similar non-code paths. This mirrors how Generate and Upload Coverage is now scoped in gate 2.
Acceptance criteria
Require AI Unit Tests reports not-applicable (not a warning) on a pull request whose diff contains no code files.
It still applies unchanged to diffs touching code paths.
Enforce Copilot Verification is reviewed for the same treatment, or explicitly left as-is with a reason.
Scope
CodeRabbit custom-checks configuration only. No workflow or application code.
Provenance
Raised during the review of #1410. CodeRabbit initially reported the requirement as a mandatory merge prerequisite, then verified the configuration, withdrew that finding, and confirmed this scoping follow-up as the valid remainder.
Problem
The CodeRabbit custom check
Require AI Unit Testswarns on every pull request that lacks AI-generated unit tests and thecopilot-rabbitlabel, including diffs that contain no code.On #1410 — a one-file change to
MERGE_POLICY.md— it asked for unit tests for a markdown document. There is no function, branch, or return value to assert against, so the only way to clear it is a vacuous test. That is work directly against #1122 ("de-vacuify its security tests") and #1236 ("make OpenAI DNS-validation tests non-vacuous"), both open specifically to delete tests of that kind.Why this is worth fixing rather than ignoring
This is the same failure mode
MERGE_POLICY.mdwas written to name — a gate naming an action the author cannot take — reproduced inside the repository's own custom checks. It is the third instance found on that file's lineage:Generate and Upload Coveragedoes not report on documentation-only diffs, so requiring it unconditionally would leave those PRs permanently pending (fixed in docs: replace Merge Gate v1 with a satisfiable merge policy #1408 by scoping it).Require AI Unit Testson a markdown-only diff — this issue.Gate 2 already carries the remedy pattern and the rule that catches it:
Severity: low — it is advisory today
Confirmed on #1410: the active CodeRabbit configuration sets both
Require AI Unit TestsandEnforce Copilot Verificationtomode: warning, so neither blocks a merge. #1408 merged withdocumentation+ci-cdlabels, nocopilot-rabbit, and zero reviews.So this is noise reduction and signal hygiene, not an unblocking task. It matters because a check that warns on every docs PR trains people to ignore it, which is how a real finding gets missed later — the same dynamic that made
agent-completion/truth-gateworthless by being red on ~100% of pull requests.Proposed change
Scope the check to diffs that contain code —
src/**,apps/**,packages/**,mcp-servers/**,scripts/**— and have it report not-applicable for diffs limited to*.md,docs/**, and similar non-code paths. This mirrors howGenerate and Upload Coverageis now scoped in gate 2.Acceptance criteria
Require AI Unit Testsreports not-applicable (not a warning) on a pull request whose diff contains no code files.Enforce Copilot Verificationis reviewed for the same treatment, or explicitly left as-is with a reason.Scope
CodeRabbit custom-checks configuration only. No workflow or application code.
Provenance
Raised during the review of #1410. CodeRabbit initially reported the requirement as a mandatory merge prerequisite, then verified the configuration, withdrew that finding, and confirmed this scoping follow-up as the valid remainder.