Tell a broken normalize gate apart from a bad patch - #88
Merged
Conversation
When the normalizer rejects a patch, serge feeds the failure back to the model and spends a correction. That is right when the patch is at fault and useless when the gate itself is broken: every correction is rejected by the same environment error, the budget runs out, and the task reports "the proposed patch does not pass the repository's normalizer" — which points the reader at the model instead of at the runner image. That is what happened on the 2026-08-17 nightly triage (huggingface/transformers#48037). transformers' main moved its tokenizers pin to >=0.23.1 while the task-runner image still had 0.22.2, so every checker that imports transformers died on the version guard, three groups reported "no fix", and ~3.5M input tokens were spent on an error no patch could address. The gate installs the worktree with --no-deps (it has to: the task pod's egress allowlist has no PyPI), so it cannot repair a dependency drift on its own — it can only notice it. So notice it: on a normalizer failure, re-run the normalizer once on the pristine checkout. Clean exit means the gate works and the patch really is at fault, so nothing changes. Non-zero means no patch can pass, and the task now fails loudly with NormalizeGateBroken naming the likely cause, instead of quietly blaming the model. The check is on the failure path only, so a healthy task pays nothing, and its verdict is cached for the rest of the task so the extra normalizer run is paid at most once. It is not a 422: that status makes the runner move on to the next candidate group, but a broken gate breaks every group. Both gate call sites get it — the in-loop validation and publish_task's raw-apply fallback, which produced the message quoted above. The existing "normalizer rejects the patch" tests used a normalizer that failed unconditionally, which is now the broken-gate case; they use a patch-sensitive one so they still test what they claim to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tarekziade
added a commit
that referenced
this pull request
Aug 18, 2026
) ## Why The task-runner image gets transformers' lint/checker toolchain from `huggingface/transformers-quality`, which installs `transformers[quality]` **at its own build time** (nightly, 00:06 UTC). Every pin in it therefore freezes on that date. When a pin then moves on `main`, the normalize gate fails for **every** task on a checker unrelated to the patch — and the gate cannot repair it, because it installs the worktree with `--no-deps` (mandatory: the task pod's egress allowlist has no PyPI). Twice in two days: | Date | Drift | Effect | | --- | --- | --- | | 2026-08-17 | `tokenizers` 0.22.2 in-image vs main's new `>=0.23.1` | every checker importing transformers died on the version guard — 3 groups lost, ~3.5M input tokens ([transformers#48037](huggingface/transformers#48037)) | | 2026-08-18 | `transformers-mlinter` 0.1.2 in-image vs the 0.1.4 rule schema main moved to at 06:45 UTC (`d032a44a9e`) | `ValueError: Invalid rule spec for TRF054: default_enabled must be bool` | Reproduced locally against main's `utils/rules.toml`: ``` mlinter 0.1.2 → ValueError: Invalid rule spec for TRF054: default_enabled must be bool mlinter 0.1.4 → parsed OK — 56 rules ``` The second one appeared on the very morning the image was rebuilt for the first: the base was built 00:21 UTC, main moved at 06:45 UTC. So **a rebuild alone is not a fix**, and waiting for the base's next nightly (00:06 UTC) misses tonight's 22:00 UTC triage. Worth noting the guard from #88 did its job here — instead of burning three correction budgets and reporting "the patch does not pass the normalizer", the three dispatched groups failed loudly as a broken gate and named the checker. That is how this was diagnosed in minutes. ## What Re-resolve the quality extra from transformers `main` in `Dockerfile.task-runner`, after the base. The gate's environment is then only as stale as *this* image, which a rebuild fixes on demand instead of waiting on the base. `TRANSFORMERS_REF` is a build arg so an operator can pin a ref when `main` itself is the problem. Costs a little build time and image size; buys an environment we can actually refresh. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
When the repo normalizer rejects a patch, the in-loop gate feeds the failure back to the model and spends a correction. That is correct when the patch is at fault, and useless when the gate itself is broken: every correction is rejected by the same environment error, the budget runs out, and the task reports "the proposed patch does not pass the repository's normalizer" — pointing the reader at the model rather than at the runner image.
That is exactly what the 2026-08-17 nightly triage hit (transformers#48037):
mainmoved itstokenizerspin to>=0.23.1,<0.24.0(242f5df9, 2026-08-17 01:35 UTC);tokenizers==0.22.2;dependency_versions_check— 6 of them, while the 2 that don't import transformers passed;cosmos3_omni,generation,cwm) reported 🚫 no fix, and ~3.5M input tokens were spent on an error no patch could address.The gate installs the worktree with
--no-deps, and it has to: the task pod's egress allowlist has no PyPI. So it cannot repair a dependency drift on its own — it can only notice it.The change
On a normalizer failure, re-run the normalizer once on the pristine checkout:
NormalizeGateBroken, so the task fails loudly with the likely cause named.Properties:
publish_task's raw-apply fallback (the one that produced the message quoted above).Tests
653 passed, ruff clean. The existing "normalizer rejects the patch" tests used a normalizer that failed unconditionally — which is now the broken-gate case — so they were switched to a patch-sensitive one and still test what they claim to. New: broken gate raises (with status 500 and thenormalize_errorevent), the baseline runs once per task, and a healthy gate never triggers it.Not in this PR
The image drift itself.
serge-task-runnerneeds a rebuild on a currenttransformers-qualitybase so prod stops failing ontokenizers; this PR only makes the next occurrence legible instead of expensive.🤖 Generated with Claude Code