Skip to content

Tell a broken normalize gate apart from a bad patch - #88

Merged
tarekziade merged 1 commit into
mainfrom
fix/normalize-gate-baseline
Aug 18, 2026
Merged

Tell a broken normalize gate apart from a bad patch#88
tarekziade merged 1 commit into
mainfrom
fix/normalize-gate-baseline

Conversation

@tarekziade

Copy link
Copy Markdown
Collaborator

The problem

When the repo normalizer rejects a patch, the in-loop gate feeds the failure back to the model and spends a correction. That is correct when the patch is at fault, and useless when the gate itself is broken: every correction is rejected by the same environment error, the budget runs out, and the task reports "the proposed patch does not pass the repository's normalizer" — pointing the reader at the model rather than at the runner image.

That is exactly what the 2026-08-17 nightly triage hit (transformers#48037):

  • transformers' main moved its tokenizers pin to >=0.23.1,<0.24.0 (242f5df9, 2026-08-17 01:35 UTC);
  • the task-runner image still had tokenizers==0.22.2;
  • so every checker that imports transformers died on dependency_versions_check — 6 of them, while the 2 that don't import transformers passed;
  • three groups (cosmos3_omni, generation, cwm) reported 🚫 no fix, and ~3.5M input tokens were spent on an error no patch could address.

The gate installs the worktree with --no-deps, and it has to: the task pod's egress allowlist has no PyPI. So it cannot repair a dependency drift on its own — it can only notice it.

The change

On a normalizer failure, re-run the normalizer once on the pristine checkout:

  • clean exit → the gate works, the patch really is at fault → unchanged behavior (feedback to the model);
  • non-zero → no patch can pass → raise NormalizeGateBroken, so the task fails loudly with the likely cause named.

Properties:

  • Failure path only — a healthy task pays nothing.
  • Cached per task — the extra normalizer run is paid at most once, no matter how many corrections follow.
  • Not a 422 — 422 makes the runner skip to the next candidate group, but a broken gate breaks every group.
  • Both call sites — the in-loop gate and publish_task's raw-apply fallback (the one that produced the message quoted above).
  • Exit code, not diff, is the signal: a base that isn't itself normalizer-clean legitimately produces changes, and the worktree is left pristine either way.

Tests

653 passed, ruff clean. The existing "normalizer rejects the patch" tests used a normalizer that failed unconditionally — which is now the broken-gate case — so they were switched to a patch-sensitive one and still test what they claim to. New: broken gate raises (with status 500 and the normalize_error event), the baseline runs once per task, and a healthy gate never triggers it.

Not in this PR

The image drift itself. serge-task-runner needs a rebuild on a current transformers-quality base so prod stops failing on tokenizers; this PR only makes the next occurrence legible instead of expensive.

🤖 Generated with Claude Code

When the normalizer rejects a patch, serge feeds the failure back to the
model and spends a correction. That is right when the patch is at fault
and useless when the gate itself is broken: every correction is rejected
by the same environment error, the budget runs out, and the task reports
"the proposed patch does not pass the repository's normalizer" — which
points the reader at the model instead of at the runner image.

That is what happened on the 2026-08-17 nightly triage
(huggingface/transformers#48037). transformers' main moved its tokenizers
pin to >=0.23.1 while the task-runner image still had 0.22.2, so every
checker that imports transformers died on the version guard, three groups
reported "no fix", and ~3.5M input tokens were spent on an error no patch
could address. The gate installs the worktree with --no-deps (it has to:
the task pod's egress allowlist has no PyPI), so it cannot repair a
dependency drift on its own — it can only notice it.

So notice it: on a normalizer failure, re-run the normalizer once on the
pristine checkout. Clean exit means the gate works and the patch really is
at fault, so nothing changes. Non-zero means no patch can pass, and the
task now fails loudly with NormalizeGateBroken naming the likely cause,
instead of quietly blaming the model.

The check is on the failure path only, so a healthy task pays nothing, and
its verdict is cached for the rest of the task so the extra normalizer run
is paid at most once. It is not a 422: that status makes the runner move
on to the next candidate group, but a broken gate breaks every group.

Both gate call sites get it — the in-loop validation and publish_task's
raw-apply fallback, which produced the message quoted above.

The existing "normalizer rejects the patch" tests used a normalizer that
failed unconditionally, which is now the broken-gate case; they use a
patch-sensitive one so they still test what they claim to.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tarekziade
tarekziade merged commit e23987c into main Aug 18, 2026
3 checks passed
@tarekziade
tarekziade deleted the fix/normalize-gate-baseline branch August 18, 2026 06:48
tarekziade added a commit that referenced this pull request Aug 18, 2026
)

## Why

The task-runner image gets transformers' lint/checker toolchain from
`huggingface/transformers-quality`, which installs
`transformers[quality]` **at its own build time** (nightly, 00:06 UTC).
Every pin in it therefore freezes on that date. When a pin then moves on
`main`, the normalize gate fails for **every** task on a checker
unrelated to the patch — and the gate cannot repair it, because it
installs the worktree with `--no-deps` (mandatory: the task pod's egress
allowlist has no PyPI).

Twice in two days:

| Date | Drift | Effect |
| --- | --- | --- |
| 2026-08-17 | `tokenizers` 0.22.2 in-image vs main's new `>=0.23.1` |
every checker importing transformers died on the version guard — 3
groups lost, ~3.5M input tokens
([transformers#48037](huggingface/transformers#48037))
|
| 2026-08-18 | `transformers-mlinter` 0.1.2 in-image vs the 0.1.4 rule
schema main moved to at 06:45 UTC (`d032a44a9e`) | `ValueError: Invalid
rule spec for TRF054: default_enabled must be bool` |

Reproduced locally against main's `utils/rules.toml`:

```
mlinter 0.1.2 → ValueError: Invalid rule spec for TRF054: default_enabled must be bool
mlinter 0.1.4 → parsed OK — 56 rules
```

The second one appeared on the very morning the image was rebuilt for
the first: the base was built 00:21 UTC, main moved at 06:45 UTC. So **a
rebuild alone is not a fix**, and waiting for the base's next nightly
(00:06 UTC) misses tonight's 22:00 UTC triage.

Worth noting the guard from #88 did its job here — instead of burning
three correction budgets and reporting "the patch does not pass the
normalizer", the three dispatched groups failed loudly as a broken gate
and named the checker. That is how this was diagnosed in minutes.

## What

Re-resolve the quality extra from transformers `main` in
`Dockerfile.task-runner`, after the base. The gate's environment is then
only as stale as *this* image, which a rebuild fixes on demand instead
of waiting on the base. `TRANSFORMERS_REF` is a build arg so an operator
can pin a ref when `main` itself is the problem.

Costs a little build time and image size; buys an environment we can
actually refresh.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant