Resolve the quality toolchain from transformers main at image build - #89
Merged
Merged
Conversation
The task-runner image inherits transformers' lint/checker toolchain from
huggingface/transformers-quality, which installs transformers[quality] at
its own build time. Every pin therefore freezes on the base's build date,
and when one moves on main the normalize gate fails for every task on a
checker unrelated to the patch. The gate cannot repair it: it installs the
worktree with --no-deps, mandatory because the task pod's egress allowlist
has no PyPI.
Twice in two days:
2026-08-17 tokenizers 0.22.2 in-image vs main's new >=0.23.1 pin.
Every checker that imports transformers died on the version
guard; 3 groups lost, ~3.5M input tokens (transformers#48037).
2026-08-18 transformers-mlinter 0.1.2 in-image vs the 0.1.4 rule schema
main moved to at 06:45 UTC. "Invalid rule spec for TRF054:
default_enabled must be bool" — reproduced locally: 0.1.2
raises it on main's utils/rules.toml, 0.1.4 parses 56 rules.
The second one landed the same morning the image was rebuilt for the
first, because the base was built at 00:21 UTC and main moved at 06:45 —
so a rebuild alone is not a fix, and waiting for the base's next nightly
(00:06 UTC) misses tonight's 22:00 UTC triage entirely.
Re-resolving the quality extra here makes the gate's environment only as
stale as this image, which a rebuild fixes on demand. TRANSFORMERS_REF is
a build arg so an operator can pin a ref when main itself is the problem.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The task-runner image gets transformers' lint/checker toolchain from
huggingface/transformers-quality, which installstransformers[quality]at its own build time (nightly, 00:06 UTC). Every pin in it therefore freezes on that date. When a pin then moves onmain, the normalize gate fails for every task on a checker unrelated to the patch — and the gate cannot repair it, because it installs the worktree with--no-deps(mandatory: the task pod's egress allowlist has no PyPI).Twice in two days:
tokenizers0.22.2 in-image vs main's new>=0.23.1transformers-mlinter0.1.2 in-image vs the 0.1.4 rule schema main moved to at 06:45 UTC (d032a44a9e)ValueError: Invalid rule spec for TRF054: default_enabled must be boolReproduced locally against main's
utils/rules.toml:The second one appeared on the very morning the image was rebuilt for the first: the base was built 00:21 UTC, main moved at 06:45 UTC. So a rebuild alone is not a fix, and waiting for the base's next nightly (00:06 UTC) misses tonight's 22:00 UTC triage.
Worth noting the guard from #88 did its job here — instead of burning three correction budgets and reporting "the patch does not pass the normalizer", the three dispatched groups failed loudly as a broken gate and named the checker. That is how this was diagnosed in minutes.
What
Re-resolve the quality extra from transformers
maininDockerfile.task-runner, after the base. The gate's environment is then only as stale as this image, which a rebuild fixes on demand instead of waiting on the base.TRANSFORMERS_REFis a build arg so an operator can pin a ref whenmainitself is the problem.Costs a little build time and image size; buys an environment we can actually refresh.
🤖 Generated with Claude Code