Never dispatch the verify workflow with an empty model input - #87
Merged
Conversation
`serge-verify-caller.yml` declares `model` as `required: true`, and GitHub's
workflow_dispatch API counts an empty string as absent:
422 {"message":"Required input 'model' not provided"}
`extract_verify_targets` derives the model from `tests/models/<model>/`, so any
group whose tests live elsewhere -- `tests/generation/test_utils.py`,
`tests/trainer/`, ... -- sends `model: ""` and cannot be dispatched at all.
It bit the 2026-08-16 nightly on the `generation` group (task d8cda8555f49). Both
dispatches failed: the reproduce one first, which is a fail-open path, so serge
investigated blind -- 44 turns, 50 tool calls, 904k input tokens, ending in the
tool-repeat guard -- produced a patch, committed it to
serge/fix/itf-464770b803d3-d8cda855, and then failed the verify dispatch the same
way. No PR, and the branch was never pushed, so that patch is gone.
Send "none" instead. The input only selects the optional collateral suite and
labels the run, and `run_collateral` is already forced false without a model
folder, so nothing downstream changes for groups that do have one.
Not fixed here, worth deciding separately: `dispatch_failed` still fails open into
a blind investigation. A 422 is deterministic -- no retry or blind run can help --
so treating it as a hard skip would have made this cost nothing. A timeout should
keep failing open.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
serge-verify-caller.ymldeclaresmodelasrequired: true, and GitHub'sworkflow_dispatch API counts an empty string as absent:
extract_verify_targetsderives the model fromtests/models/<model>/, so anygroup whose tests live elsewhere cannot be dispatched at all —
tests/generation/ test_utils.py,tests/trainer/,tests/quantization/, …What it cost on the 2026-08-16 nightly
The
generationgroup (taskd8cda8555f49, tracking issuehuggingface/transformers#48004) hit it twice:
dispatch_failedis notnot_reproduced, so_maybe_reproduce_firstfailed open and sergeinvestigated blind — the exact thing reproduce-first exists to prevent.
(There is no
serge reproduce generationrun in the workflow list, only thefour for florence2/phimoe/emu3/exaone4_5.)
the tool-repeat guard (
read_file×3, grep×3, grep×3) being forced to answerwithout tools.
GenerationConfig.update, one file, normalizer clean — committed toserge/fix/itf-464770b803d3-d8cda855.pushed (65
serge/fix/*branches exist on the remote; that fingerprint is notamong them), so the patch is gone.
The fix
Send
"none"instead of"". The input only selects the optional collateralsuite and labels the run, and
run_collateralis already forced false when thereis no model folder, so nothing changes for groups that do have one. A named
constant carries the 422 explanation so it does not get "simplified" back.
Tests:
extract_verify_targetsreally does returnmodel == ""for atests/generation/…block (the input the fallback exists for), both dispatchpaths send a non-empty model, and a real model still passes through with
run_collateralintact. 649 tests pass.Not fixed here — worth deciding separately
dispatch_failedstill fails open into a blind investigation. A 422 isdeterministic: no retry and no blind run can help, so treating it as a hard skip
would have made this incident cost nothing. A timeout is different and should
keep failing open. That is a behaviour change in the reproduce gate, so it is not
bundled into a bug fix.
Relatedly,
model: required: truecould be relaxed torequired: false+default: ""on bothserge-verify-caller.yml(transformers) andserge-verify-slow.yml(transformers-ci) so the whole class of bug cannot recur —but the caller is behind the maintainer allow-list, hence the serge-side fallback
first.
🤖 Generated with Claude Code