Skip to content

Never dispatch the verify workflow with an empty model input - #87

Merged
tarekziade merged 1 commit into
mainfrom
fix/verify-empty-model-input
Aug 17, 2026
Merged

Never dispatch the verify workflow with an empty model input#87
tarekziade merged 1 commit into
mainfrom
fix/verify-empty-model-input

Conversation

@tarekziade

Copy link
Copy Markdown
Collaborator

serge-verify-caller.yml declares model as required: true, and GitHub's
workflow_dispatch API counts an empty string as absent:

422 dispatching serge-verify-caller.yml on huggingface/transformers:
  {"message":"Required input 'model' not provided","status":"422"}

extract_verify_targets derives the model from tests/models/<model>/, so any
group whose tests live elsewhere cannot be dispatched at all
tests/generation/ test_utils.py, tests/trainer/, tests/quantization/, …

What it cost on the 2026-08-16 nightly

The generation group (task d8cda8555f49, tracking issue
huggingface/transformers#48004) hit it twice:

  1. The reproduce dispatch failed first. dispatch_failed is not
    not_reproduced, so _maybe_reproduce_first failed open and serge
    investigated blind — the exact thing reproduce-first exists to prevent.
    (There is no serge reproduce generation run in the workflow list, only the
    four for florence2/phimoe/emu3/exaone4_5.)
  2. That blind run: 44 turns, 50 tool calls, 904,008 input tokens, ending in
    the tool-repeat guard (read_file×3, grep×3, grep×3) being forced to answer
    without tools.
  3. It still produced a plausible patch — watermarking_config dict handling in
    GenerationConfig.update, one file, normalizer clean — committed to
    serge/fix/itf-464770b803d3-d8cda855.
  4. The verify dispatch then failed the same way → no PR. The branch was never
    pushed (65 serge/fix/* branches exist on the remote; that fingerprint is not
    among them), so the patch is gone.

The fix

Send "none" instead of "". The input only selects the optional collateral
suite and labels the run, and run_collateral is already forced false when there
is no model folder, so nothing changes for groups that do have one. A named
constant carries the 422 explanation so it does not get "simplified" back.

Tests: extract_verify_targets really does return model == "" for a
tests/generation/… block (the input the fallback exists for), both dispatch
paths send a non-empty model, and a real model still passes through with
run_collateral intact. 649 tests pass.

Not fixed here — worth deciding separately

dispatch_failed still fails open into a blind investigation. A 422 is
deterministic: no retry and no blind run can help, so treating it as a hard skip
would have made this incident cost nothing. A timeout is different and should
keep failing open. That is a behaviour change in the reproduce gate, so it is not
bundled into a bug fix.

Relatedly, model: required: true could be relaxed to required: false +
default: "" on both serge-verify-caller.yml (transformers) and
serge-verify-slow.yml (transformers-ci) so the whole class of bug cannot recur —
but the caller is behind the maintainer allow-list, hence the serge-side fallback
first.

🤖 Generated with Claude Code

`serge-verify-caller.yml` declares `model` as `required: true`, and GitHub's
workflow_dispatch API counts an empty string as absent:

    422 {"message":"Required input 'model' not provided"}

`extract_verify_targets` derives the model from `tests/models/<model>/`, so any
group whose tests live elsewhere -- `tests/generation/test_utils.py`,
`tests/trainer/`, ... -- sends `model: ""` and cannot be dispatched at all.

It bit the 2026-08-16 nightly on the `generation` group (task d8cda8555f49). Both
dispatches failed: the reproduce one first, which is a fail-open path, so serge
investigated blind -- 44 turns, 50 tool calls, 904k input tokens, ending in the
tool-repeat guard -- produced a patch, committed it to
serge/fix/itf-464770b803d3-d8cda855, and then failed the verify dispatch the same
way. No PR, and the branch was never pushed, so that patch is gone.

Send "none" instead. The input only selects the optional collateral suite and
labels the run, and `run_collateral` is already forced false without a model
folder, so nothing downstream changes for groups that do have one.

Not fixed here, worth deciding separately: `dispatch_failed` still fails open into
a blind investigation. A 422 is deterministic -- no retry or blind run can help --
so treating it as a hard skip would have made this cost nothing. A timeout should
keep failing open.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tarekziade
tarekziade merged commit 6d3d752 into main Aug 17, 2026
3 checks passed
@tarekziade
tarekziade deleted the fix/verify-empty-model-input branch August 17, 2026 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant