fix(ci): setup-micromamba failing intermittently on both OSes since the v3.2.1 bump

[bioedca]

Maintenance kind

CI

Work requested

mamba-org/setup-micromamba failed eight times in under six hours on 2026-08-12, across both
Linux and Windows and across three workflows, always in the step that calls
./.github/actions/setup-env and never in anything downstream. Seven were cleared by a plain
re-run; the eighth was superseded by the next push.

#	run created (UTC)	workflow / leg	runner	first error	run id (attempt 1)
1	17:08:13	CI → test (windows-latest)	windows	fetch failed	31621104375
2	17:41:39	schema-guard	ubuntu	fetch failed	31623900214
3	17:41:39	sidecar / parity	ubuntu	fetch failed	31623900269
4	19:05:08	CI → test (windows-latest)	windows	Could not find a micromamba binary for release 2.8.1-0 on platform win-64.	31630899080
5	19:49:52	schema-guard	ubuntu	fetch failed	31634581862
6	20:28:24	CI → test (windows-latest)	windows	fetch failed	31637812688
7	20:48:36	sidecar / parity	ubuntu	fetch failed	31639490512
8	22:58:44	schema-guard	ubuntu	fetch failed	31649126938

All eight are on agent/issue-439 (#441, merged 2026-08-13T04:07:11Z). That pull request touches no
workflow, lock or dependency file — its eleven files are agent-contract prose plus one test module —
so nothing in it can reach the CI toolchain and nothing in it can be the cause.

The step name is not constant, which matters for counting. On origin/main seven workflows
consume the composite under three different names: Restore env + install tether (ci.yml,
schema-guard.yml, deps-audit.yml, large-fixtures.yml), Restore the sidecar env + install tether (no deps) (sidecar.yml, sidecar-measure.yml) and Restore the isolated deep env + install tether (no deps) (deep.yml). A filter on the first name alone drops occurrences 3 and 7.
The substring all three share is install tether.

Every one of the eight also emitted a second error, Unexpected end of JSON input, from the
Post … companion step. That is not an independent failure mode and must not be counted as one.
src/post.ts at the pinned action SHA opens run() with
JSON.parse(core.getState('cacheDownloadsCacheHit')); the main step failed before reaching
core.saveState('cacheDownloadsCacheHit', …), so getState returns '' and JSON.parse('')
throws. It is a deterministic consequence of the main step failing early — 8 of 8.

The measured rate. Over agent/issue-439 between 2026-08-12T16:00Z and 23:05Z — every attempt
of all 39 runs in that window — the composite-setup step ran 78 times and failed 8, i.e. about
10%. That is the useful datum: frequent enough to cost a re-run on most pushes, rare enough that
no single run proves anything.

gh run list cannot recover this. A re-run overwrites a run's conclusion, so occurrences 1–7
all report success there; only occurrence 8 (never re-run) still reads failure. The rate has to
be read per attempt: repos/bioedca/tether/actions/runs/<RUN>/attempts/<n>/jobs, then
repos/bioedca/tether/actions/jobs/<JOB_ID>/logs. Both were confirmed to still serve the failed
attempt after the successful re-run.

One member of a simultaneous batch fails and the other four pass. At 20:28 and again at 20:48,
five composite-setup executions started within two seconds of each other and exactly one failed each
time. At 20:28:27–29: CI run 31637812688's test (ubuntu-latest), test (macos-latest) and
test (windows-latest) legs, schema-guard run 31637812798, and sidecar / parity run
31637812761 — the Windows leg failed (occurrence 6) and the other four passed. At 20:48:38–39:
sidecar / parity run 31639490512, schema-guard run 31639490281, and CI run 31639490286's
full three-OS matrix — the sidecar leg failed (occurrence 7) and the other four passed. A
batch-wide upstream lookup does not fail one member of a simultaneous batch and spare four, so
per-batch contention is not the pattern; if a limit is involved it is account- or runner-wide. Both
of those runs now report run_attempt: 2, but both failures are in attempt 1 — the original
run, not a re-run.

Both messages come from code that exists only in v3.2.1. At the pinned SHA
f457c30a868e4760d3a6fcea5f25dc655b8edf39 (v3.2.1) the module src/micromamba-version.ts defines:

• urlExists, which does fetch(url, { method: 'HEAD', headers: githubApiHeaders() }) with no
  try/catch — so any transport-level failure propagates verbatim as Node's fetch failed and
  fails the step;
• resolveGithubAssetUrl, which probes each candidate URL with urlExists and throws
  Could not find a micromamba binary for release ${tag} on platform ${arch}. when every probe
  returns a non-ok response.

At ce51e99f4bb8a82ab7158c4dc59ef4634c59c4f9 (v3.1.0) that module does not exist at all: src/
contains no micromamba-version.ts and no conda-package.ts. v3.1.0 builds the download URL with
getMicromambaUrl in src/util.ts — pure string construction — and hands it straight to
downloadTool from @actions/tool-cache, with no pre-flight probe. So v3.2.1 added an unretried
network round-trip in front of a download that previously had none, and both observed messages are
reachable only through the new module. That is consistent with the same version succeeding 70 times
in the same window: a transient-sensitivity regression, not a hard incompatibility.

Two corrections to the original filing, since either would mislead whoever picks this up:

• The pinned path does not query the mamba-org/micromamba-releases releases API, and no
  "empty release list" is involved. The API listing is reached only by
  fetchLatestStableMicromambaTag, on the latest path, and that call is wrapped in try/catch with
  a fallback. 2.8.1-0 is a pinned non-prerelease version, so it takes the direct-asset path.
• githubApiHeaders() already attaches Authorization: Bearer … when process.env.GITHUB_TOKEN is
  set. None of the three call sites set it, so authenticating the probe is a one-line env:
  addition rather than a fork — but the probe targets github.com/.../releases/download/…, not
  api.github.com, so whether a token changes anything there is unmeasured.

The 2.8.1-0 pin is correct, and the second message says otherwise. Release 2.8.1-0
(published 2026-06-09, prerelease: false) carries micromamba-win-64.exe, micromamba-win-64 and
micromamba-linux-64. The message names the pinned version and invites someone to "fix"
MICROMAMBA_PIN — which is guarded by
tests/test_marker_contract.py::test_every_setup_micromamba_call_site_pins_the_binary — when what
it actually reports is a failed HEAD probe that says nothing about whether the asset exists. Closing
that trap in writing is the one deliverable here that survives every outcome.

The retry shape is expressible inside a composite action; what it does under failure is not yet
established. runs.steps[*].if and runs.steps[*].continue-on-error are both documented
composite-action step keys in GitHub's Actions metadata-syntax reference
(content/actions/reference/workflows-and-actions/metadata-syntax.md, retrieved via Context7 on
2026-08-13), so a continue-on-error first uses: plus a conditional second one is a legal shape
in .github/actions/setup-env. What is not established is whether a later composite step's if:
can read the first step's steps.<id>.outcome, which is what makes a second attempt fire only after
a failure. Establish that against a recorded source before writing any retry.

As of grooming the symptom has stopped, which is why "note only" is an admissible outcome. Every
repository run created between 2026-08-12T23:01:28Z and 2026-08-13T05:56:42Z is success — 30 runs,
no occurrence after the 22:58 one — and agent/issue-439, the only branch that ever showed it,
merged at 04:07:11Z and is deleted. There may be nothing left to reproduce. That is a dated finding,
not a standing condition: re-check it when the work starts.

Grooming status — 2026-08-13

• Status: unblocked.
• Open dependencies: none. test(ci): guard the setup-micromamba action SHA, not only its binary version #448 overlaps tests/test_marker_contract.py; neither blocks the other.
• Autonomy: agent-can-do-alone.
• Chosen design: the diagnosis is closed at source level, so nothing here waits on reproducing the transient. Both error strings are emitted by urlExists/resolveGithubAssetUrl in src/micromamba-version.ts, a module absent from v3.1.0, and Unexpected end of JSON input is that failure's post.ts companion. What remains is the measurement, the durable note that stops the next person editing MICROMAMBA_PIN, and at most a mitigation that changes no standing configuration. Reverting to v3.1.0 and telling Dependabot to hold it there is deliberately outside this scope — see Scope and non-goals.
• Review risk: high. It is the lane that gates every required check, a Greptile credit is worth spending if the seat has budget, and one candidate mitigation forwards the repository token to a third-party action's fetch — call that out in the pull request rather than letting a reviewer find it.
• Terminal condition: PR-ready handoff.

Acceptance criteria

• The failure rate is measured per attempt, and the sweep is evidenced rather than
  transcribed. The pull request body carries the numerator, the denominator, the window, and —
  for each of the eight failures — the (run id, attempt number, job id) triple whose log it
  was read from. Use repos/bioedca/tether/actions/runs/<RUN>/attempts/<n>/jobs, take the job
  id from each entry's .id (.runner_name is also numeric and 404s against the logs
  endpoint), then repos/bioedca/tether/actions/jobs/<JOB_ID>/logs. gh run list is not
  admissible, because occurrences 1–7 report success there. Match the step by the substring
  install tether, which covers all three names on origin/main — an exact match on
  Restore env + install tether drops occurrences 3 and 7. Reproduce grooming's figure of
  8 failures in 78 executions over agent/issue-439, 2026-08-12T16:00Z–23:05Z, or state and
  justify a corrected one.

• Both error strings are traced to named symbols at the pinned SHA, and v3.1.0's lack of that
  code is shown by retrieval rather than asserted. Name urlExists — fetch(url, { method: 'HEAD', headers: githubApiHeaders() }) with no try/catch — and resolveGithubAssetUrl, which
  throws Could not find a micromamba binary for release ${tag} on platform ${arch}. only after
  every candidate probe returns non-ok, both in src/micromamba-version.ts at
  f457c30a868e4760d3a6fcea5f25dc655b8edf39. Then record the retrievals, their results and the
  retrieval date: repos/mamba-org/setup-micromamba/contents/src/micromamba-version.ts?ref=f457c30a868e4760d3a6fcea5f25dc655b8edf39
  returns the file, while that same path — and src/conda-package.ts — at
  ce51e99f4bb8a82ab7158c4dc59ef4634c59c4f9 returns 404, and at that SHA the URL is built by
  getMicromambaUrl in src/util.ts and handed to downloadTool with no pre-flight probe.
  (Grooming observed both 404s and both 200s on 2026-08-13; a fresh retrieval is what discharges
  this, not a quotation of that sentence.)

• Unexpected end of JSON input is recorded as the same failure's post.ts companion, and
  the failure count is stated as 8, not 16. Quote the two error lines together out of at least
  one of the attempt-1 logs read for criterion 1, and state that the pair appears in all eight.
  The mechanism: run() in src/post.ts opens with
  JSON.parse(core.getState('cacheDownloadsCacheHit')), and main.ts reaches
  core.saveState('cacheDownloadsCacheHit', …) only after the download, so a step that fails
  earlier leaves the state unset and JSON.parse('') throws. Note it is not v3.2.1-specific
  — v3.1.0's post.ts carries the same line — so it is evidence about this failure's ordering,
  never about the bump.

• The MICROMAMBA_PIN trap is closed where the next person will look.
  git grep -n "Could not find a micromamba binary for release" -- .github/actions/setup-env/action.yml
  returns a comment line. It returns nothing on origin/main today, so this criterion is red
  until the comment lands. The comment must state that the message is emitted when every HEAD
  probe returns non-ok and therefore says nothing about whether the tag or its assets exist —
  citing that release 2.8.1-0 (published 2026-06-09, prerelease: false) ships
  micromamba-win-64.exe, micromamba-win-64 and micromamba-linux-64.

• No-regression check — already true on origin/main, and must still hold after the change.
  git grep -l "mamba-org/setup-micromamba@" -- .github/ lists exactly
  .github/actions/setup-env/action.yml, .github/workflows/packaging.yml and
  .github/workflows/release.yml, all three on f457c30a…, and
  pytest tests/test_marker_contract.py::test_every_setup_micromamba_call_site_pins_the_binary
  is green.

• Currently-failing demonstration, required for any guard or mitigation the pull request
  adds. State the exact mutation and the named focused test that goes red, and paste the red
  output; satisfaction by inspection does not count. Concretely: if a fallback adds a second
  uses: line to a file, it must add a second micromamba-version: line to that same file —
  delete that second pin and test_every_setup_micromamba_call_site_pins_the_binary goes red on
  its "every setup-micromamba call site needs its OWN micromamba-version pin" assertion,
  because _setup_micromamba_call_sites compares the per-file uses: count against the
  per-file pin count. A mitigation that adds an env: block adds it to
  .github/actions/setup-env/action.yml — the only call site that restores a conda-lock (that
  test's own docstring says so) and where all eight failures happened. Nothing in the test suite
  asserts anything about an env: block today, so such a mitigation must ship its own guard
  together with the mutation that turns it red when the block is deleted.

• "No mitigation, note only" is an admissible outcome and does not weaken the criteria above.
  The symptom stopped before grooming and the branch it appeared on is gone, so a mitigation may
  turn out to be unwritable without a reproduction. Re-check the green streak when the work
  starts and state what you found rather than inheriting grooming's dates as a live condition.
  If nothing lands beyond the write-up, the pull request is docs-and-comment-only and criteria
  1–4 are the whole deliverable; say so explicitly rather than leaving it implied.

• Local gates green: pre-commit run --all-files, and the matrix line for the lane —
  $env:QT_QPA_PLATFORM='offscreen'; pytest -m "not large and not sidecar and not deep" in
  native PowerShell, QT_QPA_PLATFORM=offscreen pytest -m "not large and not sidecar and not deep"
  in WSL bash. mkdocs build --strict only if a file under docs/ is touched; no schema change
  is expected, so scripts/dump_schema.py --check is not required.

• Ready condition: met. Everything in this body is verified and needs no further grooming.
  Promote status:ready and post a fresh exact-scope marker — the marker binds the SHA-256 of this
  title+body, so it must be the last action.

Scope and non-goals

In scope: measuring the rate per attempt; recording the source-level diagnosis; the durable comment
that stops the next person editing MICROMAMBA_PIN; and, at most, a mitigation that changes no
standing repository configuration — authenticating the probe with env: GITHUB_TOKEN, or a
composite-step retry in .github/actions/setup-env once its failure behaviour is established.

Out of scope:

• Changing MICROMAMBA_PIN itself. 2.8.1-0 is correct, verified above, and guarded.
• Pinning the action back to ce51e99f… (v3.1.0). That reverts two merged Dependabot pull requests
  and, to stick, needs ignore entries for mamba-org/setup-micromamba in both github-actions
  update entries of .github/dependabot.yml — keys are per-entry and are not inherited between
  blocks, as that file's own comments state, and the file validates all-or-nothing so a rejected
  block silently stops every update. chore(ci): stop Dependabot bumping the deliberate setuptools<81 sidecar pin #447 is the closest parallel — an open, unlanded proposal to
  hold a deliberate pip pin against Dependabot — not a precedent that has landed. This is a
  standing dependency-policy change with an ADR attached, so it needs a maintainer-opened issue of
  its own; the mechanics are written out here so filing it costs nothing.
• Reporting the regression upstream to mamba-org/setup-micromamba. Public posting is not this
  issue's to do.
• The action-SHA consistency guard, which is test(ci): guard the setup-micromamba action SHA, not only its binary version #448.
• The stale # v3.0.0 (node24) prose comment that still sits above the v3.2.1 uses: line in
  .github/actions/setup-env/action.yml. Criterion 4 edits a comment in that same block, and it is
  tempting to fix the neighbour while there — do not. test(ci): guard the setup-micromamba action SHA, not only its binary version #448 covers the trailing # v<X.Y.Z> label on
  the uses: line, not this prose block, so no issue currently owns it; leaving it keeps this pull
  request to the single comment criterion 4 asks for.
• Anything about the conda locks.

Size estimate

M — up to 400 added lines

Larger than it looks: a uses: step cannot be wrapped in a run:-based retry action, so a retry has
to be built out of composite-step keys with its failure behaviour established first, and the
measurement is a per-attempt sweep across 39 runs rather than a single query.

Risk level

high — the lane, and a Greptile credit is worth spending if the seat has budget

Does this need an architecture decision record?

no, for everything inside the scope above. The pin-back route would need one — it is a dependency
and isolation choice under AGENTS.md "Agile execution and definition of done" — which is part of why
it sits outside.

Dependencies / blockers

None blocking. #448 shares the setup-micromamba guard in tests/test_marker_contract.py and
should land in a compatible order; neither blocks the other. Do not claim both concurrently if this
one ends up editing that file.

Related work / file overlap

#445 (Dependabot, merged 2026-08-12T02:41:51Z — v3.1.0 → v3.2.1 in /.github/actions/setup-env),
#446 (Dependabot, merged 2026-08-12T06:24:39Z — the actions group, three updates including the
same bump, across .github/workflows), #448 (the action-SHA consistency guard), #447 (an open,
parallel proposal to hold a deliberate pin against Dependabot, in the pip ecosystem), #441 (the
pull request the failures were observed on — agent-contract prose plus one test module, no workflow,
lock or dependency file; merged 2026-08-13T04:07:11Z).

Files: .github/actions/setup-env/action.yml, .github/workflows/packaging.yml,
.github/workflows/release.yml, and tests/test_marker_contract.py only if a guard is added.

Target area

CI / repository governance

Milestone

none / cross-cutting

Required safety attestations

• I searched existing issues and this is not a duplicate.
• I am not reporting a vulnerability publicly; security reports use the private advisory flow.
• I included no secrets or private, raw, unlicensed, user, or lab data.

[bioedca]

Occurrence 5 — schema-guard, 2026-08-12T19:49:59Z, ##[error]fetch failed, same step, cleared by re-run.

Running tally, all on #441 (a prose-only diff, so none of them is caused by the change under test):

#	time (UTC)	leg	error
1	17:08	test (windows-latest)	fetch failed
2	17:41	schema-guard	fetch failed
3	17:41	sidecar / parity	fetch failed
4	19:05	test (windows-latest)	Could not find a micromamba binary for release 2.8.1-0 on platform win-64
5	19:49	schema-guard	fetch failed

Five failures in under three hours, against roughly a dozen successful restores in the same window on the same branch and the same action SHA. That ratio is the useful datum: it is frequent enough to cost a re-run on most pushes and rare enough that no single run proves anything, which is exactly the shape that wastes time when someone eventually investigates without it written down.

Still not concluding a v3.2.1 regression. Same version keeps succeeding, and in occurrence 2/3 the CI workflow passed while two other workflows failed in the same batch — but note occurrences 2, 3 and 5 are all schema-guard/sidecar, which start at the same moment as CI and would contend for the same upstream API. That is consistent with rate-limiting on the release lookup and would be worth checking first.

[bioedca]

Occurrences 6 and 7, both fetch failed, both cleared by re-run:

#	time (UTC)	leg
6	20:28	test (windows-latest)
7	20:48	sidecar / parity

Seven in under four hours. It is now costing a re-run on most pushes to #441, which is the practical reason to treat this as worth fixing rather than tolerating.

One correction to my own earlier note, since it would mislead whoever picks this up: I suggested checking rate-limiting because occurrences 2, 3 and 5 were schema-guard/sidecar starting simultaneously with CI. Occurrences 6 and 7 do not fit that — 6 was a lone Windows leg and 7 a lone sidecar re-run, neither contending with a simultaneous batch. So concurrent contention is not the pattern; if it is rate-limiting it is account- or runner-wide rather than per-batch. Worth measuring before assuming either.

Also worth recording for whoever investigates: re-runs overwrite a run's conclusion, so gh run list reports every one of these seven as success. The failure rate cannot be recovered from the run list after the fact — it has to be counted from job logs or from a deliberate repeated dispatch.

[bioedca]

Approved for agent work: this snapshot's acceptance criteria are accepted and the issue is groomed to agent-can-do-alone.

Editing the title or body after this comment voids the approval - the marker binds the SHA-256 of the exact snapshot above.