CI: macos-x86 full suite wedges at position 10 in provisioning.test.ts — host-level runner loss, mechanism unexplained (HVF)

[mobileskyfi]

Status 2026-08-02 — the suite is not what takes an hour. B8a ran four bounded hosted groups on macos-x86 (report): 4/4 green on four runners, per-file cost below the checked-in window on every file, and the 11 unique files sum to 1766 s ≈ 29.4 min against B7's local 1824 s. The hosted runner is now measured — 4 CPUs / 14336 MiB, half the laptop's cores and 22% of its RAM, and still faster per file. Memory does not drift, qemuCount is 0 at all 14 file boundaries, and disk.test.ts is flat across four positions. So 62-65 min cannot be the suite working honestly. Exactly two candidates survive: (a) a mid-suite wedge that GitHub takes tens of minutes to declare (B5 measured a 44.6 min verdict lag on linux-x86; rule 6 says do not transfer the number, but the shape fits), or (b) accumulation appearing only past the ~11 min these groups reached. Next: B8b (full sequential hosted run, ledger-backed) — the only bite that separates them. Four green bounded runs are not evidence that sharding fixes this (#110 rule 6).

Current signature

Across nine macos-15-intel full-suite legs in four runs, each job ended after 59–65 minutes with GitHub's annotation:

The hosted runner lost communication with the server.

This is not a quickchr test timeout and not the configured job timeout:

• the matrix job budget is 300 minutes;
• the test-step budget added in ci: bound the test step, correct the macos-x86 accel claim, and ground #79's www mechanism #108 is 290 minutes;
• Run integration tests and every following if: always() step have conclusion: null;
• the log blob is never finalized and no artifact is uploaded.

The actual runner failure likely precedes GitHub's annotation by its heartbeat grace period, so the useful diagnostic boundary is roughly 40–50 minutes, not 290.

B7 result — per-file cost is flat, and the ~30 minutes are still missing

Local full suite on the maintainer's Intel Mac under the CI-matching HVF baseline (lab and report in #115):

• 30.4 min wall clock, 12/12 pass.
• VM-only ratio against windows-x86: slope +0.002× per file across 10 files, median 0.72×, range 0.66×–0.94×. Cost does not inflate as the suite progresses.
• Nothing accumulates: free+inactive −194 MB over 30 min, wired+compressed −84 MB, swap down, memory pressure never left level 1, zero QEMU orphans.
• The one outlier, provisioning.test.ts at 1.98×, was 619 s of its 992 s with zero QEMU processes alive — uncached pinned images, two retries each. Excluding it: 0.74×. That is a download-deadline defect (Download deadlines are flat totals, so a healthy large transfer fails as a timeout instead of classifying #116), not a leak, and it produced essentially the entire apparent position-vs-cost trend.

platform	files	total	start-stop.test.ts
linux-x86	12	1090 s (18.2 min)	226 s
macos-arm64	12	1366 s (22.8 min)	330 s
windows-x86	12	1603 s (26.7 min)	376 s
macos-x86 (hosted)	— (never completed)	—	358 s (run 30479032655, filtered)
macos-x86 (B7, local Intel/HVF)	12	1824 s (30.4 min)	—

Read it honestly. This rules out a deterministic local Intel/HVF full-suite defect and nothing more. The laptop is 8 physical cores / 64 GiB; the hosted macos-15-intel runner is not, and carrying "no memory pressure at 64 GiB" across environments is exactly the inference #110's operating rule 6 forbids.

Update (B8a, 2026-08-02): the gap is no longer uncharacterized. The hosted runner is 4 CPUs / 14336 MiB. It was measured directly from the leg checkpoint's host snapshot, and the result inverts the worry — at half the cores and 22% of the RAM it runs every file faster than the laptop, with free memory flat across 14 boundaries and no QEMU orphans. Hosted memory pressure is now a measured non-cause up to ~11 min of sustained load; past that it remains untested, which is B8b's job.

A newly-known component: macos-x86 is permanently cold

After the #104 cache redesign (#117), the cache owner is a leg that completes — the save step has no if: always(), deliberately, so a leg torn down mid-suite cannot write partial content under the key. Consequence: while this issue stands, macos-x86 has no cache writer at all and pays a full cold image download on every run. The old rotating-key scheme masked this behind the examples leg's duplicate -ex- write.

So the cold download is part of what macos-x86 costs, not noise to subtract. Since #116 (2899be4) that cost is a measurement rather than a retry-count lottery — the download reports its own bytes/elapsed/throughput, so B8a can attribute it instead of guessing.

Corrections to the original framing

• This runner uses HVF, not TCG. ci: bound the test step, correct the macos-x86 accel claim, and ground #79's www mechanism #108 split the expected accelerator from the measured runtime-budget class and fixed the false summary line.
• There is no evidence that provisioning.test.ts itself hangs. It was simply near the point where an hour-long sequential suite lost the runner — and B7 has since shown its local outlier is download time, not VM time.
• Examples smoke on the same runner class completes in about five minutes, while full suites on other platforms complete in roughly 18–35 minutes.

Why this blocks diagnosis

Runner loss skips in-job cleanup, metrics, summary, and artifact upload. ci-data/tested-versions.json therefore contains zero macos-x86 rows: failed full-suite attempts are masked by omission rather than recorded as incomplete. The step-level cap in #108 is structurally useful for an ordinary hung step, but its 290-minute value cannot fire before this runner disappears.

Plan — two hosted phases, not one

Tracked under #110. #77 remains the instrumentation prerequisite for the second phase only; B7's result is what makes the first phase runnable earlier.

Phase 1 — B8a: bounded hosted file groups (unblocked)

Groups sized to finish comfortably below the observed death boundary, so the experiment does not need a ledger that survives runner loss. B4's watchdog landing first makes it strictly better — a hung group classifies instead of vanishing — but is not a hard blocker.

#116 was the blocker and it landed (2899be4): every group pays the cold download described above, and with the old flat per-attempt deadline a group's duration would have measured the deadline bug rather than the platform. A download now reports bytes/expected/elapsed/throughput, so that component can be counted honestly instead of inferred.

The question is what the hosted environment adds that the laptop does not: per-file duration and the resource sample at each group boundary, compared against B7's local trend on the same files. Count the cold download as a named component.

Phase 2 — B8b: full sequential hosted run (blocked by B5, and by B8a's data)

Compare cumulative sequential execution against the isolated groups. That contrast is what distinguishes one blocking file, cross-file accumulation that appears only under hosted pressure, and an external runner ceiling — three outcomes that look identical from a lost runner today. B5's ledger is what lets this survive the runner vanishing.

Then

Verify cleanup between files and identify the last healthy resource trend before considering permanent sharding. If no quickchr/resource defect is found and a hosted-runner ceiling remains, document that evidence and then design a stable sharding/budget scheme. Do not shard first in a way that hides a cumulative leak.

Done-when

• macos-x86 full-suite attempts leave a server-visible terminal class and last progress/resource marker even when the runner is lost.
• A local HVF run (done — B7, flat) and bounded CI experiments distinguish a specific quickchr/QEMU/resource defect from an external runner limit.
• The underlying defect is fixed and a full sweep completes, or an evidence-backed sharding scheme covers the full suite within the real runner boundary without hiding failures.
• tested-versions.json records macos-x86 as pass/fail/incomplete instead of silently omitting the platform.
• Whatever the outcome, macos-x86 ends up with a cache writer — either because its suite completes, or because a "file loop finished" marker lets a leg that completed the loop but failed a test own the key.

Refs #77, #116, #110. Historical observations remain in the comments below.

[mobileskyfi]

Post-mortem attempt: the runner starvation theory is now grounded — by total evidence loss

After cancellation (submitted ~18:51, job marked cancelled 18:59:03), the step timeline shows the failure mode exactly:

• Run integration tests (sequential per-file) — frozen in_progress, never concluded
• Every subsequent step stayed pending and never ran — including the three if: always() steps (Kill leftover QEMU processes, Assemble metrics, Upload integration logs)
• No artifact was uploaded, and the job's log blob does not exist (BlobNotFound, retried for several minutes after completion) — the runner never even flushed its logs

So the runner didn't just hang the test process: it stopped servicing GitHub's cancellation grace period (SIGINT → always() steps) and log shipping entirely. The only record of the leg's output is the live-streamed view captured in this issue's description.

Consequences for the investigation:

1. CI cannot answer this one — there is nothing more to extract from hosted runs. The local forced-TCG repro on the Intel Mac (done-when fix(net): try normal fetch() first, fail back to public-DNS IPv4 #2) is the only evidence path, and it's a better one anyway (we can watch top, sample the bun process, and check for Keychain prompts directly).
2. This failure mode also invalidates the workflow's own safety net in exactly this scenario: the QEMU-reap step is unreachable when orphans have already starved the runner. CI: evidence-preserving per-file watchdog and incremental run ledger #77's "reap on per-file cap-hit, inside the loop" is the structural fix — the reap must happen while the runner can still execute anything at all.
3. Timeline detail worth keeping: the leg was healthy for ~1h (sequential 40-62s boots, KVM-comparable), then went from fine to fully unresponsive with no intermediate degradation visible in test durations. Whatever starved it did so abruptly after createUser with an invalid group throws PROCESS_FAILED passed — consistent with a specific event (a boot that spun without ever failing, or a blocking prompt) rather than gradual TCG load accumulation.

[mobileskyfi]

Additional evidence from run https://github.com/tikoci/quickchr/actions/runs/28758858863:

• Two more macos/x86_64 full-suite integration legs failed: long-term job 85270515871 and testing job 85270515849.
• Both jobs died while still in Run integration tests (sequential per-file). Step metadata shows no cleanup, metrics, summary, or artifact upload steps ran.
• gh run view --job ... --log, gh run view --log-failed, and the Actions job-log API all report no log for those two failed jobs, so this is the same total-evidence-loss failure mode as the original report.
• The matrix did plan macos-x86 with timeout:300 for this run (platforms=all, routeros-targets=long-term,testing, empty test-filter, tcg-smoke=false), yet the jobs ended around 57-62 minutes. That points at hosted runner failure/lost communication or an external macOS job cap, not our matrix timeout.
• The macos/x86_64 examples smoke jobs for both targets passed in the same run, so the failure is specific to the full integration suite.
• New accelerator caveat: successful issue69-settling.probe.ts artifacts on the same macos-15-intel runner report quickchr detectAccel(x86): hvf, not TCG. The issue title/body likely over-attributes this to TCG; the current grounded bucket is “macos-x86 full-suite runner/job starvation with no logs/artifacts,” not proven TCG.
• No ci-data files exist for these failed macos-x86 integration legs because no metrics artifact was uploaded.

Related but separate: #79 tracks the new linux/aarch64 · long-term clean() post-reset boot timeout from the same run.

[mobileskyfi]

Fourth occurrence: run 29061855653 (2026-07-10)

Same total-evidence-loss signature recurred in job Integration (macos/x86_64 · stable):

• Job conclusion: failure, but every step from Run integration tests (sequential per-file) onward shows conclusion: null — never actually
  concluded, matching the earlier "frozen in_progress, always() steps never
  ran" pattern.
• GET .../actions/jobs/86265259243/logs → BlobNotFound (job never flushed
  logs), same as the prior occurrences.
• The job's check-run annotation this time is explicit and confirms the
  standing theory: "The hosted runner lost communication with the server.
  Anything in your workflow that terminates the runner process, starves it for
  CPU/Memory, or blocks its network access can cause this error." — this is
  the clearest first-party confirmation yet that it's runner-side
  starvation/disconnection, not a quickchr hang per se.
• No artifact uploaded for this leg, so — as with the prior three — there's
  nothing new to extract from this run specifically; the local forced-TCG
  repro (done-when fix(net): try normal fetch() first, fail back to public-DNS IPv4 #2) is still the only path to root-cause which test/step
  triggers the starvation.
• Unrelated, same run: windows/x86_64 · stable also failed in this run, but
  that's a distinct, newly-filed issue (CI: Windows/x86_64 host-OpenSSH batch login (batchVerified) fails 100% in first exercised run #87 — real SSH batch-login failures),
  not this flake.

[mobileskyfi]

Recurred on all three macos-x86 legs — and narrow runs prove it is the full suite, not the platform

Part of the ci:slow-platform-flake series (#76 #79 #80 #91).

Run A — 30279318703, full matrix

All three macos-15-intel Integration legs failed identically:

Job	Started → completed	Elapsed
Integration (macos/x86_64 · long-term)	15:18:47 → 16:22:54	64 min
Integration (macos/x86_64 · testing)	15:18:18 → 16:24:21	66 min
Integration (macos/x86_64 · stable)	15:43:25 → 16:47:32	64 min

Every one carries the same check annotation:

The hosted runner lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

Step conclusions show Run integration tests (sequential per-file) with conclusion: null — it never finished. No logs and no artifacts: GET /actions/jobs/<id>/logs returns BlobNotFound, and no integration-logs-macos-x86-* artifact exists for the run. The job died before the upload step, so there is nothing to post-mortem. This is the same ~64-minute signature the issue already describes as the historical macos-x86 shape, now reproduced three-for-three.

Note the matrix budget was not the limiter: macos-x86 with no filter and tcg-smoke=false gets timeout=300 minutes, so these died at ~64 min for their own reasons, not a workflow timeout.

Run B — 30289164062, narrow

test-filter=start-stop.test.ts,disk.test.ts, run-examples=false:

Job	Result
Integration (macos/x86_64 · stable)	pass
Integration (macos/x86_64 · long-term)	pass

Both green, logs and artifacts intact. So the platform runs quickchr fine under TCG — what kills it is the duration/CPU-burn of the full suite, consistent with this issue's leading theory (leftover TCG QEMU pinning the cores; the reap step only runs after the loop). It supports the starvation reading over any per-test defect.

Interaction with the rest of the series

Run A also produced a Monitor command timed out on macos/x86_64 · stable in the examples-smoke job (via the rollback example) — recorded in #80, since that leg did produce logs and is a different failure mode from the runner death here.

Bearing on #77

This is the strongest argument yet for #77 (per-file runtime caps from ci-data timing): when this leg dies there is zero diagnostic residue — no logs, no artifacts, no annotation beyond the generic runner message. A cap that fails fast and honestly, mid-suite, would convert a 64-minute silent death into an actionable failure with logs attached. Until then, tcg-smoke=true for macos-x86 bounds the cost but also bounds the coverage — worth deciding explicitly rather than by default.