Skip to content

Add a macOS Apple Silicon CI job - #247

Merged
neuromechanist merged 2 commits into
devfrom
246-macos-ci
Aug 16, 2026
Merged

Add a macOS Apple Silicon CI job#247
neuromechanist merged 2 commits into
devfrom
246-macos-ci

Conversation

@neuromechanist

Copy link
Copy Markdown
Member

Closes #246 (first half: the MLX/Accelerate gap).

Why

Every job in ci.yml was ubuntu-latest. Two things were therefore untested:

  • pamica/tests/mlx_tests/ had never run in CI. MLX is the recommended
    Apple GPU backend per AGENTS.md; pyproject.toml even excludes
    pamica/mlx_impl/* from coverage with the comment that CI "cannot exercise
    it and always measures 0%".
  • The Accelerate BLAS numerical path. Not theoretical: the
    test_mir_history_survives_keep_best_restore assertion was green on Weight nd by pre-update model weights #241's
    PR CI and failed immediately after merge on macOS. The cause was not a stale
    base — the ~1e-6 trajectory shift moved the stopping iteration from 55 to 54,
    and the assertion only held when the stop landed on a multiple of mir_step.
    Linux never saw it. Adding CI on dev pushes would not have caught this
    either; only a macOS runner would.

What

One test-macos job on macos-26 (current arm64 image; macos-14 is arm64 too
but deprecated). Standard runners are free and unmetered for public repos, so
this costs nothing — deliberately not a -large/-xlarge variant, which are
billed even here.

Verified locally on macOS 26.6.1 arm64 before pushing:

  • mlx.core.__version__ -> 0.32.0, so the import assertion is real
  • LAPACK_LIBS="-framework Accelerate" bash native/build.sh builds cleanly
  • pamica/tests/mlx_tests/ collects 11 tests and runs

The MLX import is asserted explicitly rather than left to mlx_tests'
self-skip: a silent skip would leave this job green while testing nothing,
which is the exact failure mode it exists to prevent.

--no-cov because the Linux job owns the coverage gate and the Codecov upload;
a second partial run would report a different total for the same commit.

Deliberately not in scope

  • Enabling the slow Fortran-parity tests on macOS. They drive the bundled
    x86_64 amica15mac, which needs Rosetta on an arm64 runner. Pointing them at
    the arm64 binary this job already builds is the natural follow-up; left in
    CI runs Linux only: MLX backend and the Apple Silicon numerical path are never tested #246.
  • push: branches: [dev]. Worth doing, but a separate concern, and note it
    would not have caught the failure described above.
  • A self-hosted runner. sccn/pAMICA is public and GitHub advises
    self-hosted runners only for private repos (a fork PR can run arbitrary code
    on an unsandboxed runner). Hosted macos-26 covers the same ground with no
    exposure. Self-hosted remains the only option for CUDA.

neuromechanist and others added 2 commits August 15, 2026 23:22
Every job was ubuntu-latest, so pamica/tests/mlx_tests/ had never run in CI
and the Accelerate BLAS path was untested. The second gap is not theoretical:
a test_ng_convergence.py assertion stayed green on Linux through #241 and
failed immediately on macOS, because the ~1e-6 trajectory shift moved the
stopping iteration off the multiple of mir_step the assertion assumed.

macos-26 is the current arm64 image and standard runners are free for public
repositories. The job installs the mlx extra and asserts the import rather
than letting mlx_tests self-skip into a green no-op.

Refs #246.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review findings.

Assert mx.default_device().type == gpu, not just that mlx imports. The
mlx_tests module is gated on both (test_mlx_backend.py:28-33), so a runner
that imported mlx and fell back to CPU would skip all 11 tests and still
report success. The hosted macos-26 image does expose the GPU -- all 11
passed on the first run -- but nothing in the job would have caught it if a
future image stopped.

Add PYTORCH_ENABLE_MPS_FALLBACK=1. The two Linux jobs already set it
defensively; this is the only job whose hardware can actually auto-select
MPS.

Add EXTRA_LDFLAGS=-static-libquadmath so the build really is the
release-binaries.yml macos-arm64 recipe the comment claims.

Correct the -m 'not slow' rationale: it was attributed to Rosetta and the
x86_64 amica15mac, but most slow tests never invoke a binary, and the two
that do resolve through PAMICA_NATIVE_BINARY (native/resolver.py:112), which
this job sets to the arm64 binary it just built. They are excluded for
wall-clock, and enabling them is a real follow-up rather than blocked work.

Refs #246.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@neuromechanist

Copy link
Copy Markdown
Member Author

Findings addressed in da41ebd. One correction on the Critical, with evidence.

Critical 1 — "the MLX tests will very likely still self-skip": empirically not the case. I checked the completed run's log rather than reasoning about it. On the hosted macos-26 runner, all 11 MLX tests PASSED:

PASSED pamica/tests/mlx_tests/test_mlx_backend.py::test_sufficient_stats_match_numpy_reference
PASSED pamica/tests/mlx_tests/test_mlx_backend.py::test_converged_ll_matches_torch_float32
PASSED pamica/tests/mlx_tests/test_mlx_backend.py::test_multimodel_matches_torch_float32
... (11 total)
============ 402 passed, 10 skipped, 1 xfailed in 582.91s (0:09:42) ============

The 10 skips are test_fortran_adapter.py (3) and test_ng_pdf_families.py (5) plus two others — no MLX test among them. So this runner does expose Metal to MLX and mx.default_device().type == gpu holds. The cited upstream reports are real but do not describe this image.

That said, the finding is right as insurance and I have implemented it. You correctly spotted the second gate at test_mlx_backend.py:28-33 that my import assertion did not cover — if a future runner image loses GPU passthrough, the job would go green with 11 silent skips. The step now asserts mx.default_device().type == mx.DeviceType.gpu.

Important 2 — the -m "not slow" rationale was wrong. Confirmed both halves. test_sample_data.py's slow tests never invoke a binary, and the two in test_ng_backend.py (:1096, :1179) pass binary_path=None, which resolves through PAMICA_NATIVE_BINARY first (native/resolver.py:112) — already set by this job to the arm64 binary it builds. Rosetta was never involved. Comment corrected to say they are excluded for wall-clock, and that enabling them would give CI its first Fortran parity coverage.

Important 3 — PYTORCH_ENABLE_MPS_FALLBACK. Added. This is indeed the only job whose hardware can auto-select MPS.

Important 4 — EXTRA_LDFLAGS. Added -static-libquadmath so the recipe genuinely matches release-binaries.yml. Verified the build locally with both flags: arm64 Mach-O, clean.

Suggestion — stale coverage comment. Updated to state why the omission is still required (this job runs --no-cov, so the total still comes only from the ubuntu job) rather than leaving a claim a future reader would find false.

Suggestion — release-binaries.yml still on macos-14. Good catch on the November 2 2026 EOL; filing separately rather than widening this PR.

@neuromechanist
neuromechanist merged commit f38d056 into dev Aug 16, 2026
8 checks passed
@neuromechanist
neuromechanist deleted the 246-macos-ci branch August 16, 2026 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant