CI: build the graphics configuration, and run the test suite

.github/workflows/build.yml

@@ -110,6 +110,114 @@ jobs:
             exit 1
           }
 
+  # ---------------------------------------------------------------------------
+  # THE GRAPHICS LINK, WHICH NOTHING ELSE BUILDS.
+  #
+  # Issue #65: `src/imageio.c` vendors stb_image, raylib's libraylib.a bakes in its own copy, and
+  # with USE_GRAPHICS=1 on Linux the two collided -- dozens of `multiple definition of 'stbi_load'`.
+  # It reached a user because the failing configuration was built by NOBODY: on macOS ld64 tolerates
+  # duplicate symbols across archives where GNU ld makes it an error, and the `build` job above is
+  # allowed to continue when libraylib-dev is unavailable, which on these runners it always is
+  # ("E: Unable to locate package libraylib-dev"). So every CI run built USE_GRAPHICS=0.
+  #
+  # This job exists to make that configuration real, and it must FAIL rather than degrade: a job
+  # that quietly builds the placeholder renderer is what hid the bug in the first place. raylib is
+  # built from source because it is not in this image's package set, which is also why the apt
+  # attempt in `build` kept failing.
+  build-graphics:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y --no-install-recommends \
+            libgmp-dev libmpfr-dev libreadline-dev \
+            libecm-dev liblapacke-dev libopenblas-dev \
+            libflint-dev libpcre2-dev libfftw3-dev \
+            libx11-dev libxrandr-dev libxinerama-dev libxcursor-dev libxi-dev \
+            libgl1-mesa-dev cmake
+
+      # Pinned to 5.5, the version the makefile's detection and the renderer were written against.
+      - name: Build raylib from source
+        run: |
+          git clone --depth 1 --branch 5.5 https://github.com/raysan5/raylib.git /tmp/raylib
+          cmake -S /tmp/raylib -B /tmp/raylib/build \
+            -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF \
+            -DBUILD_EXAMPLES=OFF -DCMAKE_INSTALL_PREFIX=/usr/local
+          cmake --build /tmp/raylib/build -j"$(nproc)"
+          sudo cmake --install /tmp/raylib/build
+          pkg-config --exists raylib || {
+            echo "::error::raylib.pc missing after install — this job cannot verify the graphics link"
+            exit 1
+          }
+          echo "raylib $(pkg-config --modversion raylib) installed"
+
+      - name: Report toolchain
+        run: make print-cc
+
+      # The real thing: a graphics-enabled link against a real raylib.
+      - name: Build with USE_GRAPHICS=1
+        run: make -j"$(nproc)" USE_GRAPHICS=1 EXTRA_LIBS="-llapack -lblas -lX11"
+
+      # The regression test for #65, independent of raylib's own symbol choices: our stb copy must
+      # export NOTHING. `nm` shows `t` (local) for a static function and `T` (global) for an
+      # exported one, and one exported `stbi_*` is all it takes to collide with any other library
+      # that vendors stb -- raylib today, something else tomorrow.
+      - name: Assert the vendored stb symbols are file-local (issue #65)
+        run: |
+          # `_?` because Mach-O prefixes symbols with an underscore; this job is Linux, but the
+          # same line is then usable by hand on a Mac, where ' T stbi_' silently matches nothing.
+          if nm src/imageio.o | grep -qE ' T _?stbi_'; then
+            echo "::error::src/imageio.o exports stbi_* symbols; add STB_IMAGE_STATIC / STB_IMAGE_WRITE_STATIC"
+            nm src/imageio.o | grep -E ' T _?stbi_' | head -10
+            exit 1
+          fi
+          echo "no exported stbi_* symbols in imageio.o"
+
+      - name: Smoke test the image round trip
+        run: |
+          out=$(printf '%s\n' \
+            '{"id": 1, "expr": "ImageDimensions[Import[Export[\"/tmp/ci.png\", Image[Table[N[(i + j)/32], {i, 1, 16}, {j, 1, 24}], \"Real\"]]]]"}' \
+            '{"type": "quit"}' | ./Mathilda)
+          echo "$out"
+          echo "$out" | grep -q '{24, 16}' || {
+            echo "::error::PNG round trip through Import/Export failed"
+            exit 1
+          }
+
+  # ---------------------------------------------------------------------------
+  # THE TEST SUITE, WHICH NOTHING ELSE RUNS.
+  #
+  # Until this job, "the tests pass" was only ever a human claim: CI compiled the tree and ran two
+  # source-level gates, and running the 400+ binaries was left to whoever remembered. A PR then
+  # reported them as passing on the strength of 40 of 426. A machine cannot sample by accident.
+  #
+  # tests/known_failures.txt keeps this honest without making it useless: a listed failure does not
+  # fail the job, an unlisted one does, and a listed test that starts passing is reported so the
+  # line can go. A gate that is red on arrival gets ignored.
+  tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y --no-install-recommends \
+            libgmp-dev libmpfr-dev libreadline-dev \
+            libecm-dev liblapacke-dev libopenblas-dev \
+            libflint-dev libpcre2-dev libfftw3-dev cmake
+
+      - name: Report toolchain
+        run: make print-cc
+
+      # One command, pinned configuration, every binary. See the header of the script for why the
+      # configuration is spelled out rather than inherited from whatever the tree was left in.
+      - name: Build and run every test binary
+        run: tools/run_test_suite.sh
+
   # MPFR is on by default (USE_MPFR=1) and arbitrary-precision reals (EXPR_MPFR)
   # are used across ~130 files, so the `make USE_MPFR=0` graceful-degrade config
   # rots easily: a new unguarded EXPR_MPFR case or *_dispatch_mpfr branch builds

SPEC.md

@@ -363,6 +363,8 @@ make check-nd-surfaces       # do the packed and NDArray surfaces agree?
 make check-compile-coverage  # does every numeric fast path also COMPILE?
 make check-image-packing     # does every image head hand back a packed buffer?
 make check-fastpath-sweep    # measured: is each head really on the buffer?
+make check-tests             # build and RUN every test binary, once, pinned
+make check-menu-ids          # does every native menu item reach a handler?
 ```
 
 `make check-c99` runs `tools/check_c99_portability.py`, which flags POSIX-only
@@ -449,6 +451,29 @@ system library is present (`USE_ECM=1`, autodetected); install it with
 `brew install gmp-ecm` (macOS) or `sudo apt install libecm-dev` (Debian/Ubuntu),
 or build without it via `make USE_ECM=0`.
 
+### Running the whole suite
+
+`make check-tests` (`tools/run_test_suite.sh`) builds every test target and runs every binary in one
+pinned configuration, then reports against `tests/known_failures.txt`: a listed failure is expected,
+an unlisted one fails the run, and a listed test that starts passing is named so its line can be
+deleted.
+
+It exists because the suite is 400+ separate binaries and nothing ran all of them. CI compiled the
+tree and ran two source-level gates; running the tests was left to whoever remembered, and a subset
+is indistinguishable from the whole in a summary — a pull request once reported the suite as passing
+on the strength of 40 of 426 binaries. Two further traps are baked into the script: the configuration
+is pinned rather than inherited (a tree left at `USE_FLINT=OFF` reports 36 failures, 32 of them the
+missing library, and one target that cannot even link), and the per-test timeout is resolved portably
+(`timeout` is GNU coreutils and absent on macOS, where an unguarded call is exit 127 — read by a
+pass/fail loop as a failing test, which once produced a "0 passed, 444 failed" run).
+
+The [Linux CI job](.github/workflows/build.yml) runs it on every push and pull request, alongside a
+`build-graphics` job that links against a real raylib built from source. That second job exists
+because issue #65 — the vendored `stb_image` colliding with raylib's own copy — reached a user
+through a configuration **nobody built**: macOS `ld64` tolerates the duplicate symbols GNU `ld`
+rejects, and the main CI build was allowed to continue when `libraylib-dev` was unavailable, which on
+those runners it always is. It asserts that `src/imageio.o` exports no `stbi_*` symbols at all.
+
 ### Performance regression gate
 
 `tests/bench_assoc.c` guards against the Association operations silently

makefile

@@ -433,6 +433,17 @@ check-interval:
 # happened four times; this reads the dispatch sites out of the source and
 # diffs them against the list. Same "needs python3, so not part of `all`"
 # status as check-c99.
+# `make check-tests` — build and run EVERY test binary, once, in a pinned configuration.
+#
+# Until this existed, "the tests pass" was a claim no tool backed: CI compiled the tree and ran two
+# source-level gates, and running the 400+ binaries was left to whoever remembered — so a PR once
+# reported them as passing on the strength of 40 of 426. The script removes the choice of subset and
+# pins the configuration, because a run in a tree left at USE_FLINT=OFF reported 36 failures of
+# which 32 were the missing library. tests/known_failures.txt carries the standing failures with a
+# reason each, so a NEW failure is loud and the gate is not red on arrival.
+check-tests:
+	tools/run_test_suite.sh
+
 check-packed-aware:
 	python3 tools/check_packed_aware.py
 
@@ -574,7 +585,8 @@ print-cc:
 
 .PHONY: all clean docs docs-build docs-serve check-c99 check-interval check-packed-aware \
         check-array-exactness check-nd-surfaces check-compile-coverage \
-        check-fastpath-sweep check-menu-ids bench-gap check-diophantine-heldout print-cc
+        check-fastpath-sweep check-menu-ids check-tests bench-gap \
+        check-diophantine-heldout print-cc
 
 # Pull in the auto-generated header dependencies. The leading `-` silences the
 # "no such file" notice on a fresh tree (no .d files exist until the first

tests/CMakeLists.txt

@@ -125,8 +125,29 @@ if(USE_FLINT)
         add_compile_definitions(USE_FLINT)
         include_directories(${FLINT_INCLUDE_DIRS})
         link_directories(${FLINT_LIBRARY_DIRS})
-        link_libraries(${FLINT_LIBRARIES})
+        # FLINT_LINK_LIBRARIES (absolute paths) in preference to FLINT_LIBRARIES (bare names that
+        # rely on link_directories reaching the linker). On the CI runner the bare-name form left
+        # every test binary with undefined references to fmpq_mat_det, fmpz_set_mpz and friends --
+        # USE_FLINT was defined, the bridge compiled for real, and -lflint never arrived. Absolute
+        # paths cannot be lost that way.
+        if(FLINT_LINK_LIBRARIES)
+            link_libraries(${FLINT_LINK_LIBRARIES})
+        else()
+            link_libraries(${FLINT_LIBRARIES})
+        endif()
+        message(STATUS "FLINT ${FLINT_VERSION}: ${FLINT_LINK_LIBRARIES}${FLINT_LIBRARIES}")
     else()
+        # DO NOT DEGRADE SILENTLY WHEN IT WAS ASKED FOR. `-DUSE_FLINT=ON` is a statement about what
+        # the caller wants covered; answering it by turning the feature off and printing a warning
+        # into a scrolling log is how a configuration ends up untested while looking green. That is
+        # the exact shape of issue #65. An unrequested default may still fall back.
+        if(DEFINED CACHE{USE_FLINT} AND USE_FLINT)
+            message(FATAL_ERROR
+                "USE_FLINT=ON was requested but FLINT >= 3.0 was not found by pkg-config.\n"
+                "  macOS (Homebrew): brew install flint\n"
+                "  Ubuntu/Debian:    sudo apt install libflint-dev   (needs >= 3.0)\n"
+                "Configure without -DUSE_FLINT=ON to build the tests with FLINT off instead.")
+        endif()
         message(WARNING "FLINT >= 3.0 not detected; building tests with USE_FLINT=OFF")
         message(WARNING "  macOS (Homebrew): brew install flint")
         message(WARNING "  Ubuntu/Debian:    sudo apt install libflint-dev   (needs >= 3.0)")

tests/known_failures.txt

@@ -0,0 +1,29 @@
+# Tests that are known to fail, with the reason each one fails.
+#
+# `tools/run_test_suite.sh` (and the `tests` job in .github/workflows/build.yml) treats a listed
+# failure as expected and an UNLISTED one as a failure of the run. A listed test that starts passing
+# is reported so its line can be deleted.
+#
+# WHY A LIST AND NOT ZERO. The suite is 400+ binaries and some of them fail for reasons that have
+# nothing to do with whatever change is in front of you. A gate that is red the day it lands gets
+# ignored within a week, and then it is worse than no gate: people learn that red means nothing. The
+# ratchet keeps it meaningful — new breakage is loud, standing breakage is written down.
+#
+# EVERY LINE NEEDS A REASON. A bare test name here is a test nobody will ever fix, because nobody
+# will know whether it still matters. One line: what fails, and what would resolve it.
+#
+# Format:  <binary>  <one-line reason>
+
+compiledfunction_tests   test_cf_delegated_array_heads: compiled Variance/StandardDeviation differ from the interpreter by 2 ulp (8.9e-16) against an exact === assertion. Not a packing effect (identical under MATHILDA_NO_PACK=1). Resolve by aligning the two summations or asserting a tolerance.
+iter_tests               test_iter_autocompiled_boundary: expects the string "3." where the printer writes "3.0". The three int64-boundary loops compute correctly (3, 27670116110564327418, 3.0); the expectation is stale.
+ndarray_linalg_tests     test_det_edge: Det of a NON-SQUARE matrix leaks Hold[List] row heads into its result and message and exhausts the recursion limit. Reproduces on a plain List with MATHILDA_NO_PACK=1, so the bug is in Det's non-square guard, not the array substrate.
+simplify_tests           test_simplify_algebraic_u_power_extraction: expects "(-1 + x^2)/x^2^(3/2)", but x^2^(3/2) parses as Power[x, 2 Sqrt[2]] since ^ is right-associative — a different expression. The printer's "(x^2)^(3/2)" is correct and the expectation is wrong.
+
+# Measured on main at 9ee372e3 (2026-08-17): 437 passed, 7 failed. The three below are in the
+# NMinimize/FindMinimum work landed on 2026-08-16/17 and each fails DETERMINISTICALLY (three runs,
+# same exit) — so they are not flaky tests to be re-run, they are convergence assertions the
+# optimizers do not currently meet. They are recorded rather than fixed here because they belong to
+# whoever is iterating on those methods; expected to be short-lived lines.
+basin_hopping_tests      test_styblinski_tang_2d: NMinimize with Method -> {"BasinHopping", "SearchPoints" -> 6} does not reach the Styblinski-Tang minimum (-78.33198) within 1e-2.
+cobyla_tests             test_eq_line: FindMinimum[{x^2 + y^2, x + y == 1}, Method -> "COBYLA", MaxIterations -> 2000] does not reach 0.5 within 1e-4 — the equality-constrained case.
+findmin_methods_tests    test_rosenbrock [ConjugateGradient]: FindMinimum on Rosenbrock from {-1.2, 1} with Method -> "ConjugateGradient", MaxIterations -> 5000 does not converge to (1, 1) within 5e-3.