Skip to content

Add optional LuaJIT backend behind an engine-neutral scripting seam - #83

Open
mmcdole wants to merge 9 commits into
mainfrom
luajit
Open

Add optional LuaJIT backend behind an engine-neutral scripting seam#83
mmcdole wants to merge 9 commits into
mainfrom
luajit

Conversation

@mmcdole

@mmcdole mmcdole commented Jul 22, 2026

Copy link
Copy Markdown
Owner

Summary

Rune's scripting layer now drives its Lua engine through an engine-neutral seam (script package) with two backends: gopher-lua (pure Go, the default — unchanged behavior, no cgo) and LuaJIT 2.1 behind -tags luajit. A pathfinding-shaped benchmark runs ~39x faster on the LuaJIT build (563µs → 14.3µs per iteration through the real engine path).

The seam

script.Engine and script.Call define the whole contract: values obtained during a host call die with the call scope, retaining a script function requires an explicit pin, composite data crosses the boundary as Go trees in single passes, and host functions return errors instead of manipulating a stack. All seventeen API modules, the line type, and the engine dispatch paths are ported. The existing engine test suite passes unchanged on both backends and serves as the backend conformance contract (go test ./lua/ and go test -tags luajit ./lua/).

The scoped-value contract is what keeps the LuaJIT backend simple: registry references are scope-owned and released deterministically — no finalizers, no GC coupling.

Notable fixes along the way

  • Two core-script bugs that only worked by gopher-lua leniency: hooks.call truncated argument lists at an embedded nil (GMCP messages with no body lost their raw argument), and rune.caller_source broke under real Lua 5.1 tail-call elimination. Both fixed engine-agnostically.
  • A probabilistic whole-JIT failure on macOS arm64: LuaJIT's trace machine code must sit within ±128MB of its VM, probed at randomized addresses, and in a Go process that window is often crowded — a losing process thrashes compile/fail/flush and runs slower than gopher-lua. Fixed by static linking plus a loader constructor that reserves branch-range address space before the Go runtime can claim it. TestJITMcodeAllocation reads LuaJIT's own trace log and fails on any mcode allocation failure.

Known caveat

The watchdog cannot interrupt a JIT-compiled loop that never exits its trace (compiled code does not poll debug hooks). Loops touching any host function abort traces and stay interruptible — the realistic runaway class in a scripted client. A bare while true do end does not; every shipping JIT-Lua MUD client has the same exposure. Documented in docs/luajit.md.

CI/CD

  • CI gains a luajit job (Linux, macOS, Windows) running vet and the race-enabled engine suite, so the JIT backend is validated on every push.
  • Releases now attach rune-jit archives (macOS arm64, Linux amd64/arm64, Windows amd64) alongside the goreleaser-built pure rune binaries. Each jit build runs its platform's engine suite first and is gated on being statically linked, so testers need no LuaJIT installed. Beta-style tags publish as prereleases.
  • Windows builds a static libluajit.a from source with mingw-w64 via a composite action; the mcode reservation is POSIX-gated (x64's ±2GB branch range makes the failure mode unlikely there).

Note for reviewers

The Windows lane is validated by construction but has not yet executed — this PR's CI run is its first real test.

mmcdole added 6 commits July 22, 2026 05:06
The luavm seam gains a cgo facade over the LuaJIT 2.1 C API that
implements the same surface the gopher-lua aliases re-export: registry
ref handles for tables/functions/userdata with finalizer-queued
release, a C trampoline for Go callbacks with panic-sentinel error
propagation (no longjmp across Go frames), an asynchronously armed
deadline hook that leaves the JIT enabled, and cgo.Handle-backed
userdata payloads released by __gc.

Two core scripts relied on gopher-lua quirks and are now correct 5.1:
hooks.call preserves argument counts across embedded nils instead of
unpacking through an implementation-defined table border, and
rune.caller_source tolerates tail-call frame elimination when
attributing registrations.

The full engine suite passes on both builds; go test -tags luajit
./lua/ is the backend contract. A pathfinding-shaped benchmark runs
about 36x faster under the LuaJIT build.
The lua package now drives scripting through the script package's
designed contract instead of a gopher-lua-shaped API: host functions
receive call scopes whose values die with the call, retention is an
explicit pin, composite data crosses as Go trees in single passes, and
errors are returned rather than raised through stack machinery. All
seventeen API modules, the line type, and the engine dispatch paths
are ported; the suite passes unchanged on both backends and remains
the backend conformance contract.

script/gopherlua adapts gopher-lua to the seam. script/luajit
implements it natively over the C API and is simpler than the facade
it replaces: scoped registry refs released deterministically instead
of finalizer queues, one-pass tree conversion, and pins as plain
registry slots.

The LuaJIT build also fixes a probabilistic whole-JIT failure: trace
mcode must sit within the arm64 +-128MB branch range of the VM, and
LuaJIT's hardened allocator probes randomized addresses there. In a Go
process that window is often crowded, and a losing process thrashes
compile/fail/flush, running slower than gopher-lua. LuaJIT is now
statically linked so the VM sits in our text segment, and a loader
constructor reserves branch-range address space before the Go runtime
can claim it, released at first state creation. A regression test
reads LuaJIT's trace log and fails on any mcode allocation failure.

Watchdog tests exercise a host-call-bearing runaway on LuaJIT, where
compiled traces cannot be interrupted by hooks but C calls abort
traces; the bare-loop caveat is documented.
The release stays goreleaser-driven for the pure-Go rune binaries;
a native matrix (macOS arm64, Linux amd64/arm64) then builds rune-jit
with -tags luajit, runs that platform's engine suite, verifies the
binary does not dynamically link LuaJIT, and attaches the archives and
checksums to the same release. Beta-style tags publish as prereleases.

Linux gains an explicit static-archive link file so distributed
binaries run without LuaJIT installed, matching the darwin/arm64
rationale. CI adds a luajit job running vet and the race-enabled
engine suite on Linux and macOS. Windows rune-jit is deliberately
deferred until a mingw static lane is qualified.
Windows has no system LuaJIT package, so a composite action builds a
static libluajit.a from source with mingw-w64 and stages it at the
deterministic vendor path link_windows.go expects, exposing the
toolchain to the cgo build steps. The release matrix gains
windows_amd64, packaged as a zip with the same static-linkage gate
(objdump DLL imports), and CI's luajit job now covers Windows so the
lane is validated on every push, not just at tag time.

The mcode branch-range reservation is POSIX-gated and stubbed on
Windows: x64's +-2GB range makes the allocation-failure mode unlikely
there, and sys/mman.h does not exist under mingw.
staticcheck cannot see a method reached only through an anonymous
interface assertion; PinReleaser is now a named, exported part of the
contract. The mcode regression test splices its log path as a Lua
long-bracket string so Windows backslashes are not parsed as escapes.
@codecov

codecov Bot commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 70.89552% with 312 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.06%. Comparing base (8c40c15) to head (ea96d68).

Files with missing lines Patch % Lines
script/gopherlua/engine.go 69.12% 73 Missing and 19 partials ⚠️
script/value.go 68.75% 29 Missing and 6 partials ⚠️
lua/api_ui.go 44.06% 33 Missing ⚠️
script/call.go 43.63% 25 Missing and 6 partials ⚠️
lua/api_store.go 58.53% 14 Missing and 3 partials ⚠️
lua/api_core.go 62.79% 16 Missing ⚠️
lua/api_bar.go 71.15% 10 Missing and 5 partials ⚠️
lua/api_gmcp.go 56.25% 12 Missing and 2 partials ⚠️
lua/engine.go 82.50% 4 Missing and 10 partials ⚠️
script/script.go 48.14% 14 Missing ⚠️
... and 11 more
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main      #83      +/-   ##
==========================================
- Coverage   70.79%   70.06%   -0.74%     
==========================================
  Files          60       65       +5     
  Lines        4726     4981     +255     
==========================================
+ Hits         3346     3490     +144     
- Misses       1197     1309     +112     
+ Partials      183      182       -1     
Files with missing lines Coverage Δ
lua/api_session.go 100.00% <100.00%> (ø)
lua/api_state.go 100.00% <100.00%> (+9.52%) ⬆️
cmd/rune/main.go 47.05% <0.00%> (ø)
lua/backend_gopher.go 50.00% <50.00%> (ø)
lua/api_http.go 95.00% <93.75%> (+7.50%) ⬆️
lua/api_input.go 95.12% <95.00%> (+1.94%) ⬆️
lua/line.go 86.66% <86.66%> (-4.25%) ⬇️
lua/api_bind.go 70.83% <81.25%> (+9.29%) ⬆️
lua/api_log.go 87.50% <86.95%> (+2.88%) ⬆️
lua/api_picker.go 86.95% <89.47%> (+5.82%) ⬆️
... and 13 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

mmcdole added 2 commits July 22, 2026 21:17
rune.engine joins rune.version as a Go-set data field, and the
startup banner, /version, and the -version flag all report it, so
beta testers' reports identify which binary they are running.
Without the LuaJIT headers the tagged targets died inside cgo with a
bare "lua.h: No such file or directory", which says nothing about what
to install. Add a require-luajit prerequisite to the four targets that
compile cgo; it probes the same header paths as script/luajit/link_*.go,
falls back to pkg-config, and otherwise prints the per-platform install
command.

Document those install commands in docs/luajit.md, which previously
described the platform archives without saying how to get them.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant