dashboard: serialize mp3 decode to avoid libsndfile SIGBUS

[ziipo]

Symptom

The dashboard intermittently died with Bus error: 10 roughly 1.4 s after launch, before training started. Once it began happening it appeared deterministic — every restart crashed — which initially looked like a corrupt file in the dataset.

Root cause

It isn't file corruption. The faulting thread is inside libsndfile:

mpeg_init  <- libsndfile_arm64.dylib
mpeg_open
psf_open_file
sf_open
ffi_call_SYSV  <- libffi
cdata_call     <- _cffi_backend (soundfile)
...
thread_run     <- worker thread, not main

The register state is the tell:

• pc = lr = far = 0x1, ESR reports PC alignment — the CPU jumped to address 0x1 and faulted on the instruction fetch itself.
• x2 points at a symbol named jmpbuf.
• Three threads were inside libsndfile simultaneously: psf_fread, mpeg_dec_seek, and mpeg_init.

libsndfile 1.2.2's MPEG decoder reports errors by longjmp-ing through a process-global jmpbuf (libmpg123), which is not thread-safe. With concurrent mp3 decodes, thread A's setjmp context is overwritten by thread B before A's longjmp fires; A then jumps into a stack frame that no longer exists, lands on garbage, and the process takes SIGBUS.

_load_audio is reached from three concurrency sources at once:

• _spec_pool (4 workers, server.py:3007)
• the 16-worker pool at server.py:5867
• ThreadingMixIn on the HTTP server (server.py:7847)

Spectrogram rendering therefore fans out enough simultaneous decodes to hit the race regularly. Runs with more accumulated demo_*.mp3 files trip it sooner, which is what made it look dataset-specific — it was simply the run with the most demos.

Fix

Serialize sf.read behind a module-level lock. The resample below it is pure numpy/torch and stays outside the lock, so only decoding is serialized.

Verification

• 16 threads × 25 rounds (400 concurrent decodes) through the patched _load_audio, three consecutive runs — no crash.
• Sequential decode of all 127 mp3s (23 dataset + 104 demo) passes both before and after the patch, confirming the files themselves are fine.
• Ruled out individually, none of which reproduce it alone: file contents, thread concurrency over stable files, and truncated/partially-written files (every prefix length either errors cleanly or decodes).

One caveat on rigor: I was never able to trigger the crash on demand in a harness, so this is not a before/after A/B. The diagnosis rests on the crash report's register state and the three-threads-in-libsndfile evidence, not on a reproduction.

Trade-offs / alternatives

This is a workaround for an upstream libsndfile/libmpg123 limitation, not a fix for a logic error in underfit. It also serializes all audio decoding, which is a real throughput cost for a dashboard whose job is rendering many spectrograms. Two narrower options if you'd prefer:

1. Lock only mp3 paths — WAV/FLAC don't go through mpeg_init and are unaffected, so they could decode in parallel.
2. Bump the bundled libsndfile — if a newer build makes the MPEG error path thread-safe, the lock becomes unnecessary.

Happy to rework it either way.

🤖 Generated with Claude Code