Description
MusiCNN ONNX model inference crashes with SIGSEGV (signal 11) on CPUExecutionProvider using the :latest Docker image (ghcr.io/neptunehub/audiomuse-ai:latest). This happens on both AMD and Intel CPUs, rendering CPU-only operation completely unusable for library analysis.
Every single analysis job that reaches MusiCNN inference crashes immediately. Workers churn through retries but no track ever completes because the ONNX inference segfaults every time.
Environment
| Item |
Detail |
| AudioMuse-AI Image |
ghcr.io/neptunehub/audiomuse-ai:latest |
| Image variants tested |
:latest and :latest-noavx2 — both crash |
| Deployment |
Kubernetes (Talos Linux), raw manifests |
| Worker type |
SERVICE_TYPE=worker |
| Worker count |
15 RQ workers (12 default + 3 high-priority) — worker count doesn't matter, single worker also crashes |
| onnxruntime version |
1.19.2 (pinned in requirements/cpu.txt and requirements/cpu-noavx2.txt) |
| ONNX provider |
CPUExecutionProvider (no GPU available in this pod) |
CPUs Tested — Both Crash
Intel: Intel(R) Xeon(R) W-2133 @ 3.60GHz (Skylake-W, 2017)
- 6 cores / 12 threads
- AVX2 ✅, AVX-512 ✅ (F/CD/DQ/BW/VL), F16C ✅, SSE4.1/4.2 ✅
- All modern SIMD instructions available — this is NOT an AVX2-missing issue
AMD: AMD Ryzen 7 PRO 6850H (Zen 3+, 2022)
- 8 cores / 16 threads
- AVX2 ✅, F16C ✅, SSE4.1/4.2 ✅
Both CPUs have full AVX2 support. This is NOT the same bug as:
Crash Log Pattern
[INFO] Downloading track to /app/temp_audio/31055.flac
[INFO] Lazy-loading MusiCNN models for album: Summer in Siam
[INFO] ONNX provider chain: ['CPUExecutionProvider']
[INFO] OK Loaded 2 MusiCNN models for album reuse
[INFO] Starting analysis for: 31055.flac
[WARNING] Worker a0c3bcbcc796: job ffa0835e-e1d0 failed (Work-horse terminated unexpectedly; waitpid returned 11 (signal 11);)
[WARNING] Work horse killed for job ffa0835e-e1d0: retpid=176871, ret_val=11
[INFO] Job ffa0835e-e1d0: enqueued for retry, 0 remaining
Key observations:
- MusiCNN models load successfully (
OK Loaded 2 MusiCNN models for album reuse)
- The crash happens during inference — the worker subprocess is killed by onnxruntime's C++ segfault
- No Python traceback (worker subprocess dies before Python can catch it)
- PostgreSQL logs fill with
unexpected EOF on client connection with an open transaction — workers die mid-transaction
- Jobs retry but every retry crashes identically — no track ever completes
129+ crashes in a single 2000-line log window. Scale: ~1900 worker process deaths in 30 minutes across 15 workers.
ONNX Model Details
We downloaded and inspected both MusiCNN models:
musicnn_embedding.onnx: FP32 (56 FP32 initializers, 0 FP16), opset 13, IR version 7
musicnn_prediction.onnx: FP32 (6 FP32 initializers, 0 FP16), opset 13, IR version 7
Since the models are FP32 (not FP16), microsoft/onnxruntime#15494 (fp16 model + CPUExecutionProvider segfault) is NOT the cause.
Dependency
Both requirements/cpu.txt and requirements/cpu-noavx2.txt pin:
onnxruntime v1.19.2 is significantly behind. Current latest is v1.27.1 (2026-07-11). Relevant fixes in between:
- v1.20.0 — Stopped registering MLFloat16 primary Gemm kernel for CPU EP
- v1.21.0+ — Additional CPU execution stability fixes
The Dockerfile already sets ORT workaround env vars that don't prevent this:
ENV ORT_DISABLE_ALL_OPTIMIZATIONS=1
ENV ORT_EABLE_CPU_FP16_OPS=0
ENV ORT_DISABLE_AVX512=1
ENV ORT_DISABLE_MEMORY_PATTERN_OPTIMIZATION=1
Additional Context
- GPU execution (CUDAExecutionProvider) works for MusiCNN models (tested on RTX 3060) but CLAP crashes intermittently
- Worker count doesn't matter — 1 worker or 15, all workers die on first MusiCNN inference
- Both
:latest and :latest-noavx2 crash identically
- Makes AudioMuse-AI completely unusable for CPU-only deployments regardless of CPU architecture
Suggested Fix
Bump onnxruntime from 1.19.2 to >= 1.20.0 (ideally latest 1.27.1) in requirements/cpu.txt and requirements/cpu-noavx2.txt. The FP32 model + CPUExecutionProvider segfault is likely a bug fixed between 1.20.0 and 1.27.1.
Description
MusiCNN ONNX model inference crashes with SIGSEGV (signal 11) on
CPUExecutionProviderusing the:latestDocker image (ghcr.io/neptunehub/audiomuse-ai:latest). This happens on both AMD and Intel CPUs, rendering CPU-only operation completely unusable for library analysis.Every single analysis job that reaches MusiCNN inference crashes immediately. Workers churn through retries but no track ever completes because the ONNX inference segfaults every time.
Environment
ghcr.io/neptunehub/audiomuse-ai:latest:latestand:latest-noavx2— both crashSERVICE_TYPE=workerCPUExecutionProvider(no GPU available in this pod)CPUs Tested — Both Crash
Intel: Intel(R) Xeon(R) W-2133 @ 3.60GHz (Skylake-W, 2017)
AMD: AMD Ryzen 7 PRO 6850H (Zen 3+, 2022)
Both CPUs have full AVX2 support. This is NOT the same bug as:
audiomuse-ai-flask-appas of 2.0.1 (1.1.6 and 2.0.1-noavx2 both work) #561 (SIGILL on CPUs without AVX2 — our CPUs have AVX2)Crash Log Pattern
Key observations:
OK Loaded 2 MusiCNN models for album reuse)unexpected EOF on client connection with an open transaction— workers die mid-transaction129+ crashes in a single 2000-line log window. Scale: ~1900 worker process deaths in 30 minutes across 15 workers.
ONNX Model Details
We downloaded and inspected both MusiCNN models:
musicnn_embedding.onnx: FP32 (56 FP32 initializers, 0 FP16), opset 13, IR version 7musicnn_prediction.onnx: FP32 (6 FP32 initializers, 0 FP16), opset 13, IR version 7Since the models are FP32 (not FP16), microsoft/onnxruntime#15494 (fp16 model + CPUExecutionProvider segfault) is NOT the cause.
Dependency
Both
requirements/cpu.txtandrequirements/cpu-noavx2.txtpin:onnxruntime v1.19.2 is significantly behind. Current latest is v1.27.1 (2026-07-11). Relevant fixes in between:
The Dockerfile already sets ORT workaround env vars that don't prevent this:
Additional Context
:latestand:latest-noavx2crash identicallySuggested Fix
Bump
onnxruntimefrom 1.19.2 to >= 1.20.0 (ideally latest 1.27.1) inrequirements/cpu.txtandrequirements/cpu-noavx2.txt. The FP32 model + CPUExecutionProvider segfault is likely a bug fixed between 1.20.0 and 1.27.1.