Skip to content

fix: free local llama-cpp model deterministically to prevent crash on exit#31

Merged
paberr merged 1 commit into
paberr:mainfrom
kurokeita:fix/llama-cpp-deterministic-cleanup
Jun 23, 2026
Merged

fix: free local llama-cpp model deterministically to prevent crash on exit#31
paberr merged 1 commit into
paberr:mainfrom
kurokeita:fix/llama-cpp-deterministic-cleanup

Conversation

@kurokeita

@kurokeita kurokeita commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Problem

Running ownscribe with the local (llama-cpp) summarization backend crashes on exit, after the summary has already been produced and saved:

Exception ignored in: <function LlamaCppSummarizer.__del__ ...>
AttributeError: 'NoneType' object has no attribute 'suppress'
Exception ignored in: <function Llama.__del__ ...>
TypeError: 'NoneType' object is not callable
ggml-metal-device.m:622: GGML_ASSERT([rsets->data count] == 0) failed
fish: Job 1, 'ownscribe' terminated by signal SIGABRT (Abort)

Root cause

The loaded Llama model was only ever released via __del__. The summarizer object survives until interpreter shutdown, at which point module globals are torn down to None:

  • LlamaCppSummarizer.__del__ calls contextlib.suppress(...), but the module-level contextlib is already NoneAttributeError.
  • llama_cpp's own Llama.__del__ calls free_model, but its C-function pointer is already NoneTypeError.

Because free_model never runs, the Metal buffers stay allocated. When the process finally reaches exit()__cxa_finalizeggml_metal_device_free, the global device still has live residency sets, so GGML_ASSERT([rsets->data count] == 0) aborts the process with SIGABRT.

__del__ is fundamentally the wrong place for this — at shutdown the model cannot be freed because llama_cpp's C bindings are already gone.

Fix

Free the model deterministically while the interpreter is still alive, instead of relying on __del__:

  • Add an idempotent close() plus context-manager protocol (__enter__/__exit__) to the Summarizer base contract (no-op default; backends without native resources inherit it for free).
  • Implement close() in LlamaCppSummarizer to free self._llm. It uses a plain try/except rather than contextlib.suppress precisely because it may also run from __del__ at shutdown when contextlib is None.
  • Call close() deterministically from every summarization path: run_summarize and _do_transcribe_and_summarize (via try/finally) in pipeline.py, and ask (via with summarizer:) in search.py.
  • __del__ remains only as a best-effort backstop.

Testing

  • Added 6 unit tests covering close() (frees model, idempotent, no-op without load, error suppression, context manager) and the inherited no-op contract for non-local backends.
  • Full suite: 237 passed; ruff check src/ tests/ passes.
  • End-to-end: ran ownscribe summarize on a real transcript with the cached phi-4-mini model → exit code 0, none of the Exception ignored / GGML_ASSERT / SIGABRT markers.

… exit

The Llama model was only released via __del__, which runs at interpreter
shutdown when module globals (contextlib, llama_cpp's C bindings) are already
torn down to None. The model was therefore never freed, so the Metal device
finalizer aborted with GGML_ASSERT/SIGABRT, preceded by AttributeError and
TypeError from the failed __del__ paths.

Add an idempotent close() to the Summarizer contract and call it
deterministically from every summarization path (pipeline + search), so the
model is freed while the interpreter is still alive. __del__ remains only as a
best-effort backstop using plain try/except.

@paberr paberr left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR and writeup. I agree with the analysis!
It's good to merge from my side.

@paberr paberr merged commit 2b8b4d2 into paberr:main Jun 23, 2026
2 checks passed
@kurokeita kurokeita deleted the fix/llama-cpp-deterministic-cleanup branch June 23, 2026 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants