Anti-loop: forward DRY sampler + forward-only nudges (0.3.4)#38
Merged
Conversation
Root-cause investigation into model looping: the /v1/chat/completions payload forwarded ONLY temperature + repeat_penalty=1.05. The full anti-repetition config (DRY, min_p, top_k, top_p) was computed into `opts` but only reached the DEAD ollama-shaped payload (ollama was removed) — so it never hit the server. llama.cpp has DRY OFF by default, so in production the model ran with near- default sampling and NO DRY. That's a real root cause of the repeat-collapse loops (the original "gemma garbage repeat" class). Fix: forward the EOS-NEUTRAL anti-loop samplers — DRY (penalises repeated N-grams, not single tokens, so it doesn't suppress EOS like repeat_penalty) and min_p (trims the collapse-feeding low-prob tail). Deliberately NOT re-adding the aggressive repeat_penalty/top_k/top_p bundle that caused the 2026-04-29 paraphrase regression. A/B revert with LOCALCODE_SAMPLER_MINIMAL=1. Also: log the ACTUAL forwarded samplers once per session (was opt-in behind LOCALCODE_DEBUG_SAMPLERS) so looping can always be correlated with the real server config — the "control this better" record.
…tioning loop)
Log analysis of the Anki build showed the model stuck repeating "I've been
going in circles. Let me build this ENTIRE app NOW" ~10x while making tiny
progress. Root cause: our churn/spin nudges were injected as role:"user"
messages containing self-critical loop language ("you're going in circles",
"you've rewritten X N times", "planned enough"). A small model reads that as
the USER speaking and replies "You're right, I've been going in circles" — which
then sits in its own context and self-conditions the loop deeper
(arXiv:2509.09677: a model that sees its own error/loop text produces more of it).
Reworded all four churn nudges to FORWARD-ONLY imperatives: tell the model the
next concrete action, with zero echo-able language about the failure it just had.
Nothing for it to parrot back into its own context.
… on disconnect The recurring mid-stream -9 crashes were undiagnosable because the runtime restart path sent the server's stdout/stderr to /dev/null — so llama-server's own final error (ggml/Metal allocation failure, GGML_ASSERT, OOM) was discarded and we only saw httpx's generic 'connection dropped'. Only first-launch setup captured it. Now every spawn appends to ~/.local/share/localcode/server.log, and _log_disconnect_context reads the last 15 lines into the server_disconnect event + last_error.log — so the NEXT crash records the actual reason instead of leaving us guessing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root-cause fixes for model looping, found by reviewing real localcode logs of the Anki build. 634 tests pass.
Server config: actually forward DRY
The
/v1/chat/completionspayload forwarded ONLYtemperature+repeat_penalty=1.05. The full anti-repetition config (DRY, min_p) was computed intooptsbut only reached the DEAD ollama-shaped payload (ollama was removed), so it never hit the server. llama.cpp defaults DRY OFF → the model ran with no n-gram repetition penalty. That's a real root cause of the repeat-collapse loops.Fix: forward the EOS-neutral anti-loop samplers — DRY (penalizes repeated n-grams, not single tokens, so it doesn't suppress EOS like repeat_penalty) + min_p. Deliberately NOT re-adding the aggressive repeat_penalty/top_k/top_p bundle that caused the 2026-04-29 paraphrase regression. Instant A/B revert with
LOCALCODE_SAMPLER_MINIMAL=1. Also logs the actual forwarded samplers once per session so looping can be correlated with real config.Code: forward-only recovery nudges
Logs showed the model stuck repeating "I've been going in circles. Let me build this ENTIRE app NOW" while making tiny progress. Cause: churn/spin nudges were injected as
role:"user"messages with self-critical loop language ("you're going in circles", "you've rewritten X N times"). A small model reads that as the USER speaking, replies "You're right, I've been going in circles" — which then self-conditions the loop deeper (arXiv:2509.09677).Fix: reworded all four churn nudges to forward-only imperatives (next concrete action, zero echo-able language about the failure).