ai_analyst: fold the authorization context into the leading system turn - #294
Merged
Conversation
The analyst sent the approved/pending indicator list as a second system message. Strict chat templates reject any system message past index 0: Qwen3 raises "System message must be at the beginning" and returns HTTP 500, so every query that authorized an indicator failed against it while plain conversation still worked. Found against a live Qwen3 GGUF; the stubbed LLM in the tests accepts any message shape, so nothing caught it. Merging the context into the leading system message puts the same content in front of the model and is accepted by every template. History only ever carries user/assistant and the agent loop only adds tool, so after this change a system message can exist only at index 0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hotfix for #288, found immediately after it went live against a real endpoint.
Symptom
Every
@aiquery that names an indicator fails with:The log carries the real cause — an HTTP 500 from the model server:
Observed against a live Qwen3 27B GGUF via llama.cpp.
Cause
AIAnalyst.handle()sent twosystemmessages: the analyst prompt, then theauthorization context (
Approved indicators… / Pending indicators…) as a secondsystem turn. Strict chat templates reject any system message past index 0, and
Qwen3's raises exactly that.
_authorization_context()is non-empty whenever any indicator is authorized, sothe failure is categorical rather than intermittent:
@ai hello, are you there?['system', 'user']@ai what do you make of 8.8.8.8?['system', 'system', 'user']Plain conversation with the bot works; every real analyst query fails.
Fix
Fold the authorization context into the leading system message. Same content in
front of the model, and a single leading system turn is what every template
accepts — so this isn't a Qwen special-case.
The fix is complete rather than merely sufficient:
historyonly ever carriesuser/assistant(ai_analyst.py:623, 684, 697) and the agent loop only addstool, so after this change asystemmessage can exist only at index 0.Why the tests didn't catch it
FakeLLMaccepts any message shape — a stub cannot reject a chat template itdoesn't implement. This is exactly the class of defect Task 11 (live-endpoint
verification) existed to find, and it was the first thing that surfaced. The rest
of that checklist should still be treated as outstanding.
Test plan
Added
test_exactly_one_system_message_leads_the_request, which assertsroles.count('system') == 1. It fails onmain(2 != 1, roles['system', 'system', 'user', 'assistant', 'user']) and passes with this change.The two existing tests that touch message shape were already agnostic — one joins
all system messages, the other only checks
roles[0]— so neither needed editing,and the merged prompt still carries both the approved and pending lines.
Verified locally:
python -m unittest discover -s tests→ Ran 315 tests, OKruff check --select F821on the changed files → cleanpython -c "import sys; sys.modules['requests']=None; import ai_analyst"→ OKNot yet verified against a live endpoint — that needs a restart on a host with a
model attached.