Skip to content

ai_analyst: fold the authorization context into the leading system turn - #294

Merged
uforia merged 1 commit into
mainfrom
fix/ai-analyst-single-system-message
Aug 23, 2026
Merged

ai_analyst: fold the authorization context into the leading system turn#294
uforia merged 1 commit into
mainfrom
fix/ai-analyst-single-system-message

Conversation

@spierenburg

Copy link
Copy Markdown
Collaborator

Hotfix for #288, found immediately after it went live against a real endpoint.

Symptom

Every @ai query that names an indicator fails with:

I could not reach the AI backend, so I have no answer for this one.

The log carries the real cause — an HTTP 500 from the model server:

Jinja Exception: System message must be at the beginning.
  While executing CallExpression at line 106 ...
  {{- raise_exception('System message must be at the beginning...

Observed against a live Qwen3 27B GGUF via llama.cpp.

Cause

AIAnalyst.handle() sent two system messages: the analyst prompt, then the
authorization context (Approved indicators… / Pending indicators…) as a second
system turn. Strict chat templates reject any system message past index 0, and
Qwen3's raises exactly that.

_authorization_context() is non-empty whenever any indicator is authorized, so
the failure is categorical rather than intermittent:

Query Roles sent Result
@ai hello, are you there? ['system', 'user'] works
@ai what do you make of 8.8.8.8? ['system', 'system', 'user'] HTTP 500

Plain conversation with the bot works; every real analyst query fails.

Fix

Fold the authorization context into the leading system message. Same content in
front of the model, and a single leading system turn is what every template
accepts — so this isn't a Qwen special-case.

The fix is complete rather than merely sufficient: history only ever carries
user/assistant (ai_analyst.py:623, 684, 697) and the agent loop only adds
tool, so after this change a system message can exist only at index 0.

Why the tests didn't catch it

FakeLLM accepts any message shape — a stub cannot reject a chat template it
doesn't implement. This is exactly the class of defect Task 11 (live-endpoint
verification) existed to find, and it was the first thing that surfaced. The rest
of that checklist should still be treated as outstanding.

Test plan

Added test_exactly_one_system_message_leads_the_request, which asserts
roles.count('system') == 1. It fails on main (2 != 1, roles
['system', 'system', 'user', 'assistant', 'user']) and passes with this change.

The two existing tests that touch message shape were already agnostic — one joins
all system messages, the other only checks roles[0] — so neither needed editing,
and the merged prompt still carries both the approved and pending lines.

Verified locally:

  • python -m unittest discover -s testsRan 315 tests, OK
  • ruff check --select F821 on the changed files → clean
  • python -c "import sys; sys.modules['requests']=None; import ai_analyst" → OK

Not yet verified against a live endpoint — that needs a restart on a host with a
model attached.

The analyst sent the approved/pending indicator list as a second system
message. Strict chat templates reject any system message past index 0:
Qwen3 raises "System message must be at the beginning" and returns HTTP
500, so every query that authorized an indicator failed against it while
plain conversation still worked. Found against a live Qwen3 GGUF; the
stubbed LLM in the tests accepts any message shape, so nothing caught it.

Merging the context into the leading system message puts the same content
in front of the model and is accepted by every template. History only ever
carries user/assistant and the agent loop only adds tool, so after this
change a system message can exist only at index 0.
@uforia
uforia merged commit d8fe05b into main Aug 23, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants