⚗️(evalap) add a simple OpenAI compatible endpoint to test - #176
Conversation
This provide a quick and dirty way to run evaluation on the Conversations assistant.
WalkthroughThe PR adds an OpenAI-compatible chat completions endpoint ( Changes
Sequence DiagramsequenceDiagram
actor Client
participant ChatCompletionsView
participant ConversationService
participant AIAgentService
participant EventAggregator
Client->>ChatCompletionsView: POST /v1/chat/completions<br/>(messages, model, feature_flags)
ChatCompletionsView->>ChatCompletionsView: Parse request & validate messages
ChatCompletionsView->>ChatCompletionsView: Convert OpenAI messages to UIMessage format
ChatCompletionsView->>ConversationService: Create conversation instance
ChatCompletionsView->>ChatCompletionsView: Patch feature flags from request
ChatCompletionsView->>AIAgentService: Initialize with conversation
ChatCompletionsView->>AIAgentService: Start message processing
AIAgentService->>EventAggregator: Stream events (TextPart, FinishMessagePart, usage)
EventAggregator->>EventAggregator: Accumulate content & usage
EventAggregator->>ChatCompletionsView: Final aggregated result
ChatCompletionsView->>ChatCompletionsView: Build OpenAI-compatible response
ChatCompletionsView->>ChatCompletionsView: Cleanup conversation
ChatCompletionsView->>Client: Return response (id, model, content, usage)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes
Suggested labels
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Tip 📝 Customizable high-level summaries are now available in beta!You can now customize how CodeRabbit generates the high-level summary in your pull requests — including its content, structure, tone, and formatting.
Example instruction:
Note: This feature is currently in beta for Pro-tier users, and pricing will be announced later. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (4)
docs/evaluation.md (2)
3-19: Tighten wording in the intro/warning block for clarity.Minor nits only:
- Consider using “OpenAI-compatible” (no space) consistently.
- The sentence with “etc)” would read better as “etc.).” or by removing “etc” entirely.
- You may want to explicitly state here that
streamis ignored and only simple, non‑streaming chat completion requests are supported (even though it’s implied later).
32-99: Clarify how themodelparameter in/v1/chat/completionsmaps to these LLM configurations.Right now the JSON shows model HRIDs like
"mistral-medium-2508-raw"/"mistral-medium-2508-with-web-search", but the doc doesn’t explicitly say “this is what you must send as themodelvalue in the OpenAI‑like request body.”Adding a short note such as “When calling
/v1/chat/completions, setmodelto one of thesehridvalues” would make the end‑to‑end flow easier to follow and reduce guesswork when wiring EvalAP.src/backend/evaluation/views.py (2)
138-151: Global monkey‑patching ofis_feature_enabledis fragile under concurrency.Overriding
core.feature_flags.helpers.is_feature_enabledat module level per request means:
- Concurrent evaluations can interfere with each other (the last request to patch “wins”, and all in‑flight runs start using that request’s
feature_flags).- Any other code paths in the same process that rely on feature flags (e.g. normal chat UI in dev) will see the overridden behavior while an eval request is running.
Given this is labeled “quick and dirty” and dev/test‑only, you might accept the risk for strictly sequential runs, but it’s still worth at least calling out or tightening if you expect EvalAP to run multiple jobs in parallel.
Safer options (even if deferred):
- Thread/task‑local or context‑local feature‑flag overrides instead of a global monkey‑patch.
- Passing
feature_flagsdirectly intoAIAgentService/ the agent stack.- If you keep the monkey‑patch, wrap it in a
try/finallyand restore the original function at the end of the request to limit the blast radius.
158-197: Ensure temporary conversations are always cleaned up, even on errors.You create a transient
ChatConversationand delete it at the end:conversation = await ChatConversation.objects.acreate(...) ... await conversation.adelete()If
_run_agentor anything in between raises, the delete is skipped and the DB will accumulate leftover eval conversations over time.Wrapping the agent run and response construction in
try/finallywould avoid leaks:- # Create a new conversation - conversation = await ChatConversation.objects.acreate( - owner=user, - messages=[], - pydantic_messages=[], - ) + # Create a new conversation (ephemeral, for this eval request only) + conversation = await ChatConversation.objects.acreate( + owner=user, + messages=[], + pydantic_messages=[], + ) @@ - async for event in service._run_agent(ui_messages): - ... - - response_id = f"chatcmpl-{uuid.uuid4().hex[:24]}" - response = JsonResponse( - create_openai_response(response_id, model, full_content, "stop", final_usage) - ) - - # Remove the conversation to avoid accumulation - await conversation.adelete() - - return response + try: + async for event in service._run_agent(ui_messages): + ... + + response_id = f"chatcmpl-{uuid.uuid4().hex[:24]}" + response = JsonResponse( + create_openai_response(response_id, model, full_content, "stop", final_usage) + ) + return response + finally: + # Remove the conversation to avoid accumulation + await conversation.adelete()This keeps the DB clean while still treating conversations as ephemeral for eval.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
docs/evaluation.md(1 hunks)src/backend/core/urls.py(2 hunks)src/backend/evaluation/views.py(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
src/backend/evaluation/views.py (5)
src/backend/chat/ai_sdk_types.py (2)
TextUIPart(137-147)UIMessage(272-280)src/backend/chat/clients/pydantic_ai.py (2)
AIAgentService(103-811)_run_agent(354-742)src/backend/chat/models.py (1)
ChatConversation(18-75)src/backend/chat/vercel_ai_sdk/core/events_v4.py (1)
TextPart(49-61)src/backend/conversations/settings.py (1)
ENVIRONMENT(924-926)
src/backend/core/urls.py (2)
src/backend/evaluation/views.py (1)
ChatCompletionsView(101-198)src/backend/conversations/settings.py (1)
ENVIRONMENT(924-926)
🪛 LanguageTool
docs/evaluation.md
[style] ~13-~13: In American English, abbreviations like “etc.” require a period.
Context: ...tputs (no streaming, no function calls, etc). The result returned will already have...
(ETC_PERIOD)
[style] ~23-~23: You have already used this phrasing in nearby sentences. Consider replacing it to add variety to your writing.
Context: ...t you want to run. For instance, if you want to compare the Agent answer with and witho...
(REP_WANT_TO_VB)
🪛 markdownlint-cli2 (0.18.1)
docs/evaluation.md
29-29: Dollar signs used before commands without showing output
(MD014, commands-show-output)
108-108: Dollar signs used before commands without showing output
(MD014, commands-show-output)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (5)
- GitHub Check: test-e2e-chromium
- GitHub Check: test-back
- GitHub Check: lint-back
- GitHub Check: build-and-push-frontend
- GitHub Check: build-and-push-backend
🔇 Additional comments (2)
src/backend/core/urls.py (1)
13-14: Dev/test‑only routing for/v1/chat/completionslooks consistent.The import and conditional URL pattern line up with the docs and with the environment check inside
ChatCompletionsView, and the path matches the examples (/v1/chat/completionson port 8071). No issues from a routing or exposure standpoint.Also applies to: 46-51
src/backend/evaluation/views.py (1)
33-61: OpenAI‑style response builder is minimal but sufficient for non‑streaming EvalAP usage.The payload shape (
id,object,created,model, singlechoice,usage) matches what EvalAP and most OpenAI‑compatible tooling expect for a simple, non‑streaming chat completion. Given the “quick and dirty / evaluation‑only” scope, not exposing extra fields (e.g. logprobs) is reasonable.
| I needed to update the Docker compose file to add: | ||
|
|
||
| ```yaml | ||
| extra_hosts: | ||
| - "host.docker.internal:host-gateway" | ||
| ``` | ||
|
|
||
| Globally I followed the instructions in the EvalAP documentation, had a few issues with the stack initialization | ||
| but finally managed to run it with. | ||
|
|
||
| ### Create the dataset | ||
|
|
||
| Read the EvalAP documentation to create a new dataset. I did a simple dataset with only two samples to check | ||
| the evaluation works. | ||
|
|
||
| ### Create the evaluation | ||
|
|
||
| Same as before, read the EvalAP documentation to create a new evaluation. | ||
|
|
||
| The important part is to configure the model to call Conversations, and use the extra parameters to | ||
| adapt feature flags if needed. | ||
|
|
||
| ```python | ||
| import requests | ||
|
|
||
| # Replace with your Evalap API endpoint | ||
| API_URL = "http://localhost:8000/v1" | ||
|
|
||
| # Replace with your API key or authentication token | ||
| HEADERS = { | ||
| "Authorization": "Bearer YOUR_API_KEY", | ||
| "Content-Type": "application/json" | ||
| } | ||
|
|
||
| # Define your experiment set with CV schema | ||
| expset_name = "model_comparison_v1" | ||
| expset_readme = "Comparing performance of various LLMs on a QA dataset." | ||
| metrics = ["judge_precision", "output_length", "generation_time"] | ||
|
|
||
| # Parameters common to all experiments | ||
| common_params = { | ||
| "dataset": "Dataset Bidon", # assuming this dataset has been added before | ||
| #"model": {"sampling_params": {"temperature": 0.2}}, | ||
| "metrics": metrics, | ||
| "judge_model": "albert-large", | ||
| } | ||
|
|
||
| # Parameters that will vary across experiments | ||
| grid_params = { | ||
| "model": [ | ||
| { | ||
| "name": "etalab-plateform-mistral-medium-2508", | ||
| "aliased_name": "Mistral Medium", | ||
| # base_url points to Conversations API | ||
| "base_url": f"http://host.docker.internal:8071/v1", | ||
| "api_key": "plop", | ||
| "extra_params": { | ||
| "feature_flags": { | ||
| # Disable RAG tool | ||
| "tool_rag_french_public_services": "DISABLED", | ||
| }, | ||
| }, | ||
| }, | ||
| { | ||
| "name": "etalab-plateform-mistral-medium-2508", | ||
| "aliased_name": "Mistral Medium + RAG", | ||
| "base_url": f"http://host.docker.internal:8071/v1", | ||
| "api_key": "plop", | ||
| "extra_params": { | ||
| "feature_flags": { | ||
| # Enable RAG tool | ||
| "tool_rag_french_public_services": "ENABLED", | ||
| }, | ||
| }, | ||
| }, | ||
| ], | ||
| } | ||
|
|
||
| # Create the experiment set with CV schema | ||
| expset = { | ||
| "name": expset_name, | ||
| "readme": expset_readme, | ||
| "cv": { | ||
| "common_params": common_params, | ||
| "grid_params": grid_params, | ||
| "repeat": 3 # Run each combination 3 times to measure variability | ||
| } | ||
| } | ||
|
|
||
| # Launch the experiment set | ||
| requests.delete(f'{API_URL}/experiment_set/12', json=expset, headers=HEADERS) | ||
| response = requests.post(f'{API_URL}/experiment_set', json=expset, headers=HEADERS) | ||
| expset_id = response.json()["id"] | ||
| print(f"Experiment set {expset_id} is running") | ||
| ``` |
There was a problem hiding this comment.
Avoid hard‑coded DELETE /experiment_set/12 in the example script.
As written:
requests.delete(f'{API_URL}/experiment_set/12', json=expset, headers=HEADERS)anyone copy‑pasting this might accidentally delete a real experiment set in their EvalAP instance. At minimum, this deserves a strong comment or should be removed from the “happy path” example.
You could, for example, either:
- Comment it out with an explanation (“uncomment if you really want to wipe experiment_set 12”), or
- Replace
12with a placeholder and a warning in the surrounding text.
🤖 Prompt for AI Agents
In docs/evaluation.md around lines 117 to 211, the example script contains a
hard-coded destructive call requests.delete(f'{API_URL}/experiment_set/12')
which can accidentally delete real data; remove that line from the “happy path”
example (or comment it out) and instead show a safe alternative: use a clearly
named placeholder variable (e.g. EXPERIMENT_SET_ID) or wrap the delete in an
explicit conditional/confirmation and add a one-line warning in the surrounding
text explaining that delete is destructive and must be used intentionally.
Ensure the example posted for copy-paste does not perform destructive actions by
default.
| def openai_messages_to_ui_messages(openai_messages: List[dict]) -> List[UIMessage]: | ||
| """ | ||
| Convert OpenAI message format to UIMessage format for the backend view. | ||
| """ | ||
| ui_messages = [] | ||
| for msg in openai_messages: | ||
| role = msg.get("role", "user") | ||
| content = msg.get("content", "") | ||
|
|
||
| # Handle content that can be string or list of content parts | ||
| if isinstance(content, list): | ||
| parts = [] | ||
| for part in content: | ||
| if part.get("type") == "text": | ||
| parts.append(TextUIPart(type="text", text=part.get("text", ""))) | ||
| # Add handling for images, etc. as needed | ||
| ui_messages.append( | ||
| UIMessage( | ||
| id=msg.get("id", str(uuid.uuid4())), | ||
| role=role, | ||
| parts=parts, | ||
| content="".join(content), | ||
| ) | ||
| ) | ||
| else: | ||
| ui_messages.append( | ||
| UIMessage( | ||
| id=msg.get("id", str(uuid.uuid4())), | ||
| role=role, | ||
| parts=[TextUIPart(type="text", text=content or "")], | ||
| content=content or "", | ||
| ) | ||
| ) | ||
| return ui_messages |
There was a problem hiding this comment.
Bug: "".join(content) on a list of content parts will raise at runtime.
When content is a list, it’s a list of dicts (OpenAI content parts), so:
content="".join(content)will raise TypeError: sequence item 0: expected str instance, dict found.
You probably want to concatenate only the text pieces you already extracted into TextUIParts.
A safe fix would look like:
- for msg in openai_messages:
+ for msg in openai_messages:
role = msg.get("role", "user")
content = msg.get("content", "")
@@
- if isinstance(content, list):
- parts = []
- for part in content:
- if part.get("type") == "text":
- parts.append(TextUIPart(type="text", text=part.get("text", "")))
- # Add handling for images, etc. as needed
- ui_messages.append(
- UIMessage(
- id=msg.get("id", str(uuid.uuid4())),
- role=role,
- parts=parts,
- content="".join(content),
- )
- )
+ if isinstance(content, list):
+ parts: List[TextUIPart] = []
+ text_chunks: List[str] = []
+ for part in content:
+ if part.get("type") == "text":
+ text = (part.get("text") or "")
+ parts.append(TextUIPart(type="text", text=text))
+ text_chunks.append(text)
+ # TODO: handle non-text parts (images, etc.) if needed later
+
+ ui_messages.append(
+ UIMessage(
+ id=msg.get("id", str(uuid.uuid4())),
+ role=role,
+ parts=parts,
+ content="".join(text_chunks),
+ )
+ )This keeps the UIMessage parts and string content consistent for the common text‑only case.
🤖 Prompt for AI Agents
In src/backend/evaluation/views.py around lines 64 to 97, when handling the case
where message["content"] is a list the code incorrectly calls "".join(content)
on a list of dicts which raises a TypeError; instead build the parts list from
text parts (as you already do) and create a content_str by concatenating the
text values from those parts (e.g., join the TextUIPart.text values), then pass
that content_str into UIMessage.content so the UIMessage.parts and content are
consistent and no TypeError occurs.
| async def post(self, request): | ||
| """ | ||
| Handle POST requests to the chat completions endpoint. | ||
| """ | ||
| if settings.ENVIRONMENT not in ["development", "tests"]: | ||
| return JsonResponse( | ||
| {"error": "This endpoint is for evaluation purposes only."}, status=403 | ||
| ) | ||
|
|
||
| # Enforce the user | ||
| user = await User.objects.aget(email="conversations@conversations.world") | ||
|
|
||
| # Parse request body to get parameters | ||
| try: | ||
| body = json.loads(request.body) | ||
| except json.JSONDecodeError: | ||
| return JsonResponse({"error": "Invalid JSON"}, status=400) | ||
|
|
||
| # Extract parameters | ||
| model = body.get("model", "default-model") | ||
| messages = body.get("messages", []) | ||
| feature_flags = body.get("feature_flags", {}) | ||
|
|
There was a problem hiding this comment.
Handle missing seed user more gracefully.
User.objects.aget(email="conversations@conversations.world") will 500 if the demo/seed user does not exist (or is duplicated), which is easy to hit when someone spins up the stack without running the demo target first.
For a smoother DX, consider catching this and returning a clear error:
- # Enforce the user
- user = await User.objects.aget(email="conversations@conversations.world")
+ # Enforce the user used for evaluation
+ try:
+ user = await User.objects.aget(email="conversations@conversations.world")
+ except User.DoesNotExist:
+ return JsonResponse(
+ {
+ "error": (
+ "Evaluation user 'conversations@conversations.world' not found. "
+ "Did you run `make demo` to seed demo data?"
+ )
+ },
+ status=500,
+ )That makes misconfiguration obvious instead of surfacing as an opaque 500.
🤖 Prompt for AI Agents
In src/backend/evaluation/views.py around lines 115-137, the code directly
awaits User.objects.aget(email="conversations@conversations.world") which will
raise exceptions (DoesNotExist or MultipleObjectsReturned) and produce an opaque
500; wrap the aget call in a try/except that catches User.DoesNotExist and
User.MultipleObjectsReturned (importing them if needed), return a clear
JsonResponse (e.g. status 400 or 422) with a message like "Seed user
'conversations@conversations.world' not found or duplicated; run demo seeding"
and optionally log the exception for troubleshooting so the endpoint fails with
a helpful error instead of an opaque 500.
|
1 participant
Purpose
This provides a quick and dirty way to run evaluation on the Conversations assistant.
Proposal
Summary by CodeRabbit
New Features
Documentation
✏️ Tip: You can customize this high-level summary in your review settings.