Skip to content

Expand offline SDK test coverage#14

Open
lopadova wants to merge 3 commits intoregolo-ai:masterfrom
lopadova:test/offline-sdk-coverage
Open

Expand offline SDK test coverage#14
lopadova wants to merge 3 commits intoregolo-ai:masterfrom
lopadova:test/offline-sdk-coverage

Conversation

@lopadova
Copy link
Copy Markdown

Summary

This PR hardens the Python SDK test surface and removes the dependency on live API calls for the default test run.

  • Replaces the existing API-key dependent tests with deterministic offline pytest coverage backed by httpx.MockTransport.
  • Adds coverage for streaming, HTTP failures, retry behavior, malformed responses, model validation, chat history, embeddings, reranking, image/audio handling, and concurrent requests.
  • Adds a controlled retry path for transient 5xx POST responses.
  • Fixes instance audio transcription calls so they use the configured audio transcription model instead of the chat model.
  • Configures pytest collection and ignores generated cache artifacts.

Commits

  • ab0caf9 fix: retry transient POST server errors
  • b9de05a fix: use audio transcription model for instance calls
  • 575258c test: expand offline SDK coverage

Details

The previous test suite lived in a single tests/test.py file and mixed a few mocked assertions with live API calls. That made the suite hard to run reliably in CI or on contributor machines because it depended on TEST_KEY, reachable Regolo services, currently available default models, and successful image/embedding/chat responses from the live API. It also called test functions directly at module import time, so tests could execute outside pytest's normal collection and skipping behavior.

The new suite is fully offline by default. It uses httpx.MockTransport to validate the exact request payloads, authorization headers, endpoint paths, and response parsing behavior without network access. This keeps the tests deterministic while still exercising the SDK's public methods and the HTTP boundary.

Coverage was added for the gaps identified in the SDK audit:

  • Streaming server-sent events for completions and chat completions, including full-output mode and reasoning/content deltas.
  • Authentication failures and rate limits, verifying they raise immediately and do not retry.
  • Transient 5xx server errors, verifying eventual success after retries.
  • Permanent 5xx server errors, verifying failure after retry exhaustion.
  • Timeout propagation.
  • Malformed JSON responses.
  • Model-not-available validation before sending an HTTP request.
  • Multi-turn chat history maintenance through run_chat.
  • Embedding batches with multiple inputs, including empty strings, Unicode text, and long input.
  • Embedding vector dimension consistency across calls.
  • Reranking behavior for empty, single, and multiple documents, plus top_n boundary handling.
  • Structured reranking options such as rank_fields, return_documents, and max_chunks_per_doc.
  • Image generation base64 decoding.
  • Audio transcription multipart handling.
  • Concurrent completion requests sharing the same client.

While adding the tests, two small implementation issues were covered and fixed:

  • safe_post now retries transient 5xx responses a limited number of times before raising, while preserving immediate failures for 401/429 and other non-retryable errors.
  • RegoloClient.audio_transcription() now defaults to the instance audio transcription model instead of incorrectly falling back to the chat model.

Tests

python -m pytest -q
30 passed

@m4oc m4oc requested a review from FabrizioElvidio April 29, 2026 07:50
@m4oc
Copy link
Copy Markdown

m4oc commented Apr 29, 2026

@FabrizioElvidio

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants