OpenAI Responses Proxy for Chutes.ai

Production-grade Rust service translating the OpenAI Responses API into a Chat Completions request for Chutes.ai backends. Ships with streaming, tool calling, reasoning support, and operational tooling sized for live traffic.

Highlights

• 100% Responses API compatibility: Complete support for text, multimodal inputs, streamed outputs, and MCP-compliant tool calling with modern output_tool_call.* events.
• Stateless transformer: Forwards client auth, keeps no session state, easy to scale horizontally.
• Fragmentation-safe streaming: Buffers early tool arguments until headers complete, preventing event ordering violations.
• Dual event emission: Modern (output_tool_call.*) + legacy (function_call_arguments.*) events for seamless client migration.
• Safe defaults: Request validation, circuit breaker guard, bounded logging, attachment rejection.
• Observability hooks: Structured logging, optional on-disk dumps, metrics-friendly event stream.

Lightning Quick Start

Bootstrap the Chutes.ai Codex fork, config, and credential helper in one command:

curl -fsSL https://raw.githubusercontent.com/chutesai/responses-proxy/refs/heads/main/install_codex.sh | bash

The script downloads the latest Codex release for your OS, offers to replace any existing codex, and writes the recommended config.toml plus API-key helper. On Linux it auto-detects your libc (glibc vs. musl) and falls back to the musl build if the host ships an older glibc. On Windows, run it from Git Bash/Cygwin (or any POSIX shell available in PowerShell/cmd); it automatically installs the Windows .exe and writes config under %USERPROFILE%\.codex. Launching it inside WSL installs the Linux build for that distro.

For deeper background, see the companion docs in docs/ (e.g. docs/PROJECT_SUMMARY.md).

Quick Start (Codex Client)

1. Configure Codex (OpenAI) – add to your profile (~/.codex/config.toml):

   # Global
   model_provider = "chutes-ai"
   model = "Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8"
   # model = "openai/gpt-4o-mini"
   model_reasoning_effort = "high"
   
   [model_providers."chutes-ai"]
   name = "Chutes AI via responses proxy"
   base_url = "https://responses.chutes.ai/v1"
   env_key = "MY_PROVIDER_API_KEY"
   wire_api = "responses"
   
   [notice]
   hide_full_access_warning = true
   
   [features]
   #apply_patch_freeform = true
   view_image_tool = true
   web_search_request = true
   
   [experimental]
   #unified_exec = true
   #streamable_shell = true
   #experimental_sandbox_command_assessment = true
   rmcp_client = true                           # Rust MCP client

2. Export your API key (matches env_key):

   export MY_PROVIDER_API_KEY="cpk_xxx"   # example key

3. Start a Codex session:

   codex

   Pick the chutes-ai provider inside the UI; requests will flow through https://responses.chutes.ai/v1.

ℹ️ Only function tools are forwarded; Codex options such as web_search_request may fall back gracefully if the backend rejects them. Reasoning effort hints are passed through to the backend model.

Codex Configuration

The example excerpt for Codex (OpenAI) config.toml should be carefully adjusted to your needs:

• Keep env_key synced with an environment variable that stores your Chutes-compatible API token.
• The proxy only supports function tools; Codex options such as web_search_request may trigger warnings because the backend drops non-function tools.
• Reasoning effort hints are forwarded, but final behaviour depends on the selected model.

Configuration

Environment variables (see docs/QUICKSTART.md for exhaustive notes):

Variable	Default	Purpose
BACKEND_URL	https://llm.chutes.ai/v1/chat/completions	Target Chat Completions endpoint
BACKEND_TIMEOUT_SECS	600	Total request timeout against backend
HOST_PORT	8282	Axum listener port
RUST_LOG	info	Log level (error…trace)
ENABLE_LOG_VOLUME	false	When true, dumps requests/streams to LOG_DIR
LOG_DIR	logs	Base directory for optional dumps
CADDY_DOMAIN	responses.chutes.ai	TLS host for Caddy deployment
CADDY_PORT	443	Exposed HTTPS port

Logging dumps are gated behind ENABLE_LOG_VOLUME; with the flag disabled the proxy never writes request or stream bodies to disk.

API Surface

• POST /v1/responses – Accepts OpenAI Responses payloads, streams SSE events.
• GET /health – Reports circuit breaker status and readiness for load balancers.

Key behaviours:

• Request validation: Size limits on inputs, instructions, and tool counts; attachments validated and rejected with file IDs logged.
• Tool support: Forwards function tools, converts stray XML-style tool calls into native function events with full delta streaming, and emits both modern (output_tool_call.*) and legacy (function_call_arguments.*) events for client compatibility.
• MCP tool results: Accepts role:"tool" messages with content:[{type:"output", content_type, body}] per MCP spec, plus legacy function_call_output blocks for backward compat.
• Reasoning models: Captures reasoning_content, emits <think>-compatible events, and surfaces reasoning output items alongside final content.
• Responses parity: Accepts modern Responses parameters like include, stream_options, text.format, top_logprobs, and user, forwarding structured-output formats and logprob hints to the backend while warning (or rejecting) unsupported knobs such as background, prompt templates, and service_tier.
• File inputs: Rejects input_file content parts with a clear error because the Chat Completions backend cannot dereference OpenAI file IDs; clients must inline file contents before sending.
• No persistence: The optional store flag is accepted but ignored; a warning is logged when provided.
• Fragmentation safety: Buffers tool arguments arriving before function names to ensure correct event ordering.

Operational Notes

• Circuit breaker guards backend outages (5 failures → 30s cool-down).
• Model list cached in-memory and refreshed every 60 s; casing normalized automatically.
• Background tasks shut down gracefully on SIGINT/ctrl+c.
• IDs for streamed items incorporate the request identifier to prevent cross-request collisions.

Related Documentation

• docs/QUICKSTART.md – environment setup and deployment walkthroughs.
• docs/TOOL_CALLING.md – in-depth description of tool conversion and streaming semantics.
• docs/REASONING_SUPPORT.md – handling reasoning content and <think> emission.
• docs/TESTING.md – regression scripts and smoke checks.
• docs/IMPLEMENTATION_NOTES.md – architecture and internal invariants.

Keep docs authoritative: update both this README and the relevant docs/* reference when behaviour changes.