perf: route ChatGPT Codex upstream turns over responses_websockets - #1558
Merged
Conversation
The ChatGPT Codex backend serves the responses_websockets (WS) path from
a measurably faster queue than the plain SSE POST path. Measured
2026-08-12 KST (same account, same payload, strictly sequential):
gpt-5.6-luna TTFT p50 ~1.0s over WS vs ~3.9s over SSE. Codex CLI itself
defaults to WS, so requests through opencodex carried an extra 2-3s of
TTFT that direct Codex CLI usage did not.
Wrap providerFetch() so that streaming POSTs to
chatgpt.com/backend-api/codex/responses dial wss:// instead: the JSON
body goes out as a single response.create frame and returned event
frames are re-encoded as an SSE byte stream, leaving every downstream
consumer (passthrough relay, adapter parsers, usage sniffing) unchanged.
Transport selection parses the body and requires a root-level
stream === true, so nested {"metadata":{"stream":true}} or formatted
JSON cannot misroute. Failure handling: upgrade rejection, a missing
101 within 10s, and a synchronous frame-send failure all fall back to
the existing SSE path (in every case no upstream turn has started, so
the resend cannot double-generate); a socket drop after open but before
a Responses terminal event errors the stream so relaySseWithFailedTail
synthesizes a response.failed terminal instead of a terminal-less clean
EOF. Caller metadata is preserved verbatim — no originator is invented
for callers that did not send one.
After patching, local benchmarks put opencodex within ~0.2s of direct
Codex CLI (Luna TTFT p50 4364ms -> 1424ms; Terra 3322ms -> 1417ms).
Verified tool-call round-trips, 630KB-1.5MB frames, context_length
error relay, and ~600 live requests with no new failure modes.
Known trade-off: Bun's WebSocket does not expose the 101 response
headers, so x-codex-*-reset-at quota hints are not visible on this
path; the periodic quota poller still covers quota tracking.
Regression tests cover root-level stream detection, providerFetch()
routing, frame relay (including WS-only frame dropping), SSE fallback
on upgrade rejection and send failure, mid-stream drop through the
passthrough relay (synthesized failed terminal), header preservation
without originator fabrication, and pre-open abort.
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughChangesCodex WebSocket transport
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant providerFetch
participant codexWsUpstreamFetch
participant WebSocket
participant SSEClient
providerFetch->>codexWsUpstreamFetch: Route eligible Codex streaming request
codexWsUpstreamFetch->>WebSocket: Open and send response.create
WebSocket-->>codexWsUpstreamFetch: Emit Responses event
codexWsUpstreamFetch->>SSEClient: Return SSE event
WebSocket-->>codexWsUpstreamFetch: Emit terminal event
codexWsUpstreamFetch->>SSEClient: Close SSE stream
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
✅ Deterministic PR hygiene checks passed. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/ws-upstream.test.ts`:
- Around line 178-190: Add a focused test beside the existing rejected-upgrade
test for a WebSocket that emits neither open nor close. Use fake timers, invoke
codexWsUpstreamFetch with the existing streamingInit and fallback setup, advance
timers by UPGRADE_DEADLINE_MS, then assert the fallback is called exactly once
and the socket is closed.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 6cf56eae-7841-4015-b97a-6be5269e7209
📒 Files selected for processing (3)
src/server/responses/fetch-helpers.tssrc/server/responses/ws-upstream.tstests/ws-upstream.test.ts
Owner
Author
|
Merged. Big thanks to @kargnas for the original investigation and implementation in #1487, including the transport measurements that found the real source of the proxy latency gap. This is a meaningful performance win: eligible ChatGPT Codex streaming turns can now use The added regression coverage around routing, upgrade timeout, send failure, and premature socket close makes the faster path much safer to keep as the default transport optimisation. Thanks again @kargnas for the groundwork that made this possible. |
This was referenced Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Requests proxied through opencodex to the ChatGPT Codex backend consistently showed 2–3s worse TTFT than the same requests made by Codex CLI directly — even with the same account, same payload, and strictly sequential execution.
Root cause: Codex CLI talks to
chatgpt.com/backend-api/codex/responsesover theresponses_websocketstransport, while opencodex always POSTs SSE. The backend serves the WS path from a measurably faster queue.Measurements (2026-08-12, same account, same payload, sequential, alternating order)
responses_websockets)stream:true)Event timeline shows the gap is upstream scheduling, not transfer: both paths reach
response.createdin ~0.5s, but SSE then waits 2.4–4.0s beforeresponse.output_item.added(WS: 0.8–1.3s).Ruled out: HTTP/2 vs 1.1 (no change),
OpenAI-Beta: responses_websocketsheader on the SSE POST (no change),session_id/prompt_cache_key(no change), warmup effects (no decay over sequential repeats), account differences (A/B with identical account).A second finding: the fast lane keys on WS +
originatortag, not the transport alone — 60KB turns run ~1.5s withoriginator: codex_cli_rsvs ~4.6s without. The patch defaults the header for callers that don't send one (Codex CLI always does).Change
src/server/responses/ws-upstream.ts: for streaming POSTs to the Codex backend, dialwss://with the same headers, send the JSON body as a singleresponse.createframe, and re-encode returned event frames as an SSE byte stream — so the passthrough relay, adapter parsers, and usage sniffing are all unchanged.providerFetch()infetch-helpers.tswraps the provider fetch with this transport swap. Everything that isn't a Codex-backend streaming turn keeps the exact HTTP path.codex.rate_limits,responsesapi.websocket_timing) are dropped so clients see exactly the stream shape they always got.After patching (local proxy vs direct Codex CLI)
Also verified: tool-call round-trips (function_call arguments relay), 630KB / 900KB / 1.26MB / 1.5MB input frames,
context_length_exceededandserver_is_overloadederror relay, and ~600 live requests through the patched proxy with no new failure modes.Known trade-off
Bun's
WebSocketdoes not expose the 101 response headers, sox-codex-*-reset-atquota hints are not visible on this path. The periodic quota poller still covers quota tracking. If there's a preferred way to surface those, happy to adjust.Related: #1217 (stream-stage timing) would make this kind of transport gap visible in the dashboard.
Summary by CodeRabbit
New Features
Bug Fixes