fix(gateway): budget time-to-first-byte, not the whole response (OX-H2)

[ojassug]

Audit OX-H2. DECISIONS §66. Second Lane B item.

The defect is one property of fetch

forwardUpstreamRequest armed AbortSignal.timeout(30000) and handed it to fetch. A fetch
signal does not stop applying when the promise resolves — it governs the response body stream too.

So for "stream": true payloads the body reader rejected roughly 30 seconds in, and the pump in
server.ts called res.destroy(...), truncating the answer mid-generation. LLM completions
routinely run past 30 seconds, long-form Anthropic streams especially, so this broke precisely the
traffic the Gateway intercepts by default — and from the client's side it is indistinguishable from
the model simply stopping.

The fix, and the line that is the fix

An owned AbortController replaces AbortSignal.timeout, with the timer cleared in a finally on
the header phase. AbortSignal.timeout cannot be used here because it cannot be un-fired; only
a controller you own can be left permanently unaborted. Once fetch settles — resolved, timed out,
or failed — nothing can fire it again, and the body phase is governed solely by the
caller-disconnect signal.

The 504 mapping is preserved by aborting with new DOMException(…, 'TimeoutError') — the catch
matches on error.name, and that is exactly the reason AbortSignal.timeout produced.

The half a careless fix removes

params.options.abortSignal — the client-hangup signal from res.on('close') — stays combined via
AbortSignal.any. Disarming the timeout must not disarm the disconnect, or a client that walks
away leaves the Gateway pulling a response nobody will read and paying the provider for it.

Asserted, and mutation-checked: replacing the combined signal with ttfbController.signal alone
fails that test and only that test.

That test also needed a second pass to be worth anything. The first version read the upstream's flag
after the helper stopped the gateway — which closes the upstream socket anyway — so it would have
passed whether or not the hangup propagated. The value is now sampled inside the gateway's lifetime.

Measured, against a real upstream over a socket

case	before	after
headers fast, body outlives budget (120 ms budget, ~300 ms body)	truncated mid-stream	full body, [DONE], all 5 chunks
body an order of magnitude past budget (40 ms, ~400 ms)	truncated	full body, all 8 chunks
headers slower than budget	504	504 — unchanged
client hangs up mid-stream	upstream aborted	upstream aborted — unchanged

upstreamTtfbTimeoutMs

New GatewayConfig / ProxyHandlerOptions field, default 30000, as the audit suggested. It is also
what makes the defect testable at all: catching a 30-second bug required a 30-second upstream,
which is why no test caught it. This suite runs in about a second.

The tests drive a real local upstream rather than mockUpstream, because mockUpstream
short-circuits before fetch and never exercises the signal handling the defect lives in.

What this does not establish

• Nothing bounds body duration now, deliberately. If a stalled-mid-stream upstream ever needs
  bounding, that wants an idle timer on the pump — reset per chunk — not a total-duration budget,
  and it is a different change.
• Nothing about savings. This path runs after optimization; it changes what reaches the client,
  not what the Gateway elides. Invariant 8 untouched.
• Nothing about the 10 MB body cap or the session store.

Verification

npm run typecheck, npm run lint, npm run build and npx vitest run all pass: 89 files /
817 tests. Both truncation cases were confirmed failing against the unfixed tree first, with the
slow-header 504 passing throughout — which is what isolates the defect from the new knob.

🤖 Generated with Claude Code