Problem
The live launcher footer counts a request when its response completes. A target that
takes minutes to answer therefore shows nothing at all while it works, and the
session looks hung.
Why it matters
This is acute when the target is an agent. In our demo a BioMni run takes ~110
seconds of complete silence before the first byte. Users conclude the proxy is
broken and interrupt a working run — we did this repeatedly before adding the label.
Proposed behavior
Mark targets that have a dispatched-but-unanswered request:
switchyard [config] · 4 req · 183,282 in 1,294 out
bedrock-claude-opus-5 2 req · 169,391 in 536 out
biomni-agent 0 req · 0 in 0 out [running]
The mark clears when the response arrives, and the row becomes a normal stats row.
Implementation that worked for us
- Track the live
ProxyContext at the existing pre-routing hook and read the
selected target lazily at render time. Routing picks the target partway
through the chain, so reading it at entry is too early — this avoids adding a
new mid-chain hook.
- Release on the response hook.
- Wrap the ASGI app to release when the HTTP request ends. Response processors do
not run when a backend raises, so without this a failed request marks its target
as running forever. It must be a raw ASGI middleware, not a Starlette HTTP
middleware — the latter runs the app in a separate task, so a context variable
set in it never reaches the chain.
- Targets working before their first completed call need a transient row, since
they are not yet in the stats snapshot at all.
One honest limitation: for streaming targets the mark clears when the response
object resolves rather than at stream end, so it can flash briefly. For agent
targets — the case this exists for — the backend is buffered and it clears exactly
when the run finishes.
Problem
The live launcher footer counts a request when its response completes. A target that
takes minutes to answer therefore shows nothing at all while it works, and the
session looks hung.
Why it matters
This is acute when the target is an agent. In our demo a BioMni run takes ~110
seconds of complete silence before the first byte. Users conclude the proxy is
broken and interrupt a working run — we did this repeatedly before adding the label.
Proposed behavior
Mark targets that have a dispatched-but-unanswered request:
The mark clears when the response arrives, and the row becomes a normal stats row.
Implementation that worked for us
ProxyContextat the existing pre-routing hook and read theselected target lazily at render time. Routing picks the target partway
through the chain, so reading it at entry is too early — this avoids adding a
new mid-chain hook.
not run when a backend raises, so without this a failed request marks its target
as running forever. It must be a raw ASGI middleware, not a Starlette HTTP
middleware — the latter runs the app in a separate task, so a context variable
set in it never reaches the chain.
they are not yet in the stats snapshot at all.
One honest limitation: for streaming targets the mark clears when the response
object resolves rather than at stream end, so it can flash briefly. For agent
targets — the case this exists for — the backend is buffered and it clears exactly
when the run finishes.