feat(routing): quota-aware selection, per-subagent session affinity, 529/headerless-429 failover - #4674
feat(routing): quota-aware selection, per-subagent session affinity, 529/headerless-429 failover#4674johncpakin wants to merge 5 commits into
Conversation
…ldown Anthropic returns the non-standard 529 status for overloaded_error, and its OAuth 429/529 responses carry no Retry-After header. Previously a 529 failed the request immediately (status not retryable), and a headerless 429 failed on the first attempt because retry required a usable Retry-After. - shouldRetryAfterError: accept 529 alongside 429; when Retry-After is absent or non-positive, fall back to a short fixed wait (2s, capped at maxWait) so the request rolls over to another credential. - MarkResult / applyAuthFailureState: classify 529 with the transient 408/500/502/503/504 family (short cooldown, no quota flag) instead of the default branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With session affinity enabled, every Claude Code subagent inherits the parent conversation's session_id, so an entire multi-agent fan-out pins to a single credential (observed live: 20/20 requests from one fan-out on one account). Add routing.session-id-mode: "content-hash". In this mode the Claude Code session_id is combined with a hash of the first message contents, so subagents carrying different prompts bind to distinct credentials while each individual conversation stays pinned. The default mode is unchanged. The content hash uses the UNTRUNCATED system prompt and first user/assistant messages: subagents share their first several kilobytes verbatim (system reminders and project context) and diverge only later, so a truncated prefix hash would collapse them onto one binding. The legacy last-resort hash fallback keeps its historical 100-char sampling. The short-hash fallback ID preserves the existing first-turn binding inheritance. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tials Add routing.strategy: "quota-aware". A background goroutine polls the Anthropic OAuth usage endpoint (/api/oauth/usage) per Claude credential and caches percent-used for the 5h session window, the all-models weekly limit, and model-scoped weekly tiers. Pick never performs I/O: it selects among available credentials with probability proportional to remaining headroom on the tier matching the requested model. Degradation rules keep the strategy safe when quota data is missing: - Credentials with unknown or stale quota carry zero weight; when weighting is unavailable, selection round-robins over the unknown candidates only, so credentials known to be exhausted stay excluded. - Stale weekly-tier readings (up to 6h old) still exclude >=95%-used credentials, since weekly limits reset on a multi-day cadence; the 5h session window is ignored for stale exclusion because it may have reset. - If every candidate is unknown, behavior is plain round-robin. The poller is deliberately gentle with the usage endpoint, which rate-limits per-IP bursts and shares its bucket with other local consumers: 5m interval, 3s spacing between accounts, and a 15m-to-2h doubling backoff that aborts the cycle on the first 429. Its 15s HTTP client timeout applies only to this background credential-quota polling, never to a request path. The selector implements StoppableSelector; the service now stops a replaced selector on config swap, and SessionAffinitySelector.Stop cascades to its fallback, so pollers do not leak across hot reloads. quota-aware coexists with weighted-round-robin: it is a sibling strategy in the routing.strategy switch and uses the legacy selector pick path (it is not a built-in scheduler strategy), so WRR scheduler state is untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
This pull request targeted The base branch has been automatically changed to |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e943356592
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| } | ||
| if token == "" { | ||
| continue | ||
| } |
There was a problem hiding this comment.
When routing.strategy remains quota-aware and a Claude auth is removed or its token is rotated out, this method only upserts the current IDs and never deletes targets that are no longer present in the auths slice passed to Pick. The background refreshAll loop will keep polling those stale tokens forever, adding quotaPollSpacing per deleted account and potentially consuming the shared usage-endpoint rate limit before it reaches the active credentials.
Useful? React with 👍 / 👎.
| // This client only fetches credential quota metadata on a background | ||
| // goroutine, never on a request path, so a bounded timeout is safe and | ||
| // keeps a wedged poll from stalling the refresh loop indefinitely. | ||
| httpClient: &http.Client{Timeout: quotaFetchTimeout}, |
There was a problem hiding this comment.
This introduces a new Anthropic network client with a fixed timeout for quota polling, but this scoped code is not one of the repository's allowed timeout exceptions; under routing.strategy: quota-aware, slow usage-endpoint responses will be force-failed and quota data will age into unknown even though the repo convention requires avoiding new non-credential-acquisition timeouts. Consider using a stop-cancelled request context instead of http.Client.Timeout.
AGENTS.md reference: AGENTS.md:L58-L58
Useful? React with 👍 / 👎.
… with stop-cancellable context Addresses Codex review on router-for-me#4674: - updatePollTargets now stamps lastSeen and drops targets (and their cached quota) not offered to Pick for 15 minutes, so removed or rotated-out credentials stop consuming the shared usage-endpoint allowance. The window rides out temporary cooldown filtering. Covered by TestQuotaAwareSelector_StalePollTargetsExpire. - The poller's http.Client no longer carries a fixed timeout. Each fetch is bounded by a context deadline and cancelled by Stop(), keeping a wedged poll from stalling the refresh loop while aborting in-flight requests on shutdown or selector swap, in line with the repository's timeout conventions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Both Codex findings addressed in 7906a0e:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7906a0e8b6
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| req.Header.Set("Authorization", "Bearer "+token) | ||
| req.Header.Set("anthropic-beta", "oauth-2025-04-20") | ||
| req.Header.Set("anthropic-version", "2023-06-01") |
There was a problem hiding this comment.
When routing.strategy: quota-aware is enabled, each background usage poll goes out with Go's default User-Agent because this request only sets auth/beta/version headers, unlike the normal Claude OAuth request path that fingerprints as Claude Code and sets X-App: cli. The /api/oauth/usage endpoint aggressively rate-limits generic clients, so these polls can immediately get 429, enter the new 15m–2h backoff, and leave quota snapshots unknown so the strategy degenerates to round-robin; set the same Claude Code UA/X-App headers used by the executor.
Useful? React with 👍 / 👎.
Addresses Codex review on router-for-me#4674: background usage polls previously went out with Go's default User-Agent, which the /api/oauth/usage endpoint rate-limits aggressively, risking immediate 429 -> backoff -> permanent round-robin degradation. Polls now carry the same Claude Code UA/X-App/X-Stainless fingerprint the executor's OAuth request path uses. Values are mirrored as local constants because importing the executor helps package from sdk/cliproxy/auth would create an import cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Third finding addressed in 99c1fe0: usage polls now send the same Claude Code UA/X-App/X-Stainless fingerprint as the executor's OAuth request path (verified a fingerprinted request returns 200 where a generic client draws 429s). The values are mirrored as local constants in quota_selector.go because importing the executor's helps package from sdk/cliproxy/auth would create an import cycle — happy to relocate the profile constants to a shared package instead if you'd prefer a single source of truth. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 99c1fe09d7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // deadline keeps a wedged poll from stalling the refresh loop, and Stop() | ||
| // cancels an in-flight request immediately on shutdown or selector swap. | ||
| // This never runs on a request path — background quota polling only. | ||
| ctx, cancel := context.WithDeadline(context.Background(), s.now().Add(quotaFetchTimeout)) |
There was a problem hiding this comment.
The current code still imposes a fixed 15s quota-poll timeout via context.WithDeadline, so under routing.strategy: quota-aware any slow Anthropic usage response is forcibly canceled even though this poll is not credential acquisition and is not one of the documented timeout exceptions. Fresh evidence since the earlier review comment is that the timeout moved from the HTTP client to the request context but remains a deadline on the same network behavior; use stop-only cancellation instead of a wall-clock deadline.
AGENTS.md reference: AGENTS.md:L58-L58
Useful? React with 👍 / 👎.
| selector = coreauth.NewSessionAffinitySelectorWithConfig(coreauth.SessionAffinityConfig{ | ||
| Fallback: selector, | ||
| TTL: state.sessionAffinityTTL, | ||
| Fallback: selector, | ||
| TTL: state.sessionAffinityTTL, | ||
| SessionIDMode: state.sessionIDMode, |
There was a problem hiding this comment.
When users combine routing.strategy: quota-aware with session-affinity: true, this wrapper lets SessionAffinitySelector return an existing binding after only the normal availability check, before the QuotaAwareSelector can exclude an auth whose latest snapshot is exhausted. In that configuration, an active Claude Code session can stay pinned to an auth that quota-aware routing would no longer select, so it keeps sending requests to an exhausted credential until ordinary cooldown state changes; make cache hits consult quota headroom or reselect through the quota-aware fallback when the bound auth is exhausted.
Useful? React with 👍 / 👎.
|
Commit 7972f78 adds a simpler, always-on antigravity quota-aware filter to Your configurable |
Adversarial review findings1. Non-Claude providers in quota-aware mode degrade silently (medium)
Suggestion: add a one-time 2. Stale-but-healthy quota data is discarded entirely (minor)At Not a bug — deliberate safety trade-off. Worth documenting in the code comment that this is intentional. 3.
|
2 participants
The problem
I run Claude Code sessions that fan out to 50–100 subagents across several Claude Max accounts. Three compounding issues concentrated all of that load onto a single account and turned rate limits into hard failures:
session_id(inmetadata.user_id), so session affinity binds an entire fan-out to one credential. Observed: 43 of 47 requests on one binding.Retry-After;shouldRetryAfterErrorrequires one, so these fail on the first attempt without trying other credentials.The changes (3 commits, independently revertable)
fix(auth): 529 + headerless-429 failover. 529 is treated like the other transient upstream errors (cooldown via the existing transient path); a 429/529 with no usableRetry-Aftergets a short default wait instead of(0, false). ExistingRetry-Afterbehavior unchanged.feat(routing):session-id-mode: content-hash. Opt-in. For Claude Code traffic, the affinity key becomessession_id + hash(full first message content). Subagents (same session_id, different prompts) bind independently; every turn of one conversation still pins to its credential, so per-credential prompt caches stay warm. Full-content hashing is deliberate: subagents share their first few KB verbatim, so prefix hashing collapses them.feat(routing):strategy: quota-aware. A background poller fetches each Claude credential's usage from/api/oauth/usage(per model tier) and the selector picks proportionally to remaining headroom. Never argmax (avoids stampeding one account). Pick() does no I/O. Degrades to round-robin when quota is unknown, and only over quota-unknown credentials — known-exhausted ones stay excluded on stale data. The poller backs off hard (15m→2h) if the usage endpoint rate-limits, since that endpoint's per-IP bucket is shared and headerless.Both new behaviors are opt-in config (
routing.strategy: quota-aware,routing.session-id-mode: content-hash); defaults are unchanged. Composes with session affinity as its fallback selector rather than replacing it.Relation to weighted-round-robin
quota-awareis a sibling strategy to the recently-mergedweighted-round-robin, not a replacement: WRR distributes proportionally to static, operator-assigned integer weights, while quota-aware distributes proportionally to live remaining quota headroom fetched from Anthropic's OAuth usage endpoint per Claude credential (session window, weekly-all, and model-scoped weekly tiers). The two share no state — quota-aware is wired only into therouting.strategyswitch and runs through the existing legacy selector pick path rather than the built-in scheduler fast path, so the WRR scheduler machinery (credential weights, smooth-weighted state, weight validation) is untouched. When quota data is unknown or stale the strategy degrades to round-robin over unknown credentials only, keeping known-exhausted credentials excluded.One item flagged for maintainer review: the background poller uses a 15s
http.Clienttimeout, which sits in tension with the repo's timeouts-only-during-credential-acquisition rule. Rationale (in a code comment): it's background credential-quota polling that never sits on a request path, and an unbounded hang there is worse. Happy to change if you'd prefer a different mechanism.Validation (live, 4 Claude Max accounts)
session_idwith distinct prompts: 4 distinct credential bindings (previously 1). Identical repeat request: affinity cache hit on its own binding.go build ./...,go vet, package tests green; new tests cover the retry gate, cooldown marking, proportional distribution, tier exclusion, stale-data degradation, and content-hash extraction.Companion UI PR (quota visibility in the management panel): router-for-me/Cli-Proxy-API-Management-Center#363