Skip to content

fix(auth): cap upstream 429 Retry-After so quota resets are picked up - #4725

Open
artile wants to merge 1 commit into
router-for-me:devfrom
artile:fix/cooldown-reset-management
Open

fix(auth): cap upstream 429 Retry-After so quota resets are picked up#4725
artile wants to merge 1 commit into
router-for-me:devfrom
artile:fix/cooldown-reset-management

Conversation

@artile

@artile artile commented Aug 1, 2026

Copy link
Copy Markdown

Fixes #4724.

Problem

A single upstream 429 Retry-After can be huge — ChatGPT returned 561635 (~156h, a weekly quota window) on our instance. Both 429 cooldown paths honored that hint verbatim:

  • per-model MarkResult path (conductor_cooldown.go)
  • applyAuthFailureState (conductor_cooldown.go)

That parked every credential for days, and nothing re-probes a credential while its cooldown window is open. So when the account was reset upstream (manual reset-credit, or the window rolling over), the proxy kept returning 429 model_cooldown until a systemctl restart cliproxyapi cleared the in-memory cooldown.

The proxy already caps its own quota backoff at quotaBackoffMax = 30 * time.Minute (conductor_refresh.go), so re-probing at that cadence is exactly what a healthy setup already does.

Fix

Cap the honored upstream Retry-After at quotaBackoffMax (30 min) in both 429 cooldown paths. Hints at or below the cap are honored unchanged; the credential now re-probes within half an hour, so an upstream reset is picked up without a restart.

Notes

  • The management endpoint POST /v0/management/reset-quota (v7.2.100+) remains the immediate operational escape hatch for a manual reset.
  • Huge Retry-After values still work through the retry/wait path (conductor_selection.go), which is already bounded by maxWait.

Tests

  • TestCappedUpstreamRetryAfter — helper edges (zero, negative, below-cap, at-cap, above-cap).
  • TestApplyAuthFailureStateCapsUpstreamRetryAfter — 7-day hint → capped at 30 min, and auto-recovers after the capped deadline.
  • TestMarkResultCapsUpstreamRetryAfter — per-model path: 7-day hint → capped at 30 min, auto-recovers.

go build ./... and go test ./sdk/cliproxy/auth/... pass.

Upstreams occasionally return multi-day reset hints (e.g. a weekly ChatGPT
quota). Honoring them verbatim parks every credential in cooldown for days,
and nothing re-probes before the window expires, so the proxy keeps 429ing
until a service restart clears the in-memory cooldown.

Cap the honored Retry-After at quotaBackoffMax (30 min), the same ceiling
the proxy already uses for its own quota backoff, in both 429 cooldown
paths (per-model MarkResult and applyAuthFailureState). Hints at or below
the cap are honored unchanged. Fixes the scenario in router-for-me#4724.
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown

This pull request targeted main.

The base branch has been automatically changed to dev.

@github-actions
github-actions Bot changed the base branch from main to dev August 1, 2026 20:16
@jroth1111

Copy link
Copy Markdown

Clean fix — no conflict with the antigravity recovery engine (commit d5b5787). The recovery engine's antigravityRetryAfter honors upstream Retry-After for per-request retry delays inside the executor, while this PR caps the same hint at the conductor-level cooldown path. Different layers, complementary: the executor retries within a 75s budget using the hint, and the conductor re-probes the credential within 30 min regardless of how large the upstream hint was.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] model_cooldown sticks ~156h after an upstream quota reset; only a service restart wakes the proxy

2 participants