Skip to content

Connected Services: a single-member pool at a usage limit never arms wait-and-resume, and the session stays idle past the reset #348

Description

@karolzlot

Summary

With automatic fallback enabled, a Connected Services pool that holds one account never arms usage-limit wait-and-resume, so a session stopped by a provider usage limit stays idle long after the reset it already knows about.

What happened (current behavior)

The daemon parsed the usage limit and its reset time, the group switch ended as no_eligible_member because the pool's only member is the active one, no usage-limit recovery intent was ever written to the session, and the session was still idle hours after the reset had passed.

Expected behavior

A pool whose recovery mode includes waiting waits for the parsed provider reset and resumes the session, also when the pool has no second account to switch to.

Reproduction steps

  1. On a self-hosted relay, connect a Claude subscription profile and place it in a Connected Services pool whose only enabled member is that profile.
  2. In the pool policy leave Automatic fallback on and Recovery mode at its default Switch or wait.
  3. Run a session on that pool until the provider reports a usage limit that carries a reset time.
  4. In the daemon log the reactive runtime-auth switch reports "limitCategory":"usage_limit" with a parsed "resetsAtMs" and "resultStatus":"no_eligible_member", and its decision trace holds a single candidate excluded with "exclusionReason":"current_active", "retryAtMs":null and "quotaEvidence":{"status":"stale_or_missing"}.
  5. Read the session: its metadata has no sessionUsageLimitRecoveryV1, and lastRuntimeIssue.source is provider_session_error.
  6. Wait past the reset: the session stays active: true with latestTurnStatus: "failed" and an empty queue, and the daemon logs no further recovery activity for it.

Severity

high

Frequency

always

Happier version

dev @ 8cf3c40

Platform

Debian 13 (trixie), x86_64

Server version

No response

Deployment type

self-hosted

What changed recently?

Automatic fallback was turned on for the pool, which is the workaround for #276.

Diagnostics ID

No response

Additional context

Observed episode, control cases, and the source path that appears to gate the wait

Observed on the running system. Three sessions on the same single-member pool hit the limit within 20 seconds of each other and behaved identically.

The provider evidence was structured, not just screen text: the transcript row carried isApiErrorMessage: true, apiErrorStatus: 429 and error: "rate_limit".

Runtime-auth recovery retried from the reset instant for about 15 minutes, every attempt ending no_eligible_member, then stopped for good after a scheduled credential refresh, logging runtime_auth_recovery_superseded with "reason":"source_tuple_mismatch". Nothing re-armed after that, and no recovery intent for these sessions exists in the daemon's persisted recovery state.

Control cases showing the wait machinery is not provider-specific and does work: sessions on a multi-member pool in the same daemon do receive sessionUsageLimitRecoveryV1, and #264 shows a Claude session reaching status: "waiting" with a scheduled next check.

Derived from source at 8cf3c40, not confirmed by instrumentation. apps/cli/src/daemon/connectedServices/accountGroups/quotas/lifecycle.ts emits the blocked edge only when selection.reason === 'no_eligible_members' and at least one exclusion passes isQuotaLifecycleBlockingExclusion, which accepts only quota_exhausted and capacity_limited. A pool whose single member is the active profile produces neither, so the group never reaches blocked and nothing arms the wait.

A manual check cannot substitute either. The registered per-provider adapter in apps/cli/src/session/usageLimitRecoveryControls/createBackoffSessionUsageLimitRecoveryControlAdapter.ts builds an intent only from a lastRuntimeIssue whose source is usage_limit; against the observed provider_session_error it answers session_usage_limit_recovery_control_inactive.

Not verified:

  • which code path writes the generic provider_session_error for this failure, given that the rate-limit surfacing path builds a usage_limit issue
  • whether a single-member pool on another provider behaves the same
  • whether a pool with two or more members that are all limited at once arms correctly
  • no run on a stock upstream relay build

related #276: same pool shape but with automatic fallback off, ending at the policy gate as auto_switch_disabled. Its context predicted that past that gate the selection would return no_eligible_member with groupExhausted: true and a retryAtMs, and that the wait would then arm. This run is that missing control and the prediction did not hold: the selection returned no_eligible_member with no quota exclusion and no retry time.

related #277: a policy-blocked recovery stored as waiting with no retry time. Here no intent is stored at all.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs:maintainerProject review or action is required.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions