feat(queue): harden durable close queue for RUSH-2307 (v0.18.0)

[muqsitnawaz]

User-visible: linear update --done --proof survives Linear rate limits with a durable on-disk queue; v0.18.0.

What changed

• Core queue already landed in feat(queue): durable local close queue with rate-limit retry (RUSH-2308) #35 (mis-tagged RUSH-2308). This PR closes RUSH-2307 acceptance gaps:
  • File lock (fcntl.flock on ~/.linear-cli/queue/.drain.lock) so concurrent drains on one machine are safe
  • linear queue list (explicit) + linear queue drain --once
  • Retry-After surfaced from HTTP into GraphQL error extensions; drain prefers it over exponential backoff (capped)
  • Rate-limit runbook in skill.md / README (no tight-loop bulk closes)
• Version 0.18.0 + install.sh pin/SHA

Acceptance (RUSH-2307)

1. Simulated 429 → intent appears in linear queue list (not dropped)
2. linear queue drain applies after backoff / Retry-After
3. Concurrent drains safe (file lock)
4. Docs runbook present
5. Real unit tests exercise 429 path (no mock-of-success)

Run result

100/100 unittest green:

Ran 100 tests in 0.015s
OK

Simulated rate-limit path:

RUSH-DEMO -> queued for retry (rate limited / transient)
1 close intent(s) queued:
  RUSH-DEMO -> Done (attempts=0)
Retained RUSH-DEMO in queue (retry 1 in 12s)   # Retry-After: 12

Evidence:

• unittest: https://share.agents-cli.sh/muqsitnawaz/rush-2307-queue-harden-rush-2307-full-tests-36e9115961dddfc6
• demo: https://share.agents-cli.sh/muqsitnawaz/rush-2307-queue-harden-rush-2307-demo-b761cb7a1fad4168

Related

• Closes RUSH-2307
• Builds on feat(queue): durable local close queue with rate-limit retry (RUSH-2308) #35 / RUSH-2308 (agents consumer already Done)
• RUSH-2310 (fleet cache + shared limiter) out of scope

Notes

• No UI surface (CLI only) — run proof is unittest + simulated 429 transcript above

[muqsitnawaz]

Non-author review (overnight eng verification)

Verdict: APPROVE

Automated prix/code-reviewer is paused fleet-wide (#1767); CI is green and the acceptance checklist was re-verified against the diff + local unittest.

Acceptance map

Criterion	Evidence
Simulated 429 does not drop op	queue_close_and_try keeps intent; CloseQueueTest.test_queue_close_and_try_keeps_intent_on_rate_limit
Drain applies after limit	test_rate_limit_retains_intent_and_later_drain_applies
Concurrent drains safe	queue_drain_lock + fcntl.LOCK_EX|LOCK_NB; test_concurrent_drain_is_serialized_by_file_lock
Docs runbook	skill.md ## Durable closes (rate-limit runbook); README durable-closes bullet
429 path tested	16 CloseQueueTest cases incl. Retry-After and --once

Checks

• python3 -m unittest → 100 OK (CI unittest SUCCESS)
• install.sh SHA-256 matches linear at 0.18.0
• Intents never store API key (build_close_intent fields only)

Notes

• Soft-success on queue (queued for retry → exit 0) matches feat(queue): durable local close queue with rate-limit retry (RUSH-2308) #35 contract; agents may treat queue as success.
• RUSH-2310 (shared rate-limiter/cache) deliberately out of scope.

[prix-cloud]

Code Reviewer

Verdict: Ready to merge

Build: Single-file Python CLI; python3 linear queue --help / drain --help parse correctly; __version__ = 0.18.0.
Tests: 100/100 pass (python3 -m unittest test_linear). Ran against this PR's head SHA 8295515, which matches gh pr view 36 --json headRefOid.

What I verified

• Working tree matched the PR head exactly (git rev-parse HEAD = headRefOid), so the test run is against the real PR code.
• install.sh's pinned SHA (3769d233…) matches sha256sum linear on this branch — the checksum pin is consistent with the shipped file.
• New argparse wiring (queue list, queue drain --once) registered correctly; cmd_queue uses getattr(..., None) so bare queue, list, and drain dispatch cleanly.
• Concurrent-drain file lock is sound: queue_drain_lock uses flock on a dotfile (.drain.lock) inside the queue dir, which the *.json glob in list_queue_intents correctly excludes (so the lock file is never mistaken for an intent and never counts against MAX_QUEUE_SIZE). The non-blocking acquire + recursion with acquire_lock=False avoids deadlock. Test test_concurrent_drain_is_serialized_by_file_lock exercises the lock-held no-op path.
• Backoff edge cases hold: Retry-After capped at 300s, negative/invalid values fall back to exponential backoff (resolve_backoff_delay(1, -5)=2, (2,'nope')=4, (1,1e6)=300).
• drain --once accounting is correct (only one due intent processed; non-due intents aren't consumed).

Changes that work well

• Retry-After surfaced end-to-end (HTTP header → GraphQL extensions → drain) with a guard against pathological values.
• Atomic intent writes (tmp + replace) keep the queue crash-safe, and last_retry_after is never persisted — it's a per-attempt hint, popped before save — so a stale backoff can't poison a later retry.
• The tests are genuine 429-path tests (fake gql returns rate-limit error extensions), not mock-of-success.

Issues that need attention (minor, non-blocking)

• Retry-After from the first attempt is dropped when a close is queued via linear update --done. In queue_close_and_try (linear:674–684), the intent is saved to disk (with attempts=0, no last_retry_after) before apply_close_intent runs. On a transient failure, apply_close_intent sets intent["last_retry_after"] on the in-memory dict, but queue_close_and_try returns without re-saving it. So when the auto-drain on the next linear update picks the intent up, the first drain retry uses exponential backoff (2s) instead of the server's Retry-After, i.e. it retries more aggressively than the server asked — the exact thing the PR claims to fix. It self-corrects: the drain's own apply_close_intent captures a fresh Retry-After and saves it, so only the first retry ignores server guidance. No intent is lost and nothing corrupts. Fix if you want the stated guarantee airtight: re-save the intent on the transient branch of queue_close_and_try, e.g. save_queue_intent(intent) before return True — or have drain fall back to the persisted hint. Note the added tests only exercise the drain-internal path (test_drain_honors_retry_after_from_error_extensions feeds drain_queue directly), so this cross-function gap isn't covered by a test.

Documentation nit

• README:73 lists linear queue list as "same as bare queue", but the implementation makes list strictly more detailed (it shows next_attempt, via detailed=True), and print_queue_list prints a "Run linear queue drain to apply them." hint for the bare form but not the list form. Cosmetic; not worth blocking.

Things to verify manually

• Nothing end-to-end against the real Linear API was possible here (no credentials). The Retry-After-header path through gql (_header_retry_after) relies on a real 429's Retry-After header; the unit tests inject Retry-After via error extensions rather than the HTTP header. A live smoke test of the queue against a real rate limit would confirm the header-parsing path, which is otherwise unexercised.

---

Reviewed by Code Reviewer — actually ran the build and tests on this branch.