Skip to content

Retry transient failures when downloading artifacts - #585

Draft
dan-manges wants to merge 2 commits into
mainfrom
dan/retry-artifact-download
Draft

Retry transient failures when downloading artifacts#585
dan-manges wants to merge 2 commits into
mainfrom
dan/retry-artifact-download

Conversation

@dan-manges

Copy link
Copy Markdown
Member

rwx artifacts download made a single unretried request to storage, so a connection reset partway through surfaced as a hard failure:

Error: unable to download artifact: HTTP request failed: Get "https://mint-storage-...s3.us-east-2.amazonaws.com/org...
read tcp 192.168.86.51:53581->3.5.89.33:443: read: connection reset by peer

DownloadArtifact now retries with bounded exponential backoff, following the same shape as ListRuns: a downloadArtifactOnce helper returns a retryable flag, and the caller loops on retry.NewBackoff() (5 attempts, 1s→5s, ~12s worst case).

Retried:

  • transient transport errors via retry.IsTransient — resets, refused, timeouts, EOF
  • transient failures reading the response body, so a reset partway through a large artifact restarts instead of failing
  • 429 and 5xx from storage (S3 SlowDown)

Not retried: 400/403/404, so an expired presigned URL or missing key still surfaces immediately. The same presigned URL is reused across the short retry window.

Exhausted retries are wrapped with errors.ErrNetworkTransient, matching retry.RoundTripper, so telemetry buckets them as network_transient_error rather than unknown.

This also covers the parallel multi-artifact download path in service_artifacts.go, which goes through the same client method. Retries are silent under the existing "Downloading artifact..." spinner.

Still buffered, no resume

Artifacts are read fully into memory and a retry restarts from byte zero — there's no Range-based resume. A reset late in a multi-GB download still re-transfers the whole thing.

.rwx/sandbox.yml

Added, since the repo had no sandbox config. It mirrors ci.yml (codetool-versionsgo/go-deps, plus the lsp-* chain so internal/lsp's embedded bundle exists) with a sandbox task running rwx-sandbox.

Verification

  • rwx sandbox exec -- go test ./internal/... ./cmd/... — all packages pass
  • golangci-lint run ./internal/api/... — 0 issues
  • New TestAPIClient_DownloadArtifact_Retries covers reset-then-success, exhaustion after 5 attempts with the transient sentinel, 503-then-success, and 403 not retried. Resets are simulated by hijacking the connection with SetLinger(0).

🤖 Generated with Claude Code

dan-manges and others added 2 commits July 24, 2026 21:03
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…wnload

# Conflicts:
#	internal/api/client_test.go
@kylekthompson
kylekthompson marked this pull request as draft August 5, 2026 15:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant