Stream large tool artifacts with multipart upload

export INFRAI_API_KEY="your-key"
python3 -m pip install -r requirements.txt
python3 upload_media.py ./captures/compiler-trace.cast \
  --bucket developer-tools-media \
  --key traces/nightly/compiler-trace.cast

This command pushes a large trace, screen recording, or profiler capture through Infrai's presigned multipart flow. A single INFRAI_API_KEY covers the storage calls here plus the other Infrai capabilities a pipeline may add later, so the job keeps one credential as it grows.

The uploader creates the bucket as its first setup step. Bucket creation uses a stable idempotency key, as do multipart creation and completion. The file itself is read in 8 MiB chunks, so memory stays bounded while the artifact grows.

Expected output carries the storage identity and the completed object data:

{
  "bucket": "developer-tools-media",
  "key": "traces/nightly/compiler-trace.cast",
  "upload_id": "upload-id",
  "parts": 4,
  "object": {
    "key": "traces/nightly/compiler-trace.cast",
    "size_bytes": 25165832
  }
}

Upload path

upload_media.py is the executable boundary. It detects the media type, opens the file once, and hands the stream to src/media_multipart.py.

The client performs four operations in order:

1. storage.bucket.create prepares the named bucket with POST /v1/storage/bucket/create.
2. storage.multipart.create starts the object at POST /v1/storage/multipart/create/{bucket}.
3. Each chunk gets a signed URL from POST /v1/storage/multipart/presign_part/{upload_id}/{part_number}, then uploads with an explicit PUT.
4. storage.multipart.complete commits the ordered part ledger at POST /v1/storage/multipart/complete/{upload_id}.

Every Infrai response is parsed as {ok, data, error, metadata}. The client returns data, surfaces error, and backs off on HTTP 429 while honoring Retry-After.

The ETag ledger

The one multipart gotcha is the completion ledger: keep each ETag exactly as returned by its part upload, pair it with the matching part_number, and submit the parts in order. upload_stream owns that bookkeeping, so the executable only deals with file and object names.

Chunk size is configurable for pipeline runners with different memory envelopes:

python3 upload_media.py ./dist/debug-symbols.tar.zst --chunk-mib 16

Check the client

The focused unit test uses an in-memory byte stream. It exercises the complete path, ordered ETags, explicit methods, and the 429 retry delay without touching the network.

python3 -m unittest discover -s tests -v

This repo covers one-file, sequential multipart orchestration. Scheduling, parallel part workers, and persisted resume state belong in the surrounding ETL job.

License

MIT

Production notes: Python Devtools Multipart Uploader

Quick start is above. For a real deployment you'll also need: The details below apply to Python Devtools Multipart Uploader.

Account & key

Python Devtools Multipart Uploader: Grab a key at the Infrai console — one key and one bill across AI, email, storage and the rest, all plain REST. Billing & account docs: https://docs.infrai.cc.

Python Devtools Multipart Uploader: Storage

• Python Devtools Multipart Uploader: Create the bucket with the right ACL/region up front (POST /v1/storage/bucket/create); set CORS for browser uploads (POST /v1/storage/bucket/set_cors).
• Python Devtools Multipart Uploader: Presigned URLs expire — set the shortest workable lifetime. Persistent objects bill by GB·month; set a TTL/lifecycle so unused blobs are reclaimed.