Use aws cli to query S3 directly - #104
Conversation
|
Thanks, this will probably help with the likely rate limiting that we hit now with all the HEAD requests. If you run this locally, does it finish quickly? We could also run parallel head requests but maybe that is not needed. |
|
Could we use AWS.jl instead of the |
… CDN The build made thousands of unauthenticated HEAD requests through the Fastly CDN (julialang-s3.julialang.org) to determine which binaries exist, which ran into rate limits and could be misled by stale per-POP 404s. The build now talks to the julialang2 S3 bucket directly via AWS.jl, and asks for whole prefix listings instead of probing per URL: - One (paginated) list-objects-v2 per bin/<os>/<arch>/<major.minor>/ prefix answers existence, size, etag, and last-modified for every file under it, replacing the ~17 per-object checks per version. A key absent from a listed prefix is an authoritative 404; a failed listing marks its URLs as transient (kept if already published, retried next run otherwise). Binary downloads also go to the bucket, authenticated. The URLs recorded in versions.json stay on the CDN host. - CI authenticates through GitHub's OIDC provider with per-job IAM roles, so no long-lived keys are stored: full-test assumes a read-only role (list/get on bin/, 6 h sessions to outlive a full rebuild), while upload-to-s3 and deploy-schema assume a deploy role that can write exactly the two deployed files. The role ARNs are inlined in the workflows; trust and permissions documents are checked in under devdocs/aws/ with the aws iam commands to create the roles documented in devdocs/README.md. The AWS_DEPLOY_* secrets are no longer used. Without credentials (e.g. local runs) the code sends unsigned requests, which the public bucket accepts; an auth error on a signed request fails the run fast instead of silently skipping every URL. - Timestamps from S3 (ISO 8601) are normalized to the HTTP-date form versions.json has always recorded, keeping seeded entries comparable. - Dependencies: WebCacheUtilities (an unregistered git dependency pulling in Plots, DataFrames, CSV, and more) is replaced by a small local file-cache helper, and the unused TimeZones is dropped; AWS.jl is added and HTTP.jl (now only used for the GitHub tags fetch) is bumped to 2.x. Net: the Manifest shrinks by ~1400 lines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
92bf6f2 to
be3d544
Compare
|
Updated the PR to use HTTP.jl and AWS.jl instead of aws cli and WebCacheUtils. Also updated the PR to use Github OIDC (and I have deployed the policy documents already). |
|
Tangentially, and a bit provocative, but it is amazing to me that there is a need for so much infrastructure to update a text file with some new data from a few releases. Probably not a useful comment.. |
I think the main reason is that this was initially built in a way that the release process itself would not have to be changed to avoid adding additional burden and because there wasn't a reliable source of truth for the info in it. That approach hasn't really been changed since the initial prototype. #51 includes some suggestions that would significantly improve it. Given how much of the ecosystem uses the file at this point, it could be worth reconsidering the entire approach from scratch and embedding it in the release pipeline, though. |
5 participants
No description provided.