Incremental rebuilds seeded from the deployed `versions.json` by KristofferC · Pull Request #101 · JuliaLang/VersionsJSONUtil.jl

KristofferC · 2026-08-11T14:25:56Z

The seed-and-reuse idea is due to @DilumAluthge (#67); this re-implements it in main's existing plain-Dict style.

What it does

Instead of re-downloading and re-hashing every Julia binary ever released (1.5–3.5 h per run), seed from the currently deployed versions.json and only probe what's missing from it. An entry already in the seed is carried over after a cheap HEAD cross-check of its recorded size, etag, and last-modified against the live Content-Length/ETag/Last-Modified headers, following the cascade proposed in this comment: any mismatch triggers a re-download, while a field absent from an old entry is simply skipped, so the new fields can be adopted incrementally. Newly downloaded entries record etag and last-modified, so coverage grows on its own over time. A normal run downloads only newly released files and finishes in minutes.

The published versions.json format is extended additively with the optional etag and last-modified fields (schema updated); everything else is unchanged.

Robustness rules (pure, tested functions)

A non-200 HEAD for an entry already in the seed keeps the entry and warns instead of deleting it — while promoting v1.13.0-rc2 we watched Fastly serve stale per-POP 404s for objects that exist (and URL purging currently 401s), so anything that deletes on 404 would silently drop released binaries from versions.json.
A mismatch between the recorded size/etag/last-modified and the live headers means the published file was replaced → re-download and re-hash it.
A missing or unparsable seed falls back to a full rebuild (warn, never fail).
The skiplisted corrupt 0.7.0-alpha tarball is exempt from the tree-hash completeness requirement, so it isn't re-downloaded every run.

CI

full-rebuild workflow_dispatch input for a manual from-scratch rebuild. There is no scheduled full rebuild: a full run's wall time is already close to GitHub's 6-hour limit for hosted runners, and the three-header cross-check covers ongoing staleness.
Fastly purge failures surface as warning annotations instead of passing silently (they currently 401; needs fixing on the Fastly service, separately).

Closes #67.

🤖 Generated with Claude Code

Instead of re-downloading and re-hashing every Julia binary ever released (1.5-3.5h per run), seed from the currently deployed versions.json and only probe what is missing from it: an entry already in the seed is carried over after a cheap HEAD cross-check of its recorded size against Content-Length, so a normal run downloads only newly released files and finishes in minutes. The published versions.json format is completely unchanged and there is no state beyond the deployed file itself. Robustness rules, factored as pure tested functions: a non-200 HEAD for an entry already in the seed keeps the entry and warns instead of deleting it (the CDN is known to serve stale per-POP 404s); a size mismatch means the published file was replaced and forces a re-download; a missing or broken seed falls back to a full rebuild; the skiplisted corrupt tarball is exempt from the tree-hash completeness requirement so it is not re-downloaded every run. CI gains a full-rebuild workflow_dispatch input and a monthly scheduled full rebuild (which also deploys) as a backstop for anything the size check cannot detect, and Fastly purge failures now surface as warning annotations instead of passing silently. The seed-and-reuse design is due to Dilum Aluthge (#67); this is a minimal re-implementation of it on top of main. Co-authored-by: Dilum Aluthge <dilum@aluthge.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

All candidate URLs are HEADed up front with asyncmap and the build loop reads from the resulting table. ntasks matches HTTP.jl's per-host connection limit: more tasks than connections just thrash the pool with TLS setup (measured slower). A failed request (DNS, connection reset) now yields status 0 and is treated as transient instead of killing the run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

HTTP.jl's default read timeout is infinite, so a single black-holed connection (observed in the wild: Fastly tarpitting a rate-limited client by accepting the connection and never answering) hangs the whole build forever, at 0% CPU with no output. With finite timeouts the request fails, is warned about, and the transient-status policy skips or keeps the entry as appropriate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The concurrency saved ~20s on a run with minutes of fixed overhead, and the CDN has been observed to tarpit clients that make concurrent bursts -- a rate-limited CI runner would cost hours, not seconds. Keep the finite timeouts and failed-request handling; drop the prepass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

DilumAluthge · 2026-08-11T20:28:59Z

Thanks for putting this together. We should try to get this in soon.

DilumAluthge · 2026-08-11T20:29:01Z

In the future, I would like to store etag and last-modified in versions.json (in addition to size). Can you implement the following logic in preparation for that?

That way, in the future, I can begin incrementally adding etag and last-modified to old entries.
aIn the future, I would like to store etag and last-modified in versions.json (in addition to size). Can you implement the following logic in preparation for that?

That way, in the future, I can begin incrementally adding etag and last-modified to old entries.

Some capitalization conventions I'll use:

Content-Length, ETag, and Last-Modified refer to the headers we get when we make the HEAD request. These are the values for the file currently on S3, at the time when the script gets run.
size, etag, and last-modified refer to the fields in versions.json. These are the values for the old entry in versions.json, which we may or may not choose to reuse.

flowchart TD
    freshdownload(((Fresh Download)))
    reuse(((Reuse the old entry)))
    A[Get old entry from `versions.json`] --> B[Make HEAD request to S3]
    B --> C{Is the `size` field present in the old entry?}
    C -- Present --> C1match{Does `size` field match `Content-Length` header}
    C -- Absent --> freshdownload
    C1match -- Yes --> C2exist{Does old entry have `etag` field}
    C1match -- No --> freshdownload
    C2exist -- Present --> C3match{Does `etag` field match `ETag` header}
    C2exist-- Absent --> reuse
    C3match -- Yes --> C4exist{Is the `last-modified` field present in the old entry?}
    C3match -- No --> freshdownload
    C4exist -- Present --> C5match{Does `last-modified` field match `Last-Modified` header}
    C4exist -- Absent --> reuse
    C5match -- Yes --> reuse
    C5match -- No --> freshdownload

A full rebuild's wall time is already close to GitHub's 6-hour limit for hosted runners, so a scheduled full rebuild is not sustainable. The manual full-rebuild workflow_dispatch input remains for when one is needed; the ETag/Last-Modified cross-checks now take over as the ongoing staleness guard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@DilumAluthge

Implement the reuse cascade proposed by @DilumAluthge in #101: a seeded entry is compared field by field against the live HEAD response (size vs Content-Length, then etag vs ETag, then last-modified vs Last-Modified); any mismatch triggers a re-download, while a field absent from the entry ends the cascade with a reuse, so the new fields can be adopted incrementally. Newly downloaded entries record etag and last-modified from the HEAD response, and the schema allows the two new optional fields. Also use -1 instead of 0 as the sentinel for a missing Content-Length header, so a genuine zero-length response counts as a size mismatch rather than passing as unverifiable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Use nothing (uniformly with etag/last-modified) instead of a -1 sentinel for a missing Content-Length, and rewrite entry_matches_head as a field-by-field cascade where skipping an uncheckable header is spelled out, per review feedback in #101. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Blank required string fields and malformed git tree hashes now count as incomplete, so such entries get re-downloaded rather than carried over. Also make the .tar.gz check case-insensitive and note in entry_matches_head that filedict_is_complete has already required size. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Ports the test/more_tests.jl change from #80. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Makes it visible in the log what an incremental run actually changed. Both sides are key-sorted with jq since the Dict serialization order is not stable. Skipped when no seed was downloaded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

KristofferC · 2026-08-12T08:20:54Z

To be even better, we could test that file_dict["git-tree-sha1"] matches ^[0-9a-f]{40}$. And ^[0-9a-f]{64}$ for the sha256.

I can't comment on this for some reason but 5e11ba4. should have that.

DilumAluthge · 2026-08-14T04:21:32Z

        shell: julia --project {0}

+      - name: Diff against the deployed versions.json
+        # sort keys with jq first: the Dict serialization order isn't stable


I'm fine using jq to sort. But is this statement actually true? I thought JSON.jl v1 will output Dicts in sorted order (sorted by key) by default.

https://juliaio.github.io/JSON.jl/stable/reference/#JSON.json

sort_keys::Union{Bool, Nothing}=nothing: Controls whether dictionary keys are sorted before writing. If true, keys for all AbstractDict objects are sorted by their lowered string representation. If false, dictionary iteration order is preserved. If nothing, plain Dict keys are sorted by default while other dictionary-like containers preserve their iteration order.

I'm interpreting that as keys for Dict are sorted by default, but I might be misreading or misunderstanding it.

JSON v1 does not output to Dict by default though (it outputs JSON.Object). But I think the comment about the implementation details of JSON does more harm than good so I removed that. Now it says it just sorts the keys which is non-controversial.

DilumAluthge · 2026-08-14T04:22:19Z

+        # Fastly currently 401s unauthenticated purges; warn rather than fail silently
+        run: |
+          curl -fsS -X PURGE -H "Fastly-Key:${{ secrets.FASTLY_PURGE_TOKEN }}" https://julialang-s3.julialang.org/bin/versions.json \
+            || echo "::warning::Fastly purge of versions.json failed; it will be served stale until the cached copy expires"


Now that we have the Fastly token in place, should we remove that comment, and also fail loudly instead of warning?

We can remove the comment but I don't think a failure to purge should be considered a failure.

but I don't think a failure to purge should be considered a failure

I'm fine with keeping "failure to purge" as a warning (non-fatal).

Trim the jq-sort comment (JSON.jl v1 does sort plain Dict keys, so the old rationale was wrong) and drop the stale Fastly 401 comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

DilumAluthge · 2026-08-14T06:00:29Z

I'll do another review soon (hopefully this weekend). After that I think we might be good to go. Hopefully we can merge this next week.

KristofferC · 2026-08-14T06:02:42Z

I'll do another review soon (hopefully this weekend).

I kind of want to release rc3 today so ideally this could go in soonish?

DilumAluthge · 2026-08-14T06:10:08Z

I'll do another review soon (hopefully this weekend).

I kind of want to release rc3 today so ideally this could go in soonish?

Let me do the review now.

DilumAluthge

Alright, these are the only remaining comments I have.

DilumAluthge · 2026-08-14T06:15:24Z

+    )
+end
+
+# The CDN is known to serve stale per-POP 404s for files that do exist, so a non-200


Off-topic, and this doesn't block this PR, but: Do we know why this happens? It seems bad if Fastly is serving 404's for files that exist?

DilumAluthge · 2026-08-14T06:16:51Z

+                          ("last-modified", head.last_modified)]
+        # only etag/last-modified can be absent here; the filedict_is_complete check
+        # that runs before this already required size
+        haskey(file_dict, field) || return true


Presumably, if file_dict doesn't have the etag key, we should still continue to the next iteration and check last-modified?

Suggested change

haskey(file_dict, field) || return true

haskey(file_dict, field) || continue

DilumAluthge · 2026-08-14T06:19:13Z

+
+function delete_filedicts_for_url!(meta, version, url)
+    # Remove any stale entry so a re-download can't produce duplicates
+    haskey(meta, version) || return nothing


In theory, we could hit a weird case where the meta[version] key exists but the meta[version]["files"] key doesn't exist? It sound like a corner case, but should we guard against it anyway?

Suggested change

haskey(meta, version) || return nothing

haskey(meta, version) || return nothing

haskey(meta[version], "files") || return nothing

DilumAluthge · 2026-08-14T06:21:21Z

+                continue
+            end
+
+            if existing !== nothing && filedict_is_complete(existing, url) &&


Do we need an explicit check for action == :proceed?

Suggested change

if existing !== nothing && filedict_is_complete(existing, url) &&

if action == :proceed && existing !== nothing && filedict_is_complete(existing, url) &&

DilumAluthge · 2026-08-14T06:25:19Z

One comment is just a question for us to investigate later, not to block this PR.

For the other comments: @KristofferC Once you've addressed those comments, this PR LGTM and is good to merge from my point-of-view, so you can go ahead and merge, and deploy.

Before you deploy, it might be worth saving yourself a local copy of the currently-deployed (on S3) versions.json, just in case something breaks and we have to revert.

…es', explicit :proceed - entry_matches_head now skips an absent etag/last-modified field and keeps checking the remaining ones instead of reusing immediately - find_filedict/delete_filedicts_for_url! tolerate a seeded version entry without a "files" key - the carry-over check in main() states action == :proceed explicitly Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

KristofferC · 2026-08-14T08:33:19Z

Even with this, updating the versions.json is quite slow. I think we get rate limited for all the HEAD spam. Really, I want a mode where I just give the version I want to release, GHA downloads those versions and updates the file.

KristofferC force-pushed the incremental-v2 branch from d17c58c to 5d43a81 Compare August 11, 2026 14:39

KristofferC changed the title ~~Incremental rebuilds seeded from the deployed versions.json (builds on #67)~~ Incremental rebuilds seeded from the deployed versions.json Aug 11, 2026

KristofferC force-pushed the incremental-v2 branch from 5d43a81 to 7112f31 Compare August 11, 2026 14:43

KristofferC and others added 3 commits August 11, 2026 16:56

KristofferC force-pushed the incremental-v2 branch from df2d5ac to 2536ca3 Compare August 11, 2026 15:15

DilumAluthge changed the title ~~Incremental rebuilds seeded from the deployed versions.json~~ Incremental rebuilds seeded from the deployed versions.json Aug 11, 2026

DilumAluthge self-requested a review August 11, 2026 19:59

DilumAluthge reviewed Aug 11, 2026

View reviewed changes

Comment thread .github/workflows/CI.yml Outdated

DilumAluthge reviewed Aug 11, 2026

View reviewed changes

Comment thread .github/workflows/CI.yml Outdated

DilumAluthge reviewed Aug 11, 2026

View reviewed changes

Comment thread src/VersionsJSONUtil.jl Outdated

Comment thread src/VersionsJSONUtil.jl Outdated

Comment thread src/VersionsJSONUtil.jl Outdated

KristofferC and others added 4 commits August 11, 2026 22:57

Shorten comments

5519c0c

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

DilumAluthge reviewed Aug 12, 2026

View reviewed changes

Comment thread src/VersionsJSONUtil.jl Outdated

Comment thread src/VersionsJSONUtil.jl Outdated

Comment thread src/VersionsJSONUtil.jl Outdated

Comment thread src/VersionsJSONUtil.jl Outdated

Comment thread src/VersionsJSONUtil.jl Outdated

DilumAluthge reviewed Aug 12, 2026

View reviewed changes

Comment thread .github/workflows/CI.yml

Comment thread src/VersionsJSONUtil.jl Outdated

Comment thread src/VersionsJSONUtil.jl

KristofferC and others added 3 commits August 12, 2026 10:10

Allow etag and last-modified in the post-build key check

d77716d

Ports the test/more_tests.jl change from #80. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Use Fastly API token for purging

b46fef0

fredrikekre mentioned this pull request Aug 13, 2026

Get a Fastly token, and then authenticate the Fastly purge? #102

Closed

DilumAluthge reviewed Aug 14, 2026

View reviewed changes

DilumAluthge mentioned this pull request Aug 14, 2026

WIP: Start with the old versions.json, instead of starting from scratch #67

Closed

1 task

Address review comments on CI comments

a14ffdc

Trim the jq-sort comment (JSON.jl v1 does sort plain Dict keys, so the old rationale was wrong) and drop the stale Fastly 401 comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

KristofferC force-pushed the incremental-v2 branch from d993072 to a14ffdc Compare August 14, 2026 05:57

DilumAluthge reviewed Aug 14, 2026

View reviewed changes

KristofferC enabled auto-merge August 14, 2026 07:22

KristofferC merged commit e70fe88 into main Aug 14, 2026
4 checks passed

KristofferC deleted the incremental-v2 branch August 14, 2026 07:40

KristofferC mentioned this pull request Sep 3, 2026

promote: Fastly cache purge has never worked (unauthenticated PURGE returns 401) JuliaCI/julia-buildkite#625

Open

	haskey(file_dict, field) \|\| return true
	haskey(file_dict, field) \|\| continue

	haskey(meta, version) \|\| return nothing
	haskey(meta, version) \|\| return nothing
	haskey(meta[version], "files") \|\| return nothing

	if existing !== nothing && filedict_is_complete(existing, url) &&
	if action == :proceed && existing !== nothing && filedict_is_complete(existing, url) &&

Uh oh!

Uh oh!

Conversation

KristofferC commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What it does

Robustness rules (pure, tested functions)

CI

Uh oh!

DilumAluthge commented Aug 11, 2026

Uh oh!

DilumAluthge commented Aug 11, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KristofferC commented Aug 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DilumAluthge commented Aug 14, 2026

Uh oh!

KristofferC commented Aug 14, 2026

Uh oh!

DilumAluthge commented Aug 14, 2026

Uh oh!

DilumAluthge left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DilumAluthge commented Aug 14, 2026

Uh oh!

Uh oh!

KristofferC commented Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

KristofferC commented Aug 11, 2026 •

edited

Loading