WIP: Start with the old `versions.json`, instead of starting from scratch by DilumAluthge · Pull Request #67 · JuliaLang/VersionsJSONUtil.jl

DilumAluthge · 2026-04-17T06:30:34Z

🤖 Some parts written by Codex.

This addresses the first sub-item from #51:

That process is very slow, so the overall build takes a considerable amount of time: my most recent runs as of this writing took 3.5 hours each, though earlier runs had usually been more like 1.5 hours (which is still a long time). It will continue to take more time for each new version that's added.

Prior to this PR, we build the entire versions.json from scratch every time. With this PR, we re-use as much of the existing versions.json as possible. The only parts we have to do are:

New Julia versions that didn't exist previously.
If any of the old info is stale, we have to re-do that.

In order to know whether the old info is good or stale, we store the ETag and Last-Modified headers, and we discard the old info if either header has changed.

Other changes

Use custom structs for JSON.jl struct-mapping (instead of just using Dicts).

TODO before merging:

Uncomment the line marked "uncomment before merging"

🤖 Some parts written by Codex. Co-authored-by: OpenAI Codex GPT-5 <codex@openai.com>

vchuravy

Perhaps we could split the PR into the typed json changes and then the incremental rebuild.

vchuravy · 2026-07-17T15:18:08Z

@@ -0,0 +1,171 @@
+using JSON: JSON
+
+struct OurCustomStyle <: JSON.JSONStyle


Should this be something better named?

Instead of re-downloading and re-hashing every Julia binary ever released (1.5-3.5h per run), seed from the currently deployed versions.json and a new versions-meta.json sidecar that records, per URL, the ETag and Last-Modified we observed when we last hashed it (plus the URLs known not to exist). Each run HEADs every URL: entries whose recorded headers are unchanged are carried over, and only new or stale files are downloaded -- a release update becomes a few minutes. The published versions.json format is completely unchanged; the sidecar is uploaded next to it on S3. Robustness rules, factored as pure tested functions: a non-200 HEAD for an entry already in the seed keeps the entry and warns instead of deleting it (the CDN is known to serve stale per-POP 404s); a 404 on an .asc marks the .asc URL as nonexistent, not the binary; a missing or corrupt seed falls back to a full rebuild; the skiplisted corrupt tarball is exempt from the tree-hash completeness requirement so it is not re-downloaded every run. CI gains a full-rebuild workflow_dispatch input and a monthly scheduled full rebuild (which also deploys) as a backstop against drift the header comparison cannot detect, and Fastly purge failures now surface as warning annotations instead of passing silently. The incremental seeding + header-revalidation design is due to Dilum Aluthge (#67); this is a re-implementation of it on top of main without the typed-JSON refactor, so the diff stays reviewable. Co-authored-by: Dilum Aluthge <dilum@aluthge.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Instead of re-downloading and re-hashing every Julia binary ever released (1.5-3.5h per run), seed from the currently deployed versions.json and only probe what is missing from it: an entry already in the seed is carried over after a cheap HEAD cross-check of its recorded size against Content-Length, so a normal run downloads only newly released files and finishes in minutes. The published versions.json format is completely unchanged and there is no state beyond the deployed file itself. Robustness rules, factored as pure tested functions: a non-200 HEAD for an entry already in the seed keeps the entry and warns instead of deleting it (the CDN is known to serve stale per-POP 404s); a size mismatch means the published file was replaced and forces a re-download; a missing or broken seed falls back to a full rebuild; the skiplisted corrupt tarball is exempt from the tree-hash completeness requirement so it is not re-downloaded every run. CI gains a full-rebuild workflow_dispatch input and a monthly scheduled full rebuild (which also deploys) as a backstop for anything the size check cannot detect, and Fastly purge failures now surface as warning annotations instead of passing silently. The seed-and-reuse design is due to Dilum Aluthge (#67); this is a minimal re-implementation of it on top of main. Co-authored-by: Dilum Aluthge <dilum@aluthge.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

DilumAluthge · 2026-08-14T04:27:22Z

Closing in favor of #101

DilumAluthge · 2026-08-14T04:27:51Z

Perhaps we could split the PR into the typed json changes and then the incremental rebuild.

Kristoffer's #101 handles just the incremental rebuild.

@DilumAluthge

* Incremental rebuilds seeded from the deployed versions.json Instead of re-downloading and re-hashing every Julia binary ever released (1.5-3.5h per run), seed from the currently deployed versions.json and only probe what is missing from it: an entry already in the seed is carried over after a cheap HEAD cross-check of its recorded size against Content-Length, so a normal run downloads only newly released files and finishes in minutes. The published versions.json format is completely unchanged and there is no state beyond the deployed file itself. Robustness rules, factored as pure tested functions: a non-200 HEAD for an entry already in the seed keeps the entry and warns instead of deleting it (the CDN is known to serve stale per-POP 404s); a size mismatch means the published file was replaced and forces a re-download; a missing or broken seed falls back to a full rebuild; the skiplisted corrupt tarball is exempt from the tree-hash completeness requirement so it is not re-downloaded every run. CI gains a full-rebuild workflow_dispatch input and a monthly scheduled full rebuild (which also deploys) as a backstop for anything the size check cannot detect, and Fastly purge failures now surface as warning annotations instead of passing silently. The seed-and-reuse design is due to Dilum Aluthge (#67); this is a minimal re-implementation of it on top of main. Co-authored-by: Dilum Aluthge <dilum@aluthge.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Do the HEAD checks concurrently in a prepass All candidate URLs are HEADed up front with asyncmap and the build loop reads from the resulting table. ntasks matches HTTP.jl's per-host connection limit: more tasks than connections just thrash the pool with TLS setup (measured slower). A failed request (DNS, connection reset) now yields status 0 and is treated as transient instead of killing the run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Set finite timeouts on the HEAD requests HTTP.jl's default read timeout is infinite, so a single black-holed connection (observed in the wild: Fastly tarpitting a rate-limited client by accepting the connection and never answering) hangs the whole build forever, at 0% CPU with no output. With finite timeouts the request fails, is warned about, and the transient-status policy skips or keeps the entry as appropriate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Make the HEAD checks sequential again The concurrency saved ~20s on a run with minutes of fixed overhead, and the CDN has been observed to tarpit clients that make concurrent bursts -- a rate-limited CI runner would cost hours, not seconds. Keep the finite timeouts and failed-request handling; drop the prepass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * CI: remove the monthly scheduled full rebuild A full rebuild's wall time is already close to GitHub's 6-hour limit for hosted runners, so a scheduled full rebuild is not sustainable. The manual full-rebuild workflow_dispatch input remains for when one is needed; the ETag/Last-Modified cross-checks now take over as the ongoing staleness guard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Cross-check seeded entries against ETag and Last-Modified too Implement the reuse cascade proposed by @DilumAluthge in #101: a seeded entry is compared field by field against the live HEAD response (size vs Content-Length, then etag vs ETag, then last-modified vs Last-Modified); any mismatch triggers a re-download, while a field absent from the entry ends the cascade with a reuse, so the new fields can be adopted incrementally. Newly downloaded entries record etag and last-modified from the HEAD response, and the schema allows the two new optional fields. Also use -1 instead of 0 as the sentinel for a missing Content-Length header, so a genuine zero-length response counts as a size mismatch rather than passing as unverifiable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Shorten comments Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Make the can't-check-this-header logic explicit Use nothing (uniformly with etag/last-modified) instead of a -1 sentinel for a missing Content-Length, and rewrite entry_matches_head as a field-by-field cascade where skipping an uncheckable header is spelled out, per review feedback in #101. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Validate seeded entries more strictly before carrying them over Blank required string fields and malformed git tree hashes now count as incomplete, so such entries get re-downloaded rather than carried over. Also make the .tar.gz check case-insensitive and note in entry_matches_head that filedict_is_complete has already required size. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Allow etag and last-modified in the post-build key check Ports the test/more_tests.jl change from #80. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * CI: diff the rebuilt versions.json against the deployed seed Makes it visible in the log what an incremental run actually changed. Both sides are key-sorted with jq since the Dict serialization order is not stable. Skipped when no seed was downloaded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Use Fastly API token for purging * Address review comments on CI comments Trim the jq-sort comment (JSON.jl v1 does sort plain Dict keys, so the old rationale was wrong) and drop the stale Fastly 401 comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review comments: check all recorded header fields, guard 'files', explicit :proceed - entry_matches_head now skips an absent etag/last-modified field and keeps checking the remaining ones instead of reusing immediately - find_filedict/delete_filedicts_for_url! tolerate a seeded version entry without a "files" key - the carry-over check in main() states action == :proceed explicitly Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Dilum Aluthge <dilum@aluthge.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Fredrik Ekre <ekrefredrik@gmail.com>

DilumAluthge force-pushed the dpa/reuse-existing-work branch 3 times, most recently from 41138dd to d5410a9 Compare April 17, 2026 08:38

DilumAluthge force-pushed the dpa/sort-ordereddict branch 4 times, most recently from 8ac8137 to 51876ee Compare April 17, 2026 08:43

DilumAluthge changed the base branch from dpa/sort-ordereddict to main April 18, 2026 14:05

DilumAluthge mentioned this pull request Apr 18, 2026

When writing the JSON file, print the top-level keys (version numbers) in SemVer order (not lexicographical order) #66

Closed

DilumAluthge force-pushed the dpa/reuse-existing-work branch 12 times, most recently from 53f214d to b8b12f8 Compare April 19, 2026 03:44

DilumAluthge mentioned this pull request Apr 19, 2026

Record ETag and Last-Modified for each URL #76

Draft

DilumAluthge force-pushed the dpa/reuse-existing-work branch 4 times, most recently from b64e4dd to 7ce61de Compare April 19, 2026 08:15

This was referenced Apr 28, 2026

Revert "Add git-tree-{sha1,sha256} (but only for .tar.gz downloads)" #79

Closed

Temporarily disable the computation of treehashes (git-tree-sha1 and git-tree-sha256) #81

Open

DilumAluthge force-pushed the dpa/reuse-existing-work branch from faac805 to 3ab54f1 Compare May 1, 2026 04:33

Start with the old versions.json, instead of starting from scratch

9f06959

🤖 Some parts written by Codex. Co-authored-by: OpenAI Codex GPT-5 <codex@openai.com>

DilumAluthge force-pushed the dpa/reuse-existing-work branch from 3ab54f1 to 9f06959 Compare May 1, 2026 04:34

stuff

f85cdde

DilumAluthge mentioned this pull request May 3, 2026

TODO: Re-enable the computation of git-tree-sha1 and git-tree-sha256 #89

Open

vchuravy reviewed Jul 17, 2026

View reviewed changes

KristofferC mentioned this pull request Aug 11, 2026

Incremental rebuilds seeded from the deployed versions.json #101

Merged

DilumAluthge closed this Aug 14, 2026

DilumAluthge deleted the dpa/reuse-existing-work branch August 14, 2026 04:27

Navigation Menu

Uh oh!

Uh oh!

WIP: Start with the old `versions.json`, instead of starting from scratch - #67

WIP: Start with the old `versions.json`, instead of starting from scratch#67
DilumAluthge wants to merge 2 commits into
mainfrom
dpa/reuse-existing-work

DilumAluthge commented Apr 17, 2026 •

edited

Loading

Uh oh!

vchuravy left a comment

Uh oh!

vchuravy Jul 17, 2026

Uh oh!

DilumAluthge commented Aug 14, 2026

Uh oh!

DilumAluthge commented Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,171 @@
		using JSON: JSON

		struct OurCustomStyle <: JSON.JSONStyle

Uh oh!

Uh oh!

Conversation

DilumAluthge commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Other changes

TODO before merging:

Uh oh!

vchuravy left a comment

Choose a reason for hiding this comment

Uh oh!

vchuravy Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

DilumAluthge commented Aug 14, 2026

Uh oh!

DilumAluthge commented Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

DilumAluthge commented Apr 17, 2026 •

edited

Loading