Allow etag and last-modified as optional fields - #80
Open
DilumAluthge-LLM wants to merge 1 commit into
Open
Conversation
DilumAluthge
changed the title
ETag and Last-Modified as optional fields
DilumAluthge
changed the title
ETag and Last-Modified as optional fieldsetag and last-modified as optional fields
Member
|
I have manually reviewed this code. |
DilumAluthge
marked this pull request as ready for review
May 1, 2026 04:11
Co-authored-by: OpenAI Codex [GPT-5] <codex@openai.com>
DilumAluthge
force-pushed
the
codex/allow-file-etag-last-modified
branch
from
August 11, 2026 20:29
ca3d80d to
eac96c8
Compare
KristofferC
added a commit
that referenced
this pull request
Aug 12, 2026
Ports the test/more_tests.jl change from #80. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
KristofferC
added a commit
that referenced
this pull request
Aug 14, 2026
* Incremental rebuilds seeded from the deployed versions.json Instead of re-downloading and re-hashing every Julia binary ever released (1.5-3.5h per run), seed from the currently deployed versions.json and only probe what is missing from it: an entry already in the seed is carried over after a cheap HEAD cross-check of its recorded size against Content-Length, so a normal run downloads only newly released files and finishes in minutes. The published versions.json format is completely unchanged and there is no state beyond the deployed file itself. Robustness rules, factored as pure tested functions: a non-200 HEAD for an entry already in the seed keeps the entry and warns instead of deleting it (the CDN is known to serve stale per-POP 404s); a size mismatch means the published file was replaced and forces a re-download; a missing or broken seed falls back to a full rebuild; the skiplisted corrupt tarball is exempt from the tree-hash completeness requirement so it is not re-downloaded every run. CI gains a full-rebuild workflow_dispatch input and a monthly scheduled full rebuild (which also deploys) as a backstop for anything the size check cannot detect, and Fastly purge failures now surface as warning annotations instead of passing silently. The seed-and-reuse design is due to Dilum Aluthge (#67); this is a minimal re-implementation of it on top of main. Co-authored-by: Dilum Aluthge <dilum@aluthge.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Do the HEAD checks concurrently in a prepass All candidate URLs are HEADed up front with asyncmap and the build loop reads from the resulting table. ntasks matches HTTP.jl's per-host connection limit: more tasks than connections just thrash the pool with TLS setup (measured slower). A failed request (DNS, connection reset) now yields status 0 and is treated as transient instead of killing the run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Set finite timeouts on the HEAD requests HTTP.jl's default read timeout is infinite, so a single black-holed connection (observed in the wild: Fastly tarpitting a rate-limited client by accepting the connection and never answering) hangs the whole build forever, at 0% CPU with no output. With finite timeouts the request fails, is warned about, and the transient-status policy skips or keeps the entry as appropriate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Make the HEAD checks sequential again The concurrency saved ~20s on a run with minutes of fixed overhead, and the CDN has been observed to tarpit clients that make concurrent bursts -- a rate-limited CI runner would cost hours, not seconds. Keep the finite timeouts and failed-request handling; drop the prepass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * CI: remove the monthly scheduled full rebuild A full rebuild's wall time is already close to GitHub's 6-hour limit for hosted runners, so a scheduled full rebuild is not sustainable. The manual full-rebuild workflow_dispatch input remains for when one is needed; the ETag/Last-Modified cross-checks now take over as the ongoing staleness guard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Cross-check seeded entries against ETag and Last-Modified too Implement the reuse cascade proposed by @DilumAluthge in #101: a seeded entry is compared field by field against the live HEAD response (size vs Content-Length, then etag vs ETag, then last-modified vs Last-Modified); any mismatch triggers a re-download, while a field absent from the entry ends the cascade with a reuse, so the new fields can be adopted incrementally. Newly downloaded entries record etag and last-modified from the HEAD response, and the schema allows the two new optional fields. Also use -1 instead of 0 as the sentinel for a missing Content-Length header, so a genuine zero-length response counts as a size mismatch rather than passing as unverifiable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Shorten comments Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Make the can't-check-this-header logic explicit Use nothing (uniformly with etag/last-modified) instead of a -1 sentinel for a missing Content-Length, and rewrite entry_matches_head as a field-by-field cascade where skipping an uncheckable header is spelled out, per review feedback in #101. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Validate seeded entries more strictly before carrying them over Blank required string fields and malformed git tree hashes now count as incomplete, so such entries get re-downloaded rather than carried over. Also make the .tar.gz check case-insensitive and note in entry_matches_head that filedict_is_complete has already required size. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Allow etag and last-modified in the post-build key check Ports the test/more_tests.jl change from #80. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * CI: diff the rebuilt versions.json against the deployed seed Makes it visible in the log what an incremental run actually changed. Both sides are key-sorted with jq since the Dict serialization order is not stable. Skipped when no seed was downloaded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Use Fastly API token for purging * Address review comments on CI comments Trim the jq-sort comment (JSON.jl v1 does sort plain Dict keys, so the old rationale was wrong) and drop the stale Fastly 401 comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review comments: check all recorded header fields, guard 'files', explicit :proceed - entry_matches_head now skips an absent etag/last-modified field and keeps checking the remaining ones instead of reusing immediately - find_filedict/delete_filedicts_for_url! tolerate a seeded version entry without a "files" key - the carry-over check in main() states action == :proceed explicitly Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Dilum Aluthge <dilum@aluthge.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Fredrik Ekre <ekrefredrik@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
etagandlast-modifiedfields inFileobjects inschema.json.test/more_tests.jl.This is preparation for #76.
Cross-ref: #51
🤖 Generated by OpenAI Codex.