Skip to content

Generate nightlies.json: the nightly channels and their (variant) files - #107

Open
maleadt wants to merge 2 commits into
JuliaLang:mainfrom
maleadt:nightlies.json
Open

Generate nightlies.json: the nightly channels and their (variant) files#107
maleadt wants to merge 2 commits into
JuliaLang:mainfrom
maleadt:nightlies.json

Conversation

@maleadt

@maleadt maleadt commented Aug 21, 2026

Copy link
Copy Markdown
Member

versions.json describes immutable release artifacts, so the in-development builds have no machine-readable listing: juliaup, setup-julia and the website each hardcode the julia-latest-* URL layout, and a new platform or build variant has to be added to every one of them.

Add VersionsJSONUtil.nightlies, which writes nightlies.json next to versions.json: the nightly and x.y-nightly channels (in juliaup's vocabulary) with, per channel, the files that currently exist, described with the versions.json keys minus everything content-specific (version, size, hashes; the signature is referenced by asc-url), since every URL is overwritten after each build. Build variants published under julia-buildkite's bin/<os><variant>/ layout (currently the PGO+LTO+BOLT opt build and the assert build) are listed next to the standard build with a variants key, which is the shape intended for variants of releases in versions.json later.

This is one step towards JuliaLang/juliaup#1419, so cc @topolarity. I decided against changing the existing versions.json because it has a fixed schema that requires size/hashes, and this is a good way to experiment with encoding variants before rolling this out to the main database once we have releases with variants.

The generated JSON (abbreviated):

{
  "nightly": {
    "files": [
      { "arch": "x86_64", "os": "linux", "triplet": "x86_64-linux-gnu", "kind": "archive", "extension": "tar.gz",
        "url": "https://julialangnightlies-s3.julialang.org/bin/linux/x86_64/julia-latest-linux-x86_64.tar.gz",
        "asc-url": "https://julialangnightlies-s3.julialang.org/bin/linux/x86_64/julia-latest-linux-x86_64.tar.gz.asc" },
      { "arch": "i686",    "os": "linux", "triplet": "i686-linux-gnu",    "kind": "archive", "extension": "tar.gz",
        "url": ".../bin/linux/i686/julia-latest-linux-i686.tar.gz", "asc-url": ".../julia-latest-linux-i686.tar.gz.asc" },
      { "arch": "aarch64", "os": "linux", "triplet": "aarch64-linux-gnu", "kind": "archive", "extension": "tar.gz",
        "url": ".../bin/linux/aarch64/julia-latest-linux-aarch64.tar.gz", "asc-url": ".../julia-latest-linux-aarch64.tar.gz.asc" },

      { "arch": "x86_64",  "os": "mac", "triplet": "x86_64-apple-darwin14",  "kind": "archive", "extension": "dmg",
        "url": ".../bin/macos/x86_64/julia-latest-macos-x86_64.dmg" },
      { "arch": "aarch64", "os": "mac", "triplet": "aarch64-apple-darwin14", "kind": "archive", "extension": "dmg",
        "url": ".../bin/macos/aarch64/julia-latest-macos-aarch64.dmg" },
      { "arch": "x86_64",  "os": "mac", "triplet": "x86_64-apple-darwin14",  "kind": "archive", "extension": "tar.gz",
        "url": ".../bin/macos/x86_64/julia-latest-macos-x86_64.tar.gz", "asc-url": ".../julia-latest-macos-x86_64.tar.gz.asc" },
      { "arch": "aarch64", "os": "mac", "triplet": "aarch64-apple-darwin14", "kind": "archive", "extension": "tar.gz",
        "url": ".../bin/macos/aarch64/julia-latest-macos-aarch64.tar.gz", "asc-url": ".../julia-latest-macos-aarch64.tar.gz.asc" },

      { "arch": "x86_64", "os": "winnt", "triplet": "x86_64-w64-mingw32", "kind": "installer", "extension": "exe",
        "url": ".../bin/winnt/x64/julia-latest-win64.exe" },
      { "arch": "i686",   "os": "winnt", "triplet": "i686-w64-mingw32",   "kind": "installer", "extension": "exe",
        "url": ".../bin/winnt/x86/julia-latest-win32.exe" },
      { "arch": "x86_64", "os": "winnt", "triplet": "x86_64-w64-mingw32", "kind": "archive", "extension": "zip",
        "url": ".../bin/winnt/x64/julia-latest-win64.zip" },
      { "arch": "i686",   "os": "winnt", "triplet": "i686-w64-mingw32",   "kind": "archive", "extension": "zip",
        "url": ".../bin/winnt/x86/julia-latest-win32.zip" },
      { "arch": "x86_64", "os": "winnt", "triplet": "x86_64-w64-mingw32", "kind": "archive", "extension": "tar.gz",
        "url": ".../bin/winnt/x64/julia-latest-win64.tar.gz", "asc-url": ".../julia-latest-win64.tar.gz.asc" },
      { "arch": "i686",   "os": "winnt", "triplet": "i686-w64-mingw32",   "kind": "archive", "extension": "tar.gz",
        "url": ".../bin/winnt/x86/julia-latest-win32.tar.gz", "asc-url": ".../julia-latest-win32.tar.gz.asc" },

      { "arch": "x86_64", "os": "freebsd", "triplet": "x86_64-unknown-freebsd11.1", "kind": "archive", "extension": "tar.gz",
        "url": ".../bin/freebsd/x86_64/julia-latest-freebsd-x86_64.tar.gz", "asc-url": ".../julia-latest-freebsd-x86_64.tar.gz.asc" },

      { "arch": "x86_64", "os": "linux", "triplet": "x86_64-linux-gnu", "kind": "archive", "extension": "tar.gz",
        "url": "https://julialangnightlies-s3.julialang.org/bin/linuxopt/x86_64/julia-latest-linuxopt-x86_64.tar.gz",
        "asc-url": "https://julialangnightlies-s3.julialang.org/bin/linuxopt/x86_64/julia-latest-linuxopt-x86_64.tar.gz.asc",
        "variants": ["opt"] },
      { "arch": "x86_64", "os": "linux", "triplet": "x86_64-linux-gnu", "kind": "archive", "extension": "tar.gz",
        "url": "https://julialangnightlies-s3.julialang.org/bin/linuxassert/x86_64/julia-latest-linuxassert-x86_64.tar.gz",
        "asc-url": "https://julialangnightlies-s3.julialang.org/bin/linuxassert/x86_64/julia-latest-linuxassert-x86_64.tar.gz.asc",
        "variants": ["assert"] }
    ]
  },
  "1.14-nightly": { "files": [ /* same 14 standard files under bin/<os>/<arch>/1.14/..., plus the opt and assert variants */ ] },
  "1.13-nightly": { "files": [ /* 14 standard files under .../1.13/ — no variants (opt/assert are only built on master) */ ] },
  "1.12-nightly": { "files": [ /* 14 standard files under .../1.12/ */ ] },
  "1.10-nightly": { "files": [ /* 14 standard files under .../1.10/ */ ] }
}

@maleadt @claude
Generate nightlies.json: the nightly channels and their (variant) files
cfb402b 
versions.json describes immutable release artifacts, so the in-development
builds have no machine-readable listing: juliaup, setup-julia and the website
each hardcode the `julia-latest-*` URL layout, and a new platform or build
variant has to be added to every one of them.

Add `VersionsJSONUtil.nightlies`, which writes nightlies.json next to
versions.json: the `nightly` and `x.y-nightly` channels (in juliaup's
vocabulary) with, per channel, the files that currently exist, described with
the versions.json keys minus everything content-specific (version, size,
hashes; the signature is referenced by `asc-url`), since every URL is
overwritten after each build. Build variants published under julia-buildkite's
`bin/<os><variant>/` layout (currently the PGO+LTO+BOLT `opt` build and the
`assert` build) are listed next to the standard build with a `variants` key,
which is the shape intended for variants of releases in versions.json later.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@maleadt

maleadt commented Aug 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

@topolarity This does not list the nogpl builds, since those are uploaded to their own bucket. Do you know why that is? Wouldn't it be cleaner to have a nogpl variant suffix and otherwise keep using the existing buckets (just like I do for the new opt build)?

Also, the nogpl bucket doesn't appear to have a lifetime configured, so every nogpl nightly since 2022-11 is still there (1000+ objects under linuxnogpl/x86_64 alone).

@DilumAluthge

DilumAluthge commented Aug 22, 2026

Copy link
Copy Markdown
Member

My plan is to add nogpl (and other variants) to versions.json in a non-breaking way. My current idea is to preserve files the way it is, and have a separate variants list, with every entry in variants having a variant field.

@maleadt @claude
nightlies.json: separate variants list, nogpl builds, daily refresh
6ed910a 
Follow the shape proposed for variants of releases in versions.json: each
channel has `files` with the standard builds only, plus a sibling `variants`
list (absent when empty) whose entries carry the same keys and a singular
`variant` name. Consumers picking a file from `files` by platform thus never
see a variant by accident, and the same model serves both files.

Variants now map to a base URL, which adds the `nogpl` build from the
julialang-nogpl bucket next to `opt` and `assert`.

Run the workflow daily to refresh nightlies.json: the set of channels changes
with release branches, not releases. Scheduled runs skip the full
versions.json test and only deploy nightlies.json.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@maleadt

maleadt commented Sep 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

OK, I've switched to your proposed design. nightlies.json remains valuable though, since these entries are mutable and without version/hash, so they don't fit versions.json regardless of where variants end up.

  • Each channel now has files (standard builds only) and a sibling variants list, absent when the channel has none. Variant entries have the same keys as files plus a variant string. The old array-valued variants key on file entries is gone, so consumers selecting a file from files by triplet never see a variant by accident.
  • nogpl is now included alongside opt and assert. Variants map to a base URL, so the nogpl entries point at julialang-nogpl.s3.amazonaws.com/bin-nogpl/. This also means juliaup won't need to hardcode that host (or wait for the DNS alias mentioned in Fix no_GPL Windows uploads going to wrong S3 bucket JuliaCI/julia-buildkite#512).
  • The schema gained a VariantFile definition, and the post-build test asserts that only nogpl entries point at the nogpl bucket.
  • The workflow now also runs daily and, on scheduled runs, only rebuilds and deploys nightlies.json (the full versions.json test is skipped).

Generated output today: five channels (nightly, 1.14-nightly, 1.13-nightly, 1.12-nightly, 1.10-nightly), each with the 14 standard files. Master and 1.14 list opt, assert and nogpl; 1.13 and 1.12 list nogpl for Linux x86_64, macOS x86_64/aarch64 and Windows x86_64; 1.10 lists nogpl without Windows.

"nightly": {
  "files":    [ { "triplet": "x86_64-linux-gnu", "os": "linux", "arch": "x86_64", "kind": "archive", "extension": "tar.gz",
                  "url": "https://julialangnightlies-s3.julialang.org/bin/linux/x86_64/julia-latest-linux-x86_64.tar.gz",
                  "asc-url": "https://julialangnightlies-s3.julialang.org/bin/linux/x86_64/julia-latest-linux-x86_64.tar.gz.asc" }, ... ],
  "variants": [ { "variant": "opt",   "triplet": "x86_64-linux-gnu", ..., "url": "https://julialangnightlies-s3.julialang.org/bin/linuxopt/x86_64/julia-latest-linuxopt-x86_64.tar.gz" },
                { "variant": "nogpl", "triplet": "x86_64-linux-gnu", ..., "url": "https://julialang-nogpl.s3.amazonaws.com/bin-nogpl/linuxnogpl/x86_64/julia-latest-linuxnogpl-x86_64.tar.gz" }, ... ]
}

I went with a single variant string rather than a list since julia-buildkite only ever appends one suffix to the OS name.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@maleadt @DilumAluthge