Renv fix - #614
Open
MayaGans wants to merge 4 commits into
Open
Conversation
CI has failed at "Set up renv" on every branch since ~24 July 2026, including pushes straight to main. The trigger was GitHub migrating ubuntu-latest to a new image: setup-renv keys its package cache on sessionInfo()$running, so the migration invalidated every cache at once. That exposed a latent problem. renv.lock pointed at packagemanager.posit.co/cran/latest, which only ever serves the *current* version of each package, and 154 of the 374 pinned versions were no longer current. With no binary available, renv fell back to compiling old source tarballs from the CRAN archive, which fails on the new runner. A warm cache had hidden this for months. Changes: - Pin the repository to a dated snapshot (cran/2026-08-14) and re-snapshot so every recorded version matches that date. Verified that all 365 repository-sourced records resolve on macOS arm64, Windows and Linux, and that all 170 compiled packages are served as prebuilt binaries for both noble and resolute. CI now compiles nothing. - Stop recording the 15 packages that ship with R (mgcv, Matrix, survival, lattice, ...). Their versions track the R build rather than CRAN, so no snapshot date can ever satisfy them; this is also what broke local installs on macOS. utils/quarto_check_pkg_dependencies.R already handles these having no hash. - Add survRM2 1.0-4 and its dependency muhaz, to unblock PSIAIMS#598. - Drop survMisc: CRAN archived it on 17 March 2026, it is no longer a survminer dependency, and it is only mentioned in prose. km.ci, KMsurv, plyr, reshape2 and rgl go with it as transitive dependencies that current versions no longer need. - Pin runs-on to ubuntu-24.04 in both workflows so a future image migration cannot silently invalidate the caches again. Note: 138 package versions change, so the Quarto freeze cache will be largely invalidated and most pages will re-execute on the first run.
Two notes in the "Setting up CAMIS with renv" section: - Check the `renv::snapshot()` diff before committing. Snapshot records the local environment, so it rewrites requirements.txt and the Python block of renv.lock from whichever Python environment is active, and sets the recorded R version to whatever R the contributor is running. An empty local venv has previously cut requirements.txt from 124 packages to one. - The package repository is pinned to a dated PPM snapshot, so installs get that date's version and packages published after it are unavailable until the date is moved forward. Explains why the pin exists (binaries on all three platforms) and why R-bundled packages are excluded via ignored.packages.
The first CI run on the new lockfile installed every package as a binary and compiled nothing, then failed loading igraph: libglpk.so.40: cannot open shared object file vcdExtra went 0.9.1 -> 0.9.7 in this update and the newer version imports igraph, which links against GLPK. GLPK is not on the runner image, and the workflows only installed libcurl, udunits2, GDAL, GLU and Xi. Because the restore aborts on the first failure, nothing after igraph had been attempted yet, so fixing GLPK alone would likely just surface the next missing library. Instead this asks Posit Package Manager's sysreqs API what the whole 366-package set requires on ubuntu-24.04 and installs that, which is 27 apt packages. The step now carries the URL to regenerate the list. libglu1-mesa and libxi-dev are kept even though rgl (the reason they were there) is no longer in the lockfile, since they cost nothing and nothing confirms they are unused.
The lockfile refresh changes rmarkdown 2.30 -> 2.31, which 180 of the 195 pages depend on, so the pre-render script invalidates nearly the whole freeze cache and the last run spent 55.6 minutes re-executing it. That cost repeats on every run, because the render happens in a throwaway checkout and nothing is committed back. Rather than reproduce the render environment locally, capture it from the run that already does it successfully: this uploads _freeze and data/quarto_pkg_dependencies.csv so they can be downloaded and committed. Once committed, the script finds no hash changes and builds return to ~12 minutes. Remove this step before merge.
Collaborator
|
Hi @MayaGans thanks so much for contributing this PR - I've approved the GitHub actions workflow and it's run successfully which is great to see! I also wanted to clarify next steps, please could you confirm whether my understanding below is correct and if you're happy to make these updates?
Thanks so much again! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's broken
Every CI run since about 24 July has failed at the "Set up renv" step
Why
run just reused it so the "Set up renv" step took only 1.5 minutes. Then GitHub swapped
ubuntu-latestto a newer Ubuntu image in mid-July.What this PR does
mgcv,Matrix,survival,latticeand friends). Their versions follow your R installation rather than CRAN, so they can never be pinned reliably.The cost, and how we're handling it
This is an expensive change and I want to be upfront about it.
A normal PR used to take about 12 minutes: 1.5 to restore packages and 8-9 to render. The run on this branch took 65 minutes, 56 of which was rendering. Left alone this repeats forever, because the render happens in a throwaway checkout and nothing gets committed back. Every future PR would recompute the same difference and pay the same 56 minutes. So rather than leave it, this branch temporarily uploads the regenerated
_freezeanddata/quarto_pkg_dependencies.csvas an artifact, so they can be downloaded and committed. Once they're in, the pre-render script sees no change, deletes nothing, and builds go back to ~12 minutes for everyone. The temporaryupload step gets removed before merge.
So this PR is slow, but its the cost of re-executing everything once. Committing the result means the next PR reuses it, and builds go back to roughly 12 minutes for everyone.
Two caveats so nobody's surprised later:
Note on adding packages from now on: you get the version that was current on 14 August. If you need something newer we move the date forward, which is worth doing on its own rather than tacked onto a content PR.
Keep an eye on
Someone needs to move the snapshot date forward every so often. Being pinned is the point, it's what makes restores reproducible, but it does mean the gap between us and current CRAN grows quietly in the background.
I've added a note to the contributor guide for now, but doing snapshots in CI could remove the trap altogether!
One practical note
Because this comes from a fork, GitHub won't run the checks until a maintainer approves them, so the PR may look untested until someone clicks through!