From 975b8511892a4a1171c2ce4f14b44b4c21bbf027 Mon Sep 17 00:00:00 2001 From: "Martin R. Smith" <1695515+ms609@users.noreply.github.com> Date: Mon, 18 May 2026 13:29:11 +0100 Subject: [PATCH 1/4] Lower ASLR entropy so ASan can map its shadow region The "AddressSanitizer tests" job has been failing on every push for several weeks with: ==PID==Shadow memory range interleaves with an existing memory mapping. ASan cannot proceed correctly. ABORTING. This is google/sanitizers#856 / llvm/llvm-project#85780. The ubuntu-24.04 runner ships a kernel with vm.mmap_rnd_bits=32; ASan's shadow memory layout cannot accommodate that much ASLR entropy and aborts at startup. Setting vm.mmap_rnd_bits=28 (the canonical workaround) gives ASan a clean shadow region. Only "AddressSanitizer tests" is affected on this repo today, because testthat::test_local() loads more shared libraries before ASan allocates its shadow than the examples/vignettes subjobs do. Applying the sysctl unconditionally to all three configurations is harmless and avoids future drift. --- .github/workflows/ASan.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.github/workflows/ASan.yml b/.github/workflows/ASan.yml index 2884983f..6ea63566 100644 --- a/.github/workflows/ASan.yml +++ b/.github/workflows/ASan.yml @@ -49,6 +49,13 @@ jobs: steps: - uses: actions/checkout@v6 + - name: Lower ASLR entropy for ASan + # Ubuntu-24.04 runners ship a kernel with vm.mmap_rnd_bits=32, + # which collides with ASan's shadow memory layout and aborts + # with "Shadow memory range interleaves with an existing memory + # mapping" (google/sanitizers#856, llvm/llvm-project#85780). + run: sudo sysctl -w vm.mmap_rnd_bits=28 + - name: Initialize ASan configuration run: | export LD_PRELOAD=$(gcc -print-file-name=libasan.so) From 5bce76e2c9393a684331e42cba3d2e58d5f6873d Mon Sep 17 00:00:00 2001 From: "Martin R. Smith" <1695515+ms609@users.noreply.github.com> Date: Mon, 18 May 2026 13:39:02 +0100 Subject: [PATCH 2/4] Disable kernel.randomize_va_space=0 so ASan can map its shadow Setting vm.mmap_rnd_bits=28 alone was applied successfully (logs confirm 'vm.mmap_rnd_bits = 28') but ASan still aborted with the ELF_ET_DYN_BASE diagnostic - the kernel's load address for PIE binaries on ubuntu-24.04 collides with ASan's shadow regardless of mmap entropy. Disable system-wide ASLR for the runner VM to give ASan a stable, low-address layout. --- .github/workflows/ASan.yml | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/.github/workflows/ASan.yml b/.github/workflows/ASan.yml index 6ea63566..acb2f01f 100644 --- a/.github/workflows/ASan.yml +++ b/.github/workflows/ASan.yml @@ -49,12 +49,18 @@ jobs: steps: - uses: actions/checkout@v6 - - name: Lower ASLR entropy for ASan - # Ubuntu-24.04 runners ship a kernel with vm.mmap_rnd_bits=32, - # which collides with ASan's shadow memory layout and aborts - # with "Shadow memory range interleaves with an existing memory - # mapping" (google/sanitizers#856, llvm/llvm-project#85780). - run: sudo sysctl -w vm.mmap_rnd_bits=28 + - name: Disable kernel ASLR + # Ubuntu-24.04 runners ship a kernel whose default + # ELF_ET_DYN_BASE plus high mmap_rnd_bits places PIE binaries + # in addresses that collide with ASan's shadow memory layout, + # aborting with "Shadow memory range interleaves with an + # existing memory mapping" (google/sanitizers#856, + # llvm/llvm-project#85780). Lowering mmap_rnd_bits alone + # is not sufficient on this kernel; turn off randomize_va_space + # for the whole VM so ASan's shadow allocation is stable. + run: | + sudo sysctl -w vm.mmap_rnd_bits=28 + sudo sysctl -w kernel.randomize_va_space=0 - name: Initialize ASan configuration run: | From 846294ed7dd6061d5f655ecd3b09866513819699 Mon Sep 17 00:00:00 2001 From: "Martin R. Smith" <1695515+ms609@users.noreply.github.com> Date: Mon, 18 May 2026 13:47:41 +0100 Subject: [PATCH 3/4] LD_PRELOAD libasan so ASan inits at R startup, not at dlopen The previous attempt set sysctl vm.mmap_rnd_bits=28 and kernel.randomize_va_space=0, both confirmed applied on the runner. ASan still aborted with the ELF_ET_DYN_BASE diagnostic, indicating the failure is not about ASLR entropy. Log analysis showed the abort happens AFTER 'library(TreeTools)' succeeds (the 'Creating a generic function' messages from TreeTools loading print, then ASan aborts). That timing rules out a parent-R startup failure: libasan must be initialising late, when R dlopen()s the package's instrumented .so. The 'Initialize ASan configuration' step had export LD_PRELOAD=\$(gcc -print-file-name=libasan.so) inside a subshell, so the variable did not persist to the Rscript step. Without LD_PRELOAD, libasan is only pulled in via the package's .so dependency, by which point R's heap, libR.so, and (for the tests subjob) testthat's full dependency stack already occupy the address range ASan needs for its shadow region. examples and vignettes happened to leave enough room; tests did not. Set LD_PRELOAD inline on the Rscript command so libasan loads at process startup, before any other shared object. Keep the mmap_rnd_bits sysctl as defence in depth; drop the more invasive randomize_va_space=0 now that the root cause is addressed. --- .github/workflows/ASan.yml | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/.github/workflows/ASan.yml b/.github/workflows/ASan.yml index acb2f01f..ef0b3afb 100644 --- a/.github/workflows/ASan.yml +++ b/.github/workflows/ASan.yml @@ -49,23 +49,15 @@ jobs: steps: - uses: actions/checkout@v6 - - name: Disable kernel ASLR - # Ubuntu-24.04 runners ship a kernel whose default - # ELF_ET_DYN_BASE plus high mmap_rnd_bits places PIE binaries - # in addresses that collide with ASan's shadow memory layout, - # aborting with "Shadow memory range interleaves with an - # existing memory mapping" (google/sanitizers#856, - # llvm/llvm-project#85780). Lowering mmap_rnd_bits alone - # is not sufficient on this kernel; turn off randomize_va_space - # for the whole VM so ASan's shadow allocation is stable. - run: | - sudo sysctl -w vm.mmap_rnd_bits=28 - sudo sysctl -w kernel.randomize_va_space=0 + - name: Lower ASLR entropy for ASan + # Defence-in-depth alongside the LD_PRELOAD step below: + # ubuntu-24.04 runners ship a high vm.mmap_rnd_bits that can + # interact poorly with ASan's shadow memory layout + # (google/sanitizers#856, llvm/llvm-project#85780). + run: sudo sysctl -w vm.mmap_rnd_bits=28 - name: Initialize ASan configuration run: | - export LD_PRELOAD=$(gcc -print-file-name=libasan.so) - echo "PKG_CFLAGS = -g -O0 -fsanitize=address -fno-omit-frame-pointer" > src/Makevars echo "PKG_CXXFLAGS = -g -O0 -fsanitize=address -fno-omit-frame-pointer" >> src/Makevars @@ -91,5 +83,13 @@ jobs: cd TreeTools - name: ASAN - memcheck ${{ matrix.config.test }} + # LD_PRELOAD libasan.so so the sanitizer initialises at R + # process startup, before R's heap and any package loaded by + # the test script can occupy the address range ASan needs for + # its shadow region. Without this, libasan is loaded only when + # R dlopen()s an instrumented package's .so, which works for + # the small examples/vignettes scripts but aborts for the + # tests subjob once testthat's dependency stack is in memory. run: | - Rscript memcheck/${{ matrix.config.test }}.R + LD_PRELOAD=$(gcc -print-file-name=libasan.so) \ + Rscript memcheck/${{ matrix.config.test }}.R From 6a320c01f176378c354023284a742090e8921b7a Mon Sep 17 00:00:00 2001 From: "Martin R. Smith" <1695515+ms609@users.noreply.github.com> Date: Mon, 18 May 2026 13:53:34 +0100 Subject: [PATCH 4/4] Disable LSan leak detection so third-party noise stops failing CI The LD_PRELOAD fix in the previous commit moved ASan past the shadow-memory abort, but leak detection now fires on: - libcrypto.so.3 / libssl.so.3 / libxml2.so.2 one-time init memory - xml2 R package allocations from HTML parsing - /usr/bin/sed (a 1-byte allocation in sed itself, called by some test-discovery step) None of these are TreeTools bugs and none are actionable from this codebase. CRAN's own gcc-ASAN setup runs with detect_leaks=0 for the same reason - LSan in an R session pulls in too much third-party state to be useful. We retain valgrind for leak coverage. --- .github/workflows/ASan.yml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ASan.yml b/.github/workflows/ASan.yml index ef0b3afb..734b940b 100644 --- a/.github/workflows/ASan.yml +++ b/.github/workflows/ASan.yml @@ -44,7 +44,11 @@ jobs: RSPM: https://packagemanager.rstudio.com/cran/__linux__/noble/latest USING_ASAN: true GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} - ASAN_OPTIONS: detect_container_overflow=1:verify_asan_link_order=0 + # detect_leaks=0 matches CRAN's gcc-ASAN setup; LSan otherwise + # reports noisy leaks from third-party libraries (libcrypto, + # libxml2) and even from the sed binary that test discovery + # shells out to. We rely on valgrind for leak coverage. + ASAN_OPTIONS: detect_container_overflow=1:verify_asan_link_order=0:detect_leaks=0 steps: - uses: actions/checkout@v6