Failing build with `-Dasan=ON` and memory leak in minimal build

[AxelKrypton]

Check duplicate issues.

• Checked for duplicates

Description

As requested by @Axel-Naumann in #13130 this is a follow up of my note there.

Basically I recently built and linked to ROOT in an external project (in a Docker image) and I got all unit tests failing because of a memory leak detected by the address sanitiser. As ROOT was set up using CMake via

cmake -DCMAKE_INSTALL_PREFIX=../root_install ../root_src -Dxrootd=OFF

the backtrace reported in my previous note was pretty small. Hence, I tried to compile ROOT as

cmake -DCMAKE_INSTALL_PREFIX=../root_install ../root_src -Dxrootd=OFF -Dasan=ON

but this failed with this output.

Then I tried to make a minimal build via

cmake -DCMAKE_INSTALL_PREFIX=../root_install ../root_src -Dxrootd=OFF -Dasan=ON -Dminimal=ON

and this succeeded and I got my Docker image built. Inside it the tests of my application were still failing as expected, but this time the backtrace was way larger. In this file you find it all.

Reproducer

As the build fails because of a leak, I am pretty much expecting that the two issues are related and I did not invest more time in building up a MWE to reproduce the second issue. However, I would naively guess that linking to the build library in any executable with the address sanitiser on would make it.

ROOT version

In the Docker container in the minimal build ROOT binary folder, I get:

root@fc0d5912f35b:/SMASH/root_install/bin# ./root.exe -b -q
   ------------------------------------------------------------------
  | Welcome to ROOT 6.28/06                        https://root.cern |
  | (c) 1995-2023, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxx8664gcc on Aug 28 2023, 11:29:15                 |
  | From tags/v6-28-06@v6-28-06                                      |
  | With c++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0                   |
  | Try '.help'/'.?', '.demo', '.license', '.credits', '.quit'/'.q'  |
   ------------------------------------------------------------------

Installation method

Build from source with CMake version 3.22.1

Operating system

Ubuntu 22.04.3 LTS

Additional context

The same problem occurred with previous ROOT v6.28 versions, whereas switching back to v6.26.10 made the problem disappear.

[hahnjo]

see also #7968; can you try compiling with Clang?

[AxelKrypton]

Indeed, compiling with Clang after having setting up via (slightly different from above as I used a different docker image)

cmake -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=../root_install ../root_src -Dxrootd=OFF -Dasan=ON -Dbuiltin_afterimage=ON

did succeed. Here the ROOT version from its executable:

root@4bdbf4041f81:/SMASH/root_install/bin# ./root -b -q
   ------------------------------------------------------------------
  | Welcome to ROOT 6.28/06                        https://root.cern |
  | (c) 1995-2023, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxx8664gcc on Aug 28 2023, 11:29:15                 |
  | From tags/v6-28-06@v6-28-06                                      |
  | With Ubuntu clang version 13.0.1-2ubuntu2.2                      |
  | Try '.help'/'.?', '.demo', '.license', '.credits', '.quit'/'.q'  |
   ------------------------------------------------------------------

However, the tests of my application still fail with a very similar backtrace. So in my opinion the issue still exists. Feel free to rename the issue title, if you think it is appropriate.

[hahnjo]

@AxelKrypton I'm sorry, I didn't read your initial report closely enough: what you see is a leak report that we turn off due to how ROOT uses the LLVM and Clang libraries:

root/core/sanitizer/SanitizerSetup.cxx

Line 22 in 479d383

#define DETECT_LEAKS ":detect_leaks=0"

I don't remember the details, but it's possible that the method we're using to suppress the leak sanitizer doesn't work with GCC and that's why switching to Clang solved the issue during build. For your own application, you need to pull in that same code, which should happen automatically if you use ROOT_EXECUTABLE. Can you check how your application is created?

If it's not possible to use ROOT_EXECUTABLE, you can also manually link in ROOT::ROOTStaticSanitizerConfig. Alternatively you can try setting the environment variable ASAN_OPTIONS="detect_leaks=0". Hope this helps, let me know if you have further problems!

[AxelKrypton]

🤔 Just to understand, is it a known real leak or a false positive?

I initially thought it was something we in any case could not fix on our side and I also suppressed the sanitiser leak as described here (not sure this is an official documentation, but it's something google brought to my attention). In particular I added

cd root_install/etc/
printf '\n#Full libCling for SMASH\nleak:libCling\n' >> lsan-root.supp

together with

# This will let sanitizer suppress leaks according to suppressions file list of patterns
ENV LSAN_OPTIONS="suppressions=${ROOTSYS}/etc/lsan-root.supp:print_suppressions=0"

to my Dockerfile.

Doing so, all the unit tests pass, but I then monitored memory usage over a long run of our application and a linear growth was evident (plot left below). Nothing extreme, sure, but I thought it was worth reporting, especially since in v6.26.10 the problem does not occur (plot right below). 🤷‍♂️

[hahnjo]

🤔 Just to understand, is it a known real leak or a false positive?

It is a "real" leak that in my understanding we are not urgently planning to fix. Basically it comes from Clang's internal data structures that are "lost" during shutdown. With clang used as a compiler, the libraries make quite some effort to clean up all data structures and free memory allocations to make sanitizers clean. I think this mode was never implemented in Cling, also because it's not really relevant: after process termination, the OS will clean up all virtual memory anyway.

I initially thought it was something we in any case could not fix on our side and I also suppressed the sanitiser leak as described here (not sure this is an official documentation, but it's something google brought to my attention).

Yes, the upstream equivalent to that GitLab instance would be https://github.com/root-project/root/tree/master/core/sanitizer

Doing so, all the unit tests pass, but I then monitored memory usage over a long run of our application and a linear growth was evident (plot left below). Nothing extreme, sure, but I thought it was worth reporting, especially since in v6.26.10 the problem does not occur (plot right below). 🤷‍♂️

Ok, good that all unit tests pass, so you're not using uninitialized memory 👍 for the memory leak, do you have a semi-standalone program that shows the growing memory usage on 6.28, but not on 6.26? Does it involve TChains, as reported in #13130?

[AxelKrypton]

Thanks for the insight.

for the memory leak, do you have a semi-standalone program that shows the growing memory usage on 6.28, but not on 6.26?

Not really.

Does it involve TChains, as reported in #13130?

It might indirectly. I never dealt with the part of our application interacting with ROOT but after a quick look, we mostly directly use TTree. However, we link to and use HepMC, which makes heavier use of ROOT.

Now, I just realised that many (if not most) of our unit tests do not use ROOT, but only link to its libraries. In particular, I tried to add a trivial unit test file

#include "vir/test.h"  // This include has to be first

TEST(){ std::cout << "Hello world\n"; }

and compiling as the other ones, i.e. linking it to

$ ldd trivial_tests | grep -i 'root'
	libGui.so => /SMASH/root_install/lib/libGui.so (0x00007f21967ee000)
	libCore.so => /SMASH/root_install/lib/libCore.so (0x00007f2195e98000)
	libImt.so => /SMASH/root_install/lib/libImt.so (0x00007f2195e74000)
	libRIO.so => /SMASH/root_install/lib/libRIO.so (0x00007f219541f000)
	libNet.so => /SMASH/root_install/lib/libNet.so (0x00007f2195280000)
	libHist.so => /SMASH/root_install/lib/libHist.so (0x00007f21949cb000)
	libGraf.so => /SMASH/root_install/lib/libGraf.so (0x00007f21946ea000)
	libGraf3d.so => /SMASH/root_install/lib/libGraf3d.so (0x00007f21945ab000)
	libGpad.so => /SMASH/root_install/lib/libGpad.so (0x00007f2194408000)
	libROOTVecOps.so => /SMASH/root_install/lib/libROOTVecOps.so (0x00007f21941b1000)
	libTree.so => /SMASH/root_install/lib/libTree.so (0x00007f2193e65000)
	libTreePlayer.so => /SMASH/root_install/lib/libTreePlayer.so (0x00007f2193b41000)
	libRint.so => /SMASH/root_install/lib/libRint.so (0x00007f2193ae4000)
	libPostscript.so => /SMASH/root_install/lib/libPostscript.so (0x00007f2193953000)
	libMatrix.so => /SMASH/root_install/lib/libMatrix.so (0x00007f21935d5000)
	libPhysics.so => /SMASH/root_install/lib/libPhysics.so (0x00007f2193539000)
	libMathCore.so => /SMASH/root_install/lib/libMathCore.so (0x00007f2193113000)
	libThread.so => /SMASH/root_install/lib/libThread.so (0x00007f2193072000)
	libMultiProc.so => /SMASH/root_install/lib/libMultiProc.so (0x00007f219305b000)
	libROOTDataFrame.so => /SMASH/root_install/lib/libROOTDataFrame.so (0x00007f2192dd3000)
	libHepMC3rootIO.so.3 => /SMASH/hepmc3_install/lib/libHepMC3rootIO.so.3 (0x00007f2192bb0000)
	libtbb.so.2 => /SMASH/root_install/lib/libtbb.so.2 (0x00007f2184ecd000)
	libvdt.so => /SMASH/root_install/lib/libvdt.so (0x00007f2184910000)
	libGenVector.so => /SMASH/root_install/lib/libGenVector.so (0x00007f21847b9000)
	libROOTNTuple.so => /SMASH/root_install/lib/libROOTNTuple.so (0x00007f218455a000)

makes the sanitiser running the test still complain as above. 🤔 I guess I could create a standalone hello-world C++ code and compile it with the sanitiser linking to ROOT as our unit test does and I should get the same report. Let me know if you need so. Now, this behaviour (leaking just linking) should make it not too difficult to locate the issue. Do you do something with globals or static variables? Or do you have a _init and/or _fini function?

By the way, I think we might reduce which ROOT libraries we link against and we have an issue about this, although it is not a urgent aspect at the moment.

[hahnjo]

In particular, I tried to add a trivial unit test file [...] and compiling as the other ones, i.e. linking it to [...] makes the sanitiser running the test still complain as above.

Depending on how you compile it, you might get -fsanitize=address from the ROOT C++ flags. But in general yes, the libraries are sanitized and will operate no matter if your application itself is built with sanitizers or not.

I guess I could create a standalone hello-world C++ code and compile it with the sanitiser linking to ROOT as our unit test does and I should get the same report. Let me know if you need so.

That would be as expected, so I don't think it's worth the effort.

Now, this behaviour (leaking just linking) should make it not too difficult to locate the issue. Do you do something with globals or static variables? Or do you have a _init and/or _fini function?

As explained before, what the AddressSanitizer is complaining about is well understood and it also doesn't explain growing memory usage when not interacting with ROOT. This latter is what needs a clear reproducer to dig out the problem.

[AxelKrypton]

This latter is what needs a clear reproducer to dig out the problem.

I agree. This is actually what I was meaning by making an hello-world program that does not use ROOT, compile it linking to ROOT and with sanitiser on, and see if the same leak report is given. I would expect so and, if not, then there is something I definitely do not get in our application. 🙈

[ferdymercury]

Related PR #15803

[pcanal]

Doing so, all the unit tests pass, but I then monitored memory usage over a long run of our application and a linear growth was evident (plot left below). Nothing extreme, sure, but I thought it was worth reporting, especially since in v6.26.10 the problem does not occur (plot right below).

Do you see the same growth without ASAN?

[hahnjo]

@AxelKrypton any updates on this? Looking over the past discussion, we are aware of the (intentional) memory leaks in libCling (even though we fixed one case in #16150 that was a mistake; the fix will appear in 6.34). I still have trouble understanding how just linking to ROOT and not using it should introduce a memory leak in the application... Lacking more feedback, I would propose to close this issue for the moment...

[AxelKrypton]

I could not find time to provide you with more information. You will surely understand that it is not a trivial check, as I need to build a container compiling ROOT as you suggest and then build and run my application. However, IMO, you also have all the ingredients to give it a try. Following this note above, you should be able to reproduce the problem. There I wrote

I guess I could create a standalone hello-world C++ code and compile it with the sanitiser linking to ROOT as our unit test does and I should get the same report.

did you try? 🤔

It is on my to-do list to come back to this at some point, but I have not really found time yet. I plan to try to use more recent ROOT versions and check if the problem disappear. Especially as you mentioned it, 6.34 is a good first candidate. I can report here the outcome. As side remark, closing the issue does not feel useful. 🫠

[hahnjo]

However, IMO, you also have all the ingredients to give it a try. Following this note above, you should be able to reproduce the problem. There I wrote

I guess I could create a standalone hello-world C++ code and compile it with the sanitiser linking to ROOT as our unit test does and I should get the same report.

did you try? 🤔

No, because this was already discussed: The report is real, but it's a well understood problem in the way ROOT and Cling use the LLVM and Clang libraries and will not be fixed. That part of the issue has been addressed from my point of view, please let me know if there is something to clarify.

Then in #14055 (comment) you made a second report about memory usage which you clarified in #14055 (comment) happens even if only linking the ROOT libraries. As mentioned, this seemed specific to your setup and I was thus waiting for a reproducer from your side. Without that, I'm not sure what I would be able to do here?