Bug 380037

Summary: valgrind + AddressSanitizer: ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.
Product: [Developer tools] valgrind Reporter: Graham Leggett <minfrin>
Component: generalAssignee: Julian Seward <jseward>
Status: RESOLVED NOT A BUG    
Severity: normal CC: tom
Priority: NOR    
Version First Reported In: 3.12.0   
Target Milestone: ---   
Platform: Other   
OS: Linux   
Latest Commit: Version Fixed/Implemented In:
Sentry Crash Report:

Description Graham Leggett 2017-05-20 18:44:28 UTC
When trying to run valgrind on a binary that has been built linked to AddressSanitizer, valgrind immediately crashes at startup with a SIGSEGV as below.

It appears whatever valgrind does to the LD_PRELOAD environment variable breaks AddressSanitizer, which then causes valgrind to crash.

minfrin@towerofpi9:/var/www/html/stream$ GST_LEAKS_TRACER_STACK_TRACE=1 GST_LEAKS_TRACER_SIG=1 ASAN_OPTIONS=atexit=1 GST_TRACERS="leaks(stack-traces-flags=full,filters=GstMemory,check-refs=true)" GST_DEBUG_DUMP_DOT_DIR=. G_DEBUG="fatal-warnings" LD_PRELOAD=/usr/local/gcc-6.2.0/lib/libasan.so.3 LD_LIBRARY_PATH=/usr/local/lib:/usr/local/gcc-6.2.0/lib /tmp/valgrind/bin/valgrind gst-launch-1.0 --gst-debug-no-color --gst-debug=1,GST_TRACER:7,GST_REFCOUNTING:0,GST_PADS:1,GST_STATES:5,tdttsparse:5,tsdemux:1,omx:1,omxvideo:1,omxvideodec:1,omxh264dec:1,omxvideoenc:1,omxh264enc:1,videoencoder:1,videorate:1,videoscale:1,mpegtsmux:4,decodebin:1,encodebin:1,h264parse:1,transcoder:5,hlssink:5 udpsrc multicast-iface=eth0 uri=udp://239.106.0.0:1234 "caps=application/x-rtp,media=(string)video,clock-rate=(int)90000" ! rtpbin ! rtpmp2tdepay ! progressreport update-freq=5 ! transcoder name=transcoder ! capsfilter caps=audio/x-raw ! fakesink   transcoder. ! capsfilter caps=video/mpegts ! hlssink target-duration=0    transcoder. ! capsfilter caps=audio/x-raw ! fakesink
==4810== Memcheck, a memory error detector
==4810== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==4810== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info
==4810== Command: gst-launch-1.0 --gst-debug-no-color --gst-debug=1,GST_TRACER:7,GST_REFCOUNTING:0,GST_PADS:1,GST_STATES:5,tdttsparse:5,tsdemux:1,omx:1,omxvideo:1,omxvideodec:1,omxh264dec:1,omxvideoenc:1,omxh264enc:1,videoencoder:1,videorate:1,videoscale:1,mpegtsmux:4,decodebin:1,encodebin:1,h264parse:1,transcoder:5,hlssink:5 udpsrc multicast-iface=eth0 uri=udp://239.106.0.0:1234 caps=application/x-rtp,media=(string)video,clock-rate=(int)90000 ! rtpbin ! rtpmp2tdepay ! progressreport update-freq=5 ! transcoder name=transcoder ! capsfilter caps=audio/x-raw ! fakesink transcoder. ! capsfilter caps=video/mpegts ! hlssink target-duration=0 transcoder. ! capsfilter caps=audio/x-raw ! fakesink
==4810== 
==4810==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.
==4810== Jump to the invalid address stated on the next line
==4810==    at 0x0: ???
==4810==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==4810== 
==4810== 
==4810== Process terminating with default action of signal 11 (SIGSEGV)
==4810==  Bad permissions for mapped region at address 0x0
==4810==    at 0x0: ???
==4810== 
==4810== HEAP SUMMARY:
==4810==     in use at exit: 0 bytes in 0 blocks
==4810==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==4810== 
==4810== All heap blocks were freed -- no leaks are possible
==4810== 
==4810== For counts of detected and suppressed errors, rerun with: -v
==4810== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Comment 1 Tom Hughes 2017-05-20 19:25:27 UTC
Using valgrind and asan together is never likely to work anyway. I haven't seen that error before but usually it falls over because the address space manipulation each tries to do is incompatible.
Comment 2 Tom Hughes 2017-05-20 19:27:13 UTC
By the way the specific problem you're seeing here is that asan is trying to make sure it's intercepting malloc etc but of course valgrind also wants to do the same thing and they can't both do so.
Comment 3 Graham Leggett 2017-05-20 20:13:15 UTC
Having completely rebuilt both gstreamer git-master and valgrind v3.12.0 from source using gcc v6.20 with AddressSanitizer disabled, I am still getting the same error:

==26049== 
==26049==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.
==26049== Jump to the invalid address stated on the next line
==26049==    at 0x0: ???
==26049==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==26049== 
==26049== 
==26049== Process terminating with default action of signal 11 (SIGSEGV)
==26049==  Bad permissions for mapped region at address 0x0
==26049==    at 0x0: ???
==26049== 
==26049== HEAP SUMMARY:
==26049==     in use at exit: 0 bytes in 0 blocks
==26049==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==26049== 
==26049== All heap blocks were freed -- no leaks are possible
==26049== 
==26049== For counts of detected and suppressed errors, rerun with: -v
==26049== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

Is there any mechanism to switch off AddressSanitizer, or at least get valgrind to coexist peacefully with AddressSanitizer on modern compilers?
Comment 4 Graham Leggett 2017-05-20 21:10:48 UTC
Turned out the session I was using to run valgrind had libasan preloaded with LD_PRELOAD, and this turns AddressSanitizer on and triggers this crash.

Removing LD_PRELOAD produces the next crash, this time in valgrind:

vex: priv/guest_arm_toIR.c:13352 (decode_V8_instruction): Assertion `szBlg2 <= 3' failed.
vex storage: T total 243148328 bytes allocated
vex storage: P total 0 bytes allocated

valgrind: the 'impossible' happened:
   LibVEX called failure_exit().

Looks like this is this bug:

https://bugs.kde.org/show_bug.cgi?id=372794

Building bleeding edge valgrind to see if that will get further...
Comment 5 Julian Seward 2017-05-22 07:57:08 UTC
Really, this isn't likely to work, per Tom's comment 1 and comment 2.
Don't do it.
Comment 6 Graham Leggett 2017-05-22 08:34:49 UTC
For the record, bug that prevents valgrind running raised with the RPi people here:

https://bugs.kde.org/show_bug.cgi?id=380037
Comment 7 Graham Leggett 2017-05-22 23:36:41 UTC
Let's try that again. For the record, the bug with the RPi people is here:

https://github.com/RPi-Distro/repo/issues/68

At their suggestion moving /etc/ld.so.preload out of the way to prevent the loading the optimised RPi specific code causes valgrind to run successfully on the RPi.