I'm getting the Segfault when running heaptrack with attach mode (heaptrack -p) on programs compiled inside docker container (ubuntu 20.04 clang-10, heaptrack 1.1.80 and 1.2.80). I've tested it on pre-compiled programs, such as nano, and it works fine, so I'm guessing I'm doing something wrong. Even as simple program as this will fail: int main(int argc, char** argv) { while(1){ int* a = new int(10); delete a; } return 0; } I've tried to debug the issue (see ADDITIONAL INFORMATION below) but wasn't able to fix the issue. STEPS TO REPRODUCE Not sure whether it's related to my docker container or how I compile things, but anything that I compile and try to run heaptrack on it, will fail. OBSERVED RESULT heaptrack output will be written to "//heaptrack.a.out.17470.gz" injecting heaptrack into application via GDB, this might take some time... The program being debugged was signaled while in a function called from GDB. GDB remains in the frame where the signal was received. To change this behavior use "set unwindonsignal on". Evaluation of the expression containing the function (heaptrack_inject) will be abandoned. When the function is done executing, GDB will silently stop. heaptrack stats: allocations: 0 leaked allocations: 0 temporary allocations: 0 injection finished removing heaptrack injection via GDB, this might take some time... ptrace: No such process. No symbol table is loaded. Use the "file" command. The program is not being run. Heaptrack finished! Now run the following to investigate the data: heaptrack --analyze "//heaptrack.a.out.17470.gz" EXPECTED RESULT SOFTWARE/OS VERSIONS Docker image: Ubuntu 20.04 Compiler: clang-10 or gcc-9.3 heaptrack 1.1.80 and 1.2.80 ADDITIONAL INFORMATION I've tried to debug where exactly it segfaults and it seems that it fails during call to heaptrack_inject, on this line: libheaptrack.cpp:317 s_data = new LockedData(out, stopCallback); and then inside: heaptrack_inject.cpp:94 static void* hook(size_t size) noexcept Let me know if you need any other information.
can you try to run `heaptrack -d -p ...` and then get a backtrace from the gdb shell that should be kept open when it crashes? can you share a minimal docker file to reproduce the issue?
Hello Milian, thank you for replying. It seems to be reproducable with following docker image: ------------------------------------------------------------------ FROM ubuntu:latest ENV DEBIAN_FRONTEND=noninteractive RUN apt-get update && apt-get install -y heaptrack gdb clang-11 ------------------------------------------------------------------ 1. docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined 2. Create file test.cc with following content: ------------------------------- int main(int argc, char** argv) { while(1){ int* a = new int(10); delete a; } return 0; } ------------------------------- 3. Build it 'clang++-11 test.cc -o test' 4. Start it and get its PID 5. Attach heaptrack using the PID 'heaptrack -d -p PID' 6. Heaptrack output: heaptrack output will be written to "//heaptrack.test.1481.gz" injecting heaptrack into application via GDB, this might take some time... Attaching to process 1162 Reading symbols from /test... (No debugging symbols found in /test) Reading symbols from /lib/x86_64-linux-gnu/libstdc++.so.6... (No debugging symbols found in /lib/x86_64-linux-gnu/libstdc++.so.6) Reading symbols from /lib/x86_64-linux-gnu/libm.so.6... Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libm-2.31.so... Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1... (No debugging symbols found in /lib/x86_64-linux-gnu/libgcc_s.so.1) Reading symbols from /lib/x86_64-linux-gnu/libc.so.6... Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.31.so... Reading symbols from /lib64/ld-linux-x86-64.so.2... (No debugging symbols found in /lib64/ld-linux-x86-64.so.2) __GI___libc_malloc (bytes=4) at malloc.c:2940 2940 malloc.c: No such file or directory. Symbols already loaded for /lib/x86_64-linux-gnu/libc.so.6 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Symbols already loaded for /usr/lib/heaptrack/libheaptrack_inject.so Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? () The program being debugged was signaled while in a function called from GDB. GDB remains in the frame where the signal was received. To change this behavior use "set unwindonsignal on". Evaluation of the expression containing the function (heaptrack_inject) will be abandoned. When the function is done executing, GDB will silently stop. (gdb) bt #0 0x0000000000000000 in ?? () #1 0x00007f50eb626049 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) () from /lib/x86_64-linux-gnu/libstdc++.so.6 #2 0x00007f50eb1e66ff in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #3 0x00007f50eb1e7ffe in heaptrack_init () from /usr/lib/heaptrack/libheaptrack_inject.so #4 0x00007f50eb1e53f2 in heaptrack_inject () from /usr/lib/heaptrack/libheaptrack_inject.so #5 <function called from gdb> #6 __GI___libc_malloc (bytes=4) at malloc.c:2940 #7 0x00007f50eb5f9c29 in operator new(unsigned long) () from /lib/x86_64-linux-gnu/libstdc++.so.6 #8 0x0000000000401160 in main () (gdb)
Oh ok, it seems that for this example adding -pthread solves the issue. I will see whether it was also the case in my previous testings.
Sadly it doesn't help with my initial test. I will try to isolate the issue (my guess is that some libs that I'm linking with, could be somehow causing the issue) heaptrack output will be written to "/vt/heaptrack.memory_checker.31540.gz" injecting heaptrack into application via GDB, this might take some time... Attaching to process 31464 Reading symbols from /build/vt/tests/memory_checker... (No debugging symbols found in /build/vt/tests/memory_checker) Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2... Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.31.so... Reading symbols from /lib/x86_64-linux-gnu/libz.so.1... (No debugging symbols found in /lib/x86_64-linux-gnu/libz.so.1) Reading symbols from /usr/local/lib/libmpicxx.so.12... (No debugging symbols found in /usr/local/lib/libmpicxx.so.12) Reading symbols from /usr/local/lib/libmpi.so.12... (No debugging symbols found in /usr/local/lib/libmpi.so.12) Reading symbols from /lib/x86_64-linux-gnu/libstdc++.so.6... (No debugging symbols found in /lib/x86_64-linux-gnu/libstdc++.so.6) Reading symbols from /lib/x86_64-linux-gnu/libm.so.6... Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libm-2.31.so... Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1... (No debugging symbols found in /lib/x86_64-linux-gnu/libgcc_s.so.1) Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0... --Type <RET> for more, q to quit, c to continue without paging-- Reading symbols from /usr/lib/debug/.build-id/e5/4761f7b554d0fcc1562959665d93dffbebdaf0.debug... [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Reading symbols from /lib/x86_64-linux-gnu/libc.so.6... Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.31.so... Reading symbols from /lib64/ld-linux-x86-64.so.2... (No debugging symbols found in /lib64/ld-linux-x86-64.so.2) Reading symbols from /lib/x86_64-linux-gnu/librt.so.1... Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/librt-2.31.so... 0x00000000005fddd0 in main () Symbols already loaded for /lib/x86_64-linux-gnu/libc.so.6 Symbols already loaded for /usr/lib/heaptrack/libheaptrack_inject.so Program received signal SIGSEGV, Segmentation fault. 0x00007f423d9bfe76 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so The program being debugged was signaled while in a function called from GDB. GDB remains in the frame where the signal was received. To change this behavior use "set unwindonsignal on". Evaluation of the expression containing the function (heaptrack_inject) will be abandoned. When the function is done executing, GDB will silently stop. (gdb) bt #0 0x00007f423d9bfe76 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #1 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #2 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #3 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #4 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #5 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #6 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #7 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #8 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #9 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #10 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #11 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #12 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #13 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #14 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #15 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #16 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #17 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #18 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #19 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #20 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #21 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #22 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #23 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #24 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #25 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #26 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #27 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #28 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #29 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #30 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #31 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #32 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #33 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #34 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #35 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #36 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #37 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #38 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #39 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #40 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #41 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #42 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #43 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #44 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #45 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #46 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so #47 0x00007f423d9bfe83 in ?? () from /usr/lib/heaptrack/libheaptrack_inject.so --Type <RET> for more, q to quit, c to continue without paging--q Quit (gdb) q