Created attachment 159239 [details] program to demonstrate the issue SUMMARY Under valgrind the call to mremap() failed with MAP_FAILED because the overloading valgrind mremap() doesn't implement increasing the size of the memory at the same address fully. I assume this is because the valgrind calls are more malloc/free oriented and assume its ok to change the address on a resize. STEPS TO REPRODUCE 1. Compile example program which uses mmap/mremap/munmap() calls 2. Run from command line with and without valgrind. OBSERVED RESULT # valgrind -v --tool=memcheck ./a.out init mmap: size:8192 grow mmap: oldaddr:0xac00000000 oldsize:8192 newsize:16384 Failed mremap mmmap.dat (12:Cannot allocate memory) zsh: abort (core dumped) valgrind -v --tool=memcheck ./a.out EXPECTED RESULT When running the program without valgrind the system allows for the base address to be expanded in size. # ./a.out init mmap: size:8192 grow mmap: oldaddr:0xac00000000 oldsize:8192 newsize:16384 mremap: addr:0xac00000000 size:16384 That's all folks SOFTWARE/OS VERSIONS Any Linux (I ran my tests on RHEL8.8) Kernel: Linux4.18.0-425.19.2.el8_7.x86_64 #1 SMP Fri Mar 17 01:52:38 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux ADDITIONAL INFORMATION The solution this code is a key component of a in memory database.. We do out own address management so as not to conflict with HI/LO memory allocation. Hence we start at address 0xAC00000000L. When we allocate mmaps we leave enough address space between the addresses so that mremap has enough contiguous memory to succeed with no conflict. If there is a work around for avoiding this issue that would be helpful, for instance can I disable the valgrind mmap support? We don't need it to investigate mmap() issues.
Disabling mmap support would render valgrind useless - if it doesn't know what memory has been allocated that it can't track it's state. Any program has to assume that memory allocations may fail and deal with that so technically I'm not sure this is really a bug - the behaviour of memory allocation under valgrind is often very different to when running normally simply because it has to allocate extra memory to track state and that is all happening in the same address space. We certainly don't reject all mremap's that expand out of hand, as you can see at https://sourceware.org/git/?p=valgrind.git;a=blob;f=coregrind/m_syswrap/syswrap-generic.c;h=efdae60e10320cbcccd86415fabf6ef8fd7abf57;hb=HEAD#l275 which is the current implementation. A starting point would probably be to run with --trace-syscalls=yes so we can see exactly what arguments are being passed and trace out what path it is following in that handler.
Its a bug because its not fully implementing the mmap() functionality.. However yes we are using mmap/mremap so that we don't change the base address of the mmap(). That appears to be in conflict with valgrind which is implemented along the lines of expecting malloc/free block style functionally. mmap() can be used in that fashion but its not the only way... We implement out out versions of malloc/free to use the memory pools which we allocate in this fashion. Looking at the valgrind source code I can see it would be a lot of work to implement mmap() style blocks.. Which is why I'm looking for work arounds as valgrind is a valuable tool for finding memory/file handle leaks. We currently we are enable to do. I've added the system trace as an attachment
Created attachment 159240 [details] --trace-syscalls trace output of valgrind valgrind -v --trace-syscalls=yes --tool=memcheck ./a.out ==157805== Memcheck, a memory error detector ==157805== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al. ==157805== Using Valgrind-3.19.0-8d3c8034b8-20220411 and LibVEX; rerun with -h for copyright info ==157805== Command: ./a.out ==157805== --157805-- Valgrind options: --157805-- -v --157805-- --trace-syscalls=yes --157805-- --tool=memcheck --157805-- Contents of /proc/version: --157805-- Linux version 4.18.0-425.19.2.el8_7.x86_64 (mockbuild@x86-vm-09.build.eng.bos.redhat.com) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-16) (GCC)) #1 SMP Fri Mar 17 01:52:38 EDT 2023 --157805-- --157805-- Arch and hwcaps: AMD64, LittleEndian, amd64-cx16-rdtscp-sse3-ssse3-avx --157805-- Page sizes: currently 4096, max supported 4096 --157805-- Valgrind library directory: /usr/libexec/valgrind --157805-- Reading syms from /home/atcore/tmp/a.out --157805-- Reading syms from /usr/lib64/ld-2.28.so --157805-- Warning: cross-CU LIMITATION: some inlined fn names --157805-- might be shown as UnknownInlinedFun --157805-- Reading syms from /usr/libexec/valgrind/memcheck-amd64-linux --157805-- object doesn't have a symbol table --157805-- object doesn't have a dynamic symbol table --157805-- Scheduler: using generic scheduler lock implementation. --157805-- Reading suppressions file: /usr/libexec/valgrind/default.supp ==157805== embedded gdbserver: reading from /tmp/vgdb-pipe-from-vgdb-to-157805-by-atcore-on-mdev ==157805== embedded gdbserver: writing to /tmp/vgdb-pipe-to-vgdb-from-157805-by-atcore-on-mdev ==157805== embedded gdbserver: shared mem /tmp/vgdb-pipe-shared-mem-vgdb-157805-by-atcore-on-mdev ==157805== ==157805== TO CONTROL THIS PROCESS USING vgdb (which you probably ==157805== don't want to do, unless you know exactly what you're doing, ==157805== or are doing some strange experiment): ==157805== /usr/libexec/valgrind/../../bin/vgdb --pid=157805 ...command... ==157805== ==157805== TO DEBUG THIS PROCESS USING GDB: start GDB like this ==157805== /path/to/gdb ./a.out ==157805== and then give GDB the following command ==157805== target remote | /usr/libexec/valgrind/../../bin/vgdb --pid=157805 ==157805== --pid is optional if only one valgrind process is running ==157805== SYSCALL[157805,1](12) sys_brk ( 0x0 ) --> [pre-success] Success(0x4231000) SYSCALL[157805,1](158) arch_prctl ( 12289, 1fff000020 ) --> [pre-fail] Failure(0x16) --157805-- REDIR: 0x4005820 (ld-linux-x86-64.so.2:strlen) redirected to 0x580d0a02 (???) SYSCALL[157805,1](63) sys_newuname ( 0x1ffefffc00 )[sync] --> Success(0x0) --157805-- REDIR: 0x40055f0 (ld-linux-x86-64.so.2:index) redirected to 0x580d0a1c (???) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x4230a50(/usr/libexec/valgrind/vgpreload_core-amd64-linux.so), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Success(0x3) SYSCALL[157805,1](0) sys_read ( 3, 0x1ffeffe818, 832 ) --> [async] ... SYSCALL[157805,1](0) ... [async] --> Success(0x340) SYSCALL[157805,1](8) sys_lseek ( 3, 2144, 0 )[sync] --> Success(0x860) SYSCALL[157805,1](0) sys_read ( 3, 0x1ffeffe6f0, 32 ) --> [async] ... SYSCALL[157805,1](0) ... [async] --> Success(0x20) SYSCALL[157805,1](5) sys_newfstat ( 3, 0x1ffeffe6a0 )[sync] --> Success(0x0) SYSCALL[157805,1](8) sys_lseek ( 3, 2144, 0 )[sync] --> Success(0x860) SYSCALL[157805,1](0) sys_read ( 3, 0x1ffeffe3f0, 32 ) --> [async] ... SYSCALL[157805,1](0) ... [async] --> Success(0x20) SYSCALL[157805,1](9) sys_mmap ( 0x0, 2101304, 5, 2050, 3, 0 ) --> [pre-success] Success(0x4a31000) SYSCALL[157805,1](10) sys_mprotect ( 0x4a32000, 2093056, 0 )[sync] --> Success(0x0) SYSCALL[157805,1](9) sys_mmap ( 0x4c31000, 8192, 3, 2066, 3, 0 )--157805-- Reading syms from /usr/libexec/valgrind/vgpreload_core-amd64-linux.so --> [pre-success] Success(0x4c31000) SYSCALL[157805,1](3) sys_close ( 3 )[sync] --> Success(0x0) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x4230fb0(/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Success(0x3) SYSCALL[157805,1](0) sys_read ( 3, 0x1ffeffe818, 832 ) --> [async] ... SYSCALL[157805,1](0) ... [async] --> Success(0x340) SYSCALL[157805,1](8) sys_lseek ( 3, 80856, 0 )[sync] --> Success(0x13bd8) SYSCALL[157805,1](0) sys_read ( 3, 0x1ffeffe6f0, 32 ) --> [async] ... SYSCALL[157805,1](0) ... [async] --> Success(0x20) SYSCALL[157805,1](5) sys_newfstat ( 3, 0x1ffeffe6a0 )[sync] --> Success(0x0) SYSCALL[157805,1](9) sys_mmap ( 0x0, 8192, 3, 34, 4294967295, 0 ) --> [pre-success] Success(0x402e000) SYSCALL[157805,1](8) sys_lseek ( 3, 80856, 0 )[sync] --> Success(0x13bd8) SYSCALL[157805,1](0) sys_read ( 3, 0x1ffeffe3f0, 32 ) --> [async] ... SYSCALL[157805,1](0) ... [async] --> Success(0x20) SYSCALL[157805,1](9) sys_mmap ( 0x0, 2179456, 5, 2050, 3, 0 ) --> [pre-success] Success(0x4c33000) SYSCALL[157805,1](10) sys_mprotect ( 0x4c47000, 2093056, 0 )[sync] --> Success(0x0) SYSCALL[157805,1](9) sys_mmap ( 0x4e46000, 8192, 3, 2066, 3, 77824 )--157805-- Reading syms from /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so ==157805== WARNING: new redirection conflicts with existing -- ignoring it --157805-- old: 0x04005820 (strlen ) R-> (0000.0) 0x580d0a02 ??? --157805-- new: 0x04005820 (strlen ) R-> (2007.0) 0x04c3e170 strlen --> [pre-success] Success(0x4e46000) SYSCALL[157805,1](3) sys_close ( 3 )[sync] --> Success(0x0) SYSCALL[157805,1](21) sys_access ( 0x4026ad0(/etc/ld.so.preload), 4 )[sync] --> Failure(0x2) --157805-- REDIR: 0x4002000 (ld-linux-x86-64.so.2:strcmp) redirected to 0x4c3f250 (strcmp) --157805-- REDIR: 0x4005d80 (ld-linux-x86-64.so.2:mempcpy) redirected to 0x4c43180 (mempcpy) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x1ffefff120(/home/atcore/lib/glibc-hwcaps/x86-64-v2/libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Failure(0x2) SYSCALL[157805,1](4) sys_newstat ( 0x1ffefff120(/home/atcore/lib/glibc-hwcaps/x86-64-v2), 0x1ffefff1d0 )[sync] --> Failure(0x2) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x1ffefff120(/home/atcore/lib/tls/x86_64/x86_64/libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Failure(0x2) SYSCALL[157805,1](4) sys_newstat ( 0x1ffefff120(/home/atcore/lib/tls/x86_64/x86_64), 0x1ffefff1d0 )[sync] --> Failure(0x2) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x1ffefff120(/home/atcore/lib/tls/x86_64/libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Failure(0x2) SYSCALL[157805,1](4) sys_newstat ( 0x1ffefff120(/home/atcore/lib/tls/x86_64), 0x1ffefff1d0 )[sync] --> Failure(0x2) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x1ffefff120(/home/atcore/lib/tls/x86_64/libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Failure(0x2) SYSCALL[157805,1](4) sys_newstat ( 0x1ffefff120(/home/atcore/lib/tls/x86_64), 0x1ffefff1d0 )[sync] --> Failure(0x2) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x1ffefff120(/home/atcore/lib/tls/libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Failure(0x2) SYSCALL[157805,1](4) sys_newstat ( 0x1ffefff120(/home/atcore/lib/tls), 0x1ffefff1d0 )[sync] --> Failure(0x2) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x1ffefff120(/home/atcore/lib/x86_64/x86_64/libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Failure(0x2) SYSCALL[157805,1](4) sys_newstat ( 0x1ffefff120(/home/atcore/lib/x86_64/x86_64), 0x1ffefff1d0 )[sync] --> Failure(0x2) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x1ffefff120(/home/atcore/lib/x86_64/libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Failure(0x2) SYSCALL[157805,1](4) sys_newstat ( 0x1ffefff120(/home/atcore/lib/x86_64), 0x1ffefff1d0 )[sync] --> Failure(0x2) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x1ffefff120(/home/atcore/lib/x86_64/libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Failure(0x2) SYSCALL[157805,1](4) sys_newstat ( 0x1ffefff120(/home/atcore/lib/x86_64), 0x1ffefff1d0 )[sync] --> Failure(0x2) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x1ffefff120(/home/atcore/lib/libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Failure(0x2) SYSCALL[157805,1](4) sys_newstat ( 0x1ffefff120(/home/atcore/lib), 0x1ffefff1d0 )[sync] --> Success(0x0) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x1ffefff120(glibc-hwcaps/x86-64-v2/libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Failure(0x2) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x1ffefff120(tls/x86_64/x86_64/libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Failure(0x2) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x1ffefff120(tls/x86_64/libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Failure(0x2) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x1ffefff120(tls/x86_64/libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Failure(0x2) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x1ffefff120(tls/libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Failure(0x2) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x1ffefff120(x86_64/x86_64/libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Failure(0x2) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x1ffefff120(x86_64/libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Failure(0x2) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x1ffefff120(x86_64/libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Failure(0x2) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x1ffefff120(libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Failure(0x2) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x4025764(/etc/ld.so.cache), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Success(0x3) SYSCALL[157805,1](5) sys_newfstat ( 3, 0x1ffefff1d0 )[sync] --> Success(0x0) SYSCALL[157805,1](9) sys_mmap ( 0x0, 29887, 1, 2, 3, 0 ) --> [pre-success] Success(0x4030000) SYSCALL[157805,1](3) sys_close ( 3 )[sync] --> Success(0x0) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x402e520(/lib64/libc.so.6), 524288 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Success(0x3) SYSCALL[157805,1](0) sys_read ( 3, 0x1ffefff348, 832 ) --> [async] ... SYSCALL[157805,1](0) ... [async] --> Success(0x340) SYSCALL[157805,1](5) sys_newfstat ( 3, 0x1ffefff1d0 )[sync] --> Success(0x0) SYSCALL[157805,1](8) sys_lseek ( 3, 808, 0 )[sync] --> Success(0x328) SYSCALL[157805,1](0) sys_read ( 3, 0x1ffeffee90, 32 ) --> [async] ... SYSCALL[157805,1](0) ... [async] --> Success(0x20) SYSCALL[157805,1](9) sys_mmap ( 0x0, 3950816, 5, 2050, 3, 0 ) --> [pre-success] Success(0x4e48000) SYSCALL[157805,1](10) sys_mprotect ( 0x5003000, 2097152, 0 )[sync] --> Success(0x0) SYSCALL[157805,1](9) sys_mmap ( 0x5203000, 24576, 3, 2066, 3, 1814528 )--157805-- Reading syms from /usr/lib64/libc-2.28.so ==157805== WARNING: new redirection conflicts with existing -- ignoring it --157805-- old: 0x04ee4500 (memalign ) R-> (1011.0) 0x04c3d3e2 memalign --157805-- new: 0x04ee4500 (memalign ) R-> (1017.0) 0x04c3d3b5 aligned_alloc ==157805== WARNING: new redirection conflicts with existing -- ignoring it --157805-- old: 0x04ee4500 (memalign ) R-> (1011.0) 0x04c3d3e2 memalign --157805-- new: 0x04ee4500 (memalign ) R-> (1017.0) 0x04c3d388 aligned_alloc --> [pre-success] Success(0x5203000) SYSCALL[157805,1](9) sys_mmap ( 0x5209000, 14560, 3, 50, 4294967295, 0 ) --> [pre-success] Success(0x5209000) SYSCALL[157805,1](3) sys_close ( 3 )[sync] --> Success(0x0) SYSCALL[157805,1](9) sys_mmap ( 0x0, 12288, 3, 34, 4294967295, 0 ) --> [pre-success] Success(0x4038000) SYSCALL[157805,1](158) arch_prctl ( 4098, 4038740 ) --> [pre-success] Success(0x0) --157805-- REDIR: 0x4ee8780 (libc.so.6:memmove) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee7a90 (libc.so.6:strncpy) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee8ab0 (libc.so.6:strcasecmp) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee73a0 (libc.so.6:strcat) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee7af0 (libc.so.6:rindex) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4eea0e0 (libc.so.6:rawmemchr) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4f1fcc0 (libc.so.6:wmemchr) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4f1f6e0 (libc.so.6:wcscmp) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee88e0 (libc.so.6:mempcpy) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee8710 (libc.so.6:bcmp) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee7a20 (libc.so.6:strncmp) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee7450 (libc.so.6:strcmp) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee8850 (libc.so.6:memset) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4f1f6a0 (libc.so.6:wcschr) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee7980 (libc.so.6:strnlen) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee7530 (libc.so.6:strcspn) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee8b00 (libc.so.6:strncasecmp) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee74d0 (libc.so.6:strcpy) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee8c50 (libc.so.6:memcpy@@GLIBC_2.14) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4f20fc0 (libc.so.6:wcsnlen) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee7b30 (libc.so.6:strpbrk) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee7400 (libc.so.6:index) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee7940 (libc.so.6:strlen) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4eeebc0 (libc.so.6:memrchr) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee8b50 (libc.so.6:strcasecmp_l) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee86d0 (libc.so.6:memchr) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4f1f7f0 (libc.so.6:wcslen) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee7df0 (libc.so.6:strspn) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee8a50 (libc.so.6:stpncpy) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee89f0 (libc.so.6:stpcpy) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4eea120 (libc.so.6:strchrnul) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) --157805-- REDIR: 0x4ee8ba0 (libc.so.6:strncasecmp_l) redirected to 0x4a3177b (_vgnU_ifunc_wrapper) SYSCALL[157805,1](10) sys_mprotect ( 0x5203000, 16384, 1 )[sync] --> Success(0x0) SYSCALL[157805,1](10) sys_mprotect ( 0x4e46000, 4096, 1 )[sync] --> Success(0x0) SYSCALL[157805,1](10) sys_mprotect ( 0x4c31000, 4096, 1 )[sync] --> Success(0x0) SYSCALL[157805,1](10) sys_mprotect ( 0x601000, 4096, 1 )[sync] --> Success(0x0) SYSCALL[157805,1](10) sys_mprotect ( 0x422e000, 4096, 1 )[sync] --> Success(0x0) SYSCALL[157805,1](11) sys_munmap ( 0x4030000, 29887 )[sync] --> Success(0x0) --157805-- REDIR: 0x4e74bf0 (libc.so.6:__strrchr_sse2) redirected to 0x4c3db60 (__strrchr_sse2) SYSCALL[157805,1](257) sys_openat ( 4294967196, 0x400d18(mmmap.dat), 66, 420 ) --> [async] ... SYSCALL[157805,1](257) ... [async] --> Success(0x3) SYSCALL[157805,1](77) sys_ftruncate ( 3, 8192 ) --> [async] ... SYSCALL[157805,1](77) ... [async] --> Success(0x0) SYSCALL[157805,1](8) sys_lseek ( 3, 16384, 0 )[sync] --> Success(0x4000) SYSCALL[157805,1](1) sys_write ( 3, 0x400d6f, 1 ) --> [async] ... SYSCALL[157805,1](1) ... [async] --> Success(0x1) --157805-- REDIR: 0x4e749e0 (libc.so.6:__strchrnul_sse2) redirected to 0x4c42cb0 (strchrnul) SYSCALL[157805,1](5) sys_newfstat ( 1, 0x1ffefff8d0 )[sync] --> Success(0x0) --157805-- REDIR: 0x4ee3790 (libc.so.6:malloc) redirected to 0x4c380bf (malloc) --157805-- REDIR: 0x4e7ede0 (libc.so.6:__mempcpy_sse2_unaligned) redirected to 0x4c42dc0 (mempcpy) SYSCALL[157805,1](1) sys_write ( 1, 0x520d040, 21 ) --> [async] ... init mmap: size:8192 SYSCALL[157805,1](1) ... [async] --> Success(0x15) SYSCALL[157805,1](9) sys_mmap ( 0xac00000000, 8192, 3, 1, 3, 0 ) --> [pre-success] Success(0xac00000000) SYSCALL[157805,1](1) sys_write ( 1, 0x520d040, 59 ) --> [async] ... grow mmap: oldaddr:0xac00000000 oldsize:8192 newsize:16384 SYSCALL[157805,1](1) ... [async] --> Success(0x3b) SYSCALL[157805,1](25) sys_mremap ( 0xac00000000, 8192, 16384, 0x0 ) --> [pre-fail] Failure(0xc) --157805-- REDIR: 0x4e74ef0 (libc.so.6:__strlen_sse2) redirected to 0x4c3e0b0 (__strlen_sse2) --157805-- REDIR: 0x4e7b470 (libc.so.6:__stpcpy_sse2_unaligned) redirected to 0x4c41c40 (__stpcpy_sse2_unaligned) --157805-- REDIR: 0x4ee3e20 (libc.so.6:free) redirected to 0x4c3aa21 (free) SYSCALL[157805,1](1) sys_write ( 2, 0x1ffeffd9d0, 52 ) --> [async] ... Failed mremap mmmap.dat (12:Cannot allocate memory) SYSCALL[157805,1](1) ... [async] --> Success(0x34) SYSCALL[157805,1](14) sys_rt_sigprocmask ( 1, 0x1ffeffff10, 0x0, 8 ) --> [pre-success] Success(0x0) SYSCALL[157805,1](14) sys_rt_sigprocmask ( 0, 0x1ffefffe70, 0x1ffefffdf0, 8 ) --> [pre-success] Success(0x0) SYSCALL[157805,1](39) sys_getpid ()[sync] --> Success(0x2686d) SYSCALL[157805,1](186) sys_gettid ()[sync] --> Success(0x2686d) SYSCALL[157805,1](234) sys_tgkill ( 157805, 157805, 6 ) --> [async] ... SYSCALL[157805,1](234) ... [async] --> Success(0x0) SYSCALL[157805,1](14) sys_rt_sigprocmask ( 2, 0x1ffefffdf0, 0x0, 8 ) --> [pre-success] Success(0x0) ==157805== ==157805== Process terminating with default action of signal 6 (SIGABRT) ==157805== at 0x4E96ACF: raise (in /usr/lib64/libc-2.28.so) ==157805== by 0x4E69EA4: abort (in /usr/lib64/libc-2.28.so) ==157805== by 0x400C28: main (mremap_issue.c:53) ==157805== ==157805== HEAP SUMMARY: ==157805== in use at exit: 1,024 bytes in 1 blocks ==157805== total heap usage: 1 allocs, 0 frees, 1,024 bytes allocated ==157805== ==157805== Searching for pointers to 1 not-freed blocks ==157805== Checked 78,144 bytes ==157805== ==157805== LEAK SUMMARY: ==157805== definitely lost: 0 bytes in 0 blocks ==157805== indirectly lost: 0 bytes in 0 blocks ==157805== possibly lost: 0 bytes in 0 blocks ==157805== still reachable: 1,024 bytes in 1 blocks ==157805== suppressed: 0 bytes in 0 blocks ==157805== Rerun with --leak-check=full to see details of leaked memory ==157805== ==157805== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) zsh: abort (core dumped) valgrind -v --trace-syscalls=yes --tool=memcheck ./a.out
Created attachment 159241 [details] strace of the example programs running without issues
This is the call: SYSCALL[157805,1](25) sys_mremap ( 0xac00000000, 8192, 16384, 0x0 ) --> [pre-fail] Failure(0xc) so it's being rejected by valgrind with ENOMEM without being passed to the kernel. Given the flags we are reaching grow_in_place_or_fail (https://sourceware.org/git/?p=valgrind.git;a=blob;f=coregrind/m_syswrap/syswrap-generic.c;h=efdae60e10320cbcccd86415fabf6ef8fd7abf57;hb=HEAD#l479) and that is rejecting it most likely because VG_(am_get_advisory_client_simple) says that extra memory is not available for some reason. To find out why would probably need a custom build with the "if (0)" near the top of VG_(am_get_advisory) enabled so that it dumps the memory map which would show if something else was already allocated there.
Created attachment 159243 [details] Additional output from debug enabled valgrind mremap() overload The additional output from the valgrind overload of mremap().
I'd already tried outputting those segments before but I'm not sure why they failed.. My guess was that valgrind had used some of the space itself and was blocking the expansion of the existing memory segment.
It is a strange one.. The segment dump I attached shows nothing in the memory ahead.. But code walking through the do_mremap() function its not obvious to me why its returning MAP_FAILED. Something Subtle no doubt.. :-)
Actually I can see the problem there - the following segment is an SkResvn not SkFree. Now VG_(am_get_advisory_client_simple) will allow a reservation to be used so will accept the request, but VG_(am_covered_by_single_free_segment) will then reject it as it only allows a free segment. If we got past that then VG_(am_extend_map_client) would assert for the same reason... I'm not really sure about the difference between the two but my reading is that reservations are unused they're just in the top part of memory. I think the idea is to discourage allocations there so that valgrind can use it for shadow memory but still allow it if the caller insists by asking for a fixed address. Try this patch and see if it helps: --- a/coregrind/m_aspacemgr/aspacemgr-linux.c +++ b/coregrind/m_aspacemgr/aspacemgr-linux.c @@ -2213,7 +2213,7 @@ static NSegment const * VG_(am_find_free_nsegment) ( Addr a ) aspacem_assert(i >= 0 && i < nsegments_used); aspacem_assert(nsegments[i].start <= a); aspacem_assert(a <= nsegments[i].end); - if (nsegments[i].kind == SkFree) + if (nsegments[i].kind == SkFree || nsegments[i].kind == SkResvn) return &nsegments[i]; else return NULL; @@ -3203,7 +3203,7 @@ const NSegment *VG_(am_extend_map_client)( Addr addr, SizeT delta ) /* The segment following the client segment must be a free segment and it must be large enough to cover the additional memory. */ NSegment *segf = seg + 1; - aspacem_assert(segf->kind == SkFree); + aspacem_assert(segf->kind == SkFree || segf->kind == SkResvn); aspacem_assert(segf->start == xStart);
Brillant.. We have lots of smiling developers here now they can find memory leaks again :-) Yes that patch works.. Cheers..
Let's leave it open for now until that is upstream. As I say I'm not 100% sure about free vs resvn and whether that patch is completely correct so I'll try and find out before I think about committing it...