We at Infor have a custom memory allocator. I've succesfully instrumented it using valgrind macro's and we're very happy with the problems it finds. However, the bigger programs stop with an "out of memory" when running under valgrind (and run fine when running natively). I tried all steps in the manual and on the web (ulimits, stack sizes, etc). No joy. I found that a program can allocate plenty of memory, but not by using sbrk(). A simple test program: ---- cut here--- #include <stdio.h> #include <unistd.h> #include <stdlib.h> #define HUNK (32 * 1024) int main (int argc, char **argv) { int i = 0; char *p; while (1) { p = sbrk(HUNK); if (p == (char *) -1 || p == NULL) break; printf("%6d: %p - %p\n",i++,p,p + HUNK); } return(0); } --- cut here --- Shows the problem. Under valgrind: rbeerstr@nlbaldev3: valgrind ./mem | tail ==21481== Memcheck, a memory error detector ==21481== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==21481== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info ==21481== Command: ./mem ==21481== ==21481== ==21481== HEAP SUMMARY: ==21481== in use at exit: 0 bytes in 0 blocks ==21481== total heap usage: 0 allocs, 0 frees, 0 bytes allocated ==21481== ==21481== All heap blocks were freed -- no leaks are possible ==21481== ==21481== For counts of detected and suppressed errors, rerun with: -v ==21481== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2) 245: 0x49ca000 - 0x49d2000 246: 0x49d2000 - 0x49da000 247: 0x49da000 - 0x49e2000 248: 0x49e2000 - 0x49ea000 249: 0x49ea000 - 0x49f2000 250: 0x49f2000 - 0x49fa000 251: 0x49fa000 - 0x4a02000 252: 0x4a02000 - 0x4a0a000 253: 0x4a0a000 - 0x4a12000 254: 0x4a12000 - 0x4a1a000 rbeerstr@nlbaldev3: And without valgrind: rbeerstr@nlbaldev3: ./mem | tail 6210693: 0x2f62a2a000 - 0x2f62a32000 6210694: 0x2f62a32000 - 0x2f62a3a000 6210695: 0x2f62a3a000 - 0x2f62a42000 6210696: 0x2f62a42000 - 0x2f62a4a000 6210697: 0x2f62a4a000 - 0x2f62a52000 6210698: 0x2f62a52000 - 0x2f62a5a000 6210699: 0x2f62a5a000 - 0x2f62a62000 6210700: 0x2f62a62000 - 0x2f62a6a000 6210701: 0x2f62a6a000 - 0x2f62a72000 6210702: 0x2f62a72000 - 0x2f62a7a000 Many gigs when running natively, only about 800MB under valgrind. Replacing the "sbrk" call with "malloc" fixes this (test program). So it is not the amount of memory as such that is a problem, only the fact that it is acquired through sbrk() instead of malloc. Reproducible: Always Steps to Reproduce: 1. See test program above. 2.gcc mem.c -o mem 3.valgrind ./mem | tail Only about 250 block of 32KB can be allocated under valgrind. Natively, more than 6 million. Actual Results: Stop after 250 blocks of 32KB. Expected Results: About half of what a native run would allow (I understand that memcheck uses a significant amount of memory itself).
You can never know how much memory you may or may not be able to allocate with sbrk so a program which relies on that is essentially broken. There's really no good reason to use sbrk in a world with mmap available anyway... The likely cause here is simply that valgrind is arranging the memory differently (in particular it has to reserve space both for it's own code and it's data structures) and that is limiting how far the brk segment can grow before it runs into something else.
Hi, Thank you for the fast response. The sbrk() is part of a custom memory allocator written as part of the Baan ERP package (nowadays called Infor LN). It is about 30 years old (!), platform independent (Linux, HP-UX, SunOS, AIX, Windows, etc), optimized to a fare-thee-well and has lots of features not found in any other allocator. To call that "essentially broken" is a little too easy :-) Most programs using that allocator run OK under valgrind and it reports leaks and other real issues. Very valuable. But 64-bit programs that require more than about 800MB fail with out-of-memory. And those memory-intensive programs are the ones I especially want to use memcheck on. I understand about memory maps and the reservations for various structures that valgrind needs, but in a 64-bit address space there is plenty of room for all. If valgrind would allow a few extra gigabytes for the target-program's brk segment the problem would go away. I've now hacked our custom allocator to use malloc()instead of sbrk() when running under valgrind. That avoids the problem. The incurred performance penalty is only under valgrind which is OK. So I can live without a fix for this but would expect that a tool like valgrind would rather not cause problems for their target programs... Also, it took a significant effort on my part to find the cause. If valgrind could issue a decent diagnostic when this happens, that would be nice. Regards, Ruurd Beerstra -----Original Message----- From: bugzilla_noreply@kde.org [mailto:bugzilla_noreply@kde.org] On Behalf Of Tom Hughes Sent: dinsdag 15 september 2015 16:03 To: Ruurd Beerstra Subject: [valgrind] [Bug 352742] Custom allocator using sbrk() fails after about 800MB when running under memcheck https://bugs.kde.org/show_bug.cgi?id=352742 Tom Hughes <tom@compton.nu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tom@compton.nu --- Comment #1 from Tom Hughes <tom@compton.nu> --- You can never know how much memory you may or may not be able to allocate with sbrk so a program which relies on that is essentially broken. There's really no good reason to use sbrk in a world with mmap available anyway... The likely cause here is simply that valgrind is arranging the memory differently (in particular it has to reserve space both for it's own code and it's data structures) and that is limiting how far the brk segment can grow before it runs into something else. -- You are receiving this mail because: You reported the bug.
(In reply to Ruurd Beerstra from comment #2) > > The sbrk() is part of a custom memory allocator written as part of the Baan > ERP package (nowadays called Infor LN). > It is about 30 years old (!), platform independent (Linux, HP-UX, SunOS, > AIX, Windows, etc), optimized to a fare-thee-well and has lots of features > not found in any other allocator. > To call that "essentially broken" is a little too easy :-) > > Most programs using that allocator run OK under valgrind and it reports > leaks and other real issues. Very valuable. > But 64-bit programs that require more than about 800MB fail with > out-of-memory. > And those memory-intensive programs are the ones I especially want to use > memcheck on. > > I understand about memory maps and the reservations for various structures > that valgrind needs, but in a 64-bit address space there is plenty of room > for all. > If valgrind would allow a few extra gigabytes for the target-program's brk > segment the problem would go away. > > I've now hacked our custom allocator to use malloc()instead of sbrk() when > running under valgrind. That avoids the problem. > The incurred performance penalty is only under valgrind which is OK. > > So I can live without a fix for this but would expect that a tool like > valgrind would rather not cause problems for their target programs... > Also, it took a significant effort on my part to find the cause. If valgrind > could issue a decent diagnostic when this happens, that would be nice. > valgrind reserves 8MB for the brk segment. That number is currently hardwired, unfortunately. If you feel like fooling around: <valgrind>/coregrind/m_initimg/initimg-linux.c around line 995 (the line number refers to the development sources in svn). There you can change the size to something that may better serve your needs. Yes, it would be better to have some way of changing that value other than a recompile. Maybe in a future release. In the meantime, the upcoming version 3.11 will at least tell you that you've exhausted your brk segment: 252: 0x4a05000 - 0x4a0d000 253: 0x4a0d000 - 0x4a15000 254: 0x4a15000 - 0x4a1d000 ==5564== brk segment overflow in thread #1: can't grow to 0x4a25000 ==5564==
Hi, Thank you! The diagnostic will be a great help if we run into this again. One of the features of our complex allocator is that I can plug in another allocator at the very bottom. The default is sbrk(), but now a RUNNING_ON_VALGRIND test is used to switch to malloc and all is well. Hmm - I just realized I mixed up my order of magnitudes quite badly. How embarrassing :-( The limit for sbrk is not 800 MB but, as you point out, only 8MB (test program fails after of 254 blocks of 32 KB). Case of caffeine deficiency :-) That low limit is a little surprising - would not all custom allocators run into this problem? As you seem to be knowledgeable about this, a question, if I may: I've created a patch on valgrind to support our custom allocator. One of its features is that the application can create a memory pool, create objects in that pool and when done, delete the pool which auto-frees all objects in the pool. The current valgrind does not support that (it reports thousands of bogus memory leaks), but our applications rely on that behavior. From discussions I found on the net it seems this mempool behavior was a conscious design decision in valgrind. I've extended valgrind to have a VALGRIND_CREATE_AUTOFREE_MEMPOOL macro, which passes a bit of extra info to the core so when such a pool is destroyed, it auto-frees all objects. So this patch allows custom allocators to choose between 2 types of mempools: Auto-freeing or not. Works great for us (only real leaks are reported now). Is that worth submitting? Or am I entirely on the wrong track? Thank you, Ruurd Beerstra -----Original Message----- From: bugzilla_noreply@kde.org [mailto:bugzilla_noreply@kde.org] On Behalf Of Florian Krohm Sent: dinsdag 15 september 2015 16:45 To: Ruurd Beerstra Subject: [valgrind] [Bug 352742] Custom allocator using sbrk() fails after about 800MB when running under memcheck https://bugs.kde.org/show_bug.cgi?id=352742 Florian Krohm <florian@eich-krohm.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |florian@eich-krohm.de --- Comment #3 from Florian Krohm <florian@eich-krohm.de> --- (In reply to Ruurd Beerstra from comment #2) > > The sbrk() is part of a custom memory allocator written as part of the > Baan ERP package (nowadays called Infor LN). > It is about 30 years old (!), platform independent (Linux, HP-UX, > SunOS, AIX, Windows, etc), optimized to a fare-thee-well and has lots > of features not found in any other allocator. > To call that "essentially broken" is a little too easy :-) > > Most programs using that allocator run OK under valgrind and it > reports leaks and other real issues. Very valuable. > But 64-bit programs that require more than about 800MB fail with > out-of-memory. > And those memory-intensive programs are the ones I especially want to > use memcheck on. > > I understand about memory maps and the reservations for various > structures that valgrind needs, but in a 64-bit address space there is > plenty of room for all. > If valgrind would allow a few extra gigabytes for the target-program's > brk segment the problem would go away. > > I've now hacked our custom allocator to use malloc()instead of sbrk() > when running under valgrind. That avoids the problem. > The incurred performance penalty is only under valgrind which is OK. > > So I can live without a fix for this but would expect that a tool like > valgrind would rather not cause problems for their target programs... > Also, it took a significant effort on my part to find the cause. If > valgrind could issue a decent diagnostic when this happens, that would be nice. > valgrind reserves 8MB for the brk segment. That number is currently hardwired, unfortunately. If you feel like fooling around: <valgrind>/coregrind/m_initimg/initimg-linux.c around line 995 (the line number refers to the development sources in svn). There you can change the size to something that may better serve your needs. Yes, it would be better to have some way of changing that value other than a recompile. Maybe in a future release. In the meantime, the upcoming version 3.11 will at least tell you that you've exhausted your brk segment: 252: 0x4a05000 - 0x4a0d000 253: 0x4a0d000 - 0x4a15000 254: 0x4a15000 - 0x4a1d000 ==5564== brk segment overflow in thread #1: can't grow to 0x4a25000 ==5564== -- You are receiving this mail because: You reported the bug.
The problem with the mempool stuff is that I'm not sure there's anybody (or at least anybody currently working on valgrind) who actually understands it... Certainly we've never bothered with it - we had already instrumented all our custom allocators long before either the mempool or the malloclike stuff existed just using simple calls to mark stuff as undefined when it was issued from the pool and unavailable when it was returned to the pool. As to why other customer allocators don't run into this limit, well I suspect most custom allocators use mmap (or malloc/new) as the source of their memory these days. Using sbrk is hard work, especially if you ever want to have any option of returning memory to the system.
A side note: implementation of malloc() and friends from Linux glibc can cope with the limited brk segment size. If sbrk() fails, it just uses mmap(). The reason why brk cannot grow here is because address space management on Linux generally speaking allocates address space from bottom up. Address space for the brk segment is allocated early during image initialization and shortly there are segments past it for dynamically loaded libraries, for example, so it cannot grow. When we ported Valgrind to Solaris, we had to overcome this limitation of fixed (small) brk segment size. That's because inability to grow brk segment is a hard failure in Solaris libc malloc(). So the address space management on Solaris is a bit different: address space is allocated from top to bottom, like Solaris kernel does. Thus the brk segment has some (considerably large) space to grow.
Created attachment 108234 [details] collectiion of callgrind reports Collection of callgrind before and after memory leak runaway, all the way to 4GB!!