Bug 269884

Summary: overhead for huge blocks exhausts space too soon
Product: [Developer tools] valgrind Reporter: John Reiser <jreiser>
Component: memcheckAssignee: Julian Seward <jseward>
Status: RESOLVED DUPLICATE    
Severity: normal    
Priority: NOR    
Version First Reported In: 3.6.0   
Target Milestone: ---   
Platform: Compiled Sources   
OS: Linux   
Latest Commit: Version Fixed/Implemented In:
Sentry Crash Report:
Attachments: corrected test program

Description John Reiser 2011-04-01 17:24:08 UTC 
Created attachment 58504 [details]
corrected test program

This is a tracking bug for an episode that Bob Harris started in [Valgrind-users].  Bob is interested in leak detection when huge blocks are involved.  The text below originally appeared in http://sourceforge.net/mailarchive/forum.php?thread_name=4D95E414.3000108%40bitwagon.com&forum_name=valgrind-users

There is a failure, at least from the user's perspective.  When run under
memcheck, then the [corrected] test program runs out of space much sooner
than expected.  The space overhead appears to be greater than 20 to 1,
when something like 3:1 might be "normal".  When the size of individual
blocks is huge (exceeds (RAM + SWAP)/400, for example) then the default
settings tend to exhaust space very quickly, with symptoms that are
mysterious to non-experts.  This is a Usability bug.

Bart Van Assche suggests --freelist-vol=0, which is an excellent suggestion.

For better usability, then memcheck should take more care with huge blocks.
Put a bound of (RAM + SWAP)/8 on the volume of the freelist; or, keep a
separate freelist for huge blocks, and age them faster.  For a growing
realloc() of a huge block, then relax the strategy which forces malloc_new +
copy + free_old.  Give up some local error detection for the chance to
live long enough to detect more errors later.

If nothing else, then when the first huge block is detected give a warning:
    huge block detected (%lu bytes); consider adjusting --freelist-vol


[The values of MemTotal and SwapTotal are in /proc/meminfo.  Evaluating
the threshold once is enough even though /sbin/swapon or pluggable RAM
might change it.  Disregarding other processes is OK because even novices
know to minimize external contention.]
Comment 1 John Reiser 2011-04-01 17:40:49 UTC 
For additional usability, give a warning the first time that realloc() fails:
  Warning: realloc(%p, %lu) returns NULL
This warning is appropriate especially because memcheck forces realloc() into the worst case, in order to maximize error detection.
Comment 2 John Reiser 2011-04-01 18:17:26 UTC 
The underlying cause probably is https://bugs.kde.org/show_bug.cgi?id=250101 which contains a patch to fix.  Still it would be nice to get the hint about --freelist-vol when huge blocks are involved.

*** This bug has been marked as a duplicate of bug 250101 ***
Comment 3 John Reiser 2011-04-01 18:34:33 UTC 
(In reply to comment #0)
> If nothing else, then when the first huge block is detected give a warning:
>     huge block detected (%lu bytes); consider adjusting --freelist-vol

Of course this warning should be upon calling free() for a huge block, including the forced free() that memcheck employs inside realloc().