Bug 357833

I presume that this code in coregrind/m_main.c is the issue (so will affect all tools, not just memcheck):

   //--------------------------------------------------------------
   // Get the current process datasize rlimit, and set it to zero.
   // This prevents any internal uses of brk() from having any effect.
   // We remember the old value so we can restore it on exec, so that
   // child processes will have a reasonable brk value.
   VG_(getrlimit)(VKI_RLIMIT_DATA, &VG_(client_rlimit_data));
   zero.rlim_max = VG_(client_rlimit_data).rlim_max;
   VG_(setrlimit)(VKI_RLIMIT_DATA, &zero);

The logic is self-explanatory, but equally it seems it is only an attempt to detect mistakes in valgrind, so that any attempt to allocate memory with brk() would fail.

> I presume that this code in coregrind/m_main.c is the issue

indeed, after i commented out "VG_(setrlimit)(VKI_RLIMIT_DATA, &zero);" the problem was gone (as i wrote in the description).

Looks like this commit went into linux git tree recently:

commit 84638335900f1995495838fe1bd4870c43ec1f67
Author:     Konstantin Khlebnikov <koct9i@gmail.com>
AuthorDate: Thu Jan 14 15:22:07 2016 -0800
Commit:     Linus Torvalds <torvalds@linux-foundation.org>
CommitDate: Thu Jan 14 16:00:49 2016 -0800

    mm: rework virtual memory accounting

So it should be possible to replicate this now with a "normal" linux git master build. But I haven't done so yet.

Replicated it now with the fedora rawhide kernel 4.5.0-0.rc0.git6.1.fc24.i686+PAE
Not setting the RLIMIT_DATA to zero in coregrind/m_main.c (valgrind_main) does indeed work around it.

For reference here is the full commit explaining that previously the RLIMIT_DATA value indeed was mostly harmless only affecting brk, but now restricts any data area allocations:

commit 84638335900f1995495838fe1bd4870c43ec1f67
Author:     Konstantin Khlebnikov <koct9i@gmail.com>
AuthorDate: Thu Jan 14 15:22:07 2016 -0800
Commit:     Linus Torvalds <torvalds@linux-foundation.org>
CommitDate: Thu Jan 14 16:00:49 2016 -0800

    mm: rework virtual memory accounting
    
    When inspecting a vague code inside prctl(PR_SET_MM_MEM) call (which
    testing the RLIMIT_DATA value to figure out if we're allowed to assign
    new @start_brk, @brk, @start_data, @end_data from mm_struct) it's been
    commited that RLIMIT_DATA in a form it's implemented now doesn't do
    anything useful because most of user-space libraries use mmap() syscall
    for dynamic memory allocations.
    
    Linus suggested to convert RLIMIT_DATA rlimit into something suitable
    for anonymous memory accounting.  But in this patch we go further, and
    the changes are bundled together as:
    
     * keep vma counting if CONFIG_PROC_FS=n, will be used for limits
     * replace mm->shared_vm with better defined mm->data_vm
     * account anonymous executable areas as executable
     * account file-backed growsdown/up areas as stack
     * drop struct file* argument from vm_stat_account
     * enforce RLIMIT_DATA for size of data areas
    
    This way code looks cleaner: now code/stack/data classification depends
    only on vm_flags state:
    
     VM_EXEC & ~VM_WRITE            -> code  (VmExe + VmLib in proc)
     VM_GROWSUP | VM_GROWSDOWN      -> stack (VmStk)
     VM_WRITE & ~VM_SHARED & !stack -> data  (VmData)
    
    The rest (VmSize - VmData - VmStk - VmExe - VmLib) could be called
    "shared", but that might be strange beast like readonly-private or VM_IO
    area.
    
     - RLIMIT_AS            limits whole address space "VmSize"
     - RLIMIT_STACK         limits stack "VmStk" (but each vma individually)
     - RLIMIT_DATA          now limits "VmData"
    
    Signed-off-by: Konstantin Khlebnikov <koct9i@gmail.com>
    Signed-off-by: Cyrill Gorcunov <gorcunov@openvz.org>
    Cc: Quentin Casasnovas <quentin.casasnovas@oracle.com>
    Cc: Vegard Nossum <vegard.nossum@oracle.com>
    Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Willy Tarreau <w@1wt.eu>
    Cc: Andy Lutomirski <luto@amacapital.net>
    Cc: Kees Cook <keescook@google.com>
    Cc: Vladimir Davydov <vdavydov@virtuozzo.com>
    Cc: Pavel Emelyanov <xemul@virtuozzo.com>
    Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

Created attachment 96751 [details]
Remove setting zero RLIMIT_DATA

The simplest seems to be to just remove the zero data rlimit. It also gets rid of some nasty hacks in our execv and spawn wrappers.

Hi Mark, Thank you for providing the fix also for Solaris.
It works ok on Solaris 12, regression tests passed.

(In reply to Ivo Raisr from comment #7)
> Hi Mark, Thank you for providing the fix also for Solaris.
> It works ok on Solaris 12, regression tests passed.

Thanks for testing!

This has now been committed as valgrind svn r15766.