Bug 481203

Summary: mmap with MAP_STACK not handled correctly on FreeBSD
Product: [Developer tools] valgrind Reporter: Paul Floyd <pjfloyd>
Component: generalAssignee: Paul Floyd <pjfloyd>
Status: RESOLVED FIXED    
Severity: normal    
Priority: NOR    
Version: 3.22 GIT   
Target Milestone: ---   
Platform: Other   
OS: FreeBSD   
Latest Commit: Version Fixed In:
Sentry Crash Report:

Description Paul Floyd 2024-02-11 10:30:34 UTC
I've been debugging a failure with the memcheck descr_belowsp on FreeBSD. I tried running with --sanity-level=3 and got a failure. The problem occurs when allocating a stack for a thread (so quite common). I've reduced this to a small test that just allocates a stack:

#include <sys/mman.h>
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>

int main(void)
{
   void* addr;
   addr = mmap((void*)0x200000000000, 1024U*1024U, PROT_READ|PROT_WRITE, MAP_STACK, -1, 0U);
   if (addr == MAP_FAILED)
   {
      perror("mmap failed:");
      exit(1);
   }
   fprintf(stderr, "addr %p\n", addr);

   //sleep(60);
}

paulf> valgrind --sanity-level=3 ./stack
==88520== Memcheck, a memory error detector
==88520== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==88520== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==88520== Command: ./stack
==88520== 
--88520:0: aspacem segment mismatch: V's seg 1st, kernel's 2nd:
--88520:0: aspacem  52: anon 200000000000-2000000fffff 1048576 rw--- SmFixed d=0x000 i=0       o=0       (-1,-1) (none)
--88520:0: aspacem ...: .... 200000000000-2000000dffff  917504 ---.. ....... d=0x000 i=0       o=0       (.) m=. (none)
--88520:0: aspacem sync check at m_aspacemgr/aspacemgr-linux.c:2282 (Bool vgPlain_am_notify_client_mmap(Addr, SizeT, UInt, UInt, Int, Off64T)): FAILED

If I put back that sleep, turn off ASLR
paulf> sudo sysctl kern.elf64.aslr.enable=0
and then look at the memory map

paulf> procstat -v 1865                    

 1865     0x200000000000     0x2000000e0000 ---    0    0   0   0 ----- gd 
 1865     0x2000000e0000     0x200000100000 rw-    0    0   0   0 ---D- -- 

The mmap has resulted in two mappings

1M - 128k of guard page
128k of stack

The man page describes something like this, where it talks about security.bsd.stack_guard_page controlling the size of the guard page. On my system that sysctl is 1 (4k). So I think that the man page is wrong. When I was debugging descr_belowsp I saw 2 guard pages, the big one as above plus another 4k mapping.

As the memory gets touched, more 128k blocks get mapped as stack.

Adding to the test code

    int* pi = addr;
   size_t stack_begin = 1024U * 1024U / 4U - 1U;
   pi[stack_begin] = 1;

   pi[stack_begin - 32*1024] = 1;
   sleep(60);

changes the mapping

49478     0x200000000000     0x2000000c0000 ---    0    0   0   0 ----- gd 
49478     0x2000000c0000     0x2000000e0000 rw-    1    1   1   0 ---D- sw 
49478     0x2000000e0000     0x200000100000 rw-    1    1   1   0 ---D- sw 

All of that seems to be happening in the kernel. Valgrind doesn't see any extra mmaps or mprotect.
Comment 1 Paul Floyd 2024-02-11 19:29:57 UTC
I don't know how I can handle this. I've been looking at modifying parse_procselfmaps to handle the pattern

guard mapping
one or more 132k stack mappings

It's hard to tell normal mappings from these split stack mappings. parse_procselfmaps doesn't know anything about nsegments. It's just looking for mappings one at a time that it sends to sync_check_mapping_callback. And sync_check_mapping_callback mjust sees mappings one at a time.
Comment 2 Paul Floyd 2024-02-28 20:05:03 UTC
I'm closing this. I pushed a fix that works some of the time, plus a warning the --sanity-level=3 and above mail fail on FreeBSD.