Bug 383811

Valgrind should support Apple's macOS High Sierra (10.13) once a public, general release is made available in late 2017.

This is a meta bug for tracking any issues raised during the developer beta and public beta review process with Valgrind.

Reproducible: Always

Created attachment 107785 [details]
HACK: initial macOS High Sierra (10.13) support / Apple Clang 5.1+

Attached is an initial, incomplete patch that does two things:
* Supports macOS High Sierra (10.13) as an OS version; and
* Reworks the Apple Clang detection logic so all versions 5.1+ are supported

I'd welcome testing as I haven't done any yet. Note that in addition to installing the latest pre-release Xcode on macOS High Sierra (10.13), interested testers will also likely need to install the latest pre-release Commandline Tools package as well.

Patch won't land in valgrind as-is, but it is a start.

With valgrind trunk and the patch in comment 1, after having run autogen.sh in the source tree: configure && make work, and the resulting valgrind appears to work. For example, it correctly determines on invalid write in a free'd block. It also correctly detects unfree'd memory when it encounters it.

I have found, so far, two issues:

1. valgrind displays a warning about libsystem_kernel for any executable I run (valid or invalid):

==90520== Syscall param msg->desc.port.name points to uninitialised byte(s)
==90520==    at 0x100590E76: mach_msg_trap (in /usr/lib/system/libsystem_kernel.dylib)
==90520==    by 0x10059038F: mach_msg (in /usr/lib/system/libsystem_kernel.dylib)
==90520==    by 0x100589F3D: task_set_special_port (in /usr/lib/system/libsystem_kernel.dylib)
==90520==    by 0x1005F701A: _os_trace_create_debug_control_port (in /usr/lib/system/libsystem_trace.dylib)
==90520==    by 0x1005F72D0: _libtrace_init (in /usr/lib/system/libsystem_trace.dylib)
==90520==    by 0x1000BD9CC: libSystem_initializer (in /usr/lib/libSystem.B.dylib)
==90520==    by 0x10000415F: dyld3::kdebug_trace_dyld_duration(unsigned int, unsigned long long, unsigned long long, void ( block_pointer)()) (in /usr/lib/dyld)
==90520==    by 0x100019A09: ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==90520==    by 0x100019C39: ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==90520==    by 0x10001516F: ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) (in /usr/lib/dyld)
==90520==    by 0x100015102: ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) (in /usr/lib/dyld)
==90520==    by 0x1000142A5: ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) (in /usr/lib/dyld)
==90520==  Address 0x10489f19c is on thread 1's stack
==90520==  in frame #2, created by task_set_special_port (???:)

------------------------------------------------------------------------------------------------------------

2. When running "valgrind --leak-check=full", it issues a warning about dyld:

==90558== 72 bytes in 3 blocks are possibly lost in loss record 26 of 43
==90558==    at 0x1000AB542: calloc (vg_replace_malloc.c:714)
==90558==    by 0x1006C07E2: map_images_nolock (in /usr/lib/libobjc.A.dylib)
==90558==    by 0x1006D37DA: objc_object::sidetable_retainCount() (in /usr/lib/libobjc.A.dylib)
==90558==    by 0x100007C64: dyld::notifyBatchPartial(dyld_image_states, bool, char const* (*)(dyld_image_states, unsigned int, dyld_image_info const*), bool, bool) (in /usr/lib/dyld)
==90558==    by 0x100007E39: dyld::registerObjCNotifiers(void (*)(unsigned int, char const* const*, mach_header const* const*), void (*)(char const*, mach_header const*), void (*)(char const*, mach_header const*)) (in /usr/lib/dyld)
==90558==    by 0x10022284D: _dyld_objc_notify_register (in /usr/lib/system/libdyld.dylib)
==90558==    by 0x1006C0075: _objc_init (in /usr/lib/libobjc.A.dylib)
==90558==    by 0x1001ACCE0: _os_object_init (in /usr/lib/system/libdispatch.dylib)
==90558==    by 0x1001ACCC7: libdispatch_init (in /usr/lib/system/libdispatch.dylib)
==90558==    by 0x1000BD9C2: libSystem_initializer (in /usr/lib/libSystem.B.dylib)
==90558==    by 0x10000415F: dyld3::kdebug_trace_dyld_duration(unsigned int, unsigned long long, unsigned long long, void ( block_pointer)()) (in /usr/lib/dyld)
==90558==    by 0x100019A09: ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) (in /usr/lib/dyld)


------------------------------------------------------------------------------------------------------------

3. Trying to run /bin/bash under valgrind leads to this error:

--90616-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option
--90616-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 2 times)
--90616-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 4 times)
--90616-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 8 times)
==90616== Syscall param mach_msg("rcv_name") contains uninitialised byte(s)
==90616==    at 0x10076AE76: mach_msg_trap (in /usr/lib/system/libsystem_kernel.dylib)
==90616==    by 0x10076A38F: mach_msg (in /usr/lib/system/libsystem_kernel.dylib)
==90616==    by 0x1007D86F2: _os_trace_prefs_and_mode_refresh_slow (in /usr/lib/system/libsystem_trace.dylib)
==90616==    by 0x1007CEF06: _os_activity_create_addr (in /usr/lib/system/libsystem_trace.dylib)
==90616==    by 0x10055D219: ds_user_byuid (in /usr/lib/system/libsystem_info.dylib)
==90616==    by 0x10055CD5B: si_user_byuid (in /usr/lib/system/libsystem_info.dylib)
==90616==    by 0x10055CE3E: search_item_bynumber (in /usr/lib/system/libsystem_info.dylib)
==90616==    by 0x10055CD96: search_user_byuid (in /usr/lib/system/libsystem_info.dylib)
==90616==    by 0x10055CD5B: si_user_byuid (in /usr/lib/system/libsystem_info.dylib)
==90616==    by 0x10055C27E: getpwuid (in /usr/lib/system/libsystem_info.dylib)
==90616==    by 0x10002B958: setupvals (in /bin/zsh)
==90616==    by 0x10002D0C5: zsh_main (in /bin/zsh)
==90616== 
eq_SyscallStatus:
  {78 0 43}
  {78 0 40}

valgrind: m_syswrap/syswrap-main.c:438 (Bool eq_SyscallStatus(UInt, SyscallStatus *, SyscallStatus *)): the 'impossible' happened.

host stacktrace:
==90616==    at 0x2580523BB: ???
==90616==    by 0x25805274C: ???
==90616==    by 0x258052723: ???
==90616==    by 0x2580EB9D4: ???
==90616==    by 0x2580EAFB9: ???
==90616==    by 0x2580E91E0: ???
==90616==    by 0x2580E79A0: ???
==90616==    by 0x2580F985E: ???

sched status:
  running_tid=1

Thread 1: status = VgTs_Runnable (lwpid 771)
==90616==    at 0x10076AEFA: mach_generate_activity_id (in /usr/lib/system/libsystem_kernel.dylib)
==90616==    by 0x1003A7F57: _voucher_activity_id_allocate_slow (in /usr/lib/system/libdispatch.dylib)
==90616==    by 0x1003A7209: voucher_activity_create_with_data (in /usr/lib/system/libdispatch.dylib)
==90616==    by 0x1007CEFE2: _os_activity_create_addr (in /usr/lib/system/libsystem_trace.dylib)
==90616==    by 0x10055D219: ds_user_byuid (in /usr/lib/system/libsystem_info.dylib)
==90616==    by 0x10055CD5B: si_user_byuid (in /usr/lib/system/libsystem_info.dylib)
==90616==    by 0x10055CE3E: search_item_bynumber (in /usr/lib/system/libsystem_info.dylib)
==90616==    by 0x10055CD96: search_user_byuid (in /usr/lib/system/libsystem_info.dylib)
==90616==    by 0x10055CD5B: si_user_byuid (in /usr/lib/system/libsystem_info.dylib)
==90616==    by 0x10055C27E: getpwuid (in /usr/lib/system/libsystem_info.dylib)
==90616==    by 0x10002B958: setupvals (in /bin/zsh)
==90616==    by 0x10002D0C5: zsh_main (in /bin/zsh)
==90616==    by 0x1003FA144: start (in /usr/lib/system/libdyld.dylib)

Hi, valgrind folks! Mac Homebrew maintainer here.

Would you consider splitting out the Xcode 9/LLVM 9.x support from the macOS 10.13 High Sierra support? Xcode 9 has been released, and is now the current supported compiler for macOS 10.12 Sierra in addition to 10.13 High Sierra. This means that for macOS users keeping their dev tools up to date, valgrind 3.13.0 and trunk are no longer building on macOS 10.12.

Depending on your testing policy, it looks like the compiler version check in `configure` might be the only thing that would need to be updated (as in the patch supplied by Rhys Kidd), since the 10.12 libraries should stay the same with the new Xcode.

(Had this reported by a user here: https://github.com/Homebrew/homebrew-core/pull/18399)

Andrew,
I've split out and landed in valgrind git master the Xcode 9/LLVM 9.x support.

The broader macOS 10.13 support that this bug report tracks will land separately once we've ironed out a few more of the issues against the final public release (which I understand will be released tomorrow).

Created attachment 107999 [details]
HACK: initial macOS High Sierra (10.13) support

(In reply to Rhys Kidd from comment #5)
> Andrew,
> I've split out and landed in valgrind git master the Xcode 9/LLVM 9.x
> support.
> 
> The broader macOS 10.13 support that this bug report tracks will land
> separately once we've ironed out a few more of the issues against the final
> public release (which I understand will be released tomorrow).

Thank you Rhys! Building from trunk under Xcode 9 is now working for us.

Created attachment 108101 [details]
initial macOS High Sierra (10.13) support v2

So Apple released the xnu source code very quickly this time after the public release of macOS 10.13 High Sierra.

I would appreciate a few users to test the revised patch (attached) against valgrind master. Please report whether there are any issues with simple programs like 'ls' or 'date' on macOS 10.13.

If this testing doesn't identify any major issues, I'm willing to land this preliminary support in master.

(In reply to Rhys Kidd from comment #9)
> I would appreciate a few users to test the revised patch (attached) against
> valgrind master. Please report whether there are any issues with simple
> programs like 'ls' or 'date' on macOS 10.13.

Cloned the repo, applied the patched, and it built fine.
Valgrind is working as expected for me on /bin/ls and /bin/date.

However, when trying something more complex (valgrind /usr/bin/gcc -v), I get the same issue as paragraph 3 of comment #3 here (valgrind: m_syswrap/syswrap-main.c:438 (Bool eq_SyscallStatus(UInt, SyscallStatus *, SyscallStatus *)): the 'impossible' happened.)

(In reply to Adrien Bertrand from comment #10)
> However, when trying something more complex (valgrind /usr/bin/gcc -v), I
> get the same issue as paragraph 3 of comment #3 here (valgrind:
> m_syswrap/syswrap-main.c:438 (Bool eq_SyscallStatus(UInt, SyscallStatus *,
> SyscallStatus *)): the 'impossible' happened.)

Can you add '--trace-syscalls=yes' to your valgrind and report the lines prior to the one that contains eq_SyscallStatus?

(In reply to Rhys Kidd from comment #11)
> Can you add '--trace-syscalls=yes' to your valgrind and report the lines
> prior to the one that contains eq_SyscallStatus?

There you go:

SYSCALL[38163,1](unix:399) sys_close ( 3 )[sync] --> Success(0x0:0x0)
SYSCALL[38163,1](mach:43) unimplemented (by the kernel) syscall: mach:43! (ni_syscall)
 --> [pre-fail] Failure(0x4e) eq_SyscallStatus:
  {78 0 43}
  {78 0 40}

(In reply to Adrien Bertrand from comment #12)
> (In reply to Rhys Kidd from comment #11)
> > Can you add '--trace-syscalls=yes' to your valgrind and report the lines
> > prior to the one that contains eq_SyscallStatus?
> 
> There you go:
> 
> SYSCALL[38163,1](unix:399) sys_close ( 3 )[sync] --> Success(0x0:0x0)
> SYSCALL[38163,1](mach:43) unimplemented (by the kernel) syscall: mach:43!
> (ni_syscall)
>  --> [pre-fail] Failure(0x4e) eq_SyscallStatus:
>   {78 0 43}
>   {78 0 40}

Same thing trying valgrind on bash (syscall 43).
Isn't that supposed to be getegid, according to https://github.com/apple/darwin-xnu/blob/0a798f6738bc1db01281fc08ae024145e84df927/bsd/kern/syscalls.master#L93 ?

Actually, there's a whole separate listing of Mach syscalls which have negative numerals. Often they are presented as positive numbers in output, which lead to the confusion here.

Mach syscall 43 was re-added in macOS 10.12 as mach_generate_activity_id. Valgrind does not yet support it.

Anyway, tracking that unrelated issue here: https://bugs.kde.org/show_bug.cgi?id=385279. Please comment further on that issue at the newly created bug.

As the macOS 10.13 preliminary patch is otherwise passing basic smoke tests, I'll land it in valgrind master later tonight. Thanks for your testing assistance!

Comment on attachment 108101 [details]
initial macOS High Sierra (10.13) support v2

Preliminary support for Darwin 17.x (macOS 10.13) has now landed in Valgrind master with 1ce04c35c2ebbc8ea3c2b38ba69daa9dd40cde35.

Please create a new, standalone bug report for any subsequent issues that you may encounter, this bug report will be used as a meta one to track each blocker.

*** Bug 385910 has been marked as a duplicate of this bug. ***

High Sierra 10.13.1.

I was very lost on the configure step from the tarball (compilers not found?), but having pulled from the latest master (HEAD: 40f0364e1e4c9d1d4d34c99a22ff775b5199974b), I was able to complete all the listed steps in the README.

However at the end I don't have an installed binary.

```
$ valgrind ls -l
-bash: valgrind: command not found
```

These were my steps.
1. git clone git://sourceware.org/git/valgrind.git
2. brew install automake
3. cd valgrind
4. ./autogen.sh
5. ./configure --prefix=/usr/local/bin/
6. make
7. make install
8. valgrind ls -l
9. I also tried `sudo make install` because make install didn't work. No difference.

Let me know if you need me to try anything else.

This was my mistake. valgrind ended up inside of /usr/local/bin/bin.

Hi all,

thanks to @Ben Wiley's instruction, I've finally managed to install Valgrind on OSX 10.13.2 (17C205).
When I try to execute it, I get the following message :

==28351== Memcheck, a memory error detector
==28351== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==28351== Using Valgrind-3.14.0.GIT and LibVEX; rerun with -h for copyright info
==28351== Command: ls -la
==28351==
--28351-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option
--28351-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 2 times)
--28351-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 4 times)
--28351-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 8 times)
eq_SyscallStatus:
  {78 0 43}
  {78 0 40}

valgrind: m_syswrap/syswrap-main.c:438 (Bool eq_SyscallStatus(UInt, SyscallStatus *, SyscallStatus *)): the 'impossible' happened.

host stacktrace:
==28351==    at 0x2580540EB: ???
==28351==    by 0x25805448C: ???
==28351==    by 0x258054463: ???
==28351==    by 0x2580EED64: ???
==28351==    by 0x2580EE349: ???
==28351==    by 0x2580EC570: ???
==28351==    by 0x2580EAD23: ???
==28351==    by 0x2580FCBEE: ???

sched status:
  running_tid=1

Thread 1: status = VgTs_Runnable (lwpid 771)
==28351==    at 0x1005B8846: mach_generate_activity_id (in /usr/lib/system/libsystem_kernel.dylib)
==28351==    by 0x1001F4E7A: _voucher_activity_id_allocate_slow (in /usr/lib/system/libdispatch.dylib)
==28351==    by 0x1001F412C: voucher_activity_create_with_data (in /usr/lib/system/libdispatch.dylib)
==28351==    by 0x10061D042: _os_activity_create_addr (in /usr/lib/system/libsystem_trace.dylib)
==28351==    by 0x1003AA1D9: ds_user_byuid (in /usr/lib/system/libsystem_info.dylib)
==28351==    by 0x1003A9D1B: si_user_byuid (in /usr/lib/system/libsystem_info.dylib)
==28351==    by 0x1003A9DFE: search_item_bynumber (in /usr/lib/system/libsystem_info.dylib)
==28351==    by 0x1003A9D56: search_user_byuid (in /usr/lib/system/libsystem_info.dylib)
==28351==    by 0x1003A9D1B: si_user_byuid (in /usr/lib/system/libsystem_info.dylib)
==28351==    by 0x1003A923E: getpwuid (in /usr/lib/system/libsystem_info.dylib)
==28351==    by 0x10000B6E5: getuser (in /usr/local/opt/coreutils/libexec/gnubin/ls)
==28351==    by 0x100006621: format_user_width (in /usr/local/opt/coreutils/libexec/gnubin/ls)
==28351==    by 0x100004034: gobble_file (in /usr/local/opt/coreutils/libexec/gnubin/ls)
==28351==    by 0x100003262: main (in /usr/local/opt/coreutils/libexec/gnubin/ls)


Note: see also the FAQ in the source distribution.
It contains workarounds to several common problems.
In particular, if Valgrind aborted or crashed after
identifying problems in your program, there's a good chance
that fixing those problems will prevent Valgrind aborting or
crashing, especially if it happened in m_mallocfree.c.

If that doesn't help, please report this bug to: www.valgrind.org

In the bug report, send all the above text, the valgrind
version, and what OS and version you are using.  Thanks.

Hope it can helps...

Created attachment 111941 [details]
vki_semid64_ds type incomplete when compiling Mac OS 10.13.4

I got the following errors when compiling valgrind on Mac OS 10.13.4:

m_syswrap/syswrap-generic.c:1797:26: error: variable has incomplete type 'struct vki_semid64_ds'
   struct vki_semid64_ds buf;
                         ^
m_syswrap/syswrap-generic.c:1797:11: note: forward declaration of 'struct vki_semid64_ds'
   struct vki_semid64_ds buf;
          ^
m_syswrap/syswrap-generic.c:1798:8: error: no member named 'buf64' in 'union semun'
   arg.buf64 = &buf;
   ~~~ ^
2 errors generated.

See:

https://github.com/apple/darwin-xnu/blob/master/bsd/sys/sem.h

https://wiki.freebsd.org/Valgrind

Thanks -- that looks fairly straight forward to fix. Only added in a git version of Valgrind, so would like to resolve well before the next release reaches users.

A variant for the build break related to "vki_semid64_ds" on macOS has been committed to Git in 06cb991bc.

Please update your local repository to at least that version.

so I just cloned and built git://sourceware.org/git/valgrind.git on 10.13.4, and when running valgrind against any binary I get:

amohr@ip-192-168-16-6:~/dev/third_party/valgrind/coregrind$ ./valgrind `which touch`
valgrind: mmap-FIXED(0x7fff5f400000, 8388608) failed in UME (load_unixthread1) with error 22 (Invalid argument).

so it seems there's something else not working right

(In reply to Alex from comment #23)
> so I just cloned and built git://sourceware.org/git/valgrind.git on 10.13.4,
> and when running valgrind against any binary I get:
> 
> amohr@ip-192-168-16-6:~/dev/third_party/valgrind/coregrind$ ./valgrind
> `which touch`
> valgrind: mmap-FIXED(0x7fff5f400000, 8388608) failed in UME
> (load_unixthread1) with error 22 (Invalid argument).
> 
> so it seems there's something else not working right

hmm works when I install via brew install valgrind --HEAD so can ignore this

Homebrew maintainers speak about deleting valgrind formula: https://github.com/Homebrew/homebrew-core/pull/30490

Is there a release including the fix discussed here in sight?

Homebrew maintainers speak about deleting valgrind formula: https://github.com/Homebrew/homebrew-core/pull/30490

Is there a release including the fix discussed here in sight?

Hi -- Preliminary work is being undertaken to prepare for Valgrind's next numbered release. Unfortunately I don't have a specific date that I can communicate, but it is in train.
Appreciate this might not be the immediate answer you wanted -- but thought best to get it out there.

I'm hitting this today on macOS Catalina 10.15.2

% valgrind $(which touch)
valgrind: mmap-FIXED(0x7fff5f400000, 8388608) failed in UME (load_unixthread1) with error 22 (Invalid argument).


This is using 3.13.0 from homebrew. Homebrew won't let me update valgrind

% brew upgrade valgrind
==> Upgrading 1 outdated package:
valgrind 3.13.0 -> 3.15.0
valgrind: This formula either does not compile or function as expected on macOS
versions newer than High Sierra due to an upstream incompatibility.
Error: valgrind: An unsatisfied requirement failed this build.
==> Checking for dependents of upgraded formulae...
==> No dependents found!
 

What's my way forward? Is this fixed yet?