Created attachment 90879 [details] valgrind mysqld log file Platform: 64bit Ubuntu 14.04 Valgrind: 3.10.1 compiled from sources MySQL: 5.6.22 compiled from sources Running valgrind on mysqld and sending a vgdb help commands makes valgrind/mysqld hang. The attached log files have been generated as follows: # Running valgrind on mysqld sudo .../valgrind-3.10.1/installation/bin/valgrind -v -v -v -d -d -d --log-file=mysqld_sudo_verbose.log bin/mysqld # Running vgdb help sudo .../valgrind-3.10.1/installation/bin/vgdb -d -d -d --pid=9156 help &> vgdb_sudo_verbose.log The lines beginning with ==9156== Remote side has terminated connection. GDBserver will reopen the connection. at the end of the mysqld_sudo_verbose.log file resulted from stopping the vgdb process. The behavior is independent from the valgrind tool being used (similar results with callgrind).
Created attachment 90880 [details] vgdb help log file
It looks like the -v -v -v -d -d -d for Valgrind did not work (as the valgrind log file contains not the expected trace). I guess you have used VALGRIND_OPTS to pass these. -v and -d are not working properly when given via VALGRIND_OPTS or other techniques: the only working technique is to pass them directly on the command line launching valgrind. So, would be nice to redo the trial, but giving -v -v -v -d -d -d to valgrind 'directly as argument'. In any case, we see in the trace the below: ==9156== error 13 Permission denied ==9156== open fifo /tmp/vgdb-pipe-from-vgdb-to-9156-by-root-on-??? ==9156== valgrind: fatal error: vgdb FIFO cannot be opened. ==9156== could not unlink /tmp/vgdb-pipe-from-vgdb-to-9156-by-root-on-??? ==9156== could not unlink /tmp/vgdb-pipe-to-vgdb-from-9156-by-root-on-??? ==9156== could not unlink /tmp/vgdb-pipe-shared-mem-vgdb-9156-by-root-on-??? So, it looks like one way or another, the FIFO file (created on the valgrind side) cannot be opened and cannot be unlinked. So, would be nice to do an 'ls -l' of this file, to see the owner and the protections. Valgrind creates the FIFOs with: m = VG_(mknod) (nod, VKI_S_IFIFO|0600, 0); so these shoud be rw by the user creating it. Maybe mysqld starts initially under a certain user (root ?) and then switches to the mysql user using setuid or something like that ? So, if initially valgrind starts as root (as shown by the FIFO name) but then becomes mysql, then mysql will not have rw access to the FIFO, leading to the indicated error ?
A trial to do (if that is possible) would be to launch mysqld under Valgrind, but directly using your user or mysql user, without using sudo. It is probably the use of sudo that introduces the permission problem.
Thanks again for your support, Philippe. I also noticed the permission error but couldn't really explain to myself where it came from. I was under the impression that when running mysqld without the --user argument it would run under the user that starts the process. But, in fact, as you suspected, I did an strace on mysqld and does use setuid to switch to the mysql user. I assume this is configured in some configuration file I was not aware of. I ran both valgrind with mysql and vgdb under the mysql user and it works. I assume, this bug can get closed then! Thanks! One more comment: I did not use VALGRIND_OPTS, but passed -v -v -v -d -d -d directly as program arguments as posted in the original bug description. I have no idea why this did not work.
What is the consequence here? A vgdb hanging is not the best user experience. The user should be notified that something is going wrong. Ideas: * timeout in vgdb, with telling about possible setuid problems? * catch setuid in valgrind and change permissions of the vgdb pipes so that vgdb keeps working after the setuid call * ? Josef
(In reply to Josef Weidendorfer from comment #5) > What is the consequence here? A vgdb hanging is not the best user experience. Yes that is not nice. (this is one way to shoot yourself in the foot. There are others, e.g. launching vgdb in a context which differs by user or by TMPDIR or HOSTNAME or ...). > The user should be notified that something is going wrong. IIUC, there was an error message in the valgrind log e.g. EPERM or something like that. > > Ideas: > * timeout in vgdb, with telling about possible setuid problems? vgdb has a timeout arg: --cmd-time-out (default 99999999) tells vgdb to exit if the found Valgrind gdbserver has not processed a command after number seconds But what value do we put ? It must be low enough to be of some use when something does not work, but large enough to not make commands taking a long time to fail artificially. Also, vgdb cannot guess what goes wrong at valgrind side, so it would just report it has timed out for an unknown reason. > * catch setuid in valgrind and change permissions of the vgdb pipes so that > vgdb keeps working after the setuid call We could effectively track the effective and/or real user id change, and update the FIFOs owner. We would however have to do that before the setuid/seteuid syscall succeeds (otherwise we cannot chown anymore). And if after the syscall fails, then no way to repair. Also, I am wondering what would happen for the mmaped shared memory. The user launching vgdb would need to know with which user to launch vgdb, as that would have to match the current user id that the process is using. This is not very trivial, and valgrind already does not work properly with setuid executables. In this case, I do not understand why mysqld cannot be directly launched with the correct user : why require root access to launch mysqld under mysql user? Maybe we need to understand why this is needed?
A possibly related issue is observed with valgrind (initially running as root), instrumenting a target program that does a setuid (down to an unprivileged user). At valgrind shutdown, we get a few "could not unlink /tmp/vgdb....." errors (due to -EPERM).