Bug 319307

Summary:	Patches to let test suite run in parallel
Product:	[Developer tools] valgrind	Reporter:	Thomas Rast <trast>
Component:	general	Assignee:	Julian Seward <jseward>
Status:	REPORTED ---
Severity:	wishlist	CC:	mark, sam
Priority:	NOR
Version First Reported In:	3.9.0.SVN
Target Milestone:	---
Platform:	Compiled Sources
OS:	Linux
Latest Commit:		Version Fixed/Implemented In:
Sentry Crash Report:
Attachments:	Mass-edit massif tests to use unique output names {call,cache}grind/tests: use 'rm -f' to clean up Execute tests after scanning, from toplevel directory Implement threaded regression tests regtests: use forks instead of threads

Description Thomas Rast 2013-05-04 10:23:52 UTC

I got annoyed with the test suite being single-threaded, hence these patches. There are some preliminary "fixes" needed to make the tests work in parallel, because they would otherwise trample over their files or such. Then there is one big patch that would work in theory, but doesn't because Perl is currently broken, and a final band-aid to work around that for now.

You'll need to install the 'forks' module from CPAN for the band-aid to work. Otherwise it just ignores any request for threads.

BEFORE:
$ time perl tests/vg_regtest .
== 636 tests, 1 stderr failure, 0 stdout failures, 0 stderrB failures, 0 stdoutB failures, 0 post failures ==
./memcheck/tests/origin5-bz2             (stderr)

real    6m17.769s
user    4m14.089s
sys     0m33.888s

AFTER:
$ time perl tests/vg_regtest .
== 636 tests, 1 stderr failure, 0 stdout failures, 0 stderrB failures, 0 stdoutB failures, 0 post failures ==
./memcheck/tests/origin5-bz2             (stderr)

real    6m34.197s
user    4m18.843s
sys     0m38.544s

$ time perl tests/vg_regtest -j15 .
== 636 tests, 1 stderr failure, 0 stdout failures, 0 stderrB failures, 0 stdoutB failures, 0 post failures ==
./memcheck/tests/origin5-bz2             (stderr)

real    0m54.682s
user    6m46.404s
sys     0m41.829s

So about 7.3x speedup on a 6-core i7. (I stubbed out the mchelp test here, because it hangs. The origin5-bz2 failure seems to be a glibc-too-new issue.)

Reproducible: Always

Comment 1 Thomas Rast 2013-05-04 10:24:49 UTC

Created attachment 79692 [details]
Mass-edit massif tests to use unique output names

Comment 2 Thomas Rast 2013-05-04 10:25:06 UTC

Created attachment 79693 [details]
{call,cache}grind/tests: use 'rm -f' to clean up

Comment 3 Thomas Rast 2013-05-04 10:25:20 UTC

Created attachment 79694 [details]
Execute tests after scanning, from toplevel directory

Comment 4 Thomas Rast 2013-05-04 10:25:39 UTC

Created attachment 79695 [details]
Implement threaded regression tests

Comment 5 Thomas Rast 2013-05-04 10:26:20 UTC

Created attachment 79696 [details]
regtests: use forks instead of threads

Comment 6 Thomas Rast 2013-05-04 10:28:26 UTC

PS: the third patch actually fixes a directory-changing related bug by no longer changing dirs. If you do this in the current test suite:
$ perl tests/vg_regtest none/tests/amd64 memcheck
it fails to run any memcheck tests:
-- Finished tests in none/tests/amd64 ----------------------------------
`memcheck' neither a directory nor a readable test file/name

This happens because the recursion code uses 'chdir("..")' to go back one level, which is not sufficient if it was called to go more than one level deep (as in the none/tests/amd64 case).