Created attachment 59596 [details] run-insns.c Version: unspecified OS: Linux As discussed on valgrind-developers in a thread with subject "VEX unit testing". Attached is the run-insns.c which supports the .orig file format. The supported input format is work in progress. All architectures and virtually all VEX knobs are supported now. Reproducible: Didn't try
I've looked at vg_regtest and it is not really suitable for what I have in mind for VEX unit testing. So I've come up with something that is more flexible but still simple (~500 lines). The key features are: (1) Every testcase is a perl script. (2) Any input needed to run the testcase is generated by the testcase itself. (3) There can be more than one invocation of some tool within a testcase. (I use the term "run" to mean tool invocation.) (4) The result of a run is stored in a hash table. (5) There is a *single* "golden log" file which contains all the results of all runs that occur in the testcase. The log file is a perl script, so we can load and evaluate it easily for comparison. (6) Since the testcase is a perl script any filtering of stdout/stderr results can be done easily using standard perl programming. Example: my $file = test::write_file("FLOGR", <<'EOF'); A5 4F DE AD # R4 = 0xdead B9 83 00 14 # R1 = leftmost-bit(R4); R2 = R4 with leftmost-bit=0 00 00 EOF my $cmd = "$test::top_of_tree/auxprogs/run-insns --s390x $file --trace-flags=00000010"; test::execute($cmd); First, the testcase creates a file FLOGR with a few insns. Secondly, it builds up the command to execute. Here $test::top_of_tree denotes the root directory of VEX. Thirdly, the command is executed and the usual results: exit-code, stdout and stderr output, and a possibly caught signal are recorded behind the scenes. The log file for above testcase looks like this: $golden{run1} = { 'command' => '.../VEX/auxprogs/run-insns --s390x FLOGR --trace-flags=00000010', 'rc' => 0, 'sig' => 0, 'stderr' => '', 'stdout' => <<'EOF', ------------------------ Register-allocated code ------------------------ 0 v-loadi %r1,0xDEAD 8 bytes 1 v-store %r1,guest_r4 8 bytes 2 v-loadi %r1,48 8 bytes 3 v-store %r1,guest_r1 8 bytes 4 v-loadi %r1,0x5EAD 8 bytes 5 v-store %r1,guest_r2 8 bytes 6 v-loadi %r1,2 8 bytes 7 v-store %r1,guest_CC_OP 8 bytes 8 v-loadi %r1,0xDEAD 8 bytes 9 v-store %r1,guest_CC_DEP1 8 bytes 10 v-loadi %r1,0 8 bytes 11 v-store %r1,guest_CC_DEP2 8 bytes 12 v-loadi %r1,0 8 bytes 13 v-store %r1,guest_CC_NDEP 8 bytes 14 v-loadi %r1,8 8 bytes 15 v-store %r1,guest_IA 8 bytes 16 if (always) goto guest_r0 0 bytes EOF }; Even though it's perl it's highly readable and straight forward which is relevant for ease of proof-reading. If we had more than one run the log file would contain $golden{run2}, $golden{run3}, etc ... One thing that I would like to do is to have testcases for certain complex instructions and pipe them through VEX assuming various optimisation levels and hardware capabilities. I'm interested in whether or not they run into an assert. That is easy to do: my $file = test::write_file("FLOGR", <<'EOF'); B9 83 00 14 00 00 EOF for my $level (0, 1, 2) { for my hwcaps ("--ldisp", "--eimm", "--gie") { my $cmd = "$test::top_of_tree/auxprogs/run-insns " . "--s390x $file $hwcaps --opt-level=$level"; my $log = test::execute($cmd); $log->{"stdout"} = undef; # we're not interested } } So test::execute returns a reference to a hash that contains the results of executing the command. By setting $log->{"stdout"} to "undef" we're indicating that whatever appears on stdout is irrelevant (because a firing assert will leave its trace on stderr). It will not be written to the golden log. Since we get access to the captured results of a run via the hash reference we can filter the results using regular perl programming. No extra scripts needed. The attached runtest script has the following command line: runtest [--help] [-h] print usage information and exit [--keep] [-k] keep temporary files [--new] [-n] write new golden log file [--output FILE] [-o FILE] specify output file name [--verbose] [=v] be talkative file input file I want to keep it simple and not have something overly general, adding things on demand. There are two features that I know I want to add. Those are: (1) Ability on the runtest command line to pass additional options to the invoked tools. E.g. for the above example something like: runtest --args run-insns+=--verbose which would add --verbose to any command ending in run-insns (2) Debugging support. runtest --gdb run3 .... runtest --valgrind run4 .... would prefix the 3rd run with gdb --args and the 4th run with valgrind. Comments and suggestions are welcome.
Created attachment 59889 [details] runtest script runtest script
Julian, I'd like to make some progress with this stuff. Perhaps we can discuss the functionality of the proposed run-insns.c My primary use case is to pipe a single basic block of insns through VEX and observe what VEX does to it at various levels and under controllable conditions (host hwcaps, iropt levels etc). Your use case seemed to be to be able to process .orig files and iterate the translation some number of times The current run-insn.c (attached in comment #1) supports those use cases. It makes all configuration knobs that are relevant to these use cases controllable via command line flags and it works for all architectures. Is there anything else that you would want?