I've looked at vg_regtest and it is not really suitable for what
I have in mind for VEX unit testing. So I've come up with something
that is more flexible but still simple (~500 lines).

The key features are:

(1) Every testcase is a perl script.
(2) Any input needed to run the testcase is generated by the testcase
    itself.
(3) There can be more than one invocation of some tool within a testcase.
    (I use the term "run" to mean tool invocation.)
(4) The result of a run is stored in a hash table.
(5) There is a *single* "golden log" file which contains all the results
    of all runs that occur in the testcase. The log file is a perl script,
    so we can load and evaluate it easily for comparison.
(6) Since the testcase is a perl script any filtering of stdout/stderr
    results can be done easily using standard perl programming.


Example:

my $file = test::write_file("FLOGR", <<'EOF');
A5 4F DE AD    # R4 = 0xdead
B9 83 00 14    # R1 = leftmost-bit(R4); R2 = R4 with leftmost-bit=0
00 00
EOF

my $cmd = "$test::top_of_tree/auxprogs/run-insns --s390x $file --trace-flags=00000010";

test::execute($cmd);

First, the testcase creates a file FLOGR with a few insns.
Secondly, it builds up the command to execute. Here $test::top_of_tree 
denotes the root directory of VEX.
Thirdly, the command is executed and the usual results: exit-code, stdout
and stderr output, and a possibly caught signal are recorded behind the scenes.

The log file for above testcase looks like this:

$golden{run1} = 
{
  'command' => '.../VEX/auxprogs/run-insns --s390x FLOGR --trace-flags=00000010',
  'rc' => 0,
  'sig' => 0,
  'stderr' => '',
  'stdout' => <<'EOF',

------------------------ Register-allocated code ------------------------

  0   v-loadi  %r1,0xDEAD             8 bytes
  1   v-store  %r1,guest_r4           8 bytes
  2   v-loadi  %r1,48                 8 bytes
  3   v-store  %r1,guest_r1           8 bytes
  4   v-loadi  %r1,0x5EAD             8 bytes
  5   v-store  %r1,guest_r2           8 bytes
  6   v-loadi  %r1,2                  8 bytes
  7   v-store  %r1,guest_CC_OP        8 bytes
  8   v-loadi  %r1,0xDEAD             8 bytes
  9   v-store  %r1,guest_CC_DEP1      8 bytes
 10   v-loadi  %r1,0                  8 bytes
 11   v-store  %r1,guest_CC_DEP2      8 bytes
 12   v-loadi  %r1,0                  8 bytes
 13   v-store  %r1,guest_CC_NDEP      8 bytes
 14   v-loadi  %r1,8                  8 bytes
 15   v-store  %r1,guest_IA           8 bytes
 16   if (always) goto guest_r0       0 bytes

EOF
};


Even though it's perl it's highly readable and straight forward
which is relevant for ease of proof-reading. If we had more than 
one run the log file would contain $golden{run2}, $golden{run3}, 
etc ... 

One thing that I would like to do is to have testcases for certain 
complex instructions and pipe them through VEX assuming various
optimisation levels and hardware capabilities. I'm interested in
whether or not they run into an assert. That is easy to do:

my $file = test::write_file("FLOGR", <<'EOF');
B9 83 00 14
00 00
EOF

for my $level (0, 1, 2) {
  for my hwcaps ("--ldisp", "--eimm", "--gie") {
     my $cmd = "$test::top_of_tree/auxprogs/run-insns "
                . "--s390x $file $hwcaps --opt-level=$level";
     my $log = test::execute($cmd);
     $log->{"stdout"} = undef;  # we're not interested
  }
}

So test::execute returns a reference to a hash that contains 
the results of executing the command. By setting $log->{"stdout"}
to "undef" we're indicating that whatever appears on stdout is 
irrelevant (because a firing assert will leave its trace on stderr). 
It will not be written to the golden log. 

Since we get access to the captured results of a run via the hash 
reference we can filter the results using regular perl programming. 
No extra scripts needed.

The attached runtest script has the following command line:

  runtest

    [--help] [-h]      print usage information and exit

    [--keep] [-k]      keep temporary files

    [--new]  [-n]      write new golden log file

    [--output FILE]  [-o FILE]   specify output file name

    [--verbose] [=v]   be talkative

    file               input file


I want to keep it simple and not have something overly general, adding things on demand.

There are two features that I know I want to add. Those are:

(1) Ability on the runtest  command line to pass additional 
    options to the invoked tools. E.g. for the above example 
    something like: runtest --args run-insns+=--verbose
    which would add --verbose to any command ending in run-insns

(2) Debugging support.

    runtest --gdb run3 ....
    runtest --valgrind run4  ....

    would prefix the 3rd run with gdb --args and the 4th run with
    valgrind.

Comments and suggestions are welcome.

Created attachment 59889 [details]
runtest script

runtest script

Julian, I'd like to make some progress with this stuff. Perhaps we can discuss the functionality of the proposed run-insns.c
My primary use case is to pipe a single basic block of insns through VEX and observe what VEX does to it at various levels and under controllable conditions (host hwcaps, iropt levels etc).
Your use case seemed to be to be able to process .orig files and iterate the translation some number of times
The current run-insn.c (attached in comment #1) supports those use cases. It makes all configuration knobs that are relevant to these use cases controllable via command line flags and it works for all architectures.
Is there anything else that you would want?