Bug 390310 - Output summaries in XML files
Summary: Output summaries in XML files
Status: REPORTED
Alias: None
Product: valgrind
Classification: Developer tools
Component: general (show other bugs)
Version: unspecified
Platform: Compiled Sources Linux
: NOR wishlist
Target Milestone: ---
Assignee: Paul Floyd
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-02-12 10:27 UTC by renaultd
Modified: 2024-12-17 20:27 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In:
Sentry Crash Report:


Attachments
Patch file (8.94 KB, patch)
2018-02-12 10:27 UTC, renaultd
Details
Patch file (8.87 KB, patch)
2019-01-15 17:29 UTC, renaultd
Details
Patch file (36.54 KB, patch)
2024-11-07 15:02 UTC, renaultd
Details
Patch file (35.28 KB, patch)
2024-11-26 11:16 UTC, renaultd
Details

Note You need to log in before you can comment on or make changes to this bug.
Description renaultd 2018-02-12 10:27:03 UTC
Created attachment 110555 [details]
Patch file

The XML output of valgrind does not contain the different summaries from the text format (namely the heap, leak and error summaries). In consequence, the creation of a digest of the results (for example for an external GUI) is more difficult. This question has apparently already been asked without answers (cf. e.g https://stackoverflow.com/questions/40902804/leak-summary-is-missing-from-valgrind-log-file) and the persons I asked actually parse the text output to create their digests. I haven't found a similar question in the bugtracker. 

I propose a patch (to apply to the master git branch) that adds these summaries into the XML output. As a result of the patch, it is possible to output to XML and simply test (for example) whether all heap blocks have been freed : 

valgrind --xml=yes --xml-file=res.xml some_executable_to_test

xmllint --xpath "string(/valgrindoutput/all_heap_blocks_freed/@val)" res.xml 

# -> outputs "true" of "false"

This patch impacts a minor part of valgrind (namely just the output). I tried to follow the coding standards from the source. I am willing to discuss the form of the output if necessary. For example, at the moment, I store the results inside attributes instead of the nodes themselves, and this can be changed. I tried to run the regression and performance tests, but they fail on my machine with or without the patch. 

Example of output in XML format : 
<heap_summary>
  <memory_in_use_at_exit bytes="48" in_blocks="3"/>
  <total_heap_usage allocs="4" frees="1" bytes_allocated="1,072"/>
</heap_summary>
<leak_summary>
  <definitely_lost bytes="48" in_blocks="3"/>
  <indirectly_lost bytes="0" in_blocks="0"/>
  <possibly_lost bytes="0" in_blocks="0"/>
  <still_reachable bytes="0" in_blocks="0"/>
  <suppressed bytes="0" in_blocks="0"/>
</leak_summary>
<error_summary errors="3" from="3" suppressed="0" suppressed_from="0"/>
Comment 1 renaultd 2019-01-15 17:29:50 UTC
Created attachment 117477 [details]
Patch file

Updates and replaces the previous patch file.
Comment 2 renaultd 2019-01-15 17:33:48 UTC
To be more precise, the second patch file is an update to the previous one that applies cleanly on top of d7d82317 (the HEAD of master as of January the 12th, 2019). Fell free to comment or advise if this does not follow the coding standards of the project.
Comment 3 renaultd 2024-11-07 15:02:18 UTC
Created attachment 175624 [details]
Patch file
Comment 4 renaultd 2024-11-07 15:04:38 UTC
(In reply to renaultd from comment #3)
> Created attachment 175624 [details]
> Patch file

The third patch file is supposed to be up-to-date with the latest changes in valgrind. I know this bug report is not much, but I can assure that we are still using it.
Comment 5 Paul Floyd 2024-11-07 15:34:30 UTC
Which tool is using this?
Comment 6 renaultd 2024-11-07 15:44:05 UTC
(In reply to Paul Floyd from comment #5)
> Which tool is using this?

We use valgrind at our university to provide information on their code. Basically, we execute their code through valgrind, and digest the results. The students are not fluent with valgrind at that point, so we need to have a simple summary. And we extract this summary from the XML export.
Comment 7 renaultd 2024-11-07 15:45:52 UTC
Sorry, cannot find a way to edit a comment. We use valgrind to test student code at our university.
Comment 8 Paul Floyd 2024-11-10 09:52:45 UTC
Just out of curiosity, which university is that?

My comments so far
1. There's a FORK message which probably needs removing.
2. White space diffs.
3. There's a change to the non-xml  "possible leaks" indentation. Previously the output was aligned on the ":" of each row.
4. The existing xml looks to me to be almost all tag based rather than attribute based. So rather than

<error_summary errors="1" from="1" suppressed="0" suppressed_from="0"/>

the following would be more consistent

<errorsummary>
   <errors>1</errors>
   <suppressed>0</suppressed>
   <suppressedfrom>0</suppressedfrom>
</errorsummary>

(this isn't something I consider a requirement).

5. I get 18 regtest failures. The XML tests will need to have their expecteds updated.
6. I haven't looked at the manual but I guess that this will need documenting.
Comment 9 renaultd 2024-11-26 11:16:32 UTC
Created attachment 176131 [details]
Patch file

I have added a new patch that tries to address the different remarks from Paul (https://bugs.kde.org/show_bug.cgi?id=390310#c8). We are from the the University of Bordeaux (https://www.u-bordeaux.fr/en), in an engineering school belonging to the Bordeaux-INP group (https://www.bordeaux-inp.fr/en). 

Some things that should have been corrected :

1. The message should have been removed
2. The white space diffs should have been removed
3. The indentations should have been corrected
4. The XML output has been changed to match the existing style

Some things that are still a work-in-progress :

5. I have tried to change the regression tests, but still have
some strange behaviors on my machine (AMD64_LINUX). There are
still 2 regressions at the time :

  * memcheck/tests/mismatches_xml : there is a bunch of new errors
    appearing just by using some calls to VG_(printf_xml) I tried
    launching the test by adding and removing all my modification, but
    even putting only the modifications in
    memcheck/mc_malloc_wrappers.c induce these new errors.

  * tests/vg_regtest drd/tests/bar_bad_xml : same things, new errors
    that appear, this time only adding the modifications in
    coregrind/m_main.c, which, AFAIK, only *remove* calls to
    VG_(printf_xml)

6. I have not written anything in the documentation. I imagine one
should have a look at docs/internals/xml-output-protocol4.txt or even
docs/internals/xml-output-protocol5.txt since it looks more recent.
Comment 10 Paul Floyd 2024-11-27 07:48:55 UTC
Thanks for the updated patch. I've started going though it, I'll get back to you shortly.
Comment 11 Paul Floyd 2024-12-02 21:01:49 UTC
I've done some testing with CLion and Qt Creator (which both use XML via --xml-socket) and they seem OK.

There's one warning that I've fixed.

Some of the XML expected files seem system dependent. For instance this diff

paulf> cat memcheck/tests/sized_aligned_new_delete_misaligned1_xml*diff
--- sized_aligned_new_delete_misaligned1_xml.stderr.exp 2024-12-02 21:43:54.456599000 +0100
+++ sized_aligned_new_delete_misaligned1_xml.stderr.out 2024-12-02 21:57:55.929230000 +0100
@@ -465,9 +465,9 @@
     <in_blocks>0</in_blocks>
   </memory_in_use_at_exit>
   <total_heap_usage>
-    <allocs>6</allocs>
-    <frees>6</frees>
-    <bytes_allocated>72,864</bytes_allocated>
+    <allocs>5</allocs>
+    <frees>5</frees>
+    <bytes_allocated>160</bytes_allocated>
   </total_heap_usage>
 </heap_summary>

(produced on FreeBSD with clang++ and libc++. GCC allocates a memory block for exception handling which adds 1 to the alloc and free counts).
Comment 12 Paul Floyd 2024-12-03 07:35:04 UTC
I'm now down to
== 899 tests, 7 stderr failures, 0 stdout failures, 0 stderrB failures, 0 stdoutB failures, 0 post failures ==
memcheck/tests/long_namespace_xml        (stderr)
memcheck/tests/mismatches_xml            (stderr)
memcheck/tests/sized_aligned_new_delete_misaligned1_xml (stderr)
memcheck/tests/sized_aligned_new_delete_misaligned2_xml (stderr)
memcheck/tests/sized_aligned_new_delete_misaligned3_xml (stderr)
memcheck/tests/threadname_xml            (stderr)
memcheck/tests/xml1                      (stderr)

(on FreeBSD). I'll need to update some expecteds on Linux and also look at Solaris and Darwin.
Comment 13 Paul Floyd 2024-12-03 07:47:07 UTC
I'm fairly sure that this is a regression. On S11.3 I have a build that I did in Dec 2023 that seems OK for several of the test cases with signals.

Need to give git bisect a go.
Comment 14 Paul Floyd 2024-12-03 07:48:15 UTC
Ignore my last comment - wrong window in Firefox. It was meant for 496370.
Comment 15 Paul Floyd 2024-12-17 20:27:06 UTC
It's difficult to get the summaries to be stable across different platforms. What I'll do is filter them all out and then add a few tests that don't use filter_xml but use a filter to only look for the summary blocks.