SUMMARY When analyzing very large results (e.g. 8gb heaptrack file), waiting for parsing can take many minutes. Any subsequent runs of heaptrack_print or heaptrack_gui incur the same cost. It would be extremely useful to have an intermediate cache/artifact which could be used to jump over the parsing step, and allow faster re-loading of data. This is especially useful when sharing with collaborators, or for automated workflows where heaptrack profiling is done asynchronously and results provided later.
Addendum: * Loading takes 10s of minutes in the case of an 8GB file. * Unintentional process terminations (crashes, accidental window close) are of particular issue for heaptrack_gui
Is the 8GB file compressed or uncompressed? It sounds like it's uncompressed - try to compress it via `zstd` to create a `*.zst` file - that greatly reduces the disk-I/O time and should make things considerably faster I bet. That said, I have some ideas on how to speed parsing up more too, but that will require a complete rewrite of the data structure... I hope I get to that eventually, but cannot promise anything.
Looks like I got the original number wrong it's 4gb compressed with gz (and 65GB uncompressed! geeze) I'll try switching to zstd and see what the speed looks like. Also I started adding some information to the processing window with respect to which pass it was on and the timestamp currently being worked on for my own sake. I'll create a PR in case that's desirable more generally. Looking into if there's a way to parallelize the first pass as well.
OK that's a huge amount of data, which corresponds to a ton of allocations... I mean just for the fun of it, try to do something like this: zcat <yourfile> | grep -cE '^[\+-]' That will give us the number of allocation events. I suspect this to be a _very_ large number. As a workaround, until we find some improvements to this, here are some suggestions: - only track a smaller workload for your application, i.e. reduce the problem size, or the duration for which you are tracing - use runtime attaching to ignore the startup phase if that is allocation heavy Otherwise, I'm afraid to say that zstd is the only quick thing you could try. It's much faster at decompressing the data compared to gzip, so that will be noticeable to you. But in the end we'll still have to crunch through 65GB of data, which just takes a really long time...
this bug report isn't really actionable as-is, so I'm closing it