Bug 479612 - Massive memory leak when looking at a file
Summary: Massive memory leak when looking at a file
Status: RESOLVED FIXED
Alias: None
Product: frameworks-kimageformats
Classification: Frameworks and Libraries
Component: general (show other bugs)
Version: 5.113.0
Platform: openSUSE Linux
: NOR normal
Target Milestone: ---
Assignee: Alex Merry
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-01-10 16:12 UTC by Jonas Baecker
Modified: 2024-04-08 12:54 UTC (History)
7 users (show)

See Also:
Latest Commit:
Version Fixed In:
Sentry Crash Report:


Attachments
Head of the offending file. (1.17 KB, application/gzip)
2024-01-11 15:02 UTC, Jonas Baecker
Details
Heaptrack analysis (234.02 KB, image/png)
2024-01-11 16:25 UTC, Nicolas Fella
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jonas Baecker 2024-01-10 16:12:06 UTC
SUMMARY
A file from an old Windows video game makes Dolphin use all my RAM. It's a problem that has persisted across a couple of months of updates. I can't tell when exactly it started, or if it has always been there, as I haven't looked at these files in a very long time. I can provide the offending file for analysis, but it's 8,9MiB large so it does not fit the attachment.

STEPS TO REPRODUCE
1. Have the file visible in Dolphin
2. Wait a couple of seconds

OBSERVED RESULT
All of my 32GB of RAM consumed in just a few seconds.

EXPECTED RESULT
No excessive RAM usage.

SOFTWARE/OS VERSIONS
Linux/KDE Plasma: 6.6.6 / 5.27.10
(available in About System)
KDE Plasma Version: 5.27.10
KDE Frameworks Version: 5.113.0
Qt Version: 5.15.11
Comment 1 Jonas Baecker 2024-01-11 15:02:48 UTC
Created attachment 164813 [details]
Head of the offending file.

It seems that running the file through the head command gives a file that has the same effect. Unzip and it should make Dolphin consume all of your RAM. The file is a mix file, an archive format of sorts, from a game "Tiberian Sun".
Comment 2 Nicolas Fella 2024-01-11 16:24:40 UTC
I can reproduce the issue.

When generating a thumbnail for the file the memory usage of kioslave5 briefly peaks at 15GB, but goes down to normal again
Comment 3 Nicolas Fella 2024-01-11 16:25:11 UTC
Created attachment 164814 [details]
Heaptrack analysis
Comment 4 Nicolas Fella 2024-01-11 16:28:06 UTC
Analyzing in heaptrack suggests the file is loaded as a TGA file. Not sure that makes sense
Comment 5 Albert Astals Cid 2024-01-11 18:51:08 UTC
I'm not sure there's an actual bug, the header of the file describes an image of 41039 * 63040 in size, and thus lots of memory is needed to load it to memory.

You'd have to send us the actual file to confirm if it's a TGA file or not and if not if we could bail earlier.

As a side thing, the QImageReader API would probably welcome a function saying "fail if we try to read a file bigger than AxB in size"?
Comment 6 Jonas Baecker 2024-01-11 19:43:38 UTC
(In reply to Albert Astals Cid from comment #5)
> I'm not sure there's an actual bug, the header of the file describes an
> image of 41039 * 63040 in size, and thus lots of memory is needed to load it
> to memory.
> 
> You'd have to send us the actual file to confirm if it's a TGA file or not
> and if not if we could bail earlier.

Here's the whole file https://files.catbox.moe/fwoeqz.gz
I'm using a third party filehost here as the file is too big to be attached.
Comment 7 Nicolas Fella 2024-01-11 20:42:05 UTC
I can't reproduce it on Qt6, presumably because we run into the allocation limit in Qt (see https://invent.kde.org/frameworks/kimageformats/-/blob/master/src/imageformats/util_p.h?ref_type=heads#L19)
Comment 8 Nicolas Fella 2024-01-11 20:44:46 UTC
For the thumbnails specifically we might be able to check the dimensions using QImageReader before trying to load the whole file and bail out if it's more than N pixels
Comment 9 Jonas Baecker 2024-02-18 20:32:39 UTC
(In reply to Nicolas Fella from comment #4)
> Analyzing in heaptrack suggests the file is loaded as a TGA file. Not sure
> that makes sense

Yeah, it's not really a TGA file. Actually, quite a few of these MIX archives are detected as TGA files for me in Dolphin, though only one of them used a lot of memory. This false detection happens with all the old Command & Conquer games that use MIX files. The games are freeware and can be downloaded from here for testing: https://cnc-comm.com/
Comment 10 Mirco Miranda 2024-03-01 07:23:08 UTC
This patch should help: https://invent.kde.org/frameworks/kimageformats/-/merge_requests/209

If I remember correctly, the thumbnail generator leaves the task of determining the correct format to the plugins (regardless of the extension and mime type).
As a result, formats like TGA that do not have a unique signature can result in false positives and the plugin tries to read them. It is not a good thing.

To avoid excessive RAM usage, last year I modified the thumnailer with memory checks to prevent the system from threshing. I don't know if it still works though.
Also, while writing the patch for the TGA I saw that it uses double the necessary memory. This probably doesn't help either.
Comment 11 Mirco Miranda 2024-03-03 09:27:22 UTC
(In reply to Mirco Miranda from comment #10)

> To avoid excessive RAM usage, last year I modified the thumnailer with
> memory checks to prevent the system from threshing. I don't know if it still
> works though.

I checked the plugin and it doesn't support options. On QT 5 therefore mitigating RAM usage is not possible. As soon as I have a moment I'll add the function to the plugin.
However, with the imminent release of KDE6 there should be no problems because the Qt6 RAM limitation is correctly implemented in the plugin.
Comment 12 Mirco Miranda 2024-03-15 07:17:00 UTC
(In reply to Mirco Miranda from comment #11)
> I checked the plugin and it doesn't support options. On QT 5 therefore
> mitigating RAM usage is not possible. As soon as I have a moment I'll add
> the function to the plugin.

Done. See also https://invent.kde.org/frameworks/kimageformats/-/merge_requests/212
Comment 13 dreamsyntax 2024-03-31 06:42:26 UTC
I can confirm all Avalonia UI Desktop generated application executables result in this same issue.
Comment 14 Albert Astals Cid 2024-04-08 12:32:21 UTC
We believe this has been as fixed as it can be.

Please upgrade to the newer versions (still unreleased) and tell us if you (ideally KDE Frameworks 6 >= 6.1 and KDE Gear >= 24.05) and tell us if you still have the problem.
Comment 15 Jonas Baecker 2024-04-08 12:54:32 UTC
(In reply to Albert Astals Cid from comment #14)
> We believe this has been as fixed as it can be.
> 
> Please upgrade to the newer versions (still unreleased) and tell us if you
> (ideally KDE Frameworks 6 >= 6.1 and KDE Gear >= 24.05) and tell us if you
> still have the problem.

I recently upgraded to Plasma 6 and can confirm that it is indeed fixed for me now.