Bug 234030 - kio_thumbnail, memory leaks and high CPU usage
Summary: kio_thumbnail, memory leaks and high CPU usage
Status: RESOLVED FIXED
Alias: None
Product: kio
Classification: Frameworks and Libraries
Component: thumbnail (show other bugs)
Version: 4.4
Platform: Arch Linux Linux
: NOR normal
Target Milestone: ---
Assignee: Unassigned bugs mailing-list
URL:
Keywords: reproducible, testcase
Depends on:
Blocks:
 
Reported: 2010-04-11 11:48 UTC by Alexander
Modified: 2011-12-09 08:47 UTC (History)
7 users (show)

See Also:
Latest Commit:
Version Fixed In: 4.8.0


Attachments
Corrupted XCF file. (114.17 KB, application/x-gzip)
2011-03-13 20:07 UTC, Jonathan D.
Details
Dangerous memory leak (260.05 KB, image/png)
2011-09-03 14:21 UTC, Alexander
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Alexander 2010-04-11 11:48:47 UTC
Version:            (using KDE 4.4.2)
OS:                Linux
Installed from:    Archlinux Packages

When I start downloading some video files in ktorrent and open the target folder with enabled preview: http://img72.imageshack.us/img72/8062/91054576.png
All files, except two, that have no preview, are downloaded at 100%, but these two file makes kio_thumbnail crazy.

And another moment: the every next "mouseover" event for these files causes new kio_thumbnail process. Or when I reopen the folder, or, if I, god forbid, pressed f5 — it's tragedy. In this case my desktop freezes on few minutes:
http://img72.imageshack.us/img72/3733/42221637.png
http://img99.imageshack.us/img99/3641/73259312.png

I use ffmpegthumbnailer because mplayerthumbs are unusable at all.
Comment 1 Alexander 2010-05-27 20:29:32 UTC
No anyone who can confirm this bug? No way.
Comment 2 King_DuckZ 2011-02-23 13:31:05 UTC
I had the same issue yesterday - partially downloaded files (potentially with a valid beginning, torrent downloads file parts in random order) caused kio_thumbnail to eat mamory, and it took 10-15 minutes for me to log in in a shell and kill the process, due to slow swapping.
I suggest to set a maximum for memory allocations, if something is requesting too much memory/time just for a thumbnail it should be skipped.
Comment 3 Jonathan D. 2011-03-13 20:07:19 UTC
Created attachment 57930 [details]
Corrupted XCF file.

I've the same bug with a corrupted xcf file.
I've disabled all preview in Dolphin settings but kio_thumbnail is still called.
I attach this corrupted file in a compressed format to not freeze your computer if you download it.
Comment 4 Jonathan D. 2011-03-13 20:12:10 UTC
Note: I'm running KDE 4.6.1 from Kubuntu PPA.
Comment 5 dE 2011-04-23 14:05:55 UTC
I've noticed this too.
Comment 6 Dawit Alemayehu 2011-06-22 04:23:55 UTC
(In reply to comment #3)
> Created an attachment (id=57930) [details]
> Corrupted XCF file.
> 
> I've the same bug with a corrupted xcf file.
> I've disabled all preview in Dolphin settings but kio_thumbnail is still
> called.
> I attach this corrupted file in a compressed format to not freeze your computer
> if you download it.

Is the Information Panel in Dolphin enabled ? If it is, then Dolphin will try to generate previews regardless of your "Preview" settings.
Comment 7 Alexander 2011-09-03 14:21:58 UTC
Created attachment 63357 [details]
Dangerous memory leak

Wow, this happens in seconds and could eat all the ram and swap and cause the system to fail. I think this bug should be marked as a critical.
Comment 8 Alexander 2011-09-03 14:24:02 UTC
UPD:
https://bugs.kde.org/show_bug.cgi?id=234030#c7 happened with the corrupted XCF file, attached in the comment #3
Comment 9 Dawit Alemayehu 2011-12-08 07:10:52 UTC
Git commit af286ba6d8f8807c160f259b658cda30fdb4e393 by Dawit Alemayehu.
Committed on 08/12/2011 at 08:08.
Pushed by adawit into branch 'KDE/4.7'.

Ignore corrupted XCF images that contain individual tiles whose width
or height is set to a value larger than that of the image itself.

BUG: 234030
FIXED-IN: 4.8.0

M  +9    -0    kimgio/xcf.cpp

http://commits.kde.org/kdelibs/af286ba6d8f8807c160f259b658cda30fdb4e393
Comment 10 dE 2011-12-08 16:14:27 UTC
Don't you think this should be checked for every corrupted image format?
Comment 11 Alexander 2011-12-08 16:40:44 UTC
Also I reported the problem with the video preview. So I think it shouldn't ignore some specific files as there is many formats and it's hard to handle them all. Instead it should detect when something going wrong and stop. It should check the data for validity before process.
Comment 12 Dawit Alemayehu 2011-12-08 19:31:50 UTC
(In reply to comment #11)
> Also I reported the problem with the video preview. So I think it shouldn't
> ignore some specific files as there is many formats and it's hard to handle
> them all. Instead it should detect when something going wrong and stop. It
> should check the data for validity before process.

(In reply to comment #10)
> Don't you think this should be checked for every corrupted image format?

Possibly, but I personally do not maintain any of this stuff. I just fixed this one because there was a sample provided.
Comment 13 Dawit Alemayehu 2011-12-08 19:46:28 UTC
(In reply to comment #11)
> Also I reported the problem with the video preview. So I think it shouldn't
> ignore some specific files as there is many formats and it's hard to handle
> them all. Instead it should detect when something going wrong and stop. 

Ahh... The bug was not in the thumbnail generator. It was in the QImage plugin that reads the XCF file formats. As such there is no way the thumbnail generator can workaround or detect bugs in a completely different piece of code. 

> It should check the data for validity before process.

How should it exactly do that ? Understand every image, text and other content format intimately by itself ? It cannot. Anyhow, I closed this bug report because there are several others that deal with problems generating thumbail for video files. You yourself seem to have commented on those ; so this is at best simply a duplicate. 

BTW, the fact that dolphin attempts to generate a preview for files that are in the process of being downloaded is an issue of dolphin itself and the fact that ffmpegthumbnailer generator eats all the CPU attempting to generate a preview image in such circumstances is is something that needs to be fixed in the kdemultimedia module, which is where the ffmpegthumbnail generator plugin is located.
Comment 14 Alexander 2011-12-09 08:11:05 UTC
Agree. But perhaps it could detect when the process consumes too many memory and CPU time basis on the input file size. For example it's definitely clear when process goes to eat over 10Gb of RAM when processing the 1Mb file. I mean to add something in thumbnail system that will prevent dangerous situations like on the screenshot.
Comment 15 King_DuckZ 2011-12-09 08:47:43 UTC
I disagree. Killing a process is not a solution, will make the user experience mysterious slowdowns with no visible result and delay the correct fix. Also, there is no way an algorithm can decide when memory consumption is too high for an arbitrary file. Here's a good example of a 59kb file expanding to over 48mb in memory: http://upload.wikimedia.org/wikipedia/commons/c/c4/All_24-bit_RGB_colors.png
In theory I could write a program that saves one single byte in a file and produces huge outputs by looking up in a table of pre-made video files, for example.
The problem is that memory allocation doesn't fail until you run out of virtual memory. The real fix is that each plugin should take into account bogus data and refrain from loading it if no partial recovery is possible, or stop at once.