Bug 185881 - kio_thumbnail tries to create thumbnail of way to big files and eats up memory, cpu, swap
Summary: kio_thumbnail tries to create thumbnail of way to big files and eats up memor...
Status: RESOLVED FIXED
Alias: None
Product: kio
Classification: Frameworks and Libraries
Component: thumbnail (show other bugs)
Version: 4.2.0
Platform: Ubuntu Linux
: NOR normal
Target Milestone: ---
Assignee: Malte Starostik
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-03-01 16:42 UTC by Thomas Weissel
Modified: 2018-07-19 16:45 UTC (History)
23 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Thomas Weissel 2009-03-01 16:42:30 UTC
Version:            (using KDE 4.2.0)
OS:                Linux
Installed from:    Ubuntu Packages

for example in gwenview (the thumbnailbar) or when copying a big image file in konqueror that already exists .. (when the window to compare the two versions is started)  kiothumnailer is startet and it takes one minute and the whole system is unusable.. this process should NEVER get (or take) that much resources.. even if i had dolphin set up to show 300MB thumnails..  (i have not)
Comment 1 Rosetzky Cedric 2009-05-01 11:54:20 UTC
I'm having the same problem with latest SVN. kio_thumbnail uses a lot of memory and CPU, and takes a lot of time to free the used memory.
Comment 2 Andrew Muhametshin 2009-07-06 08:42:20 UTC
$ LANG=C date
Mon Jul  6 09:13:30 MSD 2009

$ ps xo pid,%cpu,%mem,rss,comm,start_time | grep -E " kio_thumbnail| dolphin"
24297  0.0  0.0   872 kio_thumbnail   06:11
31624  3.5  0.8 17056 dolphin         08:58
31772 13.5 10.0 209292 kio_thumbnail  09:00
31840 16.4  2.1 43632 kio_thumbnail   09:01
31843 13.4 13.4 278508 kio_thumbnail  09:01
31862 14.4 11.3 236172 kio_thumbnail  09:01
31881  6.2 15.6 324528 kio_thumbnail  09:02
32059  2.3 12.2 254344 kio_thumbnail  09:02
32067  1.7  9.3 194044 kio_thumbnail  09:02
32158  0.1  0.2  5264 kio_thumbnail   09:03

$ kde4-config --version
Qt: 4.5.2
KDE: 4.2.95 (KDE 4.2.95 (KDE 4.3 RC1))
kde4-config: 1.0

$ uname -rms
Linux 2.6.30-gentoo-r1 i686
Comment 3 Fredrik Klasson 2009-08-10 12:32:57 UTC
and here's my 2 {$COIN_DENOMINATOR}
Summary:
 kio_thumbnails really missbehaves by stealing memory for me too (+1.7G DATA as reported by top this morning). Can't say about the CPU usage as it fills RAM and swap causing massive swapping => skyrocketing my load avg to +30.0, top says mostly (+95%) io wait untill I manage to kill the kio_thumnail process (or processes)), this probably due to thrashing caused by the exess memory theft. (I'm puzzled as to why it's not OOM killed!)
Versions:
 OS: Gentoo
 kde-base/kdebase-kioslaves-4.3.0 
 (build with USE=debug so it should be possible to get some
  useful debuggging info, not sure how though)

 $ kde4-config --version
     Qt: 4.5.1
     KDE: 4.3.00 (KDE 4.3.0)
     kde4-config: 1.0

Reproduce:
 It usually starts by hovering big movie files or folders with big movies files. Though it shouldn't thumbnail them, max previev file size is set to 5 MB (the movies as much bigger than that), the thumbnails button isn't checked. Further more only "Grafik" (I'm 'guessing' the non localized string is "graphics") and Jpeg is selected. And "Använd miniatyribilder inbäddade i filer" (~= "use embedded thumbnails"/"use thumbnails embedded in files").
 * Clearly kio_thumbnails/dolphin isn't respecting those options. - maybe this is a konqueror/dolphin bug? (otho, imo thumbnailing a file shouldn't consume more than _tops_ 10M RAM, or perhaps I'm living in a fantasy world?)

Work-around:
 I've temporarily renamed kio_thumbnail.so to kio_thumnail.so.disabled, since it's a hazard to the system's stabillity right now.

Wish:
 A kill-switch for kio_thumbnail, i.e. a "Never generate thumbnails" option, if so only accessible via manually editing ~/.kde4/share/config/kio_thumbnailrc. Or a hard memory limit option there, say 100M (which imo is too much, but at least a limit that should prevent grand-theft-ram).
 Or better yet, a real fix that makes kio_thumbnail use a reasonable amount or RAM when generating thumbs.
Comment 4 Neil Skrypuch 2009-08-12 03:35:41 UTC
I can confirm this. I turned on previews/thumbnails for most file types (as it's a very cool feature) and then went about normal usage with Konqueror. After not long I found several kio_thumbnail processes chewing through resources like it's going out of style, the most flagrant of which goes something like this in top:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
11053 neil      20   0 4396m 1.2g  44m S   25 59.0   6:52.47 kio_thumbnail

...and my 2G RAM machine is suddenly 2.7G into swap (and rising fast).

Since among other things, I opened up my home directory, I checked to see if thumbnails were being generated recursively (which lsof suggested they were not), which is good. However, It's absolutely *INSANE* that kio_thumbnail is now approaching 5G of memory used, it's not like it's thumbnailing files anywhere close to that big!

Since I wrote the first bit, my swap usage has grown to 3.2G and I'm killing kio_thumbnail manually... Much better, close to 4G of RAM + swap freed.
Comment 5 Neil Skrypuch 2009-08-12 03:36:59 UTC
Oh, and I'm using KDE 4.3.
Comment 6 Mark 2009-08-22 22:10:56 UTC
*** This bug has been confirmed by popular vote. ***
Comment 7 Jakub Januszkiewicz 2009-10-01 23:49:57 UTC
I was also hit by this bug, twice.

In both cases some kio_thumbnail processes suddenly started hogging the memory like crazy, causing massive swap activity.

The first time it happened I couldn't do much at all - the system got so unresponsive that I only managed to run 'top' in konsole which, after refreshing a few times, froze completely (even ATL+CTRL+F1 didn't work). I left the computer on for the night thinking it would settle down, but after a few hours the disk was still swapping like crazy and I couldn't do anything. A hard reboot was needed.

The second time happened today and I was quick enough to see some more details before everything froze (I didn't manage to kill the offending processes though - mem usage skyrocketed and froze the machine in literally several seconds!). There were 3 'kdeinit4: kio_thumbnail' processes, one of which consumed 3900MB VIRT and 2 others consuming over 1500MB VIRT each. I know I should have also looked at RES and/or SHR, but well... I didn't, sorry :/

The only thumbnail-related activity within the last few hours before today's freeze I can recall is copying several large video files from DVDs to hard disk. Although the directory to which I copied the files has thumbnails turned off, Dolphin still shows the thumbnail in tooltip and in the information panel on the right. This might be related, but I have closed all Dolphin windows at least 2 hours before the bug occured, so it might as well be not related at all. I also have a large (22GB) photo collection, but I haven't touched it today at all.

I have 4GB RAM + 4GB swap.
I run KDE 4.3.1 on Gentoo.
I have all thumbnailing plugins enabled (also video via MPlayerThumbs) with max file size set to 5MB.
I use Dolphin as the file manager.
If you need any additional info, please let me know.
If I encounter this bug again, I'll try to provide more useful info.
Comment 8 Neil Skrypuch 2009-10-08 00:43:50 UTC
So, this morning, I had <100M of swap used. I left and came back in the afternoon to find this:

top - 18:33:11 up 23 days, 19:31, 66 users,  load average: 0.40, 0.58, 0.45
Tasks: 185 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.0%us,  2.7%sy,  0.0%ni, 93.9%id,  0.3%wa,  0.2%hi,  0.0%si,  0.0%st
Mem:   2057212k total,  2040096k used,    17116k free,     8264k buffers
Swap:  7807224k total,  4206896k used,  3600328k free,   221012k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
25695 neil      20   0 5037m 1.2g 5824 S    4 61.5  48:39.43 kio_thumbnail
...

Note the swap usage and the memory usage for kio_thumbnail. Though I do have a Konqueror file manager open with a few tabs, I haven't interacted with it whatsoever since yesterday, and I don't use Dolphin.

Killing kio_thumbnail returns a few resources:

top - 18:38:49 up 23 days, 19:37, 66 users,  load average: 0.23, 0.38, 0.40
Tasks: 182 total,   1 running, 181 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.8%us,  2.0%sy,  0.0%ni, 94.0%id,  0.5%wa,  0.3%hi,  0.3%si,  0.0%st
Mem:   2057212k total,   765492k used,  1291720k free,     8852k buffers
Swap:  7807224k total,  1013696k used,  6793528k free,   242908k cached

...meaning it really was consuming nearly 5G of RAM, and completely unprovoked at that.
Comment 9 Dan Reidy 2009-10-20 18:52:57 UTC
confirmed over here...
dubkat@stargazer ~ $ kde4-config --version
Qt: 4.5.3
KDE: 4.3.2 (KDE 4.3.2)
kde4-config: 1.0

i frequently have to keep a terminal window open running htop so that i can kill off any runaway kio_thumbnail processes. failure to do so causes the system to freeze, and the disk to thrash. I have 4 gigs of memory, and about 8 gigs of swap space. All easily eaten up by kio_thumbnail.
Comment 10 Federico Cuello 2009-12-07 01:43:19 UTC
It is still happening,

$ kde4-config --version
Qt: 4.6.0
KDE: 4.3.4 (KDE 4.3.4)
kde4-config: 1.0
Comment 11 Ralph 2010-03-14 21:14:51 UTC
Still happening on:

kde4-config --version
Qt: 4.6.2
KDE: 4.3.5 (KDE 4.3.5)
kde4-config: 1.0
Comment 13 Alexander 2010-06-10 14:24:55 UTC
I can't understand, how "this" could to get in the KDE trunk and stay there for so long time? Moreover, it is still there, without considering the fact, that the question already is about the loss of user data! I just have no words...

http://bugs.kde.org/buglist.cgi?quicksearch=kio_thumbnail
Comment 14 FiNeX 2010-06-10 15:27:45 UTC
Probably the reason is because developers doesn't use thumbnails or doesn't have big video files.

Anyway, I'm using KDE 4.4.x (now I've 4.4.4) and it is a couple of months that I don't have that problem, maybe it has been partially fixed in the latest release... I hope :-)
Comment 15 Rosetzky Cedric 2010-06-10 19:03:25 UTC
I don't know about KDE 4.4.4 but I'm sure it still happens in trunk.
Comment 16 Alexander 2010-06-10 19:28:08 UTC
This isn't the only one problem with kio_thumbnail. For example, you can try to start downloading few video files in ktorrent and open the target folder with preview. Or try to browse a remote folder that contains video files, etc. Seems that this component wasn't tested completely enough.
Comment 17 Igor 2010-06-27 16:37:11 UTC
i have same problem

$ kde4-config --version
Qt: 4.6.2
KDE: 4.4.4 (KDE 4.4.4)
kde4-config: 1.0
Comment 18 Marcel Hasler 2010-06-27 16:55:00 UTC
I've completely removed mplayerthumbs from my systems and installed kffmpegthumbnailer instead. No more memory problems for me. Plus, kffmpegthumbnailer is really snappy, even on a netbook.
Comment 19 Alexander 2010-06-27 18:44:18 UTC
(In reply to comment #18)
> I've completely removed mplayerthumbs from my systems and installed
> kffmpegthumbnailer instead. No more memory problems for me. Plus,
> kffmpegthumbnailer is really snappy, even on a netbook.

+1, but it has its own problems, and from time to time generates poor quality thumbnails. Are you using bittorrent? :)
Comment 20 Marcel Hasler 2010-06-27 19:05:20 UTC
Can't say I do. But at least video clips downloaded "normally" don't cause any problems, the thumbnails are just regenerated regularly during the download.
Comment 21 karl.r.ernst 2010-08-19 12:32:42 UTC
I have this problem too with kde 4.4.5, even thought I have disabled video previews and limited previews to files smaller than 5 MB.

It seems to be related to the folder preview, as if  I disable folder preview, the problem goes away.
Comment 22 Melendro 2010-10-17 23:02:38 UTC
To disable video thumbnails run the command "mplayerthumbsconfig" and add all video extensions (avi, mpg, mkv, mp4, etc.) to the "Blacklisted file extensions".

This solved the problem for me.
Comment 23 Nate Graham 2018-04-13 21:57:30 UTC
It's been a few years... does anybody still see this issue with KDE Frameworks 5.44 or greater?
Comment 24 Nate Graham 2018-07-19 16:45:52 UTC
Haven't heard anything from the reporter or anyone else who experienced this at some point in the past. Considering it fixed in KDE Frameworks 5.