| Summary: | kio_thumbnail tries to create thumbnail of way to big files and eats up memory, cpu, swap | ||
|---|---|---|---|
| Product: | [Unmaintained] kio | Reporter: | Thomas Weissel <xapient> |
| Component: | thumbnail | Assignee: | Malte Starostik <malte> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | aspotashev, chrschmitt, DJm00n, dubkat, fedux, finex, forenhampel, gokcen.eraslan, inspirra, jakub.januszkiewicz, karl.r.ernst, kde, ken, kovyrlo, loacoon, m.debruijne, mahasler, melendro, nate, R.Clark.01, scientica, vo.zaeb, xenonism |
| Priority: | NOR | ||
| Version: | 4.2.0 | ||
| Target Milestone: | --- | ||
| Platform: | Ubuntu | ||
| OS: | Linux | ||
| Latest Commit: | Version Fixed In: | ||
| Sentry Crash Report: | |||
|
Description
Thomas Weissel
2009-03-01 16:42:30 UTC
I'm having the same problem with latest SVN. kio_thumbnail uses a lot of memory and CPU, and takes a lot of time to free the used memory. $ LANG=C date Mon Jul 6 09:13:30 MSD 2009 $ ps xo pid,%cpu,%mem,rss,comm,start_time | grep -E " kio_thumbnail| dolphin" 24297 0.0 0.0 872 kio_thumbnail 06:11 31624 3.5 0.8 17056 dolphin 08:58 31772 13.5 10.0 209292 kio_thumbnail 09:00 31840 16.4 2.1 43632 kio_thumbnail 09:01 31843 13.4 13.4 278508 kio_thumbnail 09:01 31862 14.4 11.3 236172 kio_thumbnail 09:01 31881 6.2 15.6 324528 kio_thumbnail 09:02 32059 2.3 12.2 254344 kio_thumbnail 09:02 32067 1.7 9.3 194044 kio_thumbnail 09:02 32158 0.1 0.2 5264 kio_thumbnail 09:03 $ kde4-config --version Qt: 4.5.2 KDE: 4.2.95 (KDE 4.2.95 (KDE 4.3 RC1)) kde4-config: 1.0 $ uname -rms Linux 2.6.30-gentoo-r1 i686 and here's my 2 {$COIN_DENOMINATOR}
Summary:
kio_thumbnails really missbehaves by stealing memory for me too (+1.7G DATA as reported by top this morning). Can't say about the CPU usage as it fills RAM and swap causing massive swapping => skyrocketing my load avg to +30.0, top says mostly (+95%) io wait untill I manage to kill the kio_thumnail process (or processes)), this probably due to thrashing caused by the exess memory theft. (I'm puzzled as to why it's not OOM killed!)
Versions:
OS: Gentoo
kde-base/kdebase-kioslaves-4.3.0
(build with USE=debug so it should be possible to get some
useful debuggging info, not sure how though)
$ kde4-config --version
Qt: 4.5.1
KDE: 4.3.00 (KDE 4.3.0)
kde4-config: 1.0
Reproduce:
It usually starts by hovering big movie files or folders with big movies files. Though it shouldn't thumbnail them, max previev file size is set to 5 MB (the movies as much bigger than that), the thumbnails button isn't checked. Further more only "Grafik" (I'm 'guessing' the non localized string is "graphics") and Jpeg is selected. And "Använd miniatyribilder inbäddade i filer" (~= "use embedded thumbnails"/"use thumbnails embedded in files").
* Clearly kio_thumbnails/dolphin isn't respecting those options. - maybe this is a konqueror/dolphin bug? (otho, imo thumbnailing a file shouldn't consume more than _tops_ 10M RAM, or perhaps I'm living in a fantasy world?)
Work-around:
I've temporarily renamed kio_thumbnail.so to kio_thumnail.so.disabled, since it's a hazard to the system's stabillity right now.
Wish:
A kill-switch for kio_thumbnail, i.e. a "Never generate thumbnails" option, if so only accessible via manually editing ~/.kde4/share/config/kio_thumbnailrc. Or a hard memory limit option there, say 100M (which imo is too much, but at least a limit that should prevent grand-theft-ram).
Or better yet, a real fix that makes kio_thumbnail use a reasonable amount or RAM when generating thumbs.
I can confirm this. I turned on previews/thumbnails for most file types (as it's a very cool feature) and then went about normal usage with Konqueror. After not long I found several kio_thumbnail processes chewing through resources like it's going out of style, the most flagrant of which goes something like this in top: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11053 neil 20 0 4396m 1.2g 44m S 25 59.0 6:52.47 kio_thumbnail ...and my 2G RAM machine is suddenly 2.7G into swap (and rising fast). Since among other things, I opened up my home directory, I checked to see if thumbnails were being generated recursively (which lsof suggested they were not), which is good. However, It's absolutely *INSANE* that kio_thumbnail is now approaching 5G of memory used, it's not like it's thumbnailing files anywhere close to that big! Since I wrote the first bit, my swap usage has grown to 3.2G and I'm killing kio_thumbnail manually... Much better, close to 4G of RAM + swap freed. Oh, and I'm using KDE 4.3. *** This bug has been confirmed by popular vote. *** I was also hit by this bug, twice. In both cases some kio_thumbnail processes suddenly started hogging the memory like crazy, causing massive swap activity. The first time it happened I couldn't do much at all - the system got so unresponsive that I only managed to run 'top' in konsole which, after refreshing a few times, froze completely (even ATL+CTRL+F1 didn't work). I left the computer on for the night thinking it would settle down, but after a few hours the disk was still swapping like crazy and I couldn't do anything. A hard reboot was needed. The second time happened today and I was quick enough to see some more details before everything froze (I didn't manage to kill the offending processes though - mem usage skyrocketed and froze the machine in literally several seconds!). There were 3 'kdeinit4: kio_thumbnail' processes, one of which consumed 3900MB VIRT and 2 others consuming over 1500MB VIRT each. I know I should have also looked at RES and/or SHR, but well... I didn't, sorry :/ The only thumbnail-related activity within the last few hours before today's freeze I can recall is copying several large video files from DVDs to hard disk. Although the directory to which I copied the files has thumbnails turned off, Dolphin still shows the thumbnail in tooltip and in the information panel on the right. This might be related, but I have closed all Dolphin windows at least 2 hours before the bug occured, so it might as well be not related at all. I also have a large (22GB) photo collection, but I haven't touched it today at all. I have 4GB RAM + 4GB swap. I run KDE 4.3.1 on Gentoo. I have all thumbnailing plugins enabled (also video via MPlayerThumbs) with max file size set to 5MB. I use Dolphin as the file manager. If you need any additional info, please let me know. If I encounter this bug again, I'll try to provide more useful info. So, this morning, I had <100M of swap used. I left and came back in the afternoon to find this: top - 18:33:11 up 23 days, 19:31, 66 users, load average: 0.40, 0.58, 0.45 Tasks: 185 total, 1 running, 184 sleeping, 0 stopped, 0 zombie Cpu(s): 3.0%us, 2.7%sy, 0.0%ni, 93.9%id, 0.3%wa, 0.2%hi, 0.0%si, 0.0%st Mem: 2057212k total, 2040096k used, 17116k free, 8264k buffers Swap: 7807224k total, 4206896k used, 3600328k free, 221012k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 25695 neil 20 0 5037m 1.2g 5824 S 4 61.5 48:39.43 kio_thumbnail ... Note the swap usage and the memory usage for kio_thumbnail. Though I do have a Konqueror file manager open with a few tabs, I haven't interacted with it whatsoever since yesterday, and I don't use Dolphin. Killing kio_thumbnail returns a few resources: top - 18:38:49 up 23 days, 19:37, 66 users, load average: 0.23, 0.38, 0.40 Tasks: 182 total, 1 running, 181 sleeping, 0 stopped, 0 zombie Cpu(s): 2.8%us, 2.0%sy, 0.0%ni, 94.0%id, 0.5%wa, 0.3%hi, 0.3%si, 0.0%st Mem: 2057212k total, 765492k used, 1291720k free, 8852k buffers Swap: 7807224k total, 1013696k used, 6793528k free, 242908k cached ...meaning it really was consuming nearly 5G of RAM, and completely unprovoked at that. confirmed over here... dubkat@stargazer ~ $ kde4-config --version Qt: 4.5.3 KDE: 4.3.2 (KDE 4.3.2) kde4-config: 1.0 i frequently have to keep a terminal window open running htop so that i can kill off any runaway kio_thumbnail processes. failure to do so causes the system to freeze, and the disk to thrash. I have 4 gigs of memory, and about 8 gigs of swap space. All easily eaten up by kio_thumbnail. It is still happening, $ kde4-config --version Qt: 4.6.0 KDE: 4.3.4 (KDE 4.3.4) kde4-config: 1.0 Still happening on: kde4-config --version Qt: 4.6.2 KDE: 4.3.5 (KDE 4.3.5) kde4-config: 1.0 http://img72.imageshack.us/img72/3733/42221637.png http://img99.imageshack.us/img99/3641/73259312.png I can't understand, how "this" could to get in the KDE trunk and stay there for so long time? Moreover, it is still there, without considering the fact, that the question already is about the loss of user data! I just have no words... http://bugs.kde.org/buglist.cgi?quicksearch=kio_thumbnail Probably the reason is because developers doesn't use thumbnails or doesn't have big video files. Anyway, I'm using KDE 4.4.x (now I've 4.4.4) and it is a couple of months that I don't have that problem, maybe it has been partially fixed in the latest release... I hope :-) I don't know about KDE 4.4.4 but I'm sure it still happens in trunk. This isn't the only one problem with kio_thumbnail. For example, you can try to start downloading few video files in ktorrent and open the target folder with preview. Or try to browse a remote folder that contains video files, etc. Seems that this component wasn't tested completely enough. i have same problem $ kde4-config --version Qt: 4.6.2 KDE: 4.4.4 (KDE 4.4.4) kde4-config: 1.0 I've completely removed mplayerthumbs from my systems and installed kffmpegthumbnailer instead. No more memory problems for me. Plus, kffmpegthumbnailer is really snappy, even on a netbook. (In reply to comment #18) > I've completely removed mplayerthumbs from my systems and installed > kffmpegthumbnailer instead. No more memory problems for me. Plus, > kffmpegthumbnailer is really snappy, even on a netbook. +1, but it has its own problems, and from time to time generates poor quality thumbnails. Are you using bittorrent? :) Can't say I do. But at least video clips downloaded "normally" don't cause any problems, the thumbnails are just regenerated regularly during the download. I have this problem too with kde 4.4.5, even thought I have disabled video previews and limited previews to files smaller than 5 MB. It seems to be related to the folder preview, as if I disable folder preview, the problem goes away. To disable video thumbnails run the command "mplayerthumbsconfig" and add all video extensions (avi, mpg, mkv, mp4, etc.) to the "Blacklisted file extensions". This solved the problem for me. It's been a few years... does anybody still see this issue with KDE Frameworks 5.44 or greater? Haven't heard anything from the reporter or anyone else who experienced this at some point in the past. Considering it fixed in KDE Frameworks 5. |
