Version: (using KDE KDE 3.4.2) Installed from: Compiled From Sources Compiler: GCC 3.4.3-20050110 OS: Linux Konqueror seems to do a lot of file IO while only browsing the web, despite web caching off, as well as swapping. If I put my IO scheduler under a load, for example, by copying a large file to another location on my hard drive, konqueror will freeze loading web pages. The copy will fall from 40MB/Sec to under 1MB/Sec, sometimes as low as 10k/sec, until konqueror unlocks. It may remain locked for as long as 5 seconds. During periods of low IO, konqueror sometimes jitters, but rarely locks. My hard drive has a seek time of something like 8ms, it's a Maxtor 7200RPM ATA100 drive. I don't BELIEVE the problem is hardware. It happens with both the CFQ and Deadline IO schedulers, which I find odd, since I almost suspect the IO scheduler. During periods of no IO, I can load web pages up in konqueror. I see my hard drive light flickering, despite Web Caching and Swapping off. What is konqueror doing with my hard drive? Firefox uses no file IO loading web pages, the light does not go on at all, nor does it have these locking problems. It almost seems that konqueror is not only accessing my hard drive when it shoulden't need to, but perhaps also doing a lot of IO, which is why it kills the speed of a file copy so much. What exactly is it doing, and is there any fix? Thanks.
I forgot to say, I'm running reiser4, perhaps FS latencies are part of the problem. Maybe it's also just killing the speed of my copy because my write_expire is set to 5000ms and my read to 500, and konqueror is reading. Still, it woulden't lock up at all if konqueror wasn't trying to access my drive while web browsing. Hrm.
open("/home/diefool/.kde/share/config/konquerorrctOlaQa.new", O_RDWR|O_CREAT|O_EXCL, 0600) = 13 umask(0) = 022 umask(022) = 0 fchmod(13, 0600) = 0 getgid32() = 100 getuid32() = 1000 fchown32(13, 1000, 100) = 0 fcntl64(13, F_SETFD, FD_CLOEXEC) = 0 stat64("/home/diefool/.kde/share/config/konquerorrc", {st_mode=S_IFREG|0600, st_size=3565, ...}) = 0 getuid32() = 1000 getgid32() = 100 fchmod(13, 0100600) = 0 fchmod(13, 0600) = 0 fcntl64(13, F_GETFL) = 0x2 (flags O_RDWR) fstat64(13, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 mmap2(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6445000 _llseek(13, 0, [0], SEEK_CUR) = 0 write(13, "[$Version]\nupdate_info=kfmclient"..., 3565) = 3565 fdatasync(13) = 0 close(13) = 0 It seems to be looking at konquerorrctOlaQa.new when it locks and unlocks. I wonder why.
open("/home/diefool/.kde/share/config/konq_historyVFRsda.new", O_RDWR|O_CREAT|O_EXCL, 0600) = 12 umask(0) = 022 umask(022) = 0 fchmod(12, 0600) = 0 getgid32() = 100 getuid32() = 1000 fchown32(12, 1000, 100) = 0 fcntl64(12, F_SETFD, FD_CLOEXEC) = 0 stat64("/home/diefool/.kde/share/config/konq_history", {st_mode=S_IFREG|0600, st_size=241, ...}) = 0 getuid32() = 1000 getgid32() = 100 fchmod(12, 0100600) = 0 fchmod(12, 0600) = 0 fcntl64(12, F_GETFL) = 0x2 (flags O_RDWR) fstat64(12, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 mmap2(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb64d4000 _llseek(12, 0, [0], SEEK_CUR) = 0 write(12, "[History]\nCompletionItems=unused"..., 241) = 241 fdatasync(freeze) also happens with konq_history
It looks like the problem stems from abusing the KConfig class. Konq access the HD a lot, when adding new history elements, and for some reason rereads/writes konquerorrc when I switch tabs. It would be best if this was all done in ram, and then dumped to the files when konqueror closes. The KConfig class probably doesn't directly support that....is there an API that could be used to generate a file on disk that would actually be stored in ram? then we could use like...konquerorrctemp and konq_historytemp or whatnot, those would be in ram, and then dump those files to konquerorrc and konq_history on exit. An easy hack.
For giggles, I thought I'd edit my startkde script. I take ~/.kde/share, rename it to sharebak copy it to /dev/shm, and softlink it to ~/.kde/.share Upon exit, it copies the data from shm back to ~/.kde/share So, I tried it out. KDE STARTS AT LEAST 3x AS FAST And konqueror also runs much much better, no freezing at all under heavy IO. If the startkde script hack was a bit more robust than what I posted, for example, determined if KDE was already running and didn't copy over the share directory since it's already done, is there any reason NOT to copy that stuff over to /dev/shm? Is it insecure in any way?
I've confirmed the freezing behavior here as well. Using Ubuntu breezy with KDE 3.4.2. My primary drive, hda, is a UDMA133 PATA Maxtor drive (hdparm -Tt /dev/hda: ~55MB/s) with hdparm tweaks (-c1 -d1 -u1 -m16). Kernel is 2.6.13-rc4-mm1. If I copy any large, _uncached_ file to another location on the same drive, Konqueror will effectively freeze up while it's waiting for fdatasync() to sync data updates to disk.
http://www.mail-archive.com/reiserfs-list%40namesys.com/msg19013.html Apparently, it's a known problem with reiser4 that fsync() is slow. I stumbled upon this.
> Apparently, it's a known problem with reiser4 that fsync() is slow. I stumbled upon this. I'm using ext3, so apparently it's slow with that as well. I still think it's a problem with Konqueror/KHTML/KDE, because Firefox is much bulkier and it most certainly doesn't suffer from the same issue. Probably because it doesn't needlessly call fsync/fdatasync.
I have made an utility that monitors file activity with inotify, here is the output when just refreshing www.google.com (same url that was already loaded). This may not be that much file activity, but because KConfig issues a sync() after every close, it degrades the overall system performance as stated by others earlier. ~/.kde3.4/share/config/kdeglobals opened ~/.kde3.4/share/config/kdeglobals closed ~/.kde3.4/share/config/kwalletrc opened ~/.kde3.4/share/config/kwalletrc closed ~/.kde3.4/share/apps/kwallet/ opened ~/.kde3.4/share/apps/kwallet/ closed ~/.kde3.4/share/apps/konqueror/konq_history4Hgpha.new created ~/.kde3.4/share/apps/konqueror/konq_history4Hgpha.new opened ~/.kde3.4/share/apps/konqueror/konq_history4Hgpha.new metadata changed ~/.kde3.4/share/apps/konqueror/konq_history4Hgpha.new metadata changed ~/.kde3.4/share/apps/konqueror/konq_history4Hgpha.new metadata changed ~/.kde3.4/share/apps/konqueror/konq_history4Hgpha.new modified ~/.kde3.4/share/apps/konqueror/konq_history4Hgpha.new modified ~/.kde3.4/share/apps/konqueror/konq_history4Hgpha.new modified ~/.kde3.4/share/apps/konqueror/konq_history4Hgpha.new saved & closed ~/.kde3.4/share/apps/konqueror/konq_history4Hgpha.new moved from here (cookie: 3037) ~/.kde3.4/share/apps/konqueror/konq_history moved to here (cookie: 3037) ~/.kde3.4/share/config/kdeglobals opened ~/.kde3.4/share/config/kdeglobals closed ~/.kde3.4/share/config/kwalletrc opened ~/.kde3.4/share/config/kwalletrc closed ~/.kde3.4/share/apps/kwallet/ opened ~/.kde3.4/share/apps/kwallet/ closed ~/.kde3.4/share/config/kdeglobals opened ~/.kde3.4/share/config/kdeglobals closed ~/.kde3.4/share/config/kwalletrc opened ~/.kde3.4/share/config/kwalletrc closed ~/.kde3.4/share/apps/kwallet/ opened ~/.kde3.4/share/apps/kwallet/ closed ~/.kde3.4/share/config/kdeglobals opened ~/.kde3.4/share/config/kdeglobals closed ~/.kde3.4/share/config/kwalletrc opened ~/.kde3.4/share/config/kwalletrc closed ~/.kde3.4/share/apps/kwallet/ opened ~/.kde3.4/share/apps/kwallet/ closed ~/.kde3.4/share/config/kdeglobals opened ~/.kde3.4/share/config/kdeglobals closed ~/.kde3.4/share/config/kwalletrc opened ~/.kde3.4/share/config/kwalletrc closed ~/.kde3.4/share/apps/kwallet/ opened ~/.kde3.4/share/apps/kwallet/ closed ~/.kde3.4/share/apps/konqueror/konq_historyohdCra.new created ~/.kde3.4/share/apps/konqueror/konq_historyohdCra.new opened ~/.kde3.4/share/apps/konqueror/konq_historyohdCra.new metadata changed ~/.kde3.4/share/apps/konqueror/konq_historyohdCra.new metadata changed ~/.kde3.4/share/apps/konqueror/konq_historyohdCra.new metadata changed ~/.kde3.4/share/apps/konqueror/konq_historyohdCra.new modified ~/.kde3.4/share/apps/konqueror/konq_historyohdCra.new modified ~/.kde3.4/share/apps/konqueror/konq_historyohdCra.new modified ~/.kde3.4/share/apps/konqueror/konq_historyohdCra.new saved & closed ~/.kde3.4/share/apps/konqueror/konq_historyohdCra.new moved from here (cookie: 3038) ~/.kde3.4/share/apps/konqueror/konq_history moved to here (cookie: 3038)
@richard: that's some hefty amount of files! i hope some konqi devs can have a look at this, as this would make konqi faster - isn't that what we all want?
*** This bug has been confirmed by popular vote. ***
hey, i wonder, would it matter if the 'crashes' extension is turned on or off? i don't have time to test it now, but someone could... or i can test it later today.
*** Bug 120303 has been marked as a duplicate of this bug. ***
I mounted .kde/share as tmpfs (ram disk). Konqueror (and KDE in general) has never been this fast. The difference is huge. I've got a laptop with a 5400rpm hard disk and ext3 filesystem. Disk activity caused by the fsyncs takes much longer than actually rendering a web page. The fsyncs also prevent the disk from spinning down while running on battery.
Yeah. I did that as well, but it's a hack. I really hope someone can look at this.
*** Bug 118698 has been marked as a duplicate of this bug. ***
*** Bug 125406 has been marked as a duplicate of this bug. ***
SVN commit 534549 by pfeiffer: use read-only KConfig objects, they won't sync() on destruction CCBUG: 110318 M +2 -2 kwallet.cc --- branches/KDE/3.5/kdelibs/kwallet/client/kwallet.cc #534548:534549 @@ -34,7 +34,7 @@ const QString Wallet::LocalWallet() { - KConfig cfg("kwalletrc"); + KConfig cfg("kwalletrc", true); cfg.setGroup("Wallet"); if (!cfg.readBoolEntry("Use One Wallet", true)) { QString tmp = cfg.readEntry("Local Wallet", "localwallet"); @@ -52,7 +52,7 @@ } const QString Wallet::NetworkWallet() { - KConfig cfg("kwalletrc"); + KConfig cfg("kwalletrc", true); cfg.setGroup("Wallet"); QString tmp = cfg.readEntry("Default Wallet", "kdewallet");
The above patch probably doesn't make much of a difference though, as the backend only sync()s when the KConfig object was modified (which it isn't in the above cases).
kio/misc/kwalletd heavily (ab)uses KConfig, so there might be some performance to gain.
well, its mostly kwallet, then? maybe fixing kwallet might also have an impact on other apps using it?
I have this freeze problem now in kde 3.5.2/debian with a lot of apps : Konqueror(always freezes when opening a web page), Kopete(when I start/switch to a chat), KMail(when I start viewing a mail). I think it is khtml problem.
Forgot : I have a 40 G 7200 RPM Maxtor and reiser partitions
I'm pretty sure its a sync problem because after the freeze the red led indicating hdd activity blinks for a sec
I've asked myself why is Konqueror writing to disk even when I'm just refreshing simple pages. Firefox does not do that. As noted above it was the konq_history file. I went up the chain from KonqHistoryManager::saveHistory() to KSaveFile::close() to KTempFile::sync(). Being curious what could brake I've cleared body of KTempFile::sync(). Running smoothly since then! Anyway why is konq_history so important that it needs to be synced everytime?
I created a small patch for that issue, works good for me and is already included in KDEmod of Archlinux. See attachment following...
Created attachment 21849 [details] Patch to make konqueror history saves asynchronous
This problem is even more noticable when using akregator to read feeds on my battery powered laptop. When viewing a new feed the disk have to spin up which takes some seconds and needs additional power. I've configured the laptop to use laptop-mode, so there should be no need to spin the disk up that often. The implicit fsync operation just to read and save the history shouldn't be necessary. I vote for the patch from Comment #27! I've created two Gentoo ebuilds (kdelibs-3.5.8 and libkonq-3.5.8) which include these patches. They are located in my portage overlay (http://lepetitfou.dyndns.org/projects/gentoo) for all those who what to try this patch under Gentoo. The browsing and feed-reading experience is much better now.
hm, yes, this does look sensible to me... But then again, even though I took a peek at the code, I wouldn't say I'm qualified to have a say aside from being an user ;-)
Ehh, WTF is KConfig doing fsync()ing things at all? Sorry but that's just plain stupid. Normal applications should pretty much never fsync(), the kernel will write stuff to the disk when its time comes, and before that no program notices the difference (except in speed).
This delay is also made possible thanks to KonqHistoryManager::saveHistory() being called 3 times for every page I load. That's no fun if you store a history of 5000 items and your computer is under high load. I'm going to try the patch in comment 27 and see if that improves things.
Did it work, Bram? I was thinking about this recently - it's a real shame this slows down browsing so much sometimes, considering all the work which went into performance. I now also get why the lockups are so dramatical in the face of a few concurrent readers - because the kernel gives a huge priority boost to readers, writes (syncs) can become very expensive. If you're compiling in the background, the sync will try to write out a lot of data - while reading is going on, so it'll take a while. Not good ;-)
@Jos: did you do your tests even on KDE4 or only to 3.5.x?
To be honest, I haven't really tried this lately... Besides, I run KDEMOD from Archlinux so my KDE 3.5.x has been fixed already in this regard. I haven't run KDE 4.x SVN extensively enough to know if the issue still exists. I wouldn't know how I could see if konqi sends a sync command when browsing. Maybe I could try to powerdown the harddrive and then try browsing, but I won't have time for that in the coming few days.
Unfortunately I haven't hard facts, but my impression is that it improved things.
Forgot to mention, I'm working with Konqueror 3.5.9.
In my case, it doesn't matter what browser is running. A box appears withe the message "usable auto proxy config script".....sometimes it helps to click config tools....otherwise it's just pain, disappearing/jumping mouse I suppose i shouldn't have waited until finals to report it...sry for typos, glta, l8r
KDE 3.5 is unmaintained. Is this still a problem in KDE 4? If not, this bug can be closed.
@Lemma: from kwallet side? any idea? Anyway big I/O operations are still a bottleneck in GNU/Linux enviroment, but it mostly depends from the kernel scheduler.
I've forget to write one thing: if konqueror needs to write/read the disk and a big copy operation is active, konqueror performance obviously depends from the current scheduler.
Nothing I can think of FiNeX. Of course the KWallet API reads kwalletrc but it shouldn't actually write (or even sync) it. If that's the case that would be a problem with KConfig writing back unchanged config files. Strange thing is I never really stumbled on this bug despite having used reiser4 and KDE3 for quite some time. Apart from that, if saving a large history using KConfig is a PITA, maybe switching all of it to something non-KConfig could help (KConfig syncing on saving doesn't seem like that bad an idea).
As I see it, it's the syncing that's the problem, and specifically that calling fsync() actually does something different than is actually wanted in this case (I believe). Note that what fsync() does is effectively say: Write this out as soon as possible, block until that is done. It's a tool to play on exact ordering of writes to facilitate recovery. What it seems KDE folks want, since they don't depend on the exact ordering of writes for recovery, is: Write this out as soon as possible, but also return as soon as possible (don't block until it's done). Is there a case, real one I mean, not a hypothetical one, where the first case is desired by the applications using KConfig? Unfortunately I don't think POSIX offers a better way to do the latter than spawning a new thread and calling fsync() from there while the main thread continues (and doesn't freeze Konqueror). Whether this is still a problem with Konqueror, I don't know; I switched to Firefox when Konqueror was outright unusably buggy around 4.1, and have yet to switch back. I do hope to switch one day back, since I don't really like Firefox and its bloat...
sam, you said, Sami Liedes <sliedes cc hut fi> 2009-12-01 23:11:54 --- As I see it, it's the syncing that's the problem, and specifically that calling fsync() does is effectively say: Write this out as soon as possible, block until that is done. It's a tool to play on exact ordering of writes to facilitate recovery. first question, did you access the fsync? or kconfig and how. KDE folks want, since they don't depend on the exact ordering of writes for recovery, is: Write this out as soon as possible, but also return as soon as possible (don't block until it's done). makes sense, please refer to question #1 ffox seems to try harder when erasing personal info, cach, etc when closing. I notice that something is sucking ram after browser startup. I suspect adware or spyware designed for linux. imho good luck to all, jd ____________________________________________________________ Nutrition Improve your career health. Click now to study nutrition! http://thirdpartyoffers.juno.com/TGL2141/c?cp=1un6vkbKCDop5sF6BbZcJwAAJ1C5rjx__MSrDJtOPAULkXkGAAYAAAAAAAAAAAAAAAAAAADNAAAAAAAAAAAAAAAAAAASQwAAAAA=
Can anyone confirm if this issue is valid in KDE v4.7.4 or higher. I do not see this problem on my machine at all.
If this issue is valid in KDE 4, please comment on this ticket and provide a method you used to reproduce the problem.
Closing based on comments #44 and #45