Bug 286978

Version:           4.7.3 (using KDE 4.7.3) 
OS:                Linux

Konqueror uses _far_ more memory than is reasonable for bookmarks. The more bookmarks that it has, the more memory that it consumes - which is reasonable - but the amount of memory is _not_ reasonable. And the fact that this increases the memory usage _per process_ of konqueror means that it has a major compounding effect. Even small amounts of excess memory usage can add up when many Konqueror windows are open, but the amount of excess memory usage is anything but small.

In order to quantify the problem, I ran a series of tests. One of the Konqueror windows which I had open had one tab which was open to www.google.com. With each different set of bookmarks, I rebooted the computer, measured the memory usage shortly after KDE finished loading, measured how opening and close the bookmark editor (without editing anything) affected memory usage, and measured memory usage after KDE had been running for several minutes to several hours. Fortunately, memory usage did not increase much when I wasn't using konqueror for long periods of time, but it _did_ increase if I opened the bookmark editor, and it's taking up far too much memory even if I never touch the window after KDE starts.

Each set of bookmarks that I used had folders in it with either 10 or 100 bookmarks per folder. They were all bookmarks for www.google.com with only the title tag, so each entry was actually fairly small.

With no bookmarks, the konqueror window took up 35,000KB - 37,000KB. Opening and closing the bookmark editor increased memory usage by about 100KB. After leaving Konqueror alone for a while, memory usage increased around 2000KB.

With 100 bookmarks (which ended up making the bookmarks file about 4.7KB), initial memory usage was 37,000KB - 39,000KB. Opening and closing the bookmark editor increased it from 100KB - 600KB. After leaving Konqueror alone for a while, memory usage increased 1000KB - 1500KB.

With 1000 bookmarks (around 87KB for the file), initial memory usage was around 39,000KB - 40,000KB. Opening and closing the bookmark editor increased memory usage by about 100KB. After leaving Konqueror alone for a while, memory usage had increased less than 1000K.

With 10,000 bookmarks, a much greater variance occurred dependending on whether I had 10 bookmarks or 100 bookmarks per folder. With 10 (the file was 880KB), initial memory usage was around 54,000KB. After opening and closing the bookmark editor, it was just over _80,000KB_. After leaving Konqueror alone for a while, memory had only increased about 300KB. With 100 bookmarks per folder (the file was 826KB), initial memory was somewhat lower at around 49,000KB. After opening and closing the bookmark editor, it had increased to nearly 64,000KB. After leaving Konqueror alone for a while, memory usage had increased by about 3000KB.

With 100,000 bookmarks, the 10 per folder version (8.7MB) had initial memory usage of 191,000KB - 192,000KB. After opening and closing the bookmark editor, it had increased to _430,000KB_. After leaving konqueror alone for a while, it had increased about 5000KB. With the 100 per folder version (8.2MB), initial memory consumption was around 179,000KB. Opening and closing the bookmark editor increased it to over 326,000KB. Letting it sit for a while only increased it by about 1000KB.

I also tested one bookmarks file which was the same as the 100,000KB bookmarks file with 10 bookmarks per folder - except that all of the folders except for the first one were in an XML comment, so there were actually only 10 bookmarks in the file. Its initial memory was around 54,000KB (so similar to the 10,000 bookmark file). Opening and closing the bookmark editor increased the memory usage to 75,000KB. I forgot to test how its usage increased over time, but none of the others increased all that much over time, so I don't expect that it would either.

Based on the case where there was no bookmark file, having a single konqueror window open with one tab which is open to www.google.com should take around 35,000KB. Even if we assume that the amount of memory required to manage the bookmarks in whatever structure they're held in memory is triple what the XML on disk requires, even the 8.7MB file should be increasing the memory consumption by only about 26.1MB. Instead, it looks like it generally takes in the realm of around 17 - 18 times as much memory initially, and increases to 25 - 35+ times if you open and close the bookmark editor (without actually editing anything). The memory usage does not appear to drop either - even if you remove the bookmarks file. And the fact that a bookmark file with only 10 bookmarks but enough comments to make it 8.7MB more than doubles the amount of memory whereas a bookmark file with 100 bookmarks increases it by less than 100KB means that something really off is happening when reading that bookmark file (and the fact that simply opening and closing the bookmark editor makes for such a large increase on top of that would further indicate that processing the bookmark file is either leaking memory or at least causing too much to be allocated which is not deallocated when it is no longer needed). Opening and closing the bookmark editor does not increase memory usage every time, but it always does the first time, and it always seems to if you wait a while before opening it again.

Okay. So maybe this report is a bit long, but I'm hoping that the details help make it clearer as to what is going on. Actual memory usage varies (my normal bookmarks do even worse - probably because they have longer URLs and actually have stuff like favicons listed, whereas these tests were run with bookmarks that only contained the title tag), but it's clearly quite high. Obviously, it's less noticeable if you don't have a lot of bookmarks and/or if you don't have many konqueror windows open, but if you get very many bookmarks _and_ have a lot of konqueror windows open, it starts to add up very fast. And even for those with few bookmarks and few konqueror windows open at a time, a lot of memory is being wasted percentage-wise.

Reproducible: Always

Steps to Reproduce:
It's always reproducible if you watch the memory usage of konqueror, but it's most noticeable when you have thousands of bookmarks.

Actual Results:  
Bookmarks take up around 18x as much space in memory as on disk, and opening the bookmarks editor causes it to be more like 30x and can increase several times if you open the editor several times with several minutes between each time you open and close it.

Expected Results:  
Bookmarks take up no more than maybe 2 - 3x as much space in memory as on disk (and preferably less than what's on disk, since it's in XML with no compression of the data of any kind) and opening and closing the bookmark editor should not result in a large increase in memory - at minimum, any increase in memory usage should go away after it's closed.

If it matters, I'm currently running x86-64 with Arch with an nvidia card with the proprietary drivers, but I've seen the same behavior under OpenSuSE with both the nouveau and nvidia drivers. And while I'm reporting this for KDE 4.7.3, I'm sure that it has existed for a while. The memory usage on my box has been getting steadily worse for quite some time, and I believe this bug to be the culprit.

I'm attaching the bookmarks file that I used which had 100,000 bookmarks in it with 100 per folder, so you can see what the files that I tested with looked like. They're all identical save for the number of bookmarks per folder and the total number of bookmarks in the file as described above.