Bug 174663 - excessive memory use during file operations
Summary: excessive memory use during file operations
Status: RESOLVED FIXED
Alias: None
Product: amarok
Classification: Applications
Component: Collections/Local (show other bugs)
Version: 2.3-GIT
Platform: Compiled Sources Linux
: NOR normal
Target Milestone: 2.3.1
Assignee: Amarok Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2008-11-08 21:52 UTC by Evert Vorster
Modified: 2009-11-25 04:03 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Evert Vorster 2008-11-08 21:52:57 UTC
Version:            (using KDE 4.1.2)
Compiler:          gcc 4.3.2 -march=native -O2 -pipe
OS:                Linux
Installed from:    Compiled From Sources

Compiled amarok 1.94 from source with gcc 4.3.2

I wanted to clean up a little of my disk space, and get my music collection in a little better shape. Admittedly, the collection is a little large, with about 16,000 songs. 

To reproduce:
Right-Click on my local collection, and then pick Organize files. 
Ticked Sort by file type, and Replace spaces with underscores, restrict to ASCII. 

Then clicked on continue. The amarok gui becomes completely unusable, and when switching between screens and back it does not redraw. So I leave it alone for a few hours, even though file renames and moving on the same filesystem should be fairly quick. 

Result:
When I returned to my machine, all of my 2GB of ram was claimed by amarok, the CPU was still at 100% and my SWAP was thrashing madly. While amarok did not actually crash, it did become completely unresponsive. 

Investigation:
So, I killed the amarok process. 
Then I try again with a smaller set of files, ( one band ) and the file move operation was done almost instantly. 
So I try ever bigger selections of files with the time used by amarok to sort and move them increasing linearly with the amount of files picked. 

At about ten bands or so selected I run out of ram again and the time waited increases exponentially as swap is terribly slow. 

Conclusion:
Seems to me that amarok is trying to load every song in my collection in to ram before starting to move them. This is fine for handling small amounts of data, but makes no sense when pointed at a collection of about 100GB.

Expected behaviour:
amarok should be able to deal with huge amounts of files gracefully. Check how much ram is available and only run batches that will actually fit in to the RAM in serial rather than try to move all the files at once.
Comment 1 Seb Ruiz 2008-11-08 23:53:08 UTC
Sounds like a reasonable request.
Comment 2 Dan Meltzer 2008-12-02 18:28:18 UTC
Targeting this for 2.0.1 as it's a pretty important feature (and many people are going to want to organize their entire collection, so it will probably effect the masses.)
Comment 3 Mark Kretschmann 2009-03-17 19:07:04 UTC
Is this still valid with SVN trunk? The report is super old...
Comment 4 Seb Ruiz 2009-04-13 01:37:56 UTC
Yes, this report is still valid.
Comment 5 Myriam Schweingruber 2009-08-02 13:00:39 UTC
Any news on this? Changed version.
Comment 6 Evert Vorster 2009-11-25 04:03:17 UTC
Amarok does not eat up huge amounts of memory anymore.
 
Collection management still leaves a little to be desired, though. I suppose that will have to go into usability bugs?

I was able to move around 20Gb worth of songs from a directory to my collection.