Bug 258551 - KMailCVT leaks lots of memory when importing maildir
Summary: KMailCVT leaks lots of memory when importing maildir
Status: RESOLVED FIXED
Alias: None
Product: kmail
Classification: Applications
Component: kmailcvt (show other bugs)
Version: SVN trunk (KDE 4)
Platform: Unlisted Binaries Linux
: NOR normal
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-12-02 11:50 UTC by Alex Richardson
Modified: 2012-03-05 21:10 UTC (History)
3 users (show)

See Also:
Latest Commit:
Version Fixed In: 4.8.2


Attachments
compressed Valgrind output (tested with much smaller number of mails) (90.24 KB, application/zip)
2010-12-02 12:19 UTC, Alex Richardson
Details
compressed massif output (55.47 KB, application/x-gzip)
2011-11-14 19:37 UTC, Alex Richardson
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Alex Richardson 2010-12-02 11:50:00 UTC
I imported my old maildir to kmail2. I noticed that kmailcvt memory usage went up to 1282 MB, leaking 1 megabyte every 3-10 seconds.
Comment 1 Alex Richardson 2010-12-02 12:19:36 UTC
Created attachment 53973 [details]
compressed Valgrind output (tested with much smaller number of mails)
Comment 2 Elián Hanisch 2011-10-06 22:20:14 UTC
I can confirm this in kde4.7.1, from Kubuntu 11.10

importing my 700mb mail dir to kmail2 took several hours and the memory usage got up to 1.8gb. The memory was rising even after the task was completed, so I terminated the process.

This can be a problem if kmail2's migration assistant fails, and you need to import stuff manually.
Comment 3 Alex Richardson 2011-11-14 19:37:28 UTC
Created attachment 65674 [details]
compressed massif output

It is still present with current master. May also happen when importing other formats, but I only tested with importing from KMail.

As you can see from the attached massif log the memory is probably not leaked since it is freed on exit, however it is kept way too long (until importing has finished). When importing my old emails kmailcvt used over 2 Gigabytes of memory!
IMO the memory should be freed at least after every dir if not after every mail.
Comment 4 jtscsousa 2012-02-28 12:11:12 UTC
Still present in KDE 4.8.

I have about 7Gb of kmail archives, which I want to import in kmail2 for reference.

The import progresses and by the time it reaches 11% consumes 5566Mb do Virtual memory and 2729M of resident memory. This is a 4Gb machine with 4Gb swap, so I expect the process will be killed soon, like the fist time I tried to convert the folder.
Comment 5 Laurent Montel 2012-02-28 16:27:59 UTC
you use a kmail mail dir import filter or maildir filter import ?
Comment 6 jtscsousa 2012-02-28 17:32:43 UTC
I call kmailcvt directly from the terminal (no kmail nor kontact launched) and select "Import KMail Maildirs and Folder Structure". Then I select a local folder, press next and select my ~/Mail folder. It starts to run and at about 9-11% stalls and finally is killed by the kernel for filling up both ram and swap.

The only notice on the terminal is a simple 'killed'.

Superficial verification of imported mail and folders indicates the the process was progressing fine until killed.
Comment 7 Laurent Montel 2012-03-05 21:03:24 UTC
Git commit eafff41025775a8a2c15cd3f26233bd8d50cb277 by Montel Laurent.
Committed on 05/03/2012 at 21:59.
Pushed by mlaurent into branch 'master'.

Fix Bug 258551 - KMailCVT leaks lots of memory when importing maildir

Akonadi::ItemCreateJob was never delete during importing so memory used increase a lot.
(for example during importing 2 Go it takes 2.5Go virtual memory before, after my fix it takes just 400Mo)
Too bad that it's will not in 4.8.1
FIXED-IN: 4.8.2

M  +2    -0    mailimporter/filters.cpp

http://commits.kde.org/kdepim/eafff41025775a8a2c15cd3f26233bd8d50cb277
Comment 8 Laurent Montel 2012-03-05 21:10:51 UTC
Git commit 7e4953c54752dca8a10ad29adf2e225a96f8669a by Montel Laurent.
Committed on 05/03/2012 at 21:59.
Pushed by mlaurent into branch 'KDE/4.8'.

Fix Bug 258551 - KMailCVT leaks lots of memory when importing maildir

Akonadi::ItemCreateJob was never delete during importing so memory used increase a lot.
(for example during importing 2 Go it takes 2.5Go virtual memory before, after my fix it takes just 400Mo)
Too bad that it's will not in 4.8.1
FIXED-IN: 4.8.2
(cherry picked from commit eafff41025775a8a2c15cd3f26233bd8d50cb277)

If we can't update kdepim source please add this patch to your source please.

CCMAIL: kde-packager@kde.org

M  +2    -0    kmailcvt/filters.cxx

http://commits.kde.org/kdepim/7e4953c54752dca8a10ad29adf2e225a96f8669a