Bug 318233

Summary: Remove duplicates takes lots of time and then doesn´t remove the duplicates anyway
Product: [Applications] kmail2 Reporter: Martin Steigerwald <Martin>
Component: miscAssignee: kdepim bugs <kdepim-bugs>
Status: RESOLVED DUPLICATE    
Severity: normal CC: dvratil, ianseeks
Priority: NOR    
Version: 4.11   
Target Milestone: ---   
Platform: Debian unstable   
OS: Linux   
Latest Commit: Version Fixed In:
Sentry Crash Report:

Description Martin Steigerwald 2013-04-12 10:39:51 UTC 
Due to the data loss I experienced bug #318227 I removed last seen uids from

martin@merkaba:~/.kde/share/config> ls akonadi_pop3_resource_*
akonadi_pop3_resource_0rc  akonadi_pop3_resource_2rc  akonadi_pop3_resource_4rc  akonadi_pop3_resource_6rc
akonadi_pop3_resource_1rc  akonadi_pop3_resource_3rc  akonadi_pop3_resource_5rc

and had Akonadi redownload all those mails again by triggering this via KMail.

Then I tried to remove duplicates with "Folder/Remove Duplicate Mails". It says for a long time that it tries to retrieve items. I let it sit for a while. But then, duplicates were still there, after the progress information has been gone. I even restarted KMail, since that helped, when folders aren´t updated. Ssee thread:

 [kdepim-users] KMail not updating source folders on move operations unless restarted
http://lists.kde.org/?l=kdepim-users&m=136569681820899&w=2

Reproducible: Always

Steps to Reproduce:
1. Have KMail somehow have duplicate mails, removing last seen uid list from POP3 resource works just fine for that.
2. Choose a folder of about 1000 mails or so with some duplicates in it.
3. Choose "Remove Duplicate Mails" from "Folder" menu.
Actual Results:  
Takes quite some time. Mails are not removed from the folder mail list in KMail.

Expected Results:  
Duplicate mails are removed quickly.

This was during only "light" activity of Nepomuk/Virtuoso. Since switch to KDEPIM-2 from KDE SC 4.10.2 it has never been actually idle so far, but it used only about 2 of 4 logical cores of Intel Sandybridge i5-2520M cpu in this ThinkPad T520 which is also equipped with an Intel SSD 320.  A hardware which is supposed to be able to remove some duplicate mails in a eye blink.

Some additional info:

I use the following setup:

1) I had all mails in ~/Mail

2) I moved all recent mails from ~/Mail to a newly created BTRFS subvolume ~/.Mail so that I can snapshot it. I moved all the archived mails to ~/.Mail-Archiv, as these are mbox folders.

3) Then I let the migrator run so that it puts my account information into kmail2rc.

4) For each POP3 account I use I created an extra maildir agent pointing it at the maildir and then selecting "Synchronize All" in Akonadiconsole. Thus for my main account Lichtvoll I gave it ~/.Mail/Lichtvoll as maildir directory for example:

drwx------ 1 martin martin        18 Apr 11 08:33 Lichtvoll
drwx------ 1 martin martin      4106 Apr 11 17:28 .Lichtvoll.directory
-rw------- 1 martin martin   4312743 Apr 11 08:33 .Lichtvoll.index
-rw------- 1 martin martin     33909 Apr 11 08:33 .Lichtvoll.index.ids

For local mails I created:

drwx------ 1 martin martin        18 Jan  2 11:36 Lokal
drwxr-xr-x 1 martin martin       528 Apr 11 09:50 .Lokal.directory

and copied all the local mail folders into that.

5) I imported my filter rules back to this new setup, set inboxes for mail download to the right top level maildir directories and so on.


Size of setup:

martin@merkaba:~> find ~/.Mail | wc -l
431234
martin@merkaba:~> du -sh ~/.Mail
6,5G    /home/martin/.Mail


The KDEPIM-2 packages are from Debian developers, but they are not publically available in official Debian repositories. I test them for for them.

martin@merkaba:~> apt-show-versions | egrep "(kdepim|korganizer|kmail|knode|kaddressbook|zanshin|akonadi|nepomuk|strigi|virtuoso|soprano|libqt)"   akonadi-backend-mysql/experimental uptodate 1.9.1-2
akonadi-backend-postgresql/experimental uptodate 1.9.1-2
akonadi-backend-sqlite/experimental uptodate 1.9.1-2
akonadi-server/experimental uptodate 1.9.1-2
akonadiconsole 4:4.10.2-0r4 newer than version in archive
kaddressbook 4:4.10.2-0r4 newer than version in archive
kaddressbook-mobile 4:4.10.2-0r4 installed: No available version in archive
kdegraphics-strigi-analyzer/sid uptodate 4:4.8.4-1
kdepim 4:4.10.2-0r4 newer than version in archive
kdepim-dbg 4:4.10.2-0r4 newer than version in archive
kdepim-kresources 4:4.10.2-0r4 newer than version in archive
kdepim-mobileui-data 4:4.10.2-0r4 installed: No available version in archive
kdepim-runtime/experimental uptodate 4:4.10.2-1
kdepim-strigi-plugins 4:4.10.2-0r4 newer than version in archive
kdepimlibs-dbg/experimental uptodate 4:4.10.2-1
kdepimlibs-kio-plugins/experimental uptodate 4:4.10.2-1
kdesdk-strigi-plugins/sid uptodate 4:4.8.4+dfsg-1
kmail 4:4.10.2-0r4 newer than version in archive
kmail-mobile 4:4.10.2-0r4 installed: No available version in archive
knode 4:4.10.2-0r4 newer than version in archive
korganizer 4:4.10.2-0r4 newer than version in archive
korganizer-mobile 4:4.10.2-0r4 installed: No available version in archive
libakonadi-calendar4/experimental uptodate 4:4.10.2-1
libakonadi-contact4/experimental uptodate 4:4.10.2-1
libakonadi-kabc4/experimental uptodate 4:4.10.2-1
libakonadi-kcal4/experimental uptodate 4:4.10.2-1
libakonadi-kde4/experimental uptodate 4:4.10.2-1
libakonadi-kmime4/experimental uptodate 4:4.10.2-1
libakonadi-notes4/experimental uptodate 4:4.10.2-1
libakonadi-socialutils4/experimental uptodate 4:4.10.2-1
libakonadiprotocolinternals1/experimental uptodate 1.9.1-2
libkdepim4 4:4.10.2-0r4 newer than version in archive
libkdepimdbusinterfaces4 4:4.10.2-0r4 installed: No available version in archive
libkdepimmobileui4 4:4.10.2-0r4 installed: No available version in archive
libnepomuk4/experimental uptodate 4:4.10.2-2
libnepomukcore4/experimental uptodate 4:4.10.2-1
libnepomukquery4a/experimental uptodate 4:4.10.2-2
libnepomukutils4/experimental uptodate 4:4.10.2-2
libnepomukwidgets4/experimental uptodate 4:4.10.2-1
libqt4-dbg/sid uptodate 4:4.8.2+dfsg-11
libqt4-dbus/sid uptodate 4:4.8.2+dfsg-11
libqt4-declarative/sid uptodate 4:4.8.2+dfsg-11
libqt4-designer/sid uptodate 4:4.8.2+dfsg-11
libqt4-dev/sid uptodate 4:4.8.2+dfsg-11
libqt4-dev-bin/sid uptodate 4:4.8.2+dfsg-11
libqt4-help/sid uptodate 4:4.8.2+dfsg-11
libqt4-network/sid uptodate 4:4.8.2+dfsg-11
libqt4-opengl/sid uptodate 4:4.8.2+dfsg-11
libqt4-qt3support/sid uptodate 4:4.8.2+dfsg-11
libqt4-script/sid uptodate 4:4.8.2+dfsg-11
libqt4-scripttools/sid uptodate 4:4.8.2+dfsg-11
libqt4-sql/sid uptodate 4:4.8.2+dfsg-11
libqt4-sql-mysql/sid uptodate 4:4.8.2+dfsg-11
libqt4-sql-psql/sid uptodate 4:4.8.2+dfsg-11
libqt4-sql-sqlite/sid uptodate 4:4.8.2+dfsg-11
libqt4-svg/sid uptodate 4:4.8.2+dfsg-11
libqt4-test/sid uptodate 4:4.8.2+dfsg-11
libqt4-xml/sid uptodate 4:4.8.2+dfsg-11
libqt4-xmlpatterns/sid uptodate 4:4.8.2+dfsg-11
libqtassistantclient4/sid uptodate 4.6.3-4
libqtcore4/sid uptodate 4:4.8.2+dfsg-11
libqtdbus4/sid uptodate 4:4.8.2+dfsg-11
libqtglib-2.0-0/sid uptodate 0.10.2-2
libqtgstreamer-0.10-0/sid uptodate 0.10.2-2
libqtgstreamerui-0.10-0/sid uptodate 0.10.2-2
libqtgui4/sid uptodate 4:4.8.2+dfsg-11
libqtlocation1/sid uptodate 1.2.0-3
libqtruby4shared2/sid uptodate 4:4.8.4-1
libqtscript4-core/sid uptodate 0.2.0-1
libqtscript4-gui/sid uptodate 0.2.0-1
libqtscript4-network/sid uptodate 0.2.0-1
libqtscript4-sql/sid uptodate 0.2.0-1
libqtscript4-uitools/sid uptodate 0.2.0-1
libqtscript4-xml/sid uptodate 0.2.0-1
libqtwebkit-dev/sid uptodate 2.2.1-5
libqtwebkit4/sid uptodate 2.2.1-5
libqtwebkit4-dbg/sid uptodate 2.2.1-5
libqtxdg0/sid uptodate 0.4.1-2
libsmokeakonadi3/sid uptodate 4:4.8.4-1
libsmokenepomuk3/sid uptodate 4:4.8.4-1
libsmokesoprano3/sid uptodate 4:4.8.4-1
libsoprano-dev/experimental uptodate 2.9.0+dfsg1-1
libsoprano4/experimental uptodate 2.9.0+dfsg1-1
nepomuk-core-data/experimental uptodate 4:4.10.2-1
nepomuk-core-runtime/experimental uptodate 4:4.10.2-1
ruby-akonadi/sid uptodate 4:4.8.4-1
ruby-soprano/sid uptodate 4:4.8.4-1
soprano-daemon/experimental uptodate 2.9.0+dfsg1-1
virtuoso-minimal/experimental-snapshots uptodate 6.1.6+dfsg1-0r1
virtuoso-opensource-6.1-bin/experimental-snapshots uptodate 6.1.6+dfsg1-0r1
virtuoso-opensource-6.1-common/experimental-snapshots uptodate 6.1.6+dfsg1-0r1
zanshin/sid uptodate 0.2.1-1+b1
Comment 1 Martin Steigerwald 2013-04-12 10:42:24 UTC 
Akonadi Nepomuk Feeder reported:

"Nepomuk is not operational: Nepomuk is not running."

during above stuff. I left it at that as Nepomuk was still indexing files and I want to save a bit on resources. I know I can restart it. I will do so later on if things settled down.
Comment 2 Ian Powell 2013-05-20 15:05:28 UTC 
I can confirm that they do not get removed.
If i select all emails in a folder with Ctrl-A then select "Folders/Remove 
Duplicate Mails" from the menu, its runs for a while and no duplicates are 
removed.

regards

Ian
Comment 3 Daniel Vrátil 2013-08-28 13:53:29 UTC 
Removing Duplicates is implemented in KMail, not in Akonadi. Reassigning.
Comment 4 Martin Steigerwald 2015-03-12 12:42:07 UTC 
I think I reported a duplicate here:

Bug 340759 - deleting duplicate mails has no effect
Bug 318233 - Remove duplicates takes lots of time and then doesn´t remove the duplicates anyway

I am not sure whether the takes long time aspect till is true. But last time I tried removing duplicates didn´t work reliably.

As this bug still refers Nepomuk, I mark it as a duplicate of the more recent bug 340759.

*** This bug has been marked as a duplicate of bug 340759 ***