Bug 318227

Summary: Data loss on restarting Akonadi while processing POP3 retrieval to get it responsive again
Product: [Frameworks and Libraries] Akonadi Reporter: Martin Steigerwald <Martin>
Component: generalAssignee: kdepim bugs <kdepim-bugs>
Status: RESOLVED WORKSFORME    
Severity: critical    
Priority: NOR    
Version: 4.10   
Target Milestone: ---   
Platform: Debian unstable   
OS: Linux   
Latest Commit: Version Fixed In:
Sentry Crash Report:
Attachments: .xsession-errors of this data loss occurence

Description Martin Steigerwald 2013-04-12 08:11:39 UTC
After Akonadi / MySQL / Virtuoso used up 2 to 4 logical cores of a quite capable Intel i5 Dualcore mobile CPU @ 2,5 GHz which this ThinkPad T520 can max out to 3/3,2 GHz quite consistenly for minutes after I had it get recent mails today, I thought restarting Akonadi would lead it to a sane state. In hindsight I think this wasn´t such a good idea.

But well additionally to not working out, it led to data loss.

Reproducible: Didn't try

Steps to Reproduce:
I obviously haven´t tried to reproduce this, as I prefer that my mails are safe. Tentative steps:

1.  Have a POP3 account with some mailing lists like kdepim-users and LKML or so that it has some traffic.
2. Wait a night to mails to pile up.
3. Receive and filter new mails to some sub folders
4. Type akonadictl restart while Akonadi is working
Actual Results:  
martin@merkaba:~/.Mail/.Lichtvoll.directory/.KDE.directory/pim-users-ml/cur> 
grep "KMail not updating source folders" *
1365696861.R215.merkaba:2,RS:Subject: [kdepim-users] KMail not updating 
source folders on move operations
1365697032.R861.merkaba:2,S:Subject: Re: [kdepim-users] KMail not updating 
source folders on move

While there has been an answer to my question that appeared at first, but was not there after the Akonadi restart anymore:

http://lists.kde.org/?l=kdepim-users&m=136574390903236&w=2

Expected Results:  
That answer would still be there locally and displayed just fine in KMail.

On akonadictl restart Akonadi complained it could not delete some mails via popup notice. Unfortunately these error messages are not in ~/.xsession-errors. And I cannot access them from systray anymore as well.

I try to dig out relevant parts of ~/.xsession-errors and attach these.

PS: KMail updates source folders now, so at least that seems settled for now.
Comment 1 Martin Steigerwald 2013-04-12 08:27:37 UTC
This may be related to

Bug #318224 - Tries to insert existing entries into database

(note the duplicate messages have been there before this akonadictl restart from this bug report)

Some additional info:

I use the following setup:

1) I had all mails in ~/Mail

2) I moved all recent mails from ~/Mail to a newly created BTRFS subvolume ~/.Mail so that I can snapshot it. I moved all the archived mails to ~/.Mail-Archiv, as these are mbox folders.

3) Then I let the migrator run so that it puts my account information into kmail2rc.

4) For each POP3 account I use I created an extra maildir agent pointing it at the maildir and then selecting "Synchronize All" in Akonadiconsole. Thus for my main account Lichtvoll I gave it ~/.Mail/Lichtvoll as maildir directory for example:

drwx------ 1 martin martin        18 Apr 11 08:33 Lichtvoll
drwx------ 1 martin martin      4106 Apr 11 17:28 .Lichtvoll.directory
-rw------- 1 martin martin   4312743 Apr 11 08:33 .Lichtvoll.index
-rw------- 1 martin martin     33909 Apr 11 08:33 .Lichtvoll.index.ids

For local mails I created:

drwx------ 1 martin martin        18 Jan  2 11:36 Lokal
drwxr-xr-x 1 martin martin       528 Apr 11 09:50 .Lokal.directory

and copied all the local mail folders into that.

5) I imported my filter rules back to this new setup, set inboxes for mail download to the right top level maildir directories and so on.


Size of setup:

martin@merkaba:~> find ~/.Mail | wc -l
431234
martin@merkaba:~> du -sh ~/.Mail
6,5G    /home/martin/.Mail


The KDEPIM-2 packages are from Debian developers, but they are not publically available in official Debian repositories. I test them for for them.

martin@merkaba:~> apt-show-versions | egrep "(kdepim|korganizer|kmail|knode|kaddressbook|zanshin|akonadi|nepomuk|strigi|virtuoso|soprano|libqt)"   akonadi-backend-mysql/experimental uptodate 1.9.1-2
akonadi-backend-postgresql/experimental uptodate 1.9.1-2
akonadi-backend-sqlite/experimental uptodate 1.9.1-2
akonadi-server/experimental uptodate 1.9.1-2
akonadiconsole 4:4.10.2-0r4 newer than version in archive
kaddressbook 4:4.10.2-0r4 newer than version in archive
kaddressbook-mobile 4:4.10.2-0r4 installed: No available version in archive
kdegraphics-strigi-analyzer/sid uptodate 4:4.8.4-1
kdepim 4:4.10.2-0r4 newer than version in archive
kdepim-dbg 4:4.10.2-0r4 newer than version in archive
kdepim-kresources 4:4.10.2-0r4 newer than version in archive
kdepim-mobileui-data 4:4.10.2-0r4 installed: No available version in archive
kdepim-runtime/experimental uptodate 4:4.10.2-1
kdepim-strigi-plugins 4:4.10.2-0r4 newer than version in archive
kdepimlibs-dbg/experimental uptodate 4:4.10.2-1
kdepimlibs-kio-plugins/experimental uptodate 4:4.10.2-1
kdesdk-strigi-plugins/sid uptodate 4:4.8.4+dfsg-1
kmail 4:4.10.2-0r4 newer than version in archive
kmail-mobile 4:4.10.2-0r4 installed: No available version in archive
knode 4:4.10.2-0r4 newer than version in archive
korganizer 4:4.10.2-0r4 newer than version in archive
korganizer-mobile 4:4.10.2-0r4 installed: No available version in archive
libakonadi-calendar4/experimental uptodate 4:4.10.2-1
libakonadi-contact4/experimental uptodate 4:4.10.2-1
libakonadi-kabc4/experimental uptodate 4:4.10.2-1
libakonadi-kcal4/experimental uptodate 4:4.10.2-1
libakonadi-kde4/experimental uptodate 4:4.10.2-1
libakonadi-kmime4/experimental uptodate 4:4.10.2-1
libakonadi-notes4/experimental uptodate 4:4.10.2-1
libakonadi-socialutils4/experimental uptodate 4:4.10.2-1
libakonadiprotocolinternals1/experimental uptodate 1.9.1-2
libkdepim4 4:4.10.2-0r4 newer than version in archive
libkdepimdbusinterfaces4 4:4.10.2-0r4 installed: No available version in archive
libkdepimmobileui4 4:4.10.2-0r4 installed: No available version in archive
libnepomuk4/experimental uptodate 4:4.10.2-2
libnepomukcore4/experimental uptodate 4:4.10.2-1
libnepomukquery4a/experimental uptodate 4:4.10.2-2
libnepomukutils4/experimental uptodate 4:4.10.2-2
libnepomukwidgets4/experimental uptodate 4:4.10.2-1
libqt4-dbg/sid uptodate 4:4.8.2+dfsg-11
libqt4-dbus/sid uptodate 4:4.8.2+dfsg-11
libqt4-declarative/sid uptodate 4:4.8.2+dfsg-11
libqt4-designer/sid uptodate 4:4.8.2+dfsg-11
libqt4-dev/sid uptodate 4:4.8.2+dfsg-11
libqt4-dev-bin/sid uptodate 4:4.8.2+dfsg-11
libqt4-help/sid uptodate 4:4.8.2+dfsg-11
libqt4-network/sid uptodate 4:4.8.2+dfsg-11
libqt4-opengl/sid uptodate 4:4.8.2+dfsg-11
libqt4-qt3support/sid uptodate 4:4.8.2+dfsg-11
libqt4-script/sid uptodate 4:4.8.2+dfsg-11
libqt4-scripttools/sid uptodate 4:4.8.2+dfsg-11
libqt4-sql/sid uptodate 4:4.8.2+dfsg-11
libqt4-sql-mysql/sid uptodate 4:4.8.2+dfsg-11
libqt4-sql-psql/sid uptodate 4:4.8.2+dfsg-11
libqt4-sql-sqlite/sid uptodate 4:4.8.2+dfsg-11
libqt4-svg/sid uptodate 4:4.8.2+dfsg-11
libqt4-test/sid uptodate 4:4.8.2+dfsg-11
libqt4-xml/sid uptodate 4:4.8.2+dfsg-11
libqt4-xmlpatterns/sid uptodate 4:4.8.2+dfsg-11
libqtassistantclient4/sid uptodate 4.6.3-4
libqtcore4/sid uptodate 4:4.8.2+dfsg-11
libqtdbus4/sid uptodate 4:4.8.2+dfsg-11
libqtglib-2.0-0/sid uptodate 0.10.2-2
libqtgstreamer-0.10-0/sid uptodate 0.10.2-2
libqtgstreamerui-0.10-0/sid uptodate 0.10.2-2
libqtgui4/sid uptodate 4:4.8.2+dfsg-11
libqtlocation1/sid uptodate 1.2.0-3
libqtruby4shared2/sid uptodate 4:4.8.4-1
libqtscript4-core/sid uptodate 0.2.0-1
libqtscript4-gui/sid uptodate 0.2.0-1
libqtscript4-network/sid uptodate 0.2.0-1
libqtscript4-sql/sid uptodate 0.2.0-1
libqtscript4-uitools/sid uptodate 0.2.0-1
libqtscript4-xml/sid uptodate 0.2.0-1
libqtwebkit-dev/sid uptodate 2.2.1-5
libqtwebkit4/sid uptodate 2.2.1-5
libqtwebkit4-dbg/sid uptodate 2.2.1-5
libqtxdg0/sid uptodate 0.4.1-2
libsmokeakonadi3/sid uptodate 4:4.8.4-1
libsmokenepomuk3/sid uptodate 4:4.8.4-1
libsmokesoprano3/sid uptodate 4:4.8.4-1
libsoprano-dev/experimental uptodate 2.9.0+dfsg1-1
libsoprano4/experimental uptodate 2.9.0+dfsg1-1
nepomuk-core-data/experimental uptodate 4:4.10.2-1
nepomuk-core-runtime/experimental uptodate 4:4.10.2-1
ruby-akonadi/sid uptodate 4:4.8.4-1
ruby-soprano/sid uptodate 4:4.8.4-1
soprano-daemon/experimental uptodate 2.9.0+dfsg1-1
virtuoso-minimal/experimental-snapshots uptodate 6.1.6+dfsg1-0r1
virtuoso-opensource-6.1-bin/experimental-snapshots uptodate 6.1.6+dfsg1-0r1
virtuoso-opensource-6.1-common/experimental-snapshots uptodate 6.1.6+dfsg1-0r1
zanshin/sid uptodate 0.2.1-1+b1
Comment 2 Martin Steigerwald 2013-04-12 08:29:09 UTC
Created attachment 78835 [details]
.xsession-errors of this data loss occurence

I reduced all mail addresses in there to domain names for spam / privacy reasons.
Comment 3 Martin Steigerwald 2013-04-12 11:12:35 UTC
To be fair, I just remembered that I clicked on "Abort activity" for one resource prior to restarting Akonadi. It didn´t seem that the resource would listen to me, but still this might have caused the trouble. Feel free to close, but before that it would still be good to be able to confirm that restarting Akonadi server is safe for mail data.

I fixed this data loss by removing last seen uid from ~/.kde/share/config/akonadi_pop3* which lead to

Bug 318233 - Remove duplicates takes lots of time and then doesn´t remove the duplicates anyway
Comment 4 Christoph Feck 2013-05-01 12:26:30 UTC
This bug seems to be assigned to the wrong component. Reassigning to general for inspection.
Comment 5 Martin Steigerwald 2015-03-12 12:24:07 UTC
I didn´t have any data loss like this in quite some time. As its probably difficult to trigger or reproduce it again and I like to focus on the issues I currently still see with Akonadi 1.13 and KDEPIM 4.14.x I am closing this one as well.