Again I had data loss (see bug #318290 and bug #318227). I cannot yet fully prove that the BTRFS filesystem the maildirs are stored on is okay, since scrubbing it gives a kernel backtrace. I pinged the kernel developers about this. I have no visible sign of filesystem corruption, no recorded errors in BTRFS or in syslog/kern.log, backup went through with rsync without any checksum failures which BTRFS would answer with input/output error. I read a thread in debian-kde mailing list and saw a mail vom Modestas Vainius and some other mails in there. The mail was accessible just fine but I read some other mails from that thread first. Then I tried to access the mail vom Modestas Vainius again and I got an empty mail similar to bug 318290: No Subject From: To: Date: Unknown I tried some other further mails in the thread and same result. Then I though well maybe Akonadi needs a bit to retrieve these, but then it usually shows a message about currently retrieving mails. Anyway I read some other mail in that mailing list. Then KMail removed the entries for that shown as empty mails from the message list window. And well, they remained gone. Reproducible: Sometimes Steps to Reproduce: I have no idea how this happened. Actual Results: Mail is gone: martin@merkaba:~/.Mail/.Lichtvoll.directory/.Debian.directory/kde-ml#1> grep -ir "201304152120.40048.modax@debian.org" . martin@merkaba:~/.Mail/.Lichtvoll.directory/.Debian.directory/kde-ml#1> Also see screenshot that I attached. Expected Results: In no circumstance, I repead: *in no circumstance*, KMail may remove any mail that it downloaded via POP3 in that case or that is on an IMAP server *without* my explication instruction to do so. The local maildir directories have to be authoritative over the cache. So if the cache is broken, KMail may never ever delete any mail from the mail source, be it a locally stored maildir / mbox, or an IMAP account, but it has to rebuilt the cache from scratch if need be. Since I am pretty much sure meanwhile that this is not an BTRFS filesystem issue, but an issue in Akonadi, I consider downgrading to KDEPIM 1 again right now. I use the following setup: 1) I had all mails in ~/Mail 2) I moved all recent mails from ~/Mail to a newly created BTRFS subvolume ~/.Mail so that I can snapshot it. I moved all the archived mails to ~/.Mail-Archiv, as these are mbox folders. 3) Then I let the migrator run so that it puts my account information into kmail2rc. 4) For each POP3 account I use I created an extra maildir agent pointing it at the maildir and then selecting "Synchronize All" in Akonadiconsole. Thus for my main account Lichtvoll I gave it ~/.Mail/Lichtvoll as maildir directory for example: drwx------ 1 martin martin 18 Apr 11 08:33 Lichtvoll drwx------ 1 martin martin 4106 Apr 11 17:28 .Lichtvoll.directory -rw------- 1 martin martin 4312743 Apr 11 08:33 .Lichtvoll.index -rw------- 1 martin martin 33909 Apr 11 08:33 .Lichtvoll.index.ids For local mails I created: drwx------ 1 martin martin 18 Jan 2 11:36 Lokal drwxr-xr-x 1 martin martin 528 Apr 11 09:50 .Lokal.directory and copied all the local mail folders into that. 5) I imported my filter rules back to this new setup, set inboxes for mail download to the right top level maildir directories and so on. Size of setup: martin@merkaba:~> find ~/.Mail | wc -l 431234 martin@merkaba:~> du -sh ~/.Mail 6,5G /home/martin/.Mail The KDEPIM-2 packages are from Debian developers, but they are not publically available in official Debian repositories. I test them for for them. martin@merkaba:~> apt-show-versions | egrep "(kdepim|korganizer|kmail|knode|kaddressbook|zanshin|akonadi|nepomuk|strigi|virtuoso|soprano|libqt)" akonadi-backend-mysql/experimental uptodate 1.9.1-2 akonadi-backend-postgresql/experimental uptodate 1.9.1-2 akonadi-backend-sqlite/experimental uptodate 1.9.1-2 akonadi-server/experimental uptodate 1.9.1-2 akonadiconsole 4:4.10.2-0r4 newer than version in archive kaddressbook 4:4.10.2-0r4 newer than version in archive kaddressbook-mobile 4:4.10.2-0r4 installed: No available version in archive kdegraphics-strigi-analyzer/sid uptodate 4:4.8.4-1 kdepim 4:4.10.2-0r4 newer than version in archive kdepim-dbg 4:4.10.2-0r4 newer than version in archive kdepim-kresources 4:4.10.2-0r4 newer than version in archive kdepim-mobileui-data 4:4.10.2-0r4 installed: No available version in archive kdepim-runtime/experimental uptodate 4:4.10.2-1 kdepim-strigi-plugins 4:4.10.2-0r4 newer than version in archive kdepimlibs-dbg/experimental uptodate 4:4.10.2-1 kdepimlibs-kio-plugins/experimental uptodate 4:4.10.2-1 kdesdk-strigi-plugins/sid uptodate 4:4.8.4+dfsg-1 kmail 4:4.10.2-0r4 newer than version in archive kmail-mobile 4:4.10.2-0r4 installed: No available version in archive knode 4:4.10.2-0r4 newer than version in archive korganizer 4:4.10.2-0r4 newer than version in archive korganizer-mobile 4:4.10.2-0r4 installed: No available version in archive libakonadi-calendar4/experimental uptodate 4:4.10.2-1 libakonadi-contact4/experimental uptodate 4:4.10.2-1 libakonadi-kabc4/experimental uptodate 4:4.10.2-1 libakonadi-kcal4/experimental uptodate 4:4.10.2-1 libakonadi-kde4/experimental uptodate 4:4.10.2-1 libakonadi-kmime4/experimental uptodate 4:4.10.2-1 libakonadi-notes4/experimental uptodate 4:4.10.2-1 libakonadi-socialutils4/experimental uptodate 4:4.10.2-1 libakonadiprotocolinternals1/experimental uptodate 1.9.1-2 libkdepim4 4:4.10.2-0r4 newer than version in archive libkdepimdbusinterfaces4 4:4.10.2-0r4 installed: No available version in archive libkdepimmobileui4 4:4.10.2-0r4 installed: No available version in archive libnepomuk4/experimental uptodate 4:4.10.2-2 libnepomukcore4/experimental uptodate 4:4.10.2-1 libnepomukquery4a/experimental uptodate 4:4.10.2-2 libnepomukutils4/experimental uptodate 4:4.10.2-2 libnepomukwidgets4/experimental uptodate 4:4.10.2-1 libqt4-dbg/sid uptodate 4:4.8.2+dfsg-11 libqt4-dbus/sid uptodate 4:4.8.2+dfsg-11 libqt4-declarative/sid uptodate 4:4.8.2+dfsg-11 libqt4-designer/sid uptodate 4:4.8.2+dfsg-11 libqt4-dev/sid uptodate 4:4.8.2+dfsg-11 libqt4-dev-bin/sid uptodate 4:4.8.2+dfsg-11 libqt4-help/sid uptodate 4:4.8.2+dfsg-11 libqt4-network/sid uptodate 4:4.8.2+dfsg-11 libqt4-opengl/sid uptodate 4:4.8.2+dfsg-11 libqt4-qt3support/sid uptodate 4:4.8.2+dfsg-11 libqt4-script/sid uptodate 4:4.8.2+dfsg-11 libqt4-scripttools/sid uptodate 4:4.8.2+dfsg-11 libqt4-sql/sid uptodate 4:4.8.2+dfsg-11 libqt4-sql-mysql/sid uptodate 4:4.8.2+dfsg-11 libqt4-sql-psql/sid uptodate 4:4.8.2+dfsg-11 libqt4-sql-sqlite/sid uptodate 4:4.8.2+dfsg-11 libqt4-svg/sid uptodate 4:4.8.2+dfsg-11 libqt4-test/sid uptodate 4:4.8.2+dfsg-11 libqt4-xml/sid uptodate 4:4.8.2+dfsg-11 libqt4-xmlpatterns/sid uptodate 4:4.8.2+dfsg-11 libqtassistantclient4/sid uptodate 4.6.3-4 libqtcore4/sid uptodate 4:4.8.2+dfsg-11 libqtdbus4/sid uptodate 4:4.8.2+dfsg-11 libqtglib-2.0-0/sid uptodate 0.10.2-2 libqtgstreamer-0.10-0/sid uptodate 0.10.2-2 libqtgstreamerui-0.10-0/sid uptodate 0.10.2-2 libqtgui4/sid uptodate 4:4.8.2+dfsg-11 libqtlocation1/sid uptodate 1.2.0-3 libqtruby4shared2/sid uptodate 4:4.8.4-1 libqtscript4-core/sid uptodate 0.2.0-1 libqtscript4-gui/sid uptodate 0.2.0-1 libqtscript4-network/sid uptodate 0.2.0-1 libqtscript4-sql/sid uptodate 0.2.0-1 libqtscript4-uitools/sid uptodate 0.2.0-1 libqtscript4-xml/sid uptodate 0.2.0-1 libqtwebkit-dev/sid uptodate 2.2.1-5 libqtwebkit4/sid uptodate 2.2.1-5 libqtwebkit4-dbg/sid uptodate 2.2.1-5 libqtxdg0/sid uptodate 0.4.1-2 libsmokeakonadi3/sid uptodate 4:4.8.4-1 libsmokenepomuk3/sid uptodate 4:4.8.4-1 libsmokesoprano3/sid uptodate 4:4.8.4-1 libsoprano-dev/experimental uptodate 2.9.0+dfsg1-1 libsoprano4/experimental uptodate 2.9.0+dfsg1-1 nepomuk-core-data/experimental uptodate 4:4.10.2-1 nepomuk-core-runtime/experimental uptodate 4:4.10.2-1 ruby-akonadi/sid uptodate 4:4.8.4-1 ruby-soprano/sid uptodate 4:4.8.4-1 soprano-daemon/experimental uptodate 2.9.0+dfsg1-1 virtuoso-minimal/experimental-snapshots uptodate 6.1.6+dfsg1-0r1 virtuoso-opensource-6.1-bin/experimental-snapshots uptodate 6.1.6+dfsg1-0r1 virtuoso-opensource-6.1-common/experimental-snapshots uptodate 6.1.6+dfsg1-0r1 zanshin/sid uptodate 0.2.1-1+b1
Created attachment 78959 [details] Picture that shows a clear data loss The mail from Modestas definately has been displayed by KMail. Then it displayed as empty and then KMail/Akonadi removed it from the message list view.
Unfortunately I missed storing away ~/.xsession-errors this time. I wasn´t fully aware that its overwritten on each KDE login and relogged after downgrading to KDE SC 4.8.4 which allows to be installed with KDEPIM 4.4.11 in Debian. But I will gather my exact setup configuration from the config files from an BTRFS snapshot in case its needed. My other probably related data loss related bug reports contain more data: [Akonadi] [Bug 318227] New: Data loss on restarting Akonadi to get it responsive again [Akonadi] [Bug 318290] New: Empty mails: AkonadiAgentServer(4890)/libakonadi Akonadi::ResourceBase::itemRetrieved: Item does not provide part "HEAD"/"RFC822"
I'm surprised by this bug, as I use maildir with pop3 exetnsively and don't have mail losses. Can you try to put the Mail directory onto an ext4 fs and see if you still have problems?
András, thank you. I am surprised as well, cause I did not read about any data losses since quite some time on kdepim and kdepim-users mailing lists, but I do not know how many users use POP3. So you think that some detail in BTRFS behavior may cause this? I am pretty confident that the BTRFS filesystem is okay despite that scrubbing issue I mentioned elsewhere, but even then BTRFS may implement some filesystem operation differently than other filesystems. I do have a way to test on Ext4, but I do not like to move my /home filesystem around. Symlinking the main maildir directory to an should work I think. Would you also recommend to put the MySQL database and the Virtuoso database to an Ext4 filesystem (I have about 25 GB left for one)? Would symlinking ~/.kde/share/apps/nepomuk as well as ~/.local/share/akonadi to the Ext4 partition work there as well? Is my setup as described above sane otherwise? I won´t test it this week while in Berlin and will wait for a time where I have some free time to deal with any issues, but I am willing to test it on Ext4. Thanks, Martin
Guys, this has nothing to do with data loss caused by filesystem. It's simple: if we check the bug poster's name, Martin Steigerwald, we'll notice he is German, or, better, comes from a non-English country. I've reproduced this issue, and now I know why does it happen. STEPS TO REPRODUCE. 1. Install KDE without proper language packages, but in your language (let's say, Spanish). 2. Run KDE and start Akonadi. The Akonadi Maildir resource will create folders with special names ("inbox", among others) 3. Install your proper language packages, switch your KDE language to your own (Spanish) and reboot. 4. The Akonadi Maildir resource, under certain conditions, will create again those special folders, but with LOCALIZED NAMES ("bandeja de entrada", for instance) 5. Now Akonadi will confuse itself between "inbox" and "bandeja de entrada" and will sometimes send mail to the wrong folder. Also, sometimes, all mail will disappear, since KDE will attempt to display the other Inbox folder, which is empty. The solution seems to be: wait until Akonadi displays both inboxes, change the name of the only inbox whose name can be changed, move all mail from one to another, and delete the bogus inbox folder. I've tested that, it works, and since I did that I'm not losing mails anymore. This also provokes a host of side effects, including a memory leak and a hang in Kontact. Suggestion: please stop creating localized folders and simply create "inbox" folders, and then localize the name. The code for that exists, and it currently competes with the code which creates localized folders.
This speaks by itself [elyria@localhost local-mail]$ ls -la total 68 drwxrwxr-x. 17 elyria elyria 4096 oct 4 11:38 . drwxr-xr-x. 15 elyria elyria 4096 oct 8 08:08 .. drwxrwxr-x. 5 elyria elyria 4096 oct 4 11:37 bandeja de entrada drwxrwxr-x. 5 elyria elyria 4096 oct 4 11:37 bandeja de salida drwxrwxr-x. 5 elyria elyria 4096 oct 4 11:37 borradores drwxrwxr-x. 5 elyria elyria 4096 oct 4 11:37 correo-enviado drwxrwxr-x. 2 elyria elyria 4096 oct 1 11:00 cur drwxrwxr-x. 5 elyria elyria 4096 oct 1 11:00 drafts drwxrwxr-x. 5 elyria elyria 4096 oct 1 11:00 inbox drwxrwxr-x. 2 elyria elyria 4096 oct 1 11:00 new drwxrwxr-x. 5 elyria elyria 4096 oct 1 11:00 outbox drwxrwxr-x. 5 elyria elyria 4096 oct 4 11:38 papelera drwxrwxr-x. 5 elyria elyria 4096 oct 4 11:38 plantillas drwxrwxr-x. 5 elyria elyria 4096 oct 1 11:00 sent-mail drwxrwxr-x. 5 elyria elyria 4096 oct 1 11:00 templates drwxrwxr-x. 2 elyria elyria 4096 oct 1 11:00 tmp drwxrwxr-x. 5 elyria elyria 4096 oct 1 11:00 trash
Check also how many local mail directories do you have in your ~/.local/share/ folder. I found two and you may find even more (local-mail, .local-mail.directory, akonadi-maildir-resource_?). All of those will and can produce bogus folders which can confuse Akonadi.
(In reply to comment #6) > Suggestion: please stop creating localized folders and simply create "inbox" > folders, and then localize the name. Yes, I would prefer that as well. And it was done like this in KMail1. This should prevent problems when switching locales.
I don´t think that has been my problem. Cause I do not see any localized or duplicate folder names. The data loss issue I could reproduce so far is [Akonadi] [Bug 319226] New: produces 1-byte-sized files on failed move attempts while filtering This I didn´t see this one again, I think its good too close it. If you still see localized folder names for default folders please open a separate bug report for this one. Now closing this one. Thanks, Martin