Bug 339181

I can confirm this observation. It may occur if one copies/moves many mails, but last weekend I had the same behaviour also for moving only two mails into a just newly created folder (in a local maildir).

In general, I found that it can happen that
- if one creates a new subfolder in a maildir structure, it will not be created on the hard disk, but Kmail shows it and one can put mails into it (seems that the mails stay in the akonadi cache)
- if one copies/moves mails to a folder in a maildir, the messages are correctly shown in Kmail but the are not physically on the disk in the corresponding folder (seems again that they stay in the akonadi cache).

In my example last weekend, the latter one was the case.

This behaviour surely leads to data loss sooner or later!

Workaround:
If I realize that some messages of a maildir folder are not correctly stored on the hard disk (so they seem to be only in the akonadi cache), I create a temporary new folder in the maildir and copy all mails from the other folder into it; if this worked correctly, I delete all messages of my target folder and move all mails from the temp folder to the target folder (seems to work as well).

My system is just new and fast enough; although sufficent speed should not influence reliability, so this is not an argument at all. I observe this behaviour since I started to re-use Kmail & Co., which was with 4.11.

I have already used that workaround. To recover the files. The drawback is that you will loose tags. As tags are not preserved by copy.

I believe that this is the same bug as the one I described here
https://bugs.kde.org/show_bug.cgi?id=338238

Thank you for mentioning the work around.  I had tried *moving* the emails to another maildir, but many mails were still missing, but *copying* them to another maildir has seemed to work correctly.  (At least I don't have to go through IMAP to recover them...)

I have made more experiments. The problem is reproducible and appears when moving several mails. 

I have tried to move from a disconnected imap folder to a local one. Each experiment is made 5 times. 

-Moving just one message works as expected. There is a new file in the local folder.
-Moving two messages does not work. The two messages disappear from the dimap folder. Only one message appears in the local folder and the other one resides in limbo without RID.

I confirm this terrible bug. Kubuntu 14.04, kde 4.14.2

Can you confirm that the mails won´t appear after *some* time?

I believe that Akonadi caches things in database upto SizeThreshold, standard is 4096 bytes, or in ~/.local/share/akonadi/file_db_data for larger items. Only if they are not there and are not moved after some time, where I am totally not sure what that "some time" amounts to, there is really a data loss.

I do not know, because I got rid of the trouble with the copy workaround.

I have just moved 27 mails from an imap folder two a local folder in two moves, one of 20 mails an another of 7, as expected, I have now 25 mail without RID. I will wait one week to see if they dissapear.

By the way, during the movement of the mails I get a lot of messages like:

akonadi_imap_resource_0(11063)/kdepimlibs (kimap) KIMAP::StoreJob::handleResponse: We asked for UID but the server didn't give it back, resultingFlags not stored. 

No idea if this is related with the problem at hand.

Thanks. Can you see the contents of the mail within file_db_data or db_data/akonadi with grep?

Just grep an unique ASCII word within the mail (no umlauts or UTF-8 stuff as that gets encoded). Really make sure the mail is clear text by pressing V in KMail and look. It should be if it contains only ASCII characters.

Well, after a week the number of mails without RID has not changed.

I have created a big mail wihout RID and I can find it in file_db_data. That is, the mails exists in the cache, but only there...

Luisfe, the "file_db_data" is also cache. Well the mail must be stored somethere. Is it smaller than 4 KiB. Then grep like

ms@merkaba:~/.local/share/akonadi/db_data/akonadi> LANG=C grep -i "kmail" *
Binary file parttable.ibd matches
Binary file pimitemtable.ibd matches

for an ASCII word (no umlauts or special chars) in the mail.

But well it really needs to be an ASCII mail (press V in Kmail and then check the body of the mail, use something from this body to grep for).

So or so, if you are sure the mail is *not* in the maildir, then that means that Akonadi caches some mails really long without writing them to the final location. I do not like this behavior. If it would cache a mail just for as long as an IMAP server is offline or not willing to accept it, thats one thing. But I would prefer it writes a mail as soon as possible.

Postfix for sure has a better guarentee on that. It writes the mail file, makes fsync() on it, and only then tells the sender "I have it, you can discard it".

Okay, well Akonadi has the mail somewhere in the database. So it has it… but still. At the very least it confused users. But on a database corruption it also causes mail *loss*. So I consider writing to final location quickly a robustness feature.

Dan wrote:

"Those are Items (emails, events, contacts, ...) that you created locally but were not synced to the server. 

Unfortunately we don't have a mechanism at this moment to force Akonadi to try to sync those Items again to the server. If you wipe your Akonadi database, those Items will be lost."

So that basically seems to confirm this issue. See:

[kdepim-users] Akonadictl fsck
https://mail.kde.org/pipermail/kdepim-users/2016-March/000113.html

Thanks, Martin

Okay, Dan didn´t really confirm the bug, actually there are two bugs: The one with moving mails from cached IMAP to local folder, the other is generally that on failures there are items without RID, which are only stored in database, but on in resource (IMAP, local folder or such). Yet Ingo just confirmed it:

----------------------------------------------------------------------------------------------
I'm experiencing this bug since a long time and have learned to live 
with it.

Moving a single message from an IMAP account (cached or not makes no 
difference) to a local folder (maildir in mixed resource) works.

Copying multiple messages (even 1000s) also works (although it takes 
quite some time until the copied messages that are already listed in the 
destination folder appear on disk).

But if one tries to move multiple messages then only the first message 
appears on disk in the maildir folders. All other messages only appear 
"virtually" in the folder. Usually, there's a message that moving 
failed. But the messages are still gone from the source folder. This is 
100 % reproducible.
----------------------------------------------------------------------------------------------
https://mail.kde.org/pipermail/kdepim-users/2016-March/000123.html

Added a bug report about the no rid, no option to reattempt  to push item into server issue:

Bug 360834 - no mechanism to reattempt to store items without rid (just in db) into the resource

Yes, I think that these are two separated issues, probably moving mails does something wrong that triggers the more general bug in 360834.

I can reproduce it with the following steps:

1. Create two resources, DIM: Disconnected IMAP,  LOC: local folder. Ensure that the local folder actually exists (it does in my configuration).
2. Move at least two items from the DIM account to the LOC account.
3. The first item is written to the LOC account. all the rest of the items are deleted from the DIM account, are never written to the final destinaltion of the LOC account and stay in the akonadi cache. In the cache they appear as belonging to the LOC account, one has to look at the files or run akonadictl fsck to realice the problem.

I have tested with kmail 5.1.3, Akonadi 5.1.51 (Applications/15.12) compiled from sources.

(In reply to Martin Steigerwald from comment #14)
Sorry, I just repeated this post...

Git commit 36e410afdb2fec0b118d5acb3a1de4bd148e34c4 by Daniel Vrátil.
Committed on 21/03/2016 at 23:13.
Pushed by dvratil into branch 'Applications/16.04'.

Monitor: translate batch inter-resource move notifications

When we get an inter-resource move notification with multiple Items
we must always run it through translateAndCompress() to translate it
into a series of Add notifications on destination resource and Remove
notification on the source resource.

Without the transformation any inter-resource move of more than one
Item was still being emitted as Monitor::itemsMoved() possibly
causing the resource to fail the operation. In case of non-batch
resources like Maildir resource it only emitted the first Item via
Monitor::itemMoved().

This fixes a potential data loss as the change was not handled by the
destination resource and caused the Items to be stored only in the
Akonadi database.
FIXED-IN: 16.04

M  +9    -3    src/core/monitor_p.cpp

http://commits.kde.org/akonadi/36e410afdb2fec0b118d5acb3a1de4bd148e34c4

Wow, thank you, Dan. I may even test if, if I configure my IMAP account again, I removed it cause Akonadi was too slow, yet, I found it replaced mysql.conf in a recent upgrade due to a setting deprecated in MySQL and thus it decreased my InnoDB buffer settings and thats no good, now its faster again and I could put it back in. Does it have to be a cached/disconnected IMAP for testing?

This bug is marked fixed and everything worked fine for me until a recent update. It seems that the bug has been reintroduced into the code. Checked it various times today.

Right now I'm using an up-to-date Chakra Linux with:
 
KMail 5.4.0
akonadi 16.12.0

Anything else needed?

Knut

Kurt, I was tempted to just reopen the bug report, but as… it may contain more than one issue in one report from a quick glance at the comments, I kindly ask you to open a new bug report with exactly the versions you use and concrete steps on how to reproduce it. Please add a note in the comment here so people can find the new report. Its sufficient to write a comment like "I opened bug #number" as Bugzilla will automatically generated a link. Also please reference this bug report from the new one with one sentence. Thank you, Martin

Hello Martin,

(In reply to Martin Steigerwald from comment #20)
> Kurt, I was tempted to just reopen the bug report, but as… 

I made my comment #19 here because the problem was solved for me after this bug has been marked "RESOLVED FIXED".

As you proposed I wrote a new Bug 374925 report.

But since the issue has been described in various bug reports someone with knowledge of the code base should sort out which of these bugs should be closed or marked as duplicates. These are the bugs I see related to this bug:

Bug 344242: Mails that are copied do not appear on my harddrive

Bug 341192: Moving messages does not synchronize to disk

Bug 360834: no mechanism to reattempt to store items without rid (just in db) into the resource

Bug 338238: copied or moved mails are often not processed correctly (POP3)


Thanks for looking into the matter.

Knut