Hi! I really like Kmail---it's the most powerful and configurable graphical email client for Linux that I've used. In general, it meshes really well with my work flow. Unfortunately, for the last several several days, I have had several disappearing emails. This has come to a head where I have lost at least 1 important email (and as many as 3) for several days last week. I am using Kmail 4.10.5 on Fedora 18 (kdepim-4.10.5-1.fc18.x86_64 and associated libraries---these are the latest version of kdepim for Fedora 18) to access a Microsoft Office 365 email account via IMAP. I have many folders in the mail account to keep track of various messages. After reading and responding, I usually move (via a keystroke) a message out of the Inbox to a different folder. I have never noticed a message immediately disappear, but the next day (or several hours later), I've gone to the folder to look at the message, and the message is not there. I've checked directly on the server (using Microsoft's web access), and the message is completely gone (ie, not just accidentally filed to the wrong folder). It seems that sometimes Kmail's move does not actually copy the message into the new folder before deleting the message from the Inbox. Or perhaps Kmail's view of what is in each folder does not match that of the server: perhaps Kmail thinks the message has been copied to the folder, but in reality it hasn't. Then when Kmail syncs with the folder, the message is not there and is removed from the message list. My Wifi Internet access for the past month has been somewhat unreliable. I have noticed discrepancies between Kmail's message list and the server that get corrected when the message list for the folder is updated. In one instance, the next day after copying an email to a folder, I changed to that folder, and saw my desired message in the message list. I clicked on it to read the contents, and Kmail attempted to fetch it while updating the message list. After the message list updated, the message was gone. (This makes it sound like some other email client is changing the status of messages on the email server, but I don't use any other email client. I only use the Microsoft web client to determine if a message has been deleted or not, and I don't write, move, or delete messages using that client.) My suspicion is that somehow this is related to moving messages to another folder, since I've never noticed a message being missing from the Inbox (I don't always file messages, and have several hundred in my Inbox). Thank you for any help! Stephen Reproducible: Sometimes Some other relevant bugs might be 326977 and 320621. I also posted this on the KDE forums at forum.kde.org/viewtopic.php?f=215&t=118559&p=297285
One thing that I just realized is that all of the missing emails are initial messages in a conversation. My normal workflow when I receive a message in my Inbox is to write a response; the sent copy is put into the Inbox (I have the sent folder configured to my Inbox); I then highlight the entire thread and move it into the appropriate folder. Every time that I've looked in the folder immediately afterwards, the message has been there. It's only later that I realize the first message of the thread is gone. Of course, I might be missing other messages that I haven't responded to. I know for sure that I'm missing some messages because I have the responses but not the original message.
I realized that I do know of one message that was lost that I did not respond to. I know about it since I wrote to someone else about the contents, and when I went to check the original message, it was gone. So it's not purely an issue about messages that have been responded to. Also, I don't think I've lost any messages that I've written; I think they're all by other people (as far as I can tell).
Well, I can now confirm that I've lost messages that I wrote. I wrote a message earlier this evening to some people (not in response to another message), and then copied it from the Inbox into another folder. It is now gone from that folder. I also have tried using KMail 4.11.3 on a Fedora 19 laptop (kdepim 4.11.3-1.fc19, the latest version). I copied a message I had received from the Inbox into a folder (no responses to that message), and then looked in the folder a few minutes later. The message was gone.
Okay, using KMail 4.11.3 on Fedora 19, I currently have a weird situation: I copied two messages (an incoming message and my response) from the Inbox into another folder. Looking at that folder from Microsoft's web access, only my response got copied into the folder. The other message is nowhere in my account, including the Inbox or the intended folder. However, the message appears in the folder on KMail 4.11.3. It is also in ~/.local/share/akonadi/file_db_data. Forcing a sync did not make the message get uploaded to the server. I then closed KMail 4.11.3 and then restarted it. When it synced with the folder, the message disappeared from the local copy. The file disappeared from ~/.local/share/akonadi/file_db_data. So it really seems that the disappearing messages are a result of a problem with copying or syncing to folders.
Looks like a problem with the IMAP resource. Please run following commands from konsole: export KIMAP_LOGFILE=/tmp/imap.log akonadictl restart It will cause the IMAP resource to log all communication with the server to /tmp/imap.log.* files. Then reproduce the bug, find the one file and attach it here. Remember to strip any sensitive information.
Please also look into .xsession-errors for error messages of the imap resource. I couldn't find any unsafe codepath in the imap resource, so we need some more info first. The imap resource only removes the old item if the append succeeded, so if it fails nothing should happen and the old state should be restored with the next sync. (unless we get an itemRemoved signal from the akonadi server, but I wouldn't see why that should happen). Thanks for reporting.
So I have an example of a missing email, and I know the IMAP logfile that contains the relevant stuff. Unfortunately, the logfile is about 4MB big, mainly because it has several transactions and it downloaded a message with a picture. Is there a way to filter out the relevant stuff?
Here's something perhaps unusual about my setup that might be relevant: I have two accounts in KMail, one called "Incoming" and the other "Mail Folders". Both point to the same email account and the same IMAP account. In the Incoming account, only the INBOX is subscribed to. The reason I have this setup is that I have over 300 mail folders. I couldn't figure out how to have the INBOX polled for new email without all of the 300 other folders also being polled (which takes a long time); the mail interval checking options are per account. On one of the forums, I saw a recommendation to use two accounts in KMail. So the "Incoming" account gets polled for new mail, whereas the "Mail Folders" account only gets polled when clicking on a specific folder. However, just the fact of copying from one account to the other can't be the (only) problem, because I sent a message this morning where the Sent Mail folder was set immediately to a folder of "Mail Folders". The same problem happened: KMail thought the mail was copied to the folder, but when I checked on the web access, it had not been. When KMail sync'ed the folder, it deleted the local copy of the email. Thankfully I now have a filter that backs up all email, so I didn't lose the email, but I unfortunately was not running the IMAP log at the time.
Created attachment 83937 [details] Grep of akonadi in .xsession-errors This is a grep of "akonadi" in my .xsession-errors file. I don't know if there's anything helpful there.
I posted the grep of "akonadi" from my .xsession-errors. I'm not sure if anything interesting is there. There are some lines kmail2(2926)/libakonadi Akonadi::EntityTreeModelPrivate::monitoredItemChanged: Got a stale notification for an item which was already removed. 259267 "60" It's possible that occurs in the following: I copy a message to a folder. I then check the web access to see if it arrived. If it does not, then I try to copy the backup (made by a filter, copying all received and sent messages into a backup folder in Local Folders) into that folder. Kmail then thinks that there are two copies of the message in the folder, whereas the server thinks there's only one. I then have Kmail delete the message that isn't on the server, and I presume these errors are then from the server.
Created attachment 83998 [details] IMAP log #1 for sent message R This is the first file of the IMAP log for when I sent a message. A sent copy of the message was to be put in the folder "renyi". After I sent the message, it briefly appeared in the list for the folder in Kmail, but never appeared on the server (according to the web access). A few seconds later, the folder was synced, and the message disappeared from Kmail.
Created attachment 83999 [details] IMAP log #2 for sent message R The IMAP log seems to be split into a new file at various points. This is the continuation of the previous log.
Created attachment 84000 [details] IMAP log #1 for messages moved from INBOX to "referee" folder I had 4 messages in my INBOX (in the "Incoming" account) that I moved to the "referee" folder (in the "Mail Folders" account). This is the IMAP log for appending those 4 messages to the "referee" folder. The first 3 messages had their full text appear in the log, and appeared in the folder on the server. The fourth message does not have its text appear in the log (though you can see the 4th append command) and did not appear on the server. When the folder was synced, this 4th message disappeared from the folder list in Kmail. The IMAP log seems to be split at this point; I'll upload the next part of the log.
Created attachment 84001 [details] IMAP log #2 for messages moved from INBOX to "referee" folder The second part of the log where I was looking at the contents of the "referee" folder.
The X in the logfile indicates that the session was disconnected, so it looks like your session get disconnected while trying to APPEND the message. The imap resource should in such a case of course retry to upload the message, so I'll need to investigate that. Thanks for the reports so far. I assume the disconnects are simply due to unreliable connection or alike (something we should be able to deal with).
Note to self: The code looks safe, unless we somehow miss the append error, and then assign a random RID after the UID search fails, resulting in the item getting removed during the next sync instead of getting appended again. Stephen: It looks like the debug output is not activated from your xsession-errors, can you enable it using kdebugdialog (search for imap and activate everything). The debug output should be enough to see if we're entering a codepath that isn't safe if the append failed. If the output is too large, feel free to email it to me or upload it somewhere.
Created attachment 84012 [details] IMAP log for message moved from INBOX to "mathdept" folder I moved a received message from the INBOX (in the "Incoming" account) to the "mathdept" folder (in the "Mail Folders" account). The message did not appear in the "mathdept" folder on the server when I checked through the web access a few minutes later, and when I then checked in the folder in Kmail, the message was already gone. As you can see in the IMAP log, the connection was lost. This seems to happen frequently with the wireless at my apartment. I'll upload the ~/.xsession-errors file in a moment.
Created attachment 84013 [details] .xsession-errors file for message moved from INBOX to "mathdept" folder Here is the ~/.xsession-errors file, with the following debug options: kdepimlibs (kimap); kio_imap; akonadi_imap_resource_0; akonadi_imap_resource_1. I'm not completely sure, but I think resource_1 is the "Mail Folders" account and resource_0 is the "Incoming" account. I copied the file shortly after I realized the message had not been copied to "mathdept", so the relevant part should be at the end. My guess is that the message that disappeared is 22665. I think that messages 22661, 22662, 22663 were three spam folders that I moved (successfully) to the spam folder.
Christian, thanks for your help! Please let me know if I can provide additional information.
I checked the append job, and it fails as it should if something goes awry along the way. Since this is an inter-resource move, the problem has nothing to do with moveitemstask, it's an additemtask that is executed. What I find odd is that the item's already have a remote id set when added, which probably means they are indeed not protected from being deleted during the next sync. AFAIK a new item shouldn't have the remote id set, so I need to investigate why that is.
The problem seems to be that the remote id is indeed not cleared for inter-resource moves (it works for copies).
For the record: * the imap server emits a move notification with the old remoteId still set (so the original resource can has the remoteId for it's itemRemoved notification). * The monitor translates the move to itemRemoved/Added depending on resource, but doesn't clear the RID for the itemAdded case (it probably should but that's not the problem). * The server then clears the RID right after having emitted the notification, meaning the item should actually be protected from removal during a full sync. So it's currently unclear how a moved item can get lost, even if the append failed. There might be a bug that the akonadi server doesn't try to push the item again, but then it should just sit there without RID, and not vanish entirely.
We had a bug before where the source RID was not removed for an inter-resource move. If that happens the next sync will remove the item locally if it hasn't been pushed to the IMAP server in the meantime. There should be code to handle this case in the Akonadi server already, I suggest verifying that this is still in place and does the right thing. This can easily be observed in akonadiconsole, if you move a message from a different resource into an IMAP folder (while the IMAP resource is offline, to ease observation) you should see an empty RID for that mail. If you still see the old RID, that's likely the cause for this problem.
(In reply to comment #23) > We had a bug before where the source RID was not removed for an > inter-resource move. If that happens the next sync will remove the item > locally if it hasn't been pushed to the IMAP server in the meantime. > > There should be code to handle this case in the Akonadi server already, I > suggest verifying that this is still in place and does the right thing. This > can easily be observed in akonadiconsole, if you move a message from a > different resource into an IMAP folder (while the IMAP resource is offline, > to ease observation) you should see an empty RID for that mail. If you still > see the old RID, that's likely the cause for this problem. Indeed, I just verified this with akonadi 1.10.46, and the RID is apparently not cleared.
Yup, I can confirm the the MOVE handler does not detect inter-resource move. I'll look into it ASAP.
Git commit 980b02ff4b0ba7e49f3b32d39ecd76121456c3f6 by Dan Vrátil. Committed on 12/12/2013 at 15:13. Pushed by dvratil into branch '1.11'. Correctly detect inter-resource moves When comparing the source and destination resources, the item already has the destination collection and resource set, so we need to compare it with the source resource FIXED-IN: 1.11.1 M +1 -1 server/src/handler/move.cpp http://commits.kde.org/akonadi/980b02ff4b0ba7e49f3b32d39ecd76121456c3f6
Great! Thanks Chrisitian and Daniel for your work diagnosing and fixing this bug! I look forward to getting it installed on my computer.