Bug 279297

Not sure about severity, its a major issue not to get all mails, but in my case they are not lost as I can read them via the webinterface of my ISP.

Could it be that the update to Akonadi 1.6.0 broke my configuration in some way?
Two days before there was also an update from mysql 5.5.14 to mysql 5.5.15, could this be the culprit?
After these updates (KDE SC was updated to 4.7 at the same time) the problem appeared on two machines. On one of them Akonadi also fetches mails although neither Kontact nor Kmail is started. In this case the agents should stay offline AFAIK, or is this behavior intended?

Could it be a problem with leaving the messages on the server (probably related to UID), so sometimes Akonadi thinks it has the mail already?

To see if it is a problem with the database or the configuration:
Would I lose other data apart from the configuration if I wipe out all files related to Akonadi (only keep .local/share/local-mail where my mails are stored)?

Any hints appreciated.

Created attachment 62724 [details]
mysql.err log shown after restart of akonadiserver

After restarting akonadiserver from Akonadi-Console i got a window with test results. Most of them passed but this one has errors.
Should I worry about this?

I have never seen this kind of problem. To narrow it down a bit, you need to provide a few logs. Let me tell you how.
We need to find out where in the Akonadi stack the bug is, if it is in the POP3 resource, the maildir resource, the Akonadi server or KMail itself.

Once you know you have a mail that you can see on the webmail server, but that is missing in KMail, you can do the following steps. But only do the steps in that scenario, KMail and the server need to be out of sync for the logs to provide some useful information.
So once KMail doesn't show a mail that you know is on the server, do the following:
- Stop KMail, Kontact and every other applications using Akonadi
- Open a console for some commands to type
- type "akonadictl stop" to completely stop Akonadi
- Find the UID of the mail that is on the server, but missing locally. This is a bit tricky. First, login to arcor with telnet by typing:
"telnet pop3.arcor.de
Connected to pop3.arcor.de.
Escape character is '^]'.
+OK Hello there.
USER <your username>
PASS <your password>"
At this point you should be logged in to the ARCOR pop3 server over command line. You can see the list of mails on the server with the "LIST" and the "UIDL" commands. LIST gives a list of IDs and the size of the mail. UIDL gives a list of IDs and their associated. You can read mails with "RETR <ID>". Try reading all mails until you find the missing one, and note the ID. Maybe looking at the mail sizes can help narrowing the result down. Then you can use the UIDL command to find the associated UID easily. Note down the UID!

KMail keeps a list of UIDs that it has downloaded from the POP3 server, but that are still left on the server. This list is needed so that KMail doesn't download the same mail again on the next mail check. The file keeping that list is in $HOME/.kde4/share/config/akonadi_pop3_resource_7rc. You might have .kde instead of .kde4, and a different number than 7.

Next, enable debugging for the POP3 resource with kdebugdialog. Make sure debugging is enabled in general and in specific for anything containing "pop3".

Now, on the console you are using, set the scrollback to unlimit, so we don't miss anything from the loads of debug output that will follow.
Restart Akonadi with "akonadictl start". Then, start akonadiconsole, and trigger a mail check with the POP3 resource. On your console, you should see lots of POP3-related output, especially during the mail check. Copy&Paste that output to a text file, and attach it here.

In summary, please attach to the bug report:
- The POP3 UID of the missing mail
- the akonadi_pop3_resource_Xrc config file
- the debug output on the console during a POP3 mail check

Remember that the mail needs to be available on the server, but missing in the KMail, otherwise all this won't be useful.

BTW, it is correct that mail check also happens when neither KMail nor Kontact are running.

> Would I lose other data apart from the configuration if I wipe out all files
> related to Akonadi (only keep .local/share/local-mail where my mails are
> stored)?

Note that most of your mails are probably in .local/share/.local-mail (notice the additional dot)! You will loose tags, status flags (read/unread/important etc), config settings, accounts etc. The mails themselves would survive this though. If you forget to delete some files, it could cause some inconsistency. But please lets try to narrow down the problem using the steps described above first.

*** Bug 280008 has been marked as a duplicate of this bug. ***

Created attachment 63720 [details]
akonadi_pop3_resource_rc

Created attachment 63721 [details]
Debug output

UID from telnet:

UID13454-1232885516

Console-output attached.

As Dominik was faster, I just want to provide some additional information/my opinion.

IMHO this bug is not about not downloading the mails from the server, but about not saving them locally.
1.) This bug also happens, when you don't have the pop3 resource configured to leave messages on the server, but if you have it configured to remove them after download. This is of course hard to verify, but I noticed missing mails on mailing list posts first in this configuration and only after this switched to "Leave messages on the server".
2.) Using "Leave messages on the server"-configuration: For me it is the same as for Dominik the UID of the locally missing message is in akonadi_pop3_resource_rc. And AFAIU this means the message was downloaded from the server.

This means the summary should probably better be "Not all mails downloaded via pop3 are saved locally."

Thanks for taking care of the bug!

Thanks for the log.
Seems that the POP3 resource thinks it has downloaded the message, as it appears in the seenUID list. Therefore it does not download it again.
Now, there is the possibility that there is a bug in the POP3 resource and the entry in the seenUID list is wrong, but I somehow doubt that, after looking at the code it seems sane.
So I think the bug is very likely somewhere else in the Akonadi stack, the question is just where. Unfortunately I only know the POP3 code well, I am not that involved in the other parts of Akonadi.

What we can try out now is to see whether the mail is _somewhere_ in Akonadi. For this, find out the subject of the mail, and grep for it in two places. In my case, the subject for testing was "Open Science Summit", so I first grepped the external payload store:
cd .local/share/akonadi/file_db_data/
grep "Open Science Summit" *
2_r0:Subject: Open Science Summit

Then, I looked into the actual mail dir:
cd .local/share/.local-mail.directory/
find . | xargs grep Summit
./inbox/new/{4ef20774-e0f0-493e-a1a3-8dd41e840dae}:Subject: Open Science Summit

This confirms that the mail is properly stored in the maildir.

Then, in akonadiconsole, use the "DB Console" tab to do some database queries:
SELECT * FROM parttable WHERE data LIKE '%Summit%'
If you get a result, use the pimitemid (in my case 2) for another query:
SELECT * FROM parttable WHERE pimitemId = 2

If you got a result from the file_db_data grep earlier (in this case file '2_r0'), do the following:
SELECT * FROM parttable WHERE data LIKE '%2_r0%'

Please add all results of these here. That will tell us a bit, hopefully.

Also, what kind of resource is your "Local Folders" resource? You can find that out in the "Agents" tab in akonadiconsole.
Do you have any filters? Might it be that a filter moved the mail to some other directory you were not expecting?
Do you have expiry enabled (right-click the folder and chose "Expiry")?
Do you delete the trash on exit (Configure KMail->Misc->Folders)?

As written in my duplicate report. The missing mails are neither in  ~/.local/share/.local-mail.directory nor
~/.local/share/akonadi/file_db_data

As I now already have several mails in the thread with the missing mail I searched in akonadi for the return-path (which seems to be unique for the opensuse-kde list), but no entry in the database was found. (I verified that I could actually find other mails this way.)

I am not sure if here the problem may be, that I already restarted akonadi since the "download" of the missing mail. I will verify this as soon as I recognize another missing mail.
(That no mail is in the above mentioned folders is definitely true also before a restart of akonadi.)

To answer the other questions:
- Local Folders is a maildir resource
- Most of my mail is moved via filters automatically to other folders. But I am quite sure I have seen "missing" mails also appear for mails, where no filter applies. Also the mail is not accidentally in another folder.
- Empty trash on exit is enabled.

I hope this helps to nail down the problem.

I tried all the things you suggested in Comment #9 but there were no results.
I could not find the mail.

> Also, what kind of resource is your "Local Folders" resource? You can find that
> out in the "Agents" tab in akonadiconsole.

It's a maildir resource.

> Do you have any filters? Might it be that a filter moved the mail to some other
> directory you were not expecting?

I have some filters, but they are not applied automatically. I read the mails from Inbox and after that apply all filters, but those mails don't show up in Inbox. So the filters can't be the problem. 

> Do you have expiry enabled (right-click the folder and chose "Expiry")?

Expiry is disabled.

> Do you delete the trash on exit (Configure KMail->Misc->Folders)?

Empty trash on exit is disabled.

Another thing I noticed while searching for the mail over telnet is that after using RETR the corresponding mail was marked as read in the webinterface.
As all mails (also the ones missing locally) are shown as read, akonadi ought to have retrieved them. So probably it loses the mail at some point after downloading like Christian suspected in Comment #8.

Hope this information helps.

So it doesn't seem to be related to filters or expiry/trash, hmm.
Will to try reproduce myself, for a bit, but honestly I am a bit out of ideas here...

Ok, I tried to reproduce it, and to my surprise, I could reproduce pretty reliable.
Then I updated, and could no longer reproduce.
I used git bisect to search the commit that fixed it, which seems to be https://projects.kde.org/projects/kde/kdepim-runtime/repository/revisions/069f2897d372618e0428604b6954408b17ffb6bd. I am not 100% sure yet though.

The problem did not occur prior to KDE 4.7.0, right? If you happen to build from source, you can try building kdepim-runtime from the current git master and see if that fixes the problems for you too.

I have not investigated the actual problem yet.

BTW, I wrote a mail to the kdepim mailing list, if anyone is interested in following the discussion there.

(In reply to comment #14)
> The problem did not occur prior to KDE 4.7.0, right?

Yes, I have never seen this with kmail2 prior to 4.7.0.

And thanks for your work on the bug!

Ok, the real defect was fixed by Volker today in http://commits.kde.org/kdepim-runtime/f814b46a799ea8418ae17cf77d44da473124d12d.
This evening, I will backport to the 4.7 branch and mail kde-packagers about this.

That's good news!

> The problem did not occur prior to KDE 4.7.0, right?
Right.

Thanks for your effort Thomas!

Patch backported to the 4.7 branch, that means the fix will be in KDE 4.7.1. Also mailed kde-packagers, so the distributions might have updated packages earlier than that.

Dominik, Christian, many thanks for your help!