Bug 362699 - Back end MySQL table PimItemTable shows entries with remoteID = NULL
Summary: Back end MySQL table PimItemTable shows entries with remoteID = NULL
Status: RESOLVED FIXED
Alias: None
Product: Akonadi
Classification: Frameworks and Libraries
Component: server (show other bugs)
Version: 5.2.0
Platform: Kubuntu Linux
: NOR major
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-05-05 11:05 UTC by Christoph Pospiech
Modified: 2016-06-12 15:05 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Christoph Pospiech 2016-05-05 11:05:02 UTC
The problem may or may not be related to previous disgraceful exits of Kmail. There is no reliable way to reproduce the above effect. It is not clear whether the above entries refer to lost mails.

Reproducible: Couldn't Reproduce


Actual Results:  
Back end MySQL table PimItemTable shows entries with remoteID = NULL

Expected Results:  
Those entries shouldn't exist.

I am using an external MySQL server (/usr/sbin/mysqld).
Comment 1 Daniel Vrátil 2016-05-22 13:13:50 UTC
Entries with a null RemoteID are a perfectly valid case. It means that the item has been created in Akonadi and is waiting for the owning resource to store it in the remote storage (e.g. IMAP server, maildir, Google Contacts addressbook, .etc.).

The issue you are most likely referring to is that you have data stored in Akonadi that were not synchronized to the remote storage. This is indeed a known issue, we currently lack any mechanism that would be try to "upload" those entities again if the initial upload failed for any reason (this is not affected by network connectivity or shutting down computer before the item is uploaded). A solution is being worked on though.
Comment 2 Christoph Pospiech 2016-05-22 20:02:52 UTC
I just observed another 5 occurrences of RemoteID = NULL. I did some forensic analysis. Hopefully the following observations help to pin down the problem.
1. Since I am running the akonadi database with my own MySQL server, I could use phpMyAdmin to do some queries, starting from "SELECT * FROM `pimitemtable` WHERE `remoteId` is NULL;"
I found that all these entries had a collectionID pointing to the trash can.
2. Apparently they were created by moving entries from an IMAP trash can to the main trash can. The IMAP server was davmail 4.7.1-2416-1, used as an interface to outlook.
3. There were three more entries put to the trash can by Kmail directly
    - these had a non-NULL RemoteID.
4. Emptying the trash can made the entries with RemoteID = NULL go away
    (no surprise, given what I said before).
5. I had a more emails in outlook that I could dispose. I repeated the following experiment two times.
a. Inside outlook (running in a Win7 VM), put these emails into trash (one or two at a time).
b. Wait for davmail to replicate the outlook trash into Kmail.
c. Run "SELECT * FROM `pimitemtable` WHERE `remoteId` is NULL;" and note the result.
d. Move the emails from IMAP trash to main trash.
e. Rerun "SELECT * FROM `pimitemtable` WHERE `remoteId` is NULL;" and note the difference.
=> One out of three emails disposed that way created a pimitemtable entry with RemoteID = NULL.
=> This trash can scenario may not be the only one creating this effect. But it is the least dangerous as these emails are to be disposed anyway - so no harm done !

Hope this helps !
Christoph Pospiech
Comment 3 Christoph Pospiech 2016-05-27 15:37:48 UTC
In an earlier command it was written that "It means that the item has been created in Akonadi and is waiting for the owning resource to store it in the remote storage". I just observed something that potentially points to a different direction. I had a couple of NULL entries today, not related to the trash scenario as described in my previous comment. Apparently, these entries were pointing to an non-existent entry in ~/.local/share/akonadi/file_db_data. But they also had a foreign key pointing to a collection. For one of those collections there happened to be only one mail that I touched/moved into this collection. I checked in Kmail - and the mail was still there. Means - there is another entry in the PimItemtable for this mail, and the entry with remoteID = NULL appears to be really obsolete - a stale duplicate so to speak. My other (more general) observation is that these entries don't go away, even if the connection to the IMAP servers is re-established. I did similar checks for the other mails. - And then decided to eliminate the stale entries via "DELETE FROM `pimitemtable` WHERE `remoteId` IS NULL;" - knowing that there are delete cascades into the other tables. Would that be the general solution ?

Christoph Pospiech
Comment 4 Christoph Pospiech 2016-06-12 11:20:47 UTC
I just managed to create some entries with RemoteID = NULL. I repeated the experiment - it appears to be reproducible. This is what I did -
1. In akonadi, SELECT * FROM `pimitemtable` WHERE `remoteId` IS NULL;
    Take note of the number of these entries.
2. In some IMAP inbox in Kmail , select some mails by mouse click (more than one).
3. Move them into some other folder by mouse left click and hold, moving them with the mouse
    and dropping them into the target folder.
4. In akonadi, re-check via SELECT * FROM `pimitemtable` WHERE `remoteId` IS NULL;
Observation: 
1. New NULL entries with collectionID pointing to the target folder, one less than the number of
    mails that were moved. 
2. All mails can be opened in the target folder.
3. The data seems to be there, execute
SELECT * FROM `parttable` WHERE pimItemID IN 
(SELECT id FROM pimitemtable WHERE remoteID IS NULL);
Check field `data` in parttable with a hex-editor.

Is there a way to recreate the mail as ordinary ASCII file in the target folder ?
Christoph
Comment 5 Christoph Pospiech 2016-06-12 12:13:12 UTC
Perhaps I should stress the point that I believe that there is still an issue. Daniel Vrátil said it is perfectly normal to have NULL entries, when "the item has been created in Akonadi and is waiting for the owning resource to store it in the remote storage". It appears that the mails are all completely downloaded from IMAP (at least I don't miss any part), but akonadi appears to be confused about it....and it does not happen, if I move one mail at a time.

Christoph
Comment 6 Christoph Pospiech 2016-06-12 15:05:15 UTC
Since the NULL entries by themselves are not considered to be a bug, I opened another bug report on mail moving (Bug 364243) and I will close this report.