| Summary: | no mechanism to reattempt to store items without rid (just in db) into the resource | ||
|---|---|---|---|
| Product: | [Frameworks and Libraries] Akonadi | Reporter: | Martin Steigerwald <Martin> |
| Component: | general | Assignee: | kdepim bugs <kdepim-bugs> |
| Status: | CONFIRMED --- | ||
| Severity: | normal | CC: | aaronw, bugs.kde.org, danielroschka, gael, kde-bugzilla, llyw.linux, mail, marc+bugs, piedro.kulman, post, richard.bos |
| Priority: | NOR | ||
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Platform: | Other | ||
| OS: | Linux | ||
| Latest Commit: | Version Fixed In: | ||
| Sentry Crash Report: | |||
|
Description
Martin Steigerwald
2016-03-21 21:00:03 UTC
Adding link to Dan´s mail in the thread "Akonadictl fsck" on kdepim-users where he explains this: https://mail.kde.org/pipermail/kdepim-users/2016-March/000113.html Also this report might be a duplicate of bug #344242. For the sake of clearness it may make sense to mark it the other way around. Oh and Dan later added that "server" actually means any resource, so "server" could also be a local maildir or calender file or what not. *** Bug 344242 has been marked as a duplicate of this bug. *** I marked bug 344242 as a duplicate of this one and of course this thing is confirmed, as Dan already wrote its a known issue. Hey Martin, since you marked https://bugs.kde.org/show_bug.cgi?id=344242 a duplicate of this I continue here. In the other bug report I have written about my experience with data loss and described a workaround to make sure that data - in my case emails - will be copied were they belong - in my case local mail folder. Nevertheless one problem remained, the growth of ~/.local/share/akonadi/file_db_data. After a couple of weeks it holds again tens of thousands of items. After reading what you recommended to Guido I also ran "akonadictl fsck" which brought up quite a few items without RID. But it also said it had found loads of items without a reference in the database. Unfortunately I did not redirect the output to a file and thus most of it is lost. What I still could see are many entries like this: Migrated part 1324911 to new levelled cache When I started it again I got following output: [knut@chakra-pc ~]$ akonadictl fsck Looking for resources in the DB not matching a configured resource... Looking for collections not belonging to a valid resource... Checking collection tree consistency... Looking for items not belonging to a valid collection... Looking for item parts not belonging to a valid item... Looking for item flags not belonging to a valid item... Looking for overlapping external parts... Verifying external parts... Found 1946 external files. Found 1944 external parts. Found unreferenced external file: /home/knut/.local/share/akonadi/file_db_data/771220_r2 Found unreferenced external file: /home/knut/.local/share/akonadi/file_db_data/771240_r1 Moved 2 unreferenced files to lost+found. Checking size treshold changes... Found 0 parts to be moved to external files Found 0 parts to be moved to database Looking for dirty objects... Collection "Search" (id: 1) has no RID. Collection "OpenInvitations" (id: 28495) has no RID. Collection "DeclinedInvitations" (id: 28496) has no RID. Found 3 collections without RID. Item "649909" has no RID. . here came many more RID entries . Found 87 items without RID. Item "499424" has RID and is dirty. Found 1 dirty items. Migrating parts to new cache hierarchy... Consistency check done. When I checked ~/.local/share/akonadi/file_lost+found I saw that most of the files that had been in "file_db_data" before invoking "akonadictl fsck" were moved there. Now "file_lost+found" holds more than 2GB of files whereas "file_db_data" only holds less than half a GB. Now I wonder it it is save to delete all the files in "file_lost+found" to get disk space back? Or will this cause loss of data? Deleting files in "file_db_data" caused massive data loss. Knut Knut, it is safe to remove them (but only the file_lost+found files not file_db_data!). That Akonadi leaks files in file_db_data which akonadictl fsck moves to lost+found is a known bug that has been fixed. There are references to it in this bug tracker on on kdepim-users mailinglist (I don´t have the references at hand atm and its late here). Make sure you have latest kdepim 15.12. The leakage of files in file_db_data should be fixed (maybe what you see here are old leaked files and akonadictl fsck now cleaned.) Please otherwise try to keep the bug clean and only to the original potential data loss issue it reports. Use kdepim-users mailing list for further questions. (Bugs that have comments of a lot of different issues are difficult to follow for developers that why I set the duplicates this way around). Thanks, Martin Just to mention: This should have a very high priority in my opinion as most forums, threads and websites dealing with akonadi problems explicitly recommend deleting your akonadi folder to remedy a corrupted database! So data loss will occur in many of these cases - adding to the perception that kmail is not ready for productive use. Which it probably isn't without excessive backups but I guess it's very important to put that "kmail might loose your mails" credo in the land of outdated myths... So happy you found the culprit though. thx for your work, piedro p.s.: Also this will probably be realted to many of the bugs regarding incomplete email indices created by the "akonadi-indexing-agent". p. Also see: akonadictl fsck: What messages indicate real issues and what to do about them? https://marc.info/?l=kdepim-users&m=149426877926602&w=2 (I am about to reply to this thread with a further message with references to further bug reports and further information.) Any update on this? Like others suggestes, this should have high priority as it may result in loss of data. This one seems related: 362044 I'm encountering a akonadiserver crash when running akonadictl --fsck, but don't know why, are what to do about it. In file_lost+found I found this file: -rw-r--r-- 1 user users 4238 5 aug 07:58 922307_r0 It's an email, but to which (former) folder it belonged I've no idea. I would also appreciate it if the developers could follow up on this bug report and agree that its importance should be elevated. In essence this bug reports that what was supposed to be a data cache is actually also a data storage. As a result deleting entries without a remote ID can lead to data loss that is only recoverable if a backup of ~/.local/share/akonadi exists. Such a design error must be corrected and a mechanism must be implemented that flushes this cached data to the associated remote storage. I myself currently have 422 entries in the pimitemtable without a remoteId entry, all of which seem to be emails according to the associated parttable entries with the data column for those entries that are of partTypeId 2 (RFC822) either being the actual contents of the email or the filename of a file in ~/.local/share/akonadi/file_db_data, which in turn contains the email. (The entry of partTypeId 3 (HEAD) always stores the corresponding email header as plain text in the data column.) I have manually verified for some of these emails that they are indeed not present in ~/.local/share/.local-mail.directory, so if I were to delete those entries without a remote ID I would actually lose all those emails as those are POP3 accounts. This is only made worse by the fact that people seem to propagate the deletion of (all) PIM items without a remote ID as a way of cleaning/repairing the database, see discussion in https://forum.kde.org/viewtopic.php?f=215&t=138233&start=45#p381343. For items, which are duplicates of items with remote storage, this seems to actually resolve problems, see https://forum.kde.org/viewtopic.php?f=215&t=171121#p449145 and note that this suggestion is also part of the official trouble shooting guide https://docs.kde.org/trunk5/en/kmail/kmail2/resource-conflict-error.html (assuming that this is still up-to-date, of course), but there appears to be no way to automatically determine which entries are ghosts and which are unique data whose deletion results in data loss. The only good news is that when archiving a folder PIM items with NULL remote ID also appear to be included in the resulting archive, so it should still be possible to export all emails. (This way one could migrate to another email client if one does not wish to work with the software in its current state...) I am still hitting this with the latest Akonadi. Any word on when this will be addressed? It's been over 6 years. Same issue with the latest version. I use kmail more or less on weekly basis (every weekend), and the first I need to do is to clean up emails that I processed before (days or weeks). That are emails that then again retrieved and shown in kmail. It looks like email get stuck through filtering, but what it is I don't know. Some of those emails I can't delete normally. I delete with <shift><del> and deletes the email from kmail, but I don't know what the adverse might be. This behavior removes the fun of using kmail, cause every week I've to clean-up/reread email that I already processed. I really hope this can be taken care of. |