Bug 336581

Given that one can re-import filter settings and KMail then tries to select right folders by name… I think there should be something more intelligent possible than "oops, we lost your data, fix all your filter rules and identity and pop3 folder settings manually and don´t forget any of those". But even just at least *invalidating* folder assignments if the database changes would help here to avoid data loss by executing *wrong* filter rules.

For reference:

[Akonadi] [Bug 336582] New: Sudden write interruption during akonadictl vacuum causes database corruption

I have the same problem over several installations of KMail/Akonadi, with MySQL as a standalone server and with the dedicated MySQLd, and for **years** now.  The bug appeares on different hardware, after reinstallation, etc.

Sooner or later, with no discernible reason for me, Akonadi mixes up the folders. Suddenly the sent mail folder of one account is lost, for another account the trash folder is the inbox folder of another account, or, strange enough, an ownCloud CalDAV account becomes the outbox of an Imap account. Expire dates and so on also get messed up which leads important mails getting silently deleted.

We are talking about *** data loss *** here.

Same  as bug#283345 I guess

I have a similar one with kaddressbook bug#357266 - and my guess is that the application uses short numeric ids to reference akonadi database records which, after a rebuild, have completely different ids at random. Just a hunch.

Also bug#340529

David, that is exactly what happens. It is known to the developers. Challenge is to get unique ids working with all Akonadi resources, say for example IMAP. How do uniquely identify a mail with IMAP? Always check for the message-id.

We discussed this on KDE Randa Meetings last year and we didn´t come to an easy implementable solution so far. I do not recall all of the discussion. If you want to move this further along I suggest you bring up the topic on kde-pim mailing list.

Other than that thank you that you create bug relations. There are several duplicates of it. I am pretty sure I reported it as well..

This bug has only been reported for versions older than KDEPIM 4.14 (at most akonadi-1.3). Can anyone tell if this bug still present?

If noone confirms this bug for a recent version of akonadi (part of KDE Applications 15.08 or later), it gets closed in about three months.

Just as announced in my last comment, I close this bug. If you encounter it again in a recent version (at least 5.0 aka 15.08), please open a new one unless it already exists. Thank you for all your input.

Thanks, Denis.

This is a principal issue of the current Akonadi design and still present, as I am not aware of any design change. Thus reopening.

There is no need to open a new bug report about it.

This is indeed still the case with KMail 5.2.3 (Debian Unstable unfortunately does not have any newer packages due to the difficulty to properly package Qt Webengine in order to enable newer KDEPIM versions, which likely means that Debian will go with KDEPIM 16.04 for next stable version Stretch).

The principal issue of relying on database IDs outside of the database itself and thus using wrong folders in case of a database loss, potentially causing data loss, is still present.

For mailfilters:

~/.config> grep action- akonadi_mailfilter_agentrc | head -20
[…]
action-args-0=95
action-name-0=transfer
action-args-0=209
action-name-0=transfer
action-args-0=209
action-name-0=transfer
action-args-0=208
action-name-0=transfer
action-args-0=211
action-name-0=transfer
action-args-0=422
action-name-0=transfer

Action "transfer" means to move the mail into a folder. The corresponding args-0 is the numerical id of the folder.

For folder settings:

~/.config> fgrep '[Folder-' kmail2rc | head
[Folder-1]
[Folder-10]
[Folder-100]
[Folder-101]
[Folder-102]
[Folder-103]
[Folder-104]
[Folder-105]
[Folder-106]
[Folder-107]

For default folders for mail identities:

~/.config> egrep -i "draft|template" emailidentities | head -10 
Drafts=21
Templates=31
Drafts=21
Templates=31
Drafts=21
Templates=31
Drafts=21
Templates=31
Drafts=411
Templates=378

Default inbox folder for POP mail resources:

martin@merkaba:~/.config> grep targetCollection akonadi_pop3_resource_0rc
targetCollection=361

I bet is similar for IMAP resources.


My expected result is this:

1) Either keep / rebuild / reassign references after a database loss properly *or*

2) *loose* the references and require the user to reconfigure.

Yet *never ever* use *wrong* references, just do to a database loss.

Use a unique hash for identification, not just the index number of the database entry. In that way you can at least *know* when the reference is no more valid. *Or* store *all* of these reference into the database itself, so that they are gone for good if the database is rebuilt.

For user convenience I would aim for approach 1 wherever possible which is easier to do with a hash based approach. For maildir resource, you can store the folder identification in the maildir filesystem, or IMAP it becomes more difficult tough, yet you could at least also store the name and then if the reference by hash is gone, you can show the folder name to the user and possibly even autoselect it for confirmation.


You can argue that a database loss should not happen and with proper MariaDB configuration, proper filesystem and proper storage media you might well be right about that – *yet* in practice it did happen to a lot of users. So please make Akonadi *failure proof* and more *robust* in cases like this.

Another approach would be to use something else than a database for metadata or rethink the Akonadi caching concept completely like Aaron and others do with Sink for Kubemail. I don´t see this happening for KMail from KDEPIM soon, so I think it makes sense to make it more failure proof even before that may happen.

A short comment on this, since Martin perfectly summarizes my frustration. A user's opinion, I am not a developer.

Since introduction of akonadi, I experience broken database issue about 1-2 times a Year (last time some days ago) and none of the proposed solutions anywhere in posts resolves my given issues. Because I don't have the time for extensive investigations I have to delete ~/.local/share/akonadi which then totally mixes up folder associations with the described impacts on filters etc. as described above.

Email is among the most critical applications for daily use and this weakness in architecture (as I learned from above) is a total mess.

This is not a bug, but a totally broken architecture !!!
Martin's suggestions seem already very reasonable to me, and promising.

As to broken architecture... The dreaded KMail hanging with the message "Retrieving message/ folder content screen" showing indefinitely is back. On top of that KMail now always crashes when writing a new email using autofill for the receiver (start typing the email address and when trying to select one of the suggestions made, KMail will crash). I will search if a bug report has been filed for that one and if not I will file it myself.

KMail/ Akonadi had been "OK-ish" for a couple of months (to the extent I even removed my backup solution of Thunderbird), but it is back to being near useless again.

(In reply to Frits Spieker from comment #12)

Dear Frits, please refrain from writing unrelated stuff into this bug report. The bug report is about the effects of accidental database loss on folder assignments and nothing else. It is not a discussion forum or a mailing list. Using it for any random comment makes it more difficult to read by any of the developers. Please use KDE Forum or preferably kdepim-users mailing list to vent your frustration.

Have a Merry Christmas,
Martin

Just to confirm that the issue is still there for Kmail 5.20.2. The akonadi database gets corrupted randomly (happened to me after a distro upgrade that otherwise went perfectly fine) and after removing Akonadi's corrupted data folders and syncing again, all folder associations were messed up. 

Can't way for the day when our whole company can start relying on KDE PIM's framework. It looks very promising and the vision behind akonadi is brilliant, but this data corruption issue is a deal breaker. I hope you manage to fix this soon, stability and durability are so important for this kind of application. 

If a solution is known but what is lacking is manpower, please let the community know so that we can try to organize sponsorship. This is such a critical issue (I am referring to the data corruption. Folder assignment is more a symptom. If data corruption wasn't there in the first place, folder assignment mix up wouldn't be an issue. I feel it would probably be better to fix the fragility of the storage system to make such symptoms non-issues by removing a vast class of failure modes rather than rather than trying to come up with global ids).

I have seen this again with switching from PostgreSQL to SQLite3 on KDEPIM 24.08. I was not aware I could probably have used akonadi-db-migrator. But I still believe relieving on database IDs for folder assignment to be dangerous. It would be better to use some kind of UUID at least where remote storage would support storing it.