Bug 483545 - Kmail2 silently looses mails while displaying the correct counts with the folders
Summary: Kmail2 silently looses mails while displaying the correct counts with the fol...
Status: REPORTED
Alias: None
Product: kmail2
Classification: Applications
Component: general (show other bugs)
Version: 5.22.3
Platform: openSUSE Linux
: NOR grave
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-03-14 13:07 UTC by Flossy Cat
Modified: 2024-03-15 10:42 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
Error Messages by Kmail when it is losing mails (2.81 KB, text/plain)
2024-03-14 13:07 UTC, Flossy Cat
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Flossy Cat 2024-03-14 13:07:20 UTC
Created attachment 167165 [details]
Error Messages by Kmail when it is losing mails

SUMMARY
Kmail2 silently looses mails without any error messages or warnings.
The fact is hidden by wrong counts given with the folder display.
Further a complete self-disintegration of akonadi was witnessed – potentially a follow-up error …

STEPS TO REPRODUCE
not known, as the disintegration silently continued over at least 2 month and neither start nor disintegration trigger can be reconstructed from backups.

But the core problem is obvious from the observations – see below.

OBSERVED RESULT
On an air-gapped system, physically only accessible by me (used for highly confidential work with severe contractual penalties), – this precludes any malicious external causes! – the following happened:

When sorting my old mail archives (no HTML view, no attachments!) suddenly I notice changes in the folder pane on the left:
Sequentially the number of messages in folders counted down and folders were removed.
Trusting in my backups I let the system run its course, to at least risk no further inconsistencies …
The mail resided in ~/.local/share/local-mail – when the system was finished there was a new folder ~/.local/share/.local-maildir and the akonadi-DB in ~/.local/share/akonadi seemed to be completely reset.
All mails were lost according kmail.

In the storage location ~/.local/share/local-mail roughly 2 thousand mails of originally around 12 thousand remained – surprisingly all in the "new" branch of the maildir structures, albeit none of the mails was displayed as unread.

The next nasty surprise:
Obviously there has been a creeping loss of mails beforehand.
In all the backups up to 2 month prior mails were missing, while the akonadi database (via the number of mails in the folder pane) claimed all was fine.

This 100% sure knowledge because – when embarking on this little project of triaging my mail archives – I had set apart a folder of mail correspondence with a dear friend who died much to young (to avoid losing them by any mis-triaging …).
This folder was empty even in the backups 2 month old.

Curiously all mails still present in the backups always resides in the "new" branch of the maildir structures.

I then went back to the archive, I'd used to transfer the mail-archives from the pre-cursor system (still KDE4 based – no problem as air-gapped as well) to the current system. Here all mails were still present (in their past sorting state) and all resided in the "cur" branch of the maildir structures, as was to be expected.

(So happily I could recover the complete archive but still lost many, many hours of triaging!)

All in all around 2500 mails were silently lost by Kmail2 over a period of a little more than 2 month.
I investigated a sample of roughly 10%: 
While old mails with crypto predating 2010 (e.g. inline/pgp) and the common less-than-standard-conformant M$-mails were clearly over-represented also completely harmless, standard-conformant, non-crypto mails were lost, too.

All of these mails display without problems in the Kmail-viewer-component (might not be well formatted, but complete text with all attachments etc.)

When experimenting with these I could capture some error messages of Kmail – see attachment.

The core cause seems:
»"Unable to fetch item from backend (collection -1) : Unable to retrieve item from resource: [LRCONFLICT] Resource akonadi_maildir_resource_2 tries to modify item 29196 () (in collection 102) with dirty payload, aborting STORE."«

Seemingly the akonadi-component handling storage is much pickier than the display-component and simply drops items when they are moved between folders or the branches of the maildir structures.

The machine has 8 cores and 16 GB RAM and is boring itself to death, while I'm triaging mails (there are more demanding tasks for the machine, but not during triaging …) – so there were no "out of memory" problems … (ie. the copious Swap is not even touched).

EXPECTED RESULT
1. Kmail2 not loosing mails.
2. Kmail2 giving visual warnings to the user if there are problems
3. akonadi_maildir_resource_2 not silently dropping mails when it has any problems handling them.

SOFTWARE/OS VERSIONS
[copied from System Information]
Operating System: openSUSE Leap 15.5
KDE Plasma Version: 5.27.9
KDE Frameworks Version: 5.103.0
Qt Version: 5.15.8
Kernel Version: 5.14.21-150500.55.44-default (64-bit)
Graphics Platform: X11
Processors: 8 × Intel® Core™ i5-10210U CPU @ 1.60GHz
Memory: 7.6 GiB of RAM
Graphics Processor: Mesa Intel® UHD Graphics

ADDITIONAL INFORMATION
This Kmail behavior is devastating. I have experienced been many minor issues with KDEpim over the last years, all repairable with minutes or few hours and no data or structure lost – but this behavior destroys any trust in a Personal Information Manager.

I am REALLY, REALLY and EXTREMELY disgruntled!
After a quarter of a century of KDEpim usage – because I found it superior to all alternatives! – I'm on my way to the emergency exit. And I'm running, not walking!