Bug 377070

Summary: Kmail emptied inbox, all emails deleted!!!
Product: [Applications] kmail2 Reporter: William L. Thomson Jr. <wlt-ml>
Component: generalAssignee: kdepim bugs <kdepim-bugs>
Status: RESOLVED INTENTIONAL    
Severity: critical CC: Martin
Priority: NOR    
Version: unspecified   
Target Milestone: ---   
Platform: Gentoo Packages   
OS: Linux   
Latest Commit: Version Fixed In:

Description William L. Thomson Jr. 2017-03-01 17:48:58 UTC
KMail Version 5.4.2
KDE Frameworks 5.31.0
Qt 5.7.1 (built against 5.7.1)

This is NOT the 1st time this has happened. Every email in my inbox was deleted for NO reason. This seems to be some issue between the emails on disk and what ever is in the akanodi database. I am SO sick of issues with Kmail2 and email. This whole using a db backend is NOT working and needs to end. People should have a choice on using akonadi, mysql, and any db. Emails that are on DISK should NEVER be deleted unless I am deleting them. Funny enough emails in my trash remain till I empty the trash. I have lots of folders, lots of filters, lots of emails. I have not had this problem with other folders. I have had other issues with emails from day 1 using kmail2. I am about to go back to gnome and evolution. Kmail is complete garbage!!!!

I am a database person. I have worked with database for years. There is NO reason to be integrating email with a database. For all the problems I am experience it is NOT worth any benefit on search etc. I have grep and other fast things if I need to search. My contacts are all messed up because of something that associates names with emails, and does it incorrectly. At ever angle there are problems from the software trying to do things it need not. Features that CANNOT be turned off.

This needs to stop!!! Kmail2 needs to be decoupled from this horrible mess that is akonadi. Emails on disk should not disappear and be removed due to some syncing or other issue that clearly comes from the database. Nothing else would remove emails like that. I saw them flashing and re-indexing. I saw that happen to other folders before but never lost emails. Clearly seeing stuff resulting from integration with on disk emails into some sort of database.

I do not care what is done with akonadi, but it should NEVER delete emails as part of its syncing operations. This is really unacceptable. I am very displeased all around at how people are doing experimental things with software people will heavily rely on for business, to make a living, etc.

An email client should NEVER delete emails unless a person does that or its part of a filter etc. Such serious bugs should NEVER exist. That they result from some database integration and some needless features is unacceptable. The entire point of email software is to receive and store email. If it cannot do that simple function the rest is completely pointless and moot.

The most important feature of any email client is to maintain email. No change should ever be made to kmail that could potentially delete an email without a users interaction, unless from a user created filter. There needs to be some serious code review and much better QA.

This is the WORST email software I have ever used. If I did not have so many folders, filters, etc I would switch. Plus to switch means I likely need to switch desktops envs. Only reason I am even running kmail is I left gnome. But I kept using evolution for a while, and wish I never migrated to kmail.

I used evolution for over a decade and NEVER once did I have any issue. Nor with Thunderbird ever. I have been using kmail since ~2011. I have hated it ever day since. Many on Gentoo have stopped using it including many developers who maintain KDE on Gentoo.

I do like many aspects of kmail. But I absolutely HATE akonadi, any integration, etc. That NEEDS to be optional seriously. This joke about rapidly developing PIM applications etc. It is pointless. There are no other KDE PIM based email clients. The one that was developed as an example in 2008 never went further and there has been no others in almost a decade. Please reconsider this direction.

Akonadi and any efforts there are NOT improving the quality of Kmail. They are in fact making it horrible unreliable, deleting emails, corrupting others as I have reported in other bugs. So many issues that would not exist if it was not for this akonadi bs.  This failed experiment that is akonadi needs to come to an end.

If kmail continues to be developed using akonadi and no option to not use a DB backend and just stick to plain files on disk. Then I will definitely move back to evolution or thunderbird. These sorts of problems really should not exist. I have never experienced them or others till kmail.

Unacceptable, and needs to change radically!!!!
Comment 1 William L. Thomson Jr. 2017-03-01 20:42:15 UTC
If you all continue on with adonadi, aside from not deleting emails on disk during any sync or other operation. You should STOP using mysql, and use a reliable database which has MUCH better support for embedded usage, Firebird....

I am not sure if this data loss comes from mysql or akonadi, or something between then. I know akonadi deleted the emails on disk as kmail has no access. Nor does mysql.

Furthermore if using a db backend, emails should be stored in the database. That way there is NO chance of losing sync with emails in database and on disk if they are all in the database to begin with. Trying to use a db for indexing and meta data leaving emails on disk is just asking for problems. Much of which I have experienced first hand and will continue to exist as long as these horrible methods are used.

Don't use a db backend for on disk stuff. If you are going to index that, move all data into the database. If none of that, then at least never have akonadi delete emails. If it has syncing issues, it should rebuild from what is on disk. NEVER deleting what is on disk....
Comment 2 Christoph Feck 2017-03-02 21:06:54 UTC
If this should indeed be a bug report (and not just a rant), make sure developers can follow what you see. Please add exact steps to reproduce, including configuration, ideally starting from a fresh user account. There is also the akonadiconsole and several akonadi log files that can help understanding and pinpointing your issue.

akonadiconsole is described at https://techbase.kde.org/KDE_PIM/Akonadi/Development_Tools
Comment 3 William L. Thomson Jr. 2017-03-02 21:27:55 UTC
This is not something I can replicate. I have no clue how it happened. I have had nothing but problems with akonadi. It is complete garbage all around.

This is DEFINITELY a bug report! Akonadi screwed up and wiped out my inbox and deleted all emails in the inbox maildir. There is NOTHING else that would delete files in just that one folder. Nothing else managed those maildir folders but kmail/akonadi.

My contacts are all screwed up and other issues with akonadi I do not bother reporting or looking into. Stuff that is not in the address book file. Nor that shows in my contacts as I can correct or edit that stuff. So much garbage from akonadi its not funny. Seems another Gentoo dev started Trojita, I will likely see about contributing to that and moving on.

Every for every potential benefit of akonadi there is substantially more issues. It is the WRONG direction to go and needs to be optional. Till it is not longer experimental and does not mess up.

This may have been mysql corruption, since its using a HORRIBLE RDBMS, Should be using Posgresql or Firebird. Which really only Firebird has a true embedded option and mode. Not requiring a daemon. I had other issues early on with sqlite and that was dropped entirely.

I cannot replicate. But ANY code in Akonadi to remove emails from disk should be removed immediately. It did not move the emails to the trash or anything. It literally wiped them from existence on my system. I have lots of filters, folders, hundreds of thousands of emails. No issue in other folders.

However I have seen this indexing thing, were the emails seem to populate and messed with scrollbar, as akonadi is clearly doing something with the database and syncing emails on disk.

Only KDE bugs I have ever filed are all on kmail2...
https://bugs.kde.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=NEEDSINFO&bug_status=VERIFIED&bug_status=CLOSED&email1=wlt&emailreporter1=1&emailtype1=substring&list_id=1425933&query_format=advanced

I really cannot emphasis enough the amount of issues since day 1. Across so many versions, libraries, etc. It is garbage!!!! It is extremely unreliable. It seems to be widely known but just about anyone. Not sure what it will take for KDE devs to get the message and stop this madness...

If I leave KDE for another, it will be 100% due to kmail2. I wish I never left evolution seriously! Kmail has been nothing but a headache. If migrating away was trivial I would have already.
Comment 4 Martin Steigerwald 2017-03-02 21:41:55 UTC
William, as is your report is not really actionable.

It might be a duplicate or related to:

[kmail2] [Bug 375149] New: Kmail deletes emails from folder after switching to folder

which would indicate that it may have been introduced in a quite recent version of KMail + Akonadi.

I currently use 16.04 on Debian and… since… years KMail did not loose a single mail. And I expect the same for the KDEPIM developers… so… if you want them to help you, I suggest you better create a bug report that contains enough information so that developers can really work with it. Christoph made some suggestions. I suggest you create a new bug report and we better forget about this one which isn´t really constructive.

++information, --rant … is what I suggest to you if you are really into creating a change for the better. (I certainly understand frustrations with Akonadi… I went through those as well.)
Comment 5 William L. Thomson Jr. 2017-03-02 22:08:19 UTC
I cannot replicate this. I am sorry but at the time, dealing with the lost emails was more important than going on a mission to help developers fix a bug... I am not sure I could produce anything anyway. I have seen it happen before and was not much for me to produce in the form of log output etc.

Does akonadi even have an activity log file? Nothing in /var/log/* even mentions akonadi...

mysql logs are empty with last touched date of January 18th.

Exactly where is this output I should be producing?

I write lots of software. I rarely ever find or run into bugs my users report. Unit tests do not seem to eliminate user bugs either. That KDE Pim developers do not experience issues is of no surprise. I would google kmail2, and you will find countless reports of users having issues. Not sure you can even compare a Gentoo env to a Debian one. Assuming everything is the same version, maybe something newer in Gentoo than Debian, hard to say, lots of variables.

The fact that akonadi does not log any output to system log files by default shows one of its many design flaws. Who ever came up with this concept clearly missed some very important aspects needed for debugging and fixing issues... Who writes software that does not log output by default. Much less core software like this?

This is a joke...
https://techbase.kde.org/KDE_PIM/Akonadi/Debug_IMAP#Debug_Output

How is that supposed to help in situations where you cannot replicate what happpened. Go set that up after the fact of a problem occurring. Without log activity there is no trail. Write better software that produces logs of activity. That way when something goes wrong that log can be used to determine what went wrong.

If this akonadi had logging by default, I would have included that with my bug report. You can see your giving users nothing to help developers in addressing issues. This software is horribly designed and coded all around. It is WAY to experimental and keeps radically changing. Some akonadi2/kube in the works, etc.

Need to stop messing with things. Email should be reliable, rock solid. I have lost lots of individual emails. Kmail also has no ability to handle imap T flag. It pulls in messages that the server maybe trying to delete that come with the T flag. It keeps those in the folders, grayed out. I have to go into the folder in disk and delete the email. Then it is removed from the list.

This losing all emails is another matter entirely. It not only wiped out inbox/cur but inbox/new.

I will conclude with I have never had any issues with other email clients like I have repeatedly had with Kmail. I an not a novice user by any means. If I am running into such issues I can only imagine others. Anyone I speak to does not run kmail. Most avoid it for known issues. Even most on Gentoo who maintain KDE run from kmail2.

Its you all software and reputation. Akonadi and kmail2 are very below subpar... New features added that are stupid while core features cause major issues... Some stupid area for favorite folders I cannot get rid of. Some dumb text next to emails saying no html message etc. While I lose all emails with no log output...
Comment 6 William L. Thomson Jr. 2017-03-24 16:05:04 UTC
I think this is related to Filters. I am seeing mail ending up in various folders that do not match the filters. I keep coming across old mail in the wrong folder. Not sure if its doing this as mail is coming in. Or something going on with mail already arrived.

Also there is odd handling of emails in the maildir folders. Sometimes mail stays in new permanently and is never moved to cur. Should be anytime its open or read moved to cur from new.
Comment 7 William L. Thomson Jr. 2017-04-06 19:24:11 UTC
Closing since after ~5yrs of frustration with this last scare losing my entire inbox being the last straw. It took some time, as I am not one to change. I liked many aspects of Kmail2, but to many issues and things that caused me problems.

I have switched to Claws Mail. The memory usage, or lack there of when dealing with thousands of emails is insane. Akonadi would bloat ~500MB. Claws Mail never uses more than 100MB. I am extremely pleased with its performance. It starts really fast, and I am quite pleased all around.

Thanks for the fish and headaches. Kmail2 would be great without the KDE PIM. That also messed up my calendar and other stuff. All of Akonadi is garbage and should be tossed. Akonadi is why Kmail2 sucks, not because of Kmail2 itself....

Hope that lesson is learned someday at the expense of others like myself. So much wasted memory, electricity, battery life....
Comment 8 William L. Thomson Jr. 2017-04-07 05:40:12 UTC
In going through migrated emails from Kmail2 to Claws Mail. I found all the emails that disappeared from my Inbox. They were in another folder. At first I thought I may have dragged them into that folder or something. But as I moved the emails that did not belong in that folder back to my in box I noticed lots of other emails that also did not belong in that folder.

It seems like something went horrible wrong with filters. That is the only thing that would make sense for the other emails plus my inbox one to end up that way. I have filters to move email from IMAP Inboxes to local folders. Maybe somehow that confused a remove Inbox with the local one and moved all those emails to another folder. But in doing so it also redirected emails matching other filters destined for other folders into that same folder. Which is pretty odd, and it was a random folder.

Anyway I would say this is definitely related to filters. The emails were not deleted just moved. Others were not moved to the correct folder. All ended up in the same incorrect folder. Including my missing Inbox emails. Hope that helps in some way.
Comment 9 Martin Steigerwald 2017-04-07 12:18:23 UTC
William, do you really expect someone to read this after all your previous ranting? I wouldn´t.

Also "something with filters" is *waaaaay* to unspecific in order to say anything about it. So if you have a reproducable bug regarding filters, feel free to report it, but please 1. with a different tone, and 2. in a *new* bug report.

I do have an insanely complex filter setup and it works just fine. But there are many ways to setup filters in a way that they conflict which each other and so on, so what you see can easily just be a result of misconfigured filters (i.e. mistake made by you as an user.)

Thanks, Martin (do not expect me to reply to any rants you may come up with, its not that I think Akonadi is perfect – far from it, I even agree it might be good to replace it at some time, maybe with the new Sink – but for me it basically works quite okayish meanwhile).

Thank you and good luck with Claws Mail.
Comment 10 William L. Thomson Jr. 2017-10-17 20:39:32 UTC
(In reply to Martin Steigerwald from comment #9)
> William, do you really expect someone to read this after all your previous
> ranting? I wouldn´t.

I have different expectations of others. Given the experience, I could understand if another was expressing frustration due to issues. If software I wrote caused serious problems for another. I would likely read it if they took the time to write it. If they did not care, they would not even comment right?

> Also "something with filters" is *waaaaay* to unspecific in order to say
> anything about it. So if you have a reproducable bug regarding filters, feel
> free to report it, but please 1. with a different tone, and 2. in a *new*
> bug report.

I would but I moved on and am not able to help in that process. If its not effecting others it would just be a false report.

> I do have an insanely complex filter setup and it works just fine. But there
> are many ways to setup filters in a way that they conflict which each other
> and so on, so what you see can easily just be a result of misconfigured
> filters (i.e. mistake made by you as an user.)

I would love to blame myself. But I had not messed with filters in some time. I had lots of them, and they worked fine. Once I setup them up I rarely change. They had been the same for some time, across a few versions of Kmail2. I cannot explain what caused emails to end up in all sorts of folders they did not belong. I just assumed it was a filter, as that seems to be the only think that sorts emails. It very well could have been something else.

> Thanks, Martin (do not expect me to reply to any rants you may come up with,
> its not that I think Akonadi is perfect – far from it, I even agree it might
> be good to replace it at some time, maybe with the new Sink – but for me it
> basically works quite okayish meanwhile).

Sorry for the rant, but having had numerous email issues with Kmail2. I was at my limits. I can handle issues with many things, but email is not one. More so when used to make a living and losing important work emails. Its not something others even tend to believe. My email client screwed up and deleted emails. Yeah right that was something you did. Just like my issues with filters was blamed on me, regardless if it was a user issue or not. Which I am pretty sure I did not mess up filters. They worked before and after.

I had no clue the emails were in the wrong folders. I thought they were deleted. Many did not even match any filter as they were from my inbox. It was super weird but given my numerous issues and lost emails before. It was not till migrating to Claws Mail that I found them and commented.

> Thank you and good luck with Claws Mail.


Thus far months later not a single issue with claws mail other than some very minor UI quirks. I haven't even bothered to file bugs. I may look to submit patches/PR in any bug report as part. Being more familiar with C than C++.

Part of why I moved on. I did not see myself being in a position to actually help with the issues. At least I reported them. Its not like most take time to report bugs.

I think if you put yourself in my shoes you can understand the frustration and rants....

Thanks for being receptive, dealing with the rant, and closing with a polite statement! I really liked KMail2, just not the backend stuff. I am a DB guy but more of one to put it all in the db. Syncing dbs with files on disk is difficult. But filters many avoid. Part of why I left QT was like Trojita, the alternative to Kmail2 will not get filters. The developers I believe are anti filters. I need filters :)

Maybe to filter myself at times.... :P