Bug 378189

Summary: kmail often stuck in "Retrieving Folder Contents"
Product: [Applications] kmail2 Reporter: Martin Koller <kollix>
Component: generalAssignee: kdepim bugs <kdepim-bugs>
Status: RESOLVED FIXED    
Severity: normal CC: apcomptec, cousinmarc, emilianh, kde, kdebugs, montel
Priority: NOR    
Version: 5.4.3   
Target Milestone: ---   
Platform: Other   
OS: Linux   
Latest Commit: Version Fixed In:
Sentry Crash Report:

Description Martin Koller 2017-03-28 07:11:14 UTC
I upgraded from former KDE4-based kmail to KF5 based kmail.
Since then I often encounter the problem that kmail simply does not show my LOCAL messages (using a maildir resource) but instead hangs in the "Retrieving Folder Contents Please wait . . ." display.

I did not find a pattern yet and I don't know how to analyze this further to give more details.

Sometimes it just takes a few minutes waiting and then I can use kmail normally again.

The only solution I found yet to be able to use kmail again when it hangs is to restart akonadi (akonadictl restart).

Any ideas what I can do in this situation to give more hints what happens ?
Comment 1 Paul Colquhoun 2017-03-29 04:56:01 UTC
I am also experiencing this problem.

When I start kmail from a command line, and select a mailbox where the problem is happening, I get a run of errors shown, like:

org.kde.pim.akonadicore: Got a stale notification for an item which was already removed. 36928 ""

The number run up to 36968 with a few skipped.

I have also noticed, in my attempts to fix things, that running "akonadictrl fsck" reports a list of errors, but does not fix any, as the same errors are reported on the next run.

$ akonadictl fsck
Looking for resources in the DB not matching a configured resource...
Looking for collections not belonging to a valid resource...
Checking collection tree consistency...
Looking for items not belonging to a valid collection...
Looking for item parts not belonging to a valid item...
Looking for item flags not belonging to a valid item...
Looking for overlapping external parts...
Verifying external parts...
Found 274 external files.
Found 274 external parts.
Found no unreferenced external files.
Checking size treshold changes...
Found 0 parts to be moved to external files
Found 0 parts to be moved to database
Looking for dirty objects...
Collection "Search" (id: 1) has no RID.
Found 1 collections without RID.
Item "36378" has no RID.
Item "36553" has no RID.
Item "36554" has no RID.
Item "36556" has no RID.
Item "36575" has no RID.
Item "36690" has no RID.
Item "36709" has no RID.
Item "36712" has no RID.
Item "36723" has no RID.
Item "36727" has no RID.
Item "36733" has no RID.
Item "36736" has no RID.
Item "36739" has no RID.
Item "36748" has no RID.
Item "36759" has no RID.
Item "36762" has no RID.
Item "36764" has no RID.
Item "36768" has no RID.
Item "36770" has no RID.
Item "36772" has no RID.
Item "36779" has no RID.
Item "36811" has no RID.
Item "36924" has no RID.
Found 23 items without RID.
Found 0 dirty items.
Looking for rid-duplicates not matching the content mime-type of the parent collection
Checking Local Folders
Checking Personal Contacts
Checking Personal Contacts
Found duplicates 022265b7-7fa6-4835-8b83-9afea8ccc538.vcf
Checking Search
Checking ASP
Checking Accounts
Checking Aust_Post
Checking Bank
Checking Companies
Checking DandD
Checking Friends_and_Family
Checking From_Me
Checking Getup
Checking ITunes
Checking Important
Checking Internet_Sites
Checking JUNK
Found duplicates 1490556472.R580.bluering
Checking Mailing_Lists
Checking Misc
Checking Orders
Checking Software-Order
Checking TAFE_related
Checking Telegraph
Checking Tickets
Checking Web_Sites
Checking drafts
Checking inbox
Checking muttout
Checking outbox
Checking sent-mail
Checking templates
Checking trash
Checking Amazon.com
Checking Apple
Checking Athletes_foot
Checking Australian_Unity
Checking Bigpond
Checking Bioware
Checking Borders
Checking Change.org
Checking Computerworld
Checking Epson
Checking Food_Delivery
Checking Foxtel
Checking GoodOldGames_GOG.com
Checking IceTV
Found duplicates 1490549272.R827.bluering
Checking Iolo
Checking JB-Hi-Fi
Checking Jacksons
Checking Livecode
Checking NRMA
Checking Navman
Checking New_Frog
Checking New_Scientist
Checking PacktPub
Checking Pokemon
Checking RTA
Checking Realestate_Agent
Checking Roses_Only
Checking SMH.com.au
Checking Techbuy
Checking Telstra-Notices
Checking Ticketmaster
Checking Turbine
Checking lazypatch
Checking Boystown_Lottery
Checking Facebook
Checking Flatmates
Checking Instagram
Checking Internet
Checking LinkedIn
Checking Linux_Weekly_News
Checking Littlemutt
Checking No-IP
Checking Pair.com
Checking PayPal
Checking Twitter
Checking ebay
Found duplicates 1490593373.R703.bluering
Checking makeuseof
Checking Gentoo
Found duplicates 1490520173.R377.bluering
Found duplicates 1490520474.R132.bluering
Found duplicates 1490521075.R588.bluering
Found duplicates 1490534871.R650.bluering
Found duplicates 1490539671.R833.bluering
Found duplicates 1490550472.R270.bluering
Found duplicates 1490552872.R277.bluering
Found duplicates 1490587073.R750.bluering
Checking Gentoo-amd64
Checking Job_Hunting
Found duplicates 1490561272.R98.bluering
Checking Lotto-Results
Checking NSWCC
Checking Portage-elogs
Checking RedHat_Announce
Checking Red_Hat
Checking SAGE-AU
Checking Stats-Tracker
Checking The_Troll
Checking Yahoo_mail
Checking gmail.com
Checking rkhunter
Checking News-Filters
Checking Passwords
Checking Routine_Reports
Checking muttout
Checking IT_Institute_FB
Checking TAFE
Checking Avaaz
Checking Cross-Eyed-Bear
Checking Penthouse
Checking PlentyOfFish
Checking RSVP
Checking VitalPartners
Migrating parts to new cache hierarchy...
Consistency check done.
Comment 2 Paul Colquhoun 2017-03-29 05:02:33 UTC
I forgot to include this:

$ kmail --version
kmail2 5.4.3
Comment 3 Paul Colquhoun 2017-03-31 06:57:42 UTC
I have followed the directions at
https://docs.kde.org/trunk5/en/pim/kmail2/clean-start-after-a-failed-migration.html
to rebuild the akonadi databases from scratch, but the problem has persisted.
Comment 4 Martin Koller 2017-04-19 05:02:25 UTC
Git commit 63ce354bbee698e3daa5d193e0b8c0e1725af9ab by Martin Koller.
Committed on 19/04/2017 at 05:00.
Pushed by mkoller into branch 'Applications/17.04'.

On error, still send answer via DBUS, avoiding clients waiting forever

M  +3    -2    src/agentbase/resourcebase.cpp

https://commits.kde.org/akonadi/63ce354bbee698e3daa5d193e0b8c0e1725af9ab
Comment 5 Laurent Montel 2017-04-19 05:42:15 UTC
Are you sure that it's enough ?
I still have this problem.
Comment 6 Martin Koller 2017-04-19 05:54:31 UTC
> Are you sure that it's enough ?
> I still have this problem.

Well ... how sure can I be, given that I know not much about the Akonadi internals ...

This is what I found during my analysis, and here it solves the problem.
I'm using a POP account and local maildir, no IMAP (but in fact the resource type should not matter I assume).

Maybe you have a different issue or the same bug is present somewhere else.

Check with the JobTracker in akonadiconsole if kmail is stuck with jobs never get finished. And if you encounter the same problem, then you should also see some Job error mentioning the "Multiple merge candidates" message.
Comment 7 Laurent Montel 2017-04-19 11:48:56 UTC
Yep I still can able to see "Multiple merge candidates"
I use pop3 + imap too.
Comment 8 Martin Koller 2017-04-19 13:05:24 UTC
This patch does not fix your current database nor does it solve the issue
that akonadi creates multiple ids in the first place.
It just avoid the "endless waiting".
Comment 9 emilianh 2017-04-26 15:58:44 UTC
Still experiencing this on 17.04.
Status bar shows the following:
"Unable to fetch item from backend (collection -1) : Unable to retrieve item from resource: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken."
Comment 10 Martin Koller 2017-04-26 18:11:41 UTC
What you see is a different problem.
This bug report was about not getting an answer from akonadi at all and therefore
kmail showing only a "please wait..." display

At least you GET an answer - although there is still something wrong in the DB.

What you can do to solve your problem at least until it occurs again:
Open aconadiconsole, select the folder in the Browser tab, ritgh click it,
select "Clear Akonadi cache" then either run "akonadictl restart" or logout/login again
Comment 11 emilianh 2017-04-26 20:01:57 UTC
Did that countless times, problem comes back.

Can there not be a "no caching" mode? "Mail" is already on the machine (maildir resource), and just copying the mail to some other location on the FS seems rather pointless...
Comment 12 Martin Koller 2017-04-27 04:53:04 UTC
>Can there not be a "no caching" mode? "Mail" is already on the machine (maildir >resource), and just copying the mail to some other location on the FS seems >rather pointless...

There is no copying done. Mail is received from one resource (e.g. POP) and another process (e.g. maildir resource) stores it. The Akonadi DB just holds information what is where (more or less).

The problem you and I see is somewhere due to the filtering agent inbetween.
For me it seems to be a timing issue.

P.S.:I'm not the author of any part of akonadi, but I try to find the bugs which hurt me.
Comment 13 Matt Whitlock 2017-05-11 17:56:14 UTC
(In reply to Martin Koller from comment #4)
> Git commit 63ce354bbee698e3daa5d193e0b8c0e1725af9ab by Martin Koller.
> Committed on 19/04/2017 at 05:00.
> Pushed by mkoller into branch 'Applications/17.04'.
> 
> On error, still send answer via DBUS, avoiding clients waiting forever

This patch didn't solve the problem for me. I'm on Akonadi 17.04.0 (with this patch) and KMail 17.04.0 (5.5.0).

Several times a day I find that KMail has stopped retrieving messages from my Gmail IMAP account, and there is an empty progress bar stuck at the bottom of the window. This only started happening fairly recently. (I upgraded from Akonadi 16.12.3 on 21 April, so I may try going back to that version.)

I have the Akonadi Console open with the Job Tracker enabled now, to try to see if any jobs hang in Akonadi. Is there anything else I should look for to diagnose this issue?
Comment 14 Matt Whitlock 2017-05-11 19:12:06 UTC
(In reply to Matt Whitlock from comment #13)
> I have the Akonadi Console open with the Job Tracker enabled now, to try to
> see if any jobs hang in Akonadi.

Nope. KMail has gone out to lunch again, but Akonadi Console shows all jobs have "Ended" or "Failed."
Comment 15 Brallan Aguilar 2017-06-02 10:45:35 UTC
Everyday I must restart Akonadi and KMail to use my email properly :(
Using Kontact 5.5.1