| Summary: | imap resource starts crazy connect / disconnect cycle to mail server | ||
|---|---|---|---|
| Product: | [Frameworks and Libraries] Akonadi | Reporter: | Ben <ben> |
| Component: | IMAP resource | Assignee: | Christian Mollekopf <chrigi_1> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | alan, anders.nilsson, andreas, andy, ao, aog2000a, bas-bugzilla, benoitg, casta, dvratil, freekdekruijf, harryl, itsef-admin, jamatos, jsamyth, julian, kde, kde, kde, kdebugs, kdebugs, kdepim-bugs, laurent.rineau, lynx, markus.zimmermann, Martin, miroslav, mollekopf, morten, mtmkls, nmsirolli, pdgiddie+kde, r.biegel, romainguinot, s.dm, sgh, site+org.kde, stefan.bruens, stevee, stierand, thomas, tho_public, vkrause, web, weigelt.bernd, wilderkde |
| Priority: | NOR | ||
| Version: | 1.9.0 | ||
| Target Milestone: | --- | ||
| Platform: | Ubuntu | ||
| OS: | Linux | ||
| Latest Commit: | http://commits.kde.org/kdepimlibs/4b8adabf8d6d821ad6fde20588a5ef0d72363a97 | Version Fixed In: | 4.11.4 |
| Sentry Crash Report: | |||
| Attachments: | Potential minimal fix | ||
|
Description
Ben
2013-03-16 12:25:36 UTC
upon further testing even with interval mail checking turned off the same thing happens. I have had a closer look at the mail server log files and it appears to be triggered after imap server logs a timeout message for the account. I am also encountering this problem, however I have two IMAPs, one to Google and one to a courier-imap mail server. The problem does not happen with the Google connection. One more note, the loop only starts once I receive an e-mail. it operates normally until then. I am seeing this behaviour too. I am running a local Courier IMAP server and I noticed this in mail.log - one connection from my local mail client running on the desktop on the same network, and another connecting from an external network. In both cases the client is akonadi_imap on KDE 4.10.1 desktop. Also from the same desktop I am connecting to another Courier IMAP and I was told by a sysadmin of that system that the same behaviour is noticed, ie flood of connect/disconect. I am happy to provide my Courier and my client settings if that may help. Not sure if it can corroborate your issues, but since last week i'm having an issue : I use KMail from 4.10.2, on Fedora 18. 1 IMAP account connected to a Google Apps hosted domain. Worked great, and since this week i regularly have errors saying "too many simultaneous connections"... Tried recreating the account from scratch, same issue. Workaround i have found so far is to go in the Gmail web UI, and click "sign out of all sessions" in the connection details. Then the akonadi resource is willing to fetch mail again. I'm suspecting there's some kind of connection leak, maybe similar to what you are experiencing. Same problem for me on kde-4.10.2 (on gentoo), the logging looks a lot like the logging in the first comment. Same bug here on Kubuntu 13.04 *** This bug has been confirmed by popular vote. *** Wireshark dump: http://ubuntuone.com/3uS3HYKKui3cYTvZESgeyV Switch to imap without SSL and seems to be working again... Kde 4.10.3 still has the same problem, using IMAP without SSL is not a solution for me. BTW, I am using Gentoo instead of Ubuntu (the platform this ticket is reported with). If more logging / information is needed please let me know. Kontact is totally useless right now for me, and I had to reinstall Thunderbird. (In reply to comment #3) > One more note, the loop only starts once I receive an e-mail. it operates > normally until then. Confirmed the same behaviour. The same here with freshly updated openSUSE tumbleweed KDE 4.10.3. BTW, I have seen a very similar behaviour description for bug #320180. I wonder whether it is possible to detect this situation by any log at the client side. The akonadi output at .xsession-errors does not tell me. The only way for me to detect this is to monitor the network traffic (and to wait for complaints by our mail admin). I was banned from my University IMAP server because of this! Please fix this quite sever problem. *** Bug 320180 has been marked as a duplicate of this bug. *** *** Bug 317790 has been marked as a duplicate of this bug. *** *** Bug 315382 has been marked as a duplicate of this bug. *** This issue is related to bug 319569. It does not crash due to stack overflow, but causes endless loop due to unhandled SSL error. *** Bug 321509 has been marked as a duplicate of this bug. *** *** Bug 322006 has been marked as a duplicate of this bug. *** The problem persists with an update to openSuSE tumbleweed KDE 4.10.5 It seems that switching from IMAPS to IMAP + STARTTLS fixes this. the issue has vanished since 2 days when I changed the setting. Still seeing this with 4.11, turning of SSL can NOT be an option. Come on guys, this is a real showstopper and it's persisting since the release of 4.10. Months later, nothing has changed in this regard. Even worse, people have been banned by their providers due to stressing the imap-server. I have to report, that this problem causes a 100% load on my local mail server when akonadi goes crazy (which happens about 30mins after starting this piece of junk). Fix it finally!!!! (In reply to comment #23) > Still seeing this with 4.11, turning of SSL can NOT be an option. Come on > guys, this is a real showstopper and it's persisting since the release of > 4.10. Months later, nothing has changed in this regard. Even worse, people > have been banned by their providers due to stressing the imap-server. I have > to report, that this problem causes a 100% load on my local mail server when > akonadi goes crazy (which happens about 30mins after starting this piece of > junk). > > Fix it finally!!!! I fixed it with `pkill akonadi` right after login. (In reply to comment #24) > I fixed it with `pkill akonadi` right after login. 'akonadictl stop' is more appropriate since the pkill cmd leaves you with leftover akonadi_server procs. I agree with Christian; this a major problem. I was banned twice by my provider, and it's not easy to keep asking him for oportunities, e.g. "Please, unban me again, let's see if 'pkill akonadi' works". I had to switch to Thunderbird... The log that my provider showed me is different from Ben's. What my provider gets is (tons of): host=157-120-165-181.fibertel.com.ar [181.165.120.157] Jul 16 16:25:54 mail imapd[14752]: imaps SSL service init from 181.165.120.157 Jul 16 16:25:54 mail imapd[14752]: Unexpected client disconnect, while reading line user=??? I have this problem on two systems, both 64bit with latest openSUSE-Patches and KDE v4.11. I have seen this problem since an early KDE 4.10, so i stepped over to Thunderbird. Yesterday i made some test, but akonadi filled my /var/log/mail with 35MB messages in 5 hours. i use a SSD on my local IMAP-Server and i don't want to buy a new one ;-) I can break the cycle, if i log off from KDE, but it starts again after a various time. IMHO it is a big problem. thx Bernd I'm trying to switch to imap from a bunch of pop3 accounts, and came across this problem. I have a hypothesis as to what is going on. Things work fine for a long time under normal circumstances, but if the imap server disconnects the imap client, akonadi doesn't notice and keeps trying to use the session, which is now invalid and the server rejects. somehow even after the rejection the akonadi imap client still doesn't know the session is dead, and things just stop working in kmail, all while the imap server spams its own log with "disconnect" messages. At least thats how it seems to me. I can sometimes stop the cycle by fully stopping, and restarting my imap server, but it'll eventually come back. And the other way is by fully stopping and restarting akonadi. But again, it will eventually come back. Just one instance of akonadi_imap_resource running in a OpenSuse 12.3 machine with KDE 4.11 pretty much saturated our own imap server running on our mail gateway. Akonadi was eating 50% cpu on the offending station. This is not fixed in 4.11, and its a pretty serious error guys. I've had to ban hammer kmail and send everybody back to Thunderbird. It's nice to know that switching from IMAPS to IMAP + STARTTLS fixes this, but this might not be possible in all cases: it depends on what the server supports. We are actively working on resolving this - I have a suspicion that there could actually be a bug in QSslSocket ( could be related to https://bugreports.qt-project.org/browse/QTBUG-8896 ). Hopefully we will track this down soon, I'm hit by this issue too and it's annoying :) I went through Kwallet and found numerous password entries for the IMAP account that was steamrolling my system. Have deleted all of these and re-ran Kmail to get this password reset. Krunner is showing akonadi_imap_resource now behaving - staying down in the CPU order. See what goes with this. (In reply to comment #33) > I went through Kwallet and found numerous password entries for the IMAP > account that was steamrolling my system. > Have deleted all of these and re-ran Kmail to get this password reset. > Krunner is showing akonadi_imap_resource now behaving - staying down in the > CPU order. > See what goes with this. Its when you shut down kmail that you will see it hammering the imap resource hammering the imap server. If you only have one email account it will be obvious which account it is hammering. Otherwise, its very difficult to tell. I can watch the log on my in-house mail server (cyrus imap), but that is the only one I can see. Passwords in kde-wallet are strewn in three different places, under imap, under kmail, and under mailtransports. Which ones are you deleting? I deleted the password throughout Kwallet. Of course in time the bug came back. The two IMAP accounts I use : Yahoo IMAP and owc.net IMAP. The owc account is the one that is firing up the bug. I was wondering if the server "attributes /protocols/features" might be a source of the problem. Possibly not having a handler in the SSl Resource? The backing attributes or protocols for owc are: IMAP4REV1 ACL BINARY CATENATE CHILDREN CONDSTORE ENABLE ESEARCH ID IDLE LIST-EXTENDED LITERAL+ LOGIN-REFERRALS MULTIAPPEND NAMESPACE QRESYNC QUOTA RIGHTS=EKTX SASL-IR SEARCHRES UIDPLUS UNSELECT WITHIN ======== The attributes for Yahoo IMAP are: IMAP4REV1 ID NAMESPACE X-ID-ACLID UIDPLUS LITERAL+ XAPPLEPUSHSERVICE XYMHIGHESTMODSEQ One of these adding to the problem? Note: I also use Zimbra Client, and it was having problems with Yahoo after the mail system was recently updated. It is now fixed. Experiencing this, could the IMAP server side have a hand in this bug? Hi Harry, the issue is caused by two bugs in the way we handle SSL. Partially by a wrong method of disconnecting from the socket, which Christian fixed in 4.11.3, and partially by a bug in SSL fallback implementation, which we hope to fix soon, too. It's not related to password, or handling of the IMAP protocol itself. (In reply to comment #36) > Hi Harry, > the issue is caused by two bugs in the way we handle SSL. Partially by a > wrong method of disconnecting from the socket, which Christian fixed in > 4.11.3, and partially by a bug in SSL fallback implementation, which we hope > to fix soon, too. It's not related to password, or handling of the IMAP > protocol itself. I'd like to suggest a third bug: Leaving an imap resource open (running) long after the invoking application(s) has been terminated. Most often, I see runnaway processor usage long AFTER completely close out of kmail. This is completely unrelated to this particular bug and it's not actually a bug: KMail is just a client for Akonadi. The resources (including IMAP resource) are maintained by the Akonadi Server, which runs in the background all the time. There's currently an effort (in early planning phase) to make the resources to suspend/resume on demand. (In reply to comment #38) > This is completely unrelated to this particular bug and it's not actually a > bug: KMail is just a client for Akonadi. The resources (including IMAP > resource) are maintained by the Akonadi Server, which runs in the background > all the time. There's currently an effort (in early planning phase) to make > the resources to suspend/resume on demand. I think we all understand that kmail is not directly involved, other than triggering akonadi to start a resource that was not previously running. You'll have to forgive my naiveté on this issue, because I've only been designing and programming systems for 35 years, and it hadn't yet occurred to me that one might actually design a system to initiate a daemon without a thought about how to cleanly shut it down. Sorry, I meant no offense. Shutting down the Akonadi server is not always the desired behavior :-) It allows you to receive your emails and get notifications without having to run a big client (KMail). But I agree it's arguable that some people might not welcome this behaviour, for those we are working on the suspend/resume resource on demand feature (it will not stop the Akonadi server itself, but at least the unneeded Akonadi resources to preserve some HW resources). Anyway, this is off topic of this bug report, if you want to discuss this or suggest new improvements, please open a new bug, or a forum thread. I managed to work around this bug by using stunnel in client mode. You need to add something like this to stunnel.conf: [outlook-imap] client = yes accept = 127.0.0.1:51143 connect = imap-mail.outlook.com:993 Then just connect to localhost 51143 without encryption. (In reply to comment #40) > Sorry, I meant no offense. Shutting down the Akonadi server is not always > the desired behavior :-) It allows you to receive your emails and get > notifications without having to run a big client (KMail). But I agree it's > arguable that some people might not welcome this behaviour, for those we are > working on the suspend/resume resource on demand feature (it will not stop > the Akonadi server itself, but at least the unneeded Akonadi resources Yes, I understood you to be talking about shutting down the akonadi-imap-resourse only. One additional point in favor of that: Some imap servers limit connections (Even Google does this). With an imap-resource running when no mail client is running, you consume at least one connection. (There seems to be no way to actually tell if its is really only one, not documented anywhere, I actually suspect it is more than one connection per imap-resource because 6 devices can exceed Google's 16 connection max on occasion). Since we all seem to have multiple mobile devices these days, you can run out of available connections fairly easily. I tried with akonadi-server (experimental) 1.10 under Debian (generally testing/unstable), and same result: Hammering of my courrier imap TLS server. It seems that requiring my courier daemon to force starttls solves the issue with hanging connection. In /etc/courier/imapd-ssl I now use IMAP_TLS_REQUIRED=1, while still running SSL and allowing TLS. Before, I never saw the following LOGOUT entries in my imap log: Oct 29 12:54:20 <host> imapd-ssl: LOGOUT, user=<user>, ip=[::ffff:127.0.0.1], headers=0, body=0, rcvd=13, sent=86, time=0, starttls=1 Now, all LOGIN entries are followed by such LOGOUT entries, and after observing for some hours without any high-frequency CONNECT-DISCONNECTED entries, I think I can claim that this solved the problem. I attempted this inspired by the comment from Guillaume Castagnino 2013-07-30 18:20:36 UTC. However, as indicated by Ariel Garcia 2013-09-13 20:09:24 UTC, this only solves the problem for IMAP-servers under our own control. Dang. That conclusion (at 2013-10-29 12:14:41 UTC) was premature. The Akonadi IMAP Resource is hammering away again. Sorry for the noise! I just upgraded akonadi-server to 1.10.2-2 and kmail to 4:4.11.3-1 from Debian unstable. Same bombardment on the courier imap server. Are there any other relevant packages that might be involved? As far as I understand, it is actually only akonadi-server which has the imap connection. The IMAP resource has a new maintainer, reassigning to him. Same problem here. i've just rediscovered kde/kontact but this bug could be a showstopper for me. btw. there seems to be an additional problem. I stumbled upon this error because my mailprovider changed from STARTTLS to SSL/TLS. I have 2 different IMAP-Accounts in different servers, the changed one stopped Kmail. IMAP trys to open folders and never come back to life with hammering the server. Hypothesis (sessionthread.cpp): * SSL is used, hence doSslFallback == true * connection succeeds, but with ssl errors which are ignored (e.g. certificate), in this codepath doSslFallback is not reset to false => working connection, but doSslFallback is still true. * Connection get's aborted by the server => QAbstractSocket::RemoteHostClosedError * the session goes into reconnect frenzy I'm working on a patch, a couple of things look fishy in there. Created attachment 83637 [details]
Potential minimal fix
Applies to master and should fix the problem in my hypothesis.
Git commit 4b8adabf8d6d821ad6fde20588a5ef0d72363a97 by Christian Mollekopf. Committed on 19/11/2013 at 17:29. Pushed by cmollekopf into branch 'KDE/4.11'. Always reset doSslFallback. Otherwise doSslFallback will remain true if the errors are ignored, although a connection is established. If we then receive an error from the server (e.g. due to disconnect), the session will go into a reconnect frenzy. FIXED-IN: 4.11.4 M +1 -1 kimap/sessionthread.cpp http://commits.kde.org/kdepimlibs/4b8adabf8d6d821ad6fde20588a5ef0d72363a97 This patch breaks previously working scenarios. See https://bugs.kde.org/show_bug.cgi?id=308854 and https://bugs.kde.org/show_bug.cgi?id=306964 (In reply to comment #52) > This patch breaks previously working scenarios. > See https://bugs.kde.org/show_bug.cgi?id=308854 and > https://bugs.kde.org/show_bug.cgi?id=306964 How exactly? IMO my patch shouldn't break anything, if it does please let me know why exactly. This seems to have totally fixed my problem with run-away akonadi-imap-resourse, I'm running KDE 4.11.4 from Opensuse updates repository. Kmail is usable for the first time in many months. Bug #322199 may be caused by this as well. At least it looks very similar. I will test once 4.11.4 packages come into Debian and I manage to take time for it. works here. too thx Bernd Just upgraded to KDE 4.12.0 - Kmail IMAP works as it should! Thanks Christian for the Fix!!!!! I've also just upgraded to 4.12. It works here too. I'm glad to be using KMail again, thanks!!! |