Bug 127696 - [PATCH] "Unable to complete LIST" operation with invalid messages on server
Summary: [PATCH] "Unable to complete LIST" operation with invalid messages on server
Status: RESOLVED FIXED
Alias: None
Product: kmail
Classification: Applications
Component: pop3 (show other bugs)
Version: 1.9.1
Platform: unspecified Linux
: NOR normal with 45 votes (vote)
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
: 79569 144273 148225 148368 (view as bug list)
Depends on:
Blocks:
 
Reported: 2006-05-20 05:34 UTC by John Birch
Modified: 2008-08-24 18:31 UTC (History)
9 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
patch for comment 15 (7.10 KB, patch)
2008-01-16 22:50 UTC, Patrick McManus
Details

Note You need to log in before you can comment on or make changes to this bug.
Description John Birch 2006-05-20 05:34:38 UTC
Version:           1.9.1 (using KDE 3.5.2, compiled sources)
Compiler:          gcc version 3.4.5 (Gentoo 3.4.5-r1, ssp-3.4.5-1.0, pie-8.7.9)
OS:                Linux (i686) release 2.6.16-gentoo-r7

I've had a mail message added that causes kmail to fail to retreive any messages.

This patch got around the problem but is not correct code. I include it to help pinpoint the problem.

I realise that the message on the server was incorrect but I don't believe kmail should fail with this simple error in the data. Also  the messagebox itself was a bit misleading as it wasn't failing in the LIST stage but in the UIDL stage.

Firstly here's the output from the patch  showing the incorrect message. It's message 77 and it doesn't contain a space and second part (message id and address?)

kmail: PopAccount:: stage==Uidl <2>
kmail: PopAccount:: data=<[37 36 20 3c 32 30 30 36 30 35 31 35 31 30 34 33 2e 35 33 31 39 30 2e 6a 70 65 74 73 6f 40 67 6d 78 2e 61 74 3e 00 0a ]>
kmail: PopAccount::qdata=<76 <200605151043.53190.jpetso@gmx.at>>
kmail: PopAccount::spc=2
kmail: PopAccount:: stage==Uidl <2>
kmail: PopAccount:: data=<[37 37 20 20 00 0a ]>
kmail: PopAccount::qdata=<77>
kmail: PopAccount::spc=-1
kmail: PopAccount:: spc test failed - stage==2

When  this patch is applied it ignores this error and keeps going.   As I said not a correct patch, but at least I can read my mail again     :-)

Cheers,
jbb

Index: popaccount.cpp
===================================================================
--- popaccount.cpp	(revision 542622)
+++ popaccount.cpp	(working copy)
@@ -903,8 +903,12 @@
   QString qdata = data;
   qdata = qdata.simplifyWhiteSpace(); // Workaround for Maillennium POP3/UNIBOX
   int spc = qdata.find( ' ' );
+  kdDebug(5006) << "PopAccount:: data=<" <<  data << ">" << endl;
+  kdDebug(5006) << "PopAccount::qdata=<" << qdata << ">" << endl;
+  kdDebug(5006) << "PopAccount::spc="    << spc << endl;
   if (spc > 0) {
     if (stage == List) {
+      kdDebug(5006) << "PopAccount:: stage==List <" << stage << ">" << endl;
       QString length = qdata.mid(spc+1);
       if (length.find(' ') != -1) length.truncate(length.find(' '));
       int len = length.toInt();
@@ -914,6 +918,7 @@
       mMsgsPendingDownload.insert( id, len );
     }
     else { // stage == Uidl
+      kdDebug(5006) << "PopAccount:: stage==Uidl <" << stage << ">" << endl;
       const QString id = qdata.left(spc);
       const QString uid = qdata.mid(spc + 1);
       int *size = new int; //malloc(size_of(int));
@@ -942,13 +947,18 @@
     }
   }
   else {
-    stage = Idle;
-    if (job) job->kill();
-    job = 0;
-    mSlave = 0;
-    KMessageBox::error(0, i18n( "Unable to complete LIST operation." ),
+    kdDebug(5006) << "PopAccount:: spc test failed - stage==" << stage << endl;
+    if (stage == List)
+    {
+      kdDebug(5006) << "PopAccount:: stage==" << stage << endl;
+      stage = Idle;
+      if (job) job->kill();
+      job = 0;
+      mSlave = 0;
+      KMessageBox::error(0, i18n( "Unable to complete LIST operation." ),
                           i18n("Invalid Response From Server"));
-    return;
+      return;
+    }
   }
 }
Comment 1 Thomas McGuire 2007-06-09 14:56:07 UTC
*** Bug 144273 has been marked as a duplicate of this bug. ***
Comment 2 Thomas McGuire 2007-06-15 15:44:20 UTC
*** Bug 79569 has been marked as a duplicate of this bug. ***
Comment 3 Thomas McGuire 2007-07-26 17:30:55 UTC
*** Bug 148225 has been marked as a duplicate of this bug. ***
Comment 4 Thomas McGuire 2007-07-30 19:21:45 UTC
*** Bug 148368 has been marked as a duplicate of this bug. ***
Comment 5 Ulf Dambacher 2007-11-21 07:47:19 UTC
I have the same bug on 1.9.7. It's very anoying as I always have to delete the message without uidl first, kill the pop3 task and restart kmail before geetting my email!!
Comment 6 Ulf Dambacher 2007-11-21 07:48:34 UTC
I have the same bug on 1.9.7. It's very anoying as I always have to delete the message without uidl first, kill the pop3 task and restart kmail before geetting my email!!
Comment 7 David 2007-12-02 23:49:31 UTC
Same issue here. I had to setup Thunderbird until this get's figured out. 
Comment 8 Michael Mikowski 2007-12-24 03:43:27 UTC
Same issue here for the last 6 months.  Switching to Thunderbird.
Comment 9 Michael Mikowski 2007-12-25 03:43:11 UTC
I should mention my environment: 
kde3.5.8 on Kubuntu 7.10 (Gutsy Gibbon)
Package kdepim-3.5.7enterprise20070926
Comment 10 Michael Mikowski 2007-12-25 03:44:06 UTC
Rebuilding from source now; will report on success or failure
Comment 11 Michael Mikowski 2007-12-26 03:00:45 UTC
Rebuilt packages on Kubuntu 7.10.  Results so far excellent.  Thunderbird conversion averted, at least for now.  Here are the steps:

Using: http://www.debian-administration.org/articles/20

apt-get source kmail
apt-get build-dep kmail

Then I edited kdepim-3.5.7enterprise20070926/kmail/popaccount.cpp
as illustrated as above, removing debug output statements:

 964   else {
 965     if( stage == List ){
 966       stage = Idle;
 967       if (job) job->kill();
 968       job = 0;
 969       mSlave = 0;
 970       KMessageBox::error(0, i18n( "Unable to complete LIST operation." ),
 971                             i18n("Invalid Response From Server"));
 972       return;
 973     }
 974   }
 975 }

cd kdepim-3.5.7enterprise20070926
debuild -us -uc
cd ..
sudo dpkg --install kdepim-3.5.7enterprise20070926.deb

Comment 12 Michael Mikowski 2007-12-26 03:03:08 UTC
make the last line in comment 11:

sudo dpkg --install kmail_3.5.7enterprise20070926-0ubuntu2_i386.deb

Comment 13 Michael Mikowski 2007-12-26 05:06:44 UTC
Added for local fix:

sudo vi /etc/apt/preferences

And add the lines:

Package: kmail
Pin: origin ""
Pin-Priority: 999

This prevents kmail from being overridden by regular updates in debian / ubuntu / kubuntu systems
Comment 14 Christian Tessarek 2007-12-29 10:21:12 UTC
thanks john for the patch; at least, it works (finally) again.
is there any chance for that patch to make it into the next kmail release, say 1.9.8? i really like kontact, but thats more than just a slight annoyance...
Comment 15 Patrick McManus 2008-01-16 22:50:22 UTC
Created attachment 23087 [details]
patch for comment 15
Comment 16 Patrick McManus 2008-01-16 22:51:18 UTC
The issue here is error handling when receiving a blank uid prortion of a UIDL response.. for instance:

1 1200413730.1[CRLF]
2 [CRLF]
3 1200414111.3[CRLF]
. [CRLF] 

will trigger it. John's debug data from the original bug description shows this nicely.

it seems some pop server will generate a bogus uidl response like the above based on the email containing an invalid message-id header. 

I only tested with Dovecot, and could not reproduce that behavior, but forcing the protocol trace with netcat using the above sequence did create a reproduction.

If any of the reporters could identify the pop server that is doing that, we could potentially file a bug against the server which is really the root cause here too.

In any event, without a valid UIDL identifier the message cannot be persisted on the server - otherwise, everytime a LIST operation occurred there would be no way to know whether or not there was a local copy of this message and we would keep downloading it over and over again.

The attached patch 

 * separates error handling for LIST and UIDL operations (removing the confusing error dialog here that casts doubt on LIST un-necessarily)
 * takes UIDL lines with an ID but without a valid UID portion and forces a download of those messages and a delete from the server
 * logs a debug message when it is doing this

Comment 17 Patrick McManus 2008-01-22 17:39:00 UTC
I can confirm a server that announces itself as

IMPLEMENTATION Openwave Email vM.7.08.02.00 201-2186-121-20061213

exhibits this bug.. and the patch from comment 15 works around it.

The website seems to indicate this is legacy closed source, so not much chance it will get repaired on the server end.
Comment 18 Christian Tessarek 2008-01-22 17:54:18 UTC
Whether it is a server side error or not, kmail should be able to fetch emails from the worst non standard compliant servers anyways. It cant be that a malformated header forces a mail client to manually delete the message (e.g. via webmail) and restart the client...

just my humble opinion...
Comment 19 David 2008-01-22 22:16:59 UTC
I had not had this error for awhile until couple days ago and again this morning. Went to the server to check each email address and both times it was one or two spam emails. That were left on the servers. 

Like this morning, Kmail downloaded several emails and then gave me the LIST error. I found the same two spam in two email addresses, no ham present they had been downloaded via Kmail. Deleted both emails in each address and checked server with Kmail again and didn't get any errors. 

If/when this happens again I could get the email source of each one before deleting to see which one it is. Going to try that next time, will post it if anyone thinks it might help.
Comment 20 Patrick McManus 2008-01-22 23:54:14 UTC
Christian, re comment 18: I agree it should be worked around in the client. That's what the patch in comment 15 does. In a perfect open source world, it would be fixed in the server too - that's why I noted the server.

David, re comment 19: the patch in comment #15 should fix that.
Comment 21 Thomas McGuire 2008-01-27 01:06:29 UTC
SVN commit 766946 by tmcguire:

Don't fail when the server sends an invalid UIDL entry.
Instead, generate a fake UID for and delete the mail from the
server (after downloading it).

Based on patch by Patrick McManus <mcmanus at ducksong dot com>.

BUG: 127696



 M  +71 -31    popaccount.cpp  
 M  +6 -0      popaccount.h  


WebSVN link: http://websvn.kde.org/?view=rev&revision=766946
Comment 22 Thomas McGuire 2008-01-27 01:10:11 UTC
Thanks all for tracking this down and for writing a patch.
I've now committed this to trunk, and given that it works OK there, I'll backport it to the 3.5 branch. Please remind me if I haven't done so in two weeks.

Sorry for the delay, but the few KMail developers that are there really have a lack of time.
Comment 23 Hank Ivy 2008-01-28 01:19:38 UTC
I was able to temporarily work around this problem by accessing my POP3 Server using a different email client, moving an offending message from the inbox folder to another folder created just to hold the message.  Then Kmail was able to download all of the other messages.  The specific message was not a spam message.  I will try to report more later.

My email is hankivy@hot.rr.com.
Comment 24 Hank Ivy 2008-01-29 18:25:27 UTC
I can log on to my POP3 Server using telnet, and see all of the operations that are normally hidden.  I also have an email that can cause this problem to occur when ever desired.  Does any experienced POP3 or mail client developer have any suggestions for me to try to duplicate the problem, and collect diagnosic information?
Comment 25 Thomas McGuire 2008-01-30 17:29:15 UTC
> Does any experienced POP3 or mail client developer have any suggestions for me to try to duplicate the problem, and collect diagnosic information? 
There is no need to duplicate the problem, we know what the problem is and already fixed it (in trunk, but I'll backport it soon).
 
Comment 26 Kevin Kofler 2008-02-13 02:51:08 UTC
2 weeks have passed, how's the backporting going?
Comment 27 Thomas McGuire 2008-02-14 18:31:37 UTC
>2 weeks have passed, how's the backporting going? 

Thanks for reminding my. I thought I had backported a week ago, but apparently the commit failed for some reason.

I've now backported it to the enterprise branch in rev. 775015.
It is to late for 3.5.9 though, which is already tagged. Sorry :(

The diff should apply to the 3.5 branch without problems if you want to manually include it in the release.
Comment 28 Kevin Kofler 2008-02-15 08:23:00 UTC
Thanks, I'll apply this to the Fedora 3.5.9 packages. Shouldn't this be applied to the regular 3.5 branch too?
Comment 29 Thomas McGuire 2008-02-15 16:00:30 UTC
> Thanks, I'll apply this to the Fedora 3.5.9 packages. Shouldn't this be
> applied to the regular 3.5 branch too?

Yes, it should. I'm hoping someone who has svnmerge+both branches checked out 
will do it for me, otherwise I'll do it myself.
Comment 30 Till Adam 2008-02-15 16:17:09 UTC
Enterprise changes go into 3.5 automatically, roughly once a week.
Comment 31 sac62 2008-08-24 18:31:18 UTC
Interesting.  I was trying to figure out why this problem seemed intermittent.
I have KMail set not to DL messages from the server so if I make a mistake and delete a message I wanted to keep I can retrieve it.
I now realize that my behavior of going directly to my mail server and going through my mail deleting unwanted crap it would solve the problem and KMail would work again.

I do not have enough knowledge and understanding of how to apply the patch provided here.  I have for many years wanted to use linux but could never understand it enough to make it work. With the development of PC-BSD though I am now allmost a full time user of that OS.  I love it and all that comes with it.  I must admit I still have to run Windows from time to time when needing print and scan because I still can't get those applications working.  

May I suggest that the patch be developed for PC-BSD as a PBI for us newbies?

Thanks
Steve