297198 2012-03-31 17:16:44 +0000 Violation of rfc4155: kmail2 does not use ctime format for first from line when storing in mbox format 2014-03-22 12:16:41 +0000 1 1 2 Applications kmail2 commands and actions 4.8.3 openSUSE Linux RESOLVED FIXED NOR grave --- 1 thomas pim-bugs-null martin montel thomas http://commits.kde.org/kdepimlibs/d1379fda35c9809913d5ab5432461ad5a8fe92d8 4.13 20 oldest_to_newest 1241707 0 70041 thomas 2012-03-31 17:16:44 +0000 Created attachment 70041 zipped file of a single exported message Since 4.7 the spamassassin "sa-learn --spam --mbox" command can not learn spam messages which are stored in mbox format by kmail. sa-learn always reports: Learned tokens from 0 message(s) (0 message(s) examined) sa-learn still works on older mbox files. I noticed some differences between former and current mbox exports. Now there are two empty lines between two messages. Removing the empty lines didn't not solve the problem. This same error in 4.8.0, 4.8.1 1243367 1 montel 2012-04-06 07:59:30 +0000 I confirm it. 1255100 2 71117 thomas 2012-05-15 20:11:28 +0000 Created attachment 71117 Message saved from kmail Version 1.13.6 This message was received and saved from KMail Version 1.13.6 Unter KDE 4.6.00 (4.6.0) "release 6" sa-learn --spam --mbox reports Learned tokens from 1 message(s) (1 message(s) examined) Which is what we expect! 1255103 3 71118 thomas 2012-05-15 20:15:25 +0000 Created attachment 71118 Message saved from kmail Version 4.8.3 This message was received and stored with kmail2 Version 4.8.3 sa-learn --spam --mbox gives Learned tokens from 0 message(s) (0 message(s) examined) Which we do not expect. Both checks sa-learn were run on the same computer with spamassassin 3.3.1! 1255112 4 thomas 2012-05-15 20:47:34 +0000 There is a simple difference in the messages which causes the problem. kmail1 used the following format of the "From" line: From thomas@arend-rhb.de Tue May 15 22:01:41 2012 kmail2 used the format: From thomas@arend-rhb.de Tue, 15 May 2012 22:01:41 +0200 If I change the "From"-line to the older format, sa-learn can learn the messages! kmail1 is not iaw RFC 822; which kmail2 is. Sticking to the standard the problem is with spamassassin. 1255474 5 thomas 2012-05-16 20:06:07 +0000 See bug report to SpamAssassin #6703 (https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6703) See http://tools.ietf.org/html/rfc4155 which states that "A comprehensive description of mbox database files on UNIX-like systems can be found at http://qmail.org./man/man5/mbox.html, which should be treated as mostly authoritative ..." man 5 mbox defines that the date time stamp of the From_ line shopuld be in ctime format. Therefore I propose to switch back to the old date time stamp format of kmail (Version 1.x). 1255489 6 thomas 2012-05-16 21:11:19 +0000 Comment from Mark Martinec 2012-05-16 13:42:01 UTC from https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6703#c6 > When saving messages with kmail 1 the From Line has following format > which is not iaw RFC 822: > From info@ende-18-06.com Fri Jun 17 16:03:07 2011 > With kmail 2 the format is changed to the format which is iaw RFC 822 > From thomas@arend-rhb.de Tue, 15 May 2012 22:01:41 +0200 > which is not parsed correctly by sa-learn. sa-learn --spam reports: [...] Oh, no, not yet another incompatible mbox format!!! > Proposed to change the behavior in a way that the old malformed From lines > and the new correct ones are parsed. It is the other way around, the new one differs from everybody else. The format of the mbox file (along with its separator From_ lines) is *not* governed by RFC 822 or its successors. There is no formal standard for an mbox format, the RFC 4155 comes closest: http://tools.ietf.org/html/rfc4155 See also a Wikipedia article: http://en.wikipedia.org/wiki/Mbox RFC 4155 says: | a timestamp indicating the UTC date and time when the message | was originally received, conformant with the syntax of the | traditional UNIX 'ctime' output sans timezone (note that the | use of UTC precludes the need for a timezone indicator); This matches qmail docs: http://qmail.org/qmail-manual-html/man5/mbox.html and matches Postfix and sendmail's local delivery agent. To accommodate the new incompatible format it seems that the two instances of a regexps in ArchiveIterator.pm need to be extended, or just relaxed. Not sure if the date would still be correctly parsed. Best would be to persuade kmail folks to back off the change! 1279602 7 martin 2012-07-28 22:52:24 +0000 confirmed in comment #1 1297419 8 thomas 2012-09-15 02:19:48 +0000 The bug is still in 4.9.1 1297421 9 thomas 2012-09-15 02:30:47 +0000 I changed the name of the bug so that it is clear, that this is a kmail2 issue. The mbox format kmail2 uses is incompatible to kmai1 mbox format and is a violation of rfc 4155. 1310075 10 thomas 2012-10-27 22:33:29 +0000 kmail 4.9.2 When a message with unkown date - for example spam massages storing the adress in the date field - the from_ line used the local, nationalized date time format: From user@example.com So. Okt 28 00:25:18 2012 SpamAssassin can not detect messages in such an mbox. 1322491 11 thomas 2012-12-09 16:41:25 +0000 The error is now also kmail2 4.9.4. 1379612 12 thomas 2013-06-26 19:46:40 +0000 Nice try to fix this bug, but you did it totally wrong and screwed things more than ever: The From_ line now looks as follows: From thomas@example.com Mi. Jun 26 21:26:17 2013 Why did you change the weekday to German languge? Is the month now also Englisch? 1379614 13 thomas 2013-06-26 19:47:24 +0000 I forgot the Version 4.10.4 1436916 14 martin 2014-03-22 10:53:54 +0000 patch added https://git.reviewboard.kde.org/r/116975/ 1436930 15 martin 2014-03-22 12:16:41 +0000 Git commit d1379fda35c9809913d5ab5432461ad5a8fe92d8 by Martin Koller. Committed on 22/03/2014 at 12:14. Pushed by mkoller into branch 'KDE/4.13'. write "From " delimiter line with correct dateTime format Fix writing dateTime field in the "From " delimiter line according to RFC4155 FIXED-IN: 4.13 REVIEW: 116975 M +6 -2 kmbox/mbox_p.cpp http://commits.kde.org/kdepimlibs/d1379fda35c9809913d5ab5432461ad5a8fe92d8 70041 2012-03-31 17:16:44 +0000 2012-03-31 17:16:44 +0000 zipped file of a single exported message spam.zip application/x-download 1427 thomas UEsDBBQAAAAIAE2Uf0B5ZnXH5QQAAFMIAAAMABwAc3BhbS14MS5tYm94VVQJAAMRMndPKzJ3T3V4 CwABBOgDAAAEZAAAAI1Va3PaOhD9HP+KDZ+gxY4fPJ3SWwIk5BYIhSQ3bafTEbawFYzkSDKE/Pq7 dkiaZnIfDMNDWh2d3T17fCrFGhhfik8xyZQVUpgTXQXPgTGR4NqOC47j12t+zYP3tmvbxozqTHJz SnTsw4eXRz8aN+Y8JWuzF9NgRaV5TaVigvuQr3aVIkoxDp7lWQ6UEds2bc90GhUQHFauRSTloaVX KnwCGtENTXx49+7d08pcE50pHyaiCioQknY8qw2S3mVM0rBTs+qgqdKqc9L9Opj/bNnVWffbxcz9 2RsOep+rxsGsP5n/7H+ddMfnPSCZFgklkne4gM0j207B7/k+/FjvfLCtdrtttkzTjk2npvA7NE21 8rfbrSXpQ/XfAnD/CW74hGbbNdMxzVps2hjrYuyCyqj65s4J7iDAhWQR4yQxL4UPK0ET/ikRAUli obTRpwnDBGj41u7Lys5oQDEy9GGZt/63skP5+Qh8d9ymhWws50fFOFjsXkdOMWjJ7iuwZTqGwXx8 OQUWgtd3Trpuq93oGQdLIeHDKyofj98QmIvq8u29wKDcG8wvK68yOhvfQPi4sgMtYOk6zqdofY+y e51SKlIr3+EUs3Ad13IxEwffjfaPNzIp+E8vph6Ul1QH8ZqwxGzkGmhX/iEHKKOOo4SaoRRp5X9n 9Itn+S6/BeduI1Y0BOREEkYUQr1Esdv54NUdMG27GLzf0mw7puu2TM82Hbthhbpes2QG5eFgdAE5 estru561H80KfG87WIiW5WFH7cYPA/Jb1/f2c6nK+KfR3Pczb+fxGy6AZB5dYExxnCNqnvfRBDbD XuA5wj5zRCw+P5CmcFm8q9XI9qppd08z75dD9Imm/n96zCkm6EOpu5BkrUTMS6+NJpfE03a+G0oS 3ZJgtZfER2OeLW5poAvvKMSTjx+KJyRBrIvFqdCUP6RSLBK6pn8AgNETHNe0eblLkaSm9/ooTQjj x8ZBEBOpqO6UmBJmq1Vvm07pV7wkXC3R8AY8ECFKw4fmgmmcWbzZ7HLNNkzmroVSyI0m/wNLkfGw 8iJGIUMf6lA+wUQQMqBFH4HgzO8eaOXYgD7VuNKp95tfWu2h10hrI9u+vrzpNfq1O3vyvju0p1c9 fjZyN3cXUUpOvo6chmBajWXU10eTL5N03vt22lzoOjfgr0t3OHZXurHaTs9vjpKrdFxbLW7uJ+3G n+tbdzTYZTwOsofzzW2z9qDibqdz7Rwj46vz/sgHLz0vHSZ8UDp0+l7psNHqlA4N4yzDksAlQS/D ihrnHMdWwqQo+0O2Boo/MGBIeaR+atia2BPKDSPWOvWPjvZ+SlO9lJSZG0YiYi6LMX98qbx+NIEF DSlehQiMwmeRLot7BghmYZkk5KYJKsVQDZRxCjORJBRHLo8zxjsdi0REePaR3obcCihfi2RVseA0 w7Xi0Ak+SyiLMh4BU3uggUxItuBMFUj4QFqvqUJpbUmi80Au9BbdhUWPRPbYY2w3No9DwmikQWU0 TFgQI4aCjVijlhIhSShglnscRHTBkpDmpRqRFUW1bmmSAEqmIEISBUpElBOOEpR5zEokOLmGZnRJ +SZLVgQpBjECXEmVSmSmLOQOVMZ0gQQeL890wSKj+PiBkCiYIceF2BrTBCeVZNUiZw7dBWJxpnVB +JnsPkpVc2b7Dv0NUEsBAh4DFAAAAAgATZR/QHlmdcflBAAAUwgAAAwAGAAAAAAAAQAAAKSBAAAA AHNwYW0teDEubWJveFVUBQADETJ3T3V4CwABBOgDAAAEZAAAAFBLBQYAAAAAAQABAFIAAAArBQAA AAA= 71117 2012-05-15 20:11:28 +0000 2012-05-15 20:11:28 +0000 Message saved from kmail Version 1.13.6 k2-0001.mbox application/mbox 1110 thomas RnJvbSB0aG9tYXNAYXJlbmQtcmhiLmRlIFR1ZSBNYXkgMTUgMjI6MDE6NDEgMjAxMgpSZXR1cm4t UGF0aDogPHRob21hc0BhcmVuZC1yaGIuZGU+ClgtU3BhbS1DaGVja2VyLVZlcnNpb246IFNwYW1B c3Nhc3NpbiAzLjMuMSAoMjAxMC0wMy0xNikgb24gazIuYXJlbmQudGtzZApYLVNwYW0tTGV2ZWw6 IApYLVNwYW0tU3RhdHVzOiBObywgc2NvcmU9LTEuMiByZXF1aXJlZD00LjUgdGVzdHM9QVdMLEJB WUVTXzAwLE5PX1JFTEFZUwoJYXV0b2xlYXJuPWhhbSB2ZXJzaW9uPTMuMy4xClgtU3BhbS1TcGFt bXk6IDAuOTU3LTU2NDYtLTE4NTdoLTEzNjE1cy0tMGQtLUhDb250ZW50LVRyYW5zZmVyLUVuY29k aW5nOjdiaXQsCgkwLjkyMy0xODAzLS0xNTMzaC02MDU0cy0tMmQtLUgqYzp1cy1hc2NpaQpYLVNw YW0tSGFtbXk6IDAuMDAwLTE5MjgtLTMyMDY4aC0xcy0tMzhkLS1IKnA6RCphcmVuZC1yaGIuZGUs CgkwLjAwMi0xOTA2LS0zMjA4N2gtMjRzLS0xMmQtLUgqRjpEKmFyZW5kLXJoYi5kZQpYLU9yaWdp bmFsLVRvOiB0aG9tYXMKRGVsaXZlcmVkLVRvOiB0aG9tYXNAYXJlbmQtcmhiLmRlClJlY2VpdmVk OiBieSBrMi5hcmVuZC50a3NkIChQb3N0Zml4LCBmcm9tIHVzZXJpZCAxMDAwKQoJaWQgMUEwNUUy OEE4QjsgVHVlLCAxNSBNYXkgMjAxMiAyMjowMTo0MSArMDIwMCAoQ0VTVCkKRGF0ZTogVHVlLCAx NSBNYXkgMjAxMiAyMjowMTo0MSArMDIwMApUbzogdGhvbWFzQGFyZW5kLXJoYi5kZQpTdWJqZWN0 OiB0ZXN0ClVzZXItQWdlbnQ6IEhlaXJsb29tIG1haWx4IDEyLjIgMDEvMDcvMDcKTUlNRS1WZXJz aW9uOiAxLjAKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOwogIGNoYXJzZXQ9dXMtYXNjaWkKQ29u dGVudC1UcmFuc2Zlci1FbmNvZGluZzogN2JpdApNZXNzYWdlLUlkOiA8MjAxMjA1MTUyMDAxNDIu MUEwNUUyOEE4QkBrMi5hcmVuZC50a3NkPgpGcm9tOiB0aG9tYXNAYXJlbmQtcmhiLmRlIChUaG9t YXMgQXJlbmQpClgtVUlETDogTVJGISFKV2MiIThLaCIhOWskIyEKU3RhdHVzOiBSClgtU3RhdHVz OiBOClgtS01haWwtRW5jcnlwdGlvblN0YXRlOiAgClgtS01haWwtU2lnbmF0dXJlU3RhdGU6ICAK WC1LTWFpbC1NRE4tU2VudDogIAoKdGVzdAoK 71118 2012-05-15 20:15:25 +0000 2012-05-15 20:15:25 +0000 Message saved from kmail Version 4.8.3 x1-0001.mbox application/mbox 1032 thomas RnJvbSB0aG9tYXNAYXJlbmQtcmhiLmRlIFR1ZSwgMTUgTWF5IDIwMTIgMjI6MDE6NDEgKzAyMDAK UmV0dXJuLVBhdGg6IDx0aG9tYXNAYXJlbmQtcmhiLmRlPgpYLVNwYW0tQ2hlY2tlci1WZXJzaW9u OiBTcGFtQXNzYXNzaW4gMy4zLjEgKDIwMTAtMDMtMTYpIG9uIGsyLmFyZW5kLnRrc2QKWC1TcGFt LUxldmVsOiAKWC1TcGFtLVN0YXR1czogTm8sIHNjb3JlPS0xLjIgcmVxdWlyZWQ9NC41IHRlc3Rz PUFXTCxCQVlFU18wMCxOT19SRUxBWVMKCWF1dG9sZWFybj1oYW0gdmVyc2lvbj0zLjMuMQpYLVNw YW0tU3BhbW15OiAwLjk1Ny01NjQ2LS0xODU3aC0xMzYxNXMtLTBkLS1IQ29udGVudC1UcmFuc2Zl ci1FbmNvZGluZzo3Yml0LAoJMC45MjMtMTgwMy0tMTUzM2gtNjA1NHMtLTJkLS1IKmM6dXMtYXNj aWkKWC1TcGFtLUhhbW15OiAwLjAwMC0xOTI4LS0zMjA2OGgtMXMtLTM4ZC0tSCpwOkQqYXJlbmQt cmhiLmRlLAoJMC4wMDItMTkwNi0tMzIwODdoLTI0cy0tMTJkLS1IKkY6RCphcmVuZC1yaGIuZGUK WC1PcmlnaW5hbC1UbzogdGhvbWFzCkRlbGl2ZXJlZC1UbzogdGhvbWFzQGFyZW5kLXJoYi5kZQpS ZWNlaXZlZDogYnkgazIuYXJlbmQudGtzZCAoUG9zdGZpeCwgZnJvbSB1c2VyaWQgMTAwMCkKCWlk IDFBMDVFMjhBOEI7IFR1ZSwgMTUgTWF5IDIwMTIgMjI6MDE6NDEgKzAyMDAgKENFU1QpCkRhdGU6 IFR1ZSwgMTUgTWF5IDIwMTIgMjI6MDE6NDEgKzAyMDAKVG86IHRob21hc0BhcmVuZC1yaGIuZGUK U3ViamVjdDogdGVzdApVc2VyLUFnZW50OiBIZWlybG9vbSBtYWlseCAxMi4yIDAxLzA3LzA3Ck1J TUUtVmVyc2lvbjogMS4wCkNvbnRlbnQtVHlwZTogdGV4dC9wbGFpbjsgY2hhcnNldD11cy1hc2Np aQpDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiA3Yml0Ck1lc3NhZ2UtSWQ6IDwyMDEyMDUxNTIw MDE0Mi4xQTA1RTI4QThCQGsyLmFyZW5kLnRrc2Q+CkZyb206IHRob21hc0BhcmVuZC1yaGIuZGUg KFRob21hcyBBcmVuZCkKWC1VSURMOiBNUkYhIUpXYyIhOEtoIiE5ayQjIQpTdGF0dXM6IFJPCgp0 ZXN0CgoK