Bug 400770 - Erratic Kmail filtering (startup problem?)
Summary: Erratic Kmail filtering (startup problem?)
Status: REPORTED
Alias: None
Product: kmail2
Classification: Applications
Component: sieve (show other bugs)
Version: unspecified
Platform: openSUSE Linux
: NOR normal
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-11-07 01:43 UTC by David Lochrin
Modified: 2019-05-16 02:27 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
Kmail log & filter listings (4.93 KB, text/plain)
2018-11-07 01:43 UTC, David Lochrin
Details

Note You need to log in before you can comment on or make changes to this bug.
Description David Lochrin 2018-11-07 01:43:03 UTC
Created attachment 116136 [details]
Kmail log & filter listings

SUMMARY

Kmail filters sometimes appear to be either not actioned at all or only partly so.  I think I've discovered this occurs if Kmail2 is started during the normal power-up process but not if it's started manually when the system is running, however the problem doesn't always appear.

I've attached a file containing filter-log output which demonstrates the problem and a filter-listing.

This may be related to the following bug reports: 378231, 384943, 118213.

I've also noticed another issue which isn't important but may be related.  When Kmail is started automatically during power-up, the download progress-bar is frequently displayed immediately at 0% even though no download should be happening (Retrieval Options > Check mail on startup is off), and checking mail doesn't appear to work.  Restarting Kmail then clears the problem.

The software versions are Kmail2 V5.5.2 running on OpenSuSE Leap 42.3, with all updates applied.

Regards,
David Lochrin

STEPS TO REPRODUCE
1.  See above, but problem is intermittent.
2. 
3. 

OBSERVED RESULT
Filters are not actioned, or not completely actioned.

EXPECTED RESULT
Filters should always be actioned when "Check Mail" is clicked and mail is downloaded. 

SOFTWARE VERSIONS
(available in About System)
KDE Plasma Version: 5.8.7
KDE Frameworks Version: 5.32.0
Qt Version: 5.6.2

ADDITIONAL INFORMATION
See attached file.
Comment 1 David Lochrin 2018-11-10 03:05:15 UTC
I'd like to add another observation which may be relevant.

I created a filter for testing purposes, the first to be executed, which simply copies each incoming email into a scratch folder and continues to the next filter.  A subsequent filter ("Link") tests the Return-Path header then, if true, moves the email to another folder and exits.

But sometimes two copies of the email go to the Link mailbox, one which seems perfectly OK and a second which has no bodypart, is permanently flagged as "unread", and cannot be deleted.  Other effects are unpredictable; I have an instance now where the same two copies also appeared in the scratch folder but the copy with a blank bodypart was successfully deleted.

I've just disabled the first filter (which creates a copy) and I'll see if the problem returns.

Does this sound like some sort of race condition?  I should say that one filter (there are now only three active) pipes the bodypart of flowed-format emails through the following bash script in order to remove the flowed-format line wraps, and I think / assume piped procedures are executed synchronously:
   sed ':L1; /\x20$/{N; s/\x20\x0D/\x20/g; s/\x20\x0A/\x20/g; bL1}'
   exit 0

DL
Comment 2 David Lochrin 2018-11-15 00:37:37 UTC
Please also refer to the discussion in the OpenSuSE Forum at https://forums.opensuse.org/showthread.php/533871-Kmail-filters-and-duplicated-emails?p=2886392#post2886392

There seems a possibility these problems may be associated with Plasma.

DL
Comment 3 David Lochrin 2018-11-17 00:24:54 UTC
I think this problem may also be related to Bug 283682 "KMail duplicates filtered messages" - see https://bugs.kde.org/show_bug.cgi?id=283682

I've confirmed that using successive filters to simply copy each incoming message to different folders does not cause a problem.

The discussion in the OpenSuSE Forum mentioned above seems to support the idea that erratic functioning (or non-functioning) of filtering is associated with Kmail startup.  I'm now starting Kmail manually after the system is fully booted, then waiting until Kmail is running before entering the Kwallet password, and I'll see if the problem recurs.  I've also disabled download-pipelining.

DL
Comment 4 Andy Goss 2018-12-29 00:25:45 UTC
I have a similar problem with filters.

Some filters are not applied on incoming emails. Those that are applied are the ones that redirect emails from their initial inbox (I have several email addresses) to the local inbox or to other folders, based on the To, From, and Subject fields. Other filters are not applied, even though the first group are set to not stop checking if met.

Filters cannot be run manually on a folder unless the Settings > Configure Filters utility is opened, and one filter, it doesn't matter which, is selected, and OK pressed. Then filters can be run manually on a folder, and any subsequent incoming emails during the session will be fully filtered.

The problem does not occur on every startup, just most of them. On rare occasions no filters are applied.

In addition, the initial download on startup does not always happen
Comment 5 David Lochrin 2018-12-29 23:17:20 UTC
Please also see recent discussion which may clarify this problem at https://forums.opensuse.org/showthread.php/534002-Kmail-A-filter-won-t-work-until-re-OK-d

DL
Comment 6 David Lochrin 2019-05-16 02:18:05 UTC
(In reply to Andy Goss from comment #4)

> Filters cannot be run manually on a folder unless the Settings > Configure
> Filters utility is opened, and one filter, it doesn't matter which, is
> selected, and OK pressed. Then filters can be run manually on a folder, and
> any subsequent incoming emails during the session will be fully filtered.
> 
> The problem does not occur on every startup, just most of them. On rare
> occasions no filters are applied.

I can confirm this.  My Kmail-client startup does not download immediately which gives me time to open the "apply filters" menu and clock "OK".  That reliably initialises the filters.  Clearly Kmail startup is buggy.
Comment 7 David Lochrin 2019-05-16 02:27:45 UTC
(In reply to David Lochrin from comment #1)

> But sometimes two copies of the email go to the Link mailbox, one which
> seems perfectly OK and a second which has no bodypart, is permanently
> flagged as "unread", and cannot be deleted.  Other effects are
> unpredictable; I have an instance now where the same two copies also
> appeared in the scratch folder but the copy with a blank bodypart was
> successfully deleted.

This problem still occurs, and it's extremely annoying to the point where I'm considering ditching Kmail altogether.

The obvious problem can be fixed by renaming the Kmail folder, creating a new folder with the original name, moving all emails except the offending one(s) to the new folder, deleting the renamed folder, and finally re-specifying the folder in the relevant filter.

However I have no idea in what state this leaves the Kmail database.  The problem cannot be solved by running the Kmail database utility.

Is there any plan for fixing it?