Bug 50997

bug 40647 and bug 51377 are related to this. If you implement this, please
implement moving on the server too and not only when downloading on the machine.
Downloading means you do not use the advantages of IMAP over POP3. I would like
to have my mail on the mailserver, so I can check it at home and at work.

I agree with the original bug reporter. Evolution handles the issue pretty 
well. Can we get Evoltion's code and port it to Kmail? 
 
Thanks in avance. 

This feature should be implemented before Kolab is released. I do not know any
company that does not use IMAP for their internal mail. If you have to tell
them, that they cannot filter their mail *into IMAP folders*, KMail is not an
option for them.

*** Bug 54040 has been marked as a duplicate of this bug. ***

At my company we don't have any quota issues, but like above, we don't support
server-side filtering.  Due to this, we use either Netscrape 4.61 or Mozilla for
mail (we're using Solaris), both of which do support filtering imap traffic and
placing it into imap folders.  When the client-side imap filtering is
implemented it should also be possible to specify the target folder as imap.

It would also be nice if kmail could import and export Mozilla filter rules.

This is an important feature that i miss too:
To filter incomming mails on an imap account!
I connect to an exchange 2000 server via imap. And i have made a filter to pipe all mails trough /usr/bin/korganizer. If i get any invitation to a meeting with a meeting.ics attachment, this script will handle the invitation to korganizer. In korganizer i can choose to accept or decline the invitation, and my calender is automaticle updated. Everything works fine, exept that kmail does not filter my incoming e-mails! 
Workaround: I have to mark my emails, and chooce to filter them.

If KDE want to have support for Exhange 2000, i think it is important to fix this feature.

But filtering of an imap folder did work! 
I used it with my SuSE 8.0 and 8.1 (kde 3.0.1 I think). 

To re-iterate, evolution does this quite well.  With all the mailing lists I'm
on, it's great to have the email filtered before I try to read it, and the imap
ability allows me to read it anywhere.  If there's a user-friendly way to do
server side filtering, I have not found it yet.

I have read a lot of arguments w.r.t. to not allowing filtering of *online* IMAP folders. 
However, there are still a few things to be said: 
 
1. It should be possible to specify an IMAP folder as the target of a move operation 
in a filter (I might retrieve mails from different  POP accounts, filter them locally and 
then upload them to an IMAP account). 
 
2. Filtering should be allowed for cachedimap. In this case all mails already reside 
on the local system and can thus be easily searched. Only the moving needs to be 
performed on the server side too but as I will otherwise filter them by hand that's no 
additional traffic. 
 
Cheers, 
Volker 

I hereby strongly second above wishes.

My Mail Server now finally has a working SPAM Assasin installation which is
currently the best working SPAM Filter I have seen in a production enviroment.

It is very easy to filter out SPAM now using the modified headers but this has
to be done automatically since manual filtering would trigger all my other
filter rules.

I know there is Sieve which unfortenatly is not an option, since it is not
supported by most ISPs besides KMail stable does not have the Sieve either...


I know too that running automatic filters on an IMAP server is not really the
smartest idea however I don't see a killer argument not to do it.

Maybe adding one additional header to IMAP filtered mails would help to avoid
nasty loops.


Cbeers,
Thorsten

*** Bug 59551 has been marked as a duplicate of this bug. ***

*** Bug 59685 has been marked as a duplicate of this bug. ***

*** Bug 62745 has been marked as a duplicate of this bug. ***

*** Bug 30825 has been marked as a duplicate of this bug. ***

This is a really important feature! 
My clients (who I help to migrate from Win to Linux/KDE) constantly complain 
that this feature is missing.. 

Isn't it possible to vote with money? :) People who make money with KDE 
could vote with hard cash for certain features or bugs, and if these bugs 
are fixed or features are implemented the money will flow into the KDE 
project. 

I'm pretty sure that this has already been implemented in KDE's CVS and will 
be included in the upcoming KDE 3.2 release. 

Subject: Re:  IMAP Filters and the ability to download email

On Thursday 25 September 2003 02:54, Kasper.Souren@ircam.fr wrote:
> Isn't it possible to vote with money? :) People who
> make money with KDE could vote with hard cash for certain features
> or bugs, and if these bugs are fixed or features are implemented
> the money will flow into the KDE project.

I think it's a nice idea, and a natural extension of the bug tracking 
system, I really do. But I don't think it can work for KDE as a whole 
because KDE has no decision making process other than near unanimous 
agreement. And unfortunately from my experience I believe there is 
very little chance that all KDE contributors are going to unanimously 
agree on how to distribute funds. 

Don.

I don't think distributing the money will cause a lot of problems. The best 
solution to me seems to let the money be a donation to the entire KDE 
project. I don't guess a lot of people will object to this. 
 
But for bigger features it could be done different. For a company it could 
be nice to have an alternative to renting a programmer who implements it. 

Subject: Re:  IMAP Filters and the ability to download email

On Friday 26 September 2003 02:29, Kasper.Souren@ircam.fr wrote:
> I don't think distributing the money will cause a lot
> of problems. The best solution to me seems to let the money be a
> donation to the entire KDE project. 

I'm speaking as someone who has advocated this idea, your idea of 
voting with money on the core development list, been rejected and 
spent several years thinking about, and working towards it.

There are two main problems involved. The first problem is there is no 
one person, no one decision making entity who can accept your idea 
and say ok let's do it. If something is to be done on behalf of all 
KDE then all KDE should agree to it, that's the cost of default right 
of veto.

The second problem is that KDE doesn't have a bank account, it has no 
way to accept donations. KDE e.V. is the closest thing to that.

> I don't guess a lot of people will object to this.

I'm not guessing, I've proposed the idea and people have objected, 
some KDE contributors don't want anything to do with a vote with 
money scheme. Some say no significant amount of money will be raised 
and they don't want to be involved with an idea that they believe is 
bound to fail. Others argue about how the money should be 
distributed, what is equitable, what is moral, what is just? These 
are ancient philosphical questions that human societies have been 
trying to answer since before recorded history.

The KDE way is to preserve harmony at all costs, and that means 
avoiding discussing controversial subjects such as politics, religion 
and money.

For instance this conversation is off topic and should not be taking 
place on this list. (I would know I'm the list admin). By discussing 
it here we are violating unwritten KDE policy. To violate the mos 
maiorum is to threaten KDE civilization itself. By discussing this 
issue with you instead of ignoring you I'm violating unwritten KDE 
policy.

> But for bigger features it could be done different. For a company
> it could be nice to have an alternative to renting a programmer who
> implements it.

I agree it would be nice. But how many people are making money from 
using KDE, how much revenue would such a company receive? These are 
questions no one seems to know the answers to, but today without 
official KDE support I guess such a company is not financially 
viable.

Again I do think the idea has merit. I'm actually working on something 
similar, but just on a per KDE component basis. This way the number 
of participants that require involvement is kept to a manageable 
level. If others don't like the idea they can ignore it, if the idea 
is successful and other application maintainers like they idea then 
they can adopt it.

Don.

Server side scripting applies before kmail ever accesses the mailbox.  Being able to use filters on any mailbox accessable from kmail should be a no-brainer.  This ability is available in any decent gui mail client.  What would be really nice is to be able to store the actual filters themselves in an imap folder, so that your filters can be used from kmail at different locations.  Without this ability, kmail seems half complete.

Please add the IMAP folder filtering feature into Kmail. It's really half useful without it. It compells me to use sylpheed.

>I'm pretty sure that this has already been implemented in KDE's CVS
>and will be included in the upcoming KDE 3.2 release. 

I've installed Mandrake version 10.0 RC1 with KDE 3.2 (though I'm not sure if it's a final version of 3.2 or some beta), and this feature is still not implemented. :(

I recently subscribed to a mailing list and I want to have filters which move mail coming from the list to a special imap folder. I like KMail but now I have to use Thunderbird... 

I will second Jakub's comments.  I'm also running Mandrake 10 RC1 with KDE 3.2 and this feature is not implemented.

Assigned? Hip Hip horray!

ETA of implementation? This is the only reason I can't move to KMail

What might be nice is adding a 'trigger' feature to a search virtual folder.  I have a few search virtual folders defined, which works ok for me because I've configured them to scan only headers, but what would be even better would be a behavior to run the search each time a mailbox is polled and have a behavior run on the results of the search automatically.

confirmed still present in Kmail 1.6.4 on KDE 3.2.3 ... i really NEED Kmail as it's the only mail client in the freakin whole wide world that properly renders complex devanagari script ... but our people get thousands of emails and we can't do without filters.

PLEASE PLEASE add this feature

yeah what's the big deal with implementing this. to filter imap mail i must download all mails to a local folder, select all the mails then press ctrl-j. is it so hard to automate this ?

Hi,


no, this is not good ! the folder is on the server with backup's. 


regards Marco



Am Mo, den 12.07.2004 schrieb h um 15:05:
> ------- You are receiving this mail because: -------
> You are a voter for the bug, or are watching someone who is.
>       
> http://bugs.kde.org/show_bug.cgi?id=50997      
> 
> 
> 
> 
> ------- Additional Comments From h llorien org  2004-07-12 15:05 -------
> yeah what's the big deal with implementing this. to filter imap mail i must download all mails to a local folder, select all the mails then press ctrl-j. is it so hard to automate this ?
> 

whatever, we need the filters to process remote imap boxes however we want.


On Monday 12 July 2004 15:30, Marco Löwl wrote:
> no, this is not good ! the folder is on the server with backup's.
>
>
> regards Marco
>
> Am Mo, den 12.07.2004 schrieb h um 15:05:
> > ------- You are receiving this mail because: -------
> > You are a voter for the bug, or are watching someone who is.
> >
> > http://bugs.kde.org/show_bug.cgi?id=50997
> >
> >
> >
> >
> > ------- Additional Comments From h llorien org  2004-07-12 15:05 -------
> > yeah what's the big deal with implementing this. to filter imap mail i
> > must download all mails to a local folder, select all the mails then
> > press ctrl-j. is it so hard to automate this ?
>
> !DSPAM:40f29276290541810121039!

As it is, I use a search folder to search headers, then just select-all and delete.

If we can search headers and turn them into search folders, then why not apply that to the filters process?  You would be saving me 3-5 keystrokes.

Let's put it this way: Mozilla/Tbird does it.  LookOut does it.  Why doesn't KMail?  At this point, the only thing keeping me in KMail is its KDE/Qt theme integration, and the jokey way Tbird/Ffox share mozilla prefs.  They're curing the prefs nonsense and GTK-Qt is pretty close enough...

(and the integrated bayes filter spam catcher?  That's a feature request ;)

Also, check the Most Requested list:
http://bugs.kde.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_severity=wishlist&votes=21&order=bugs.votes

There appears to be a disconnect between what developers feel is the Right Way To Do Things and what users want to Get Their Jobs Done.  If your goal is to do things the Right Way and damn the users, don't be surprised if your userbase is small and shrinking, and if you don't care, fine.  If you want KMail to be more useful and gain more users (and get more feedback, etc), then you should hit at least the top 5 feature requests.

Guess with feature request is #1?

Oh, and one last thing, lots of people are not using Cyrus IMAP or other IMAPs that support Sieve filtering.  When you say this bug isn't important, you're basically telling us to go f??? ourselves.

Also, even if we had Sieve filtering, where's the Sieve interface?

Is Sieve remote filter rule configuration SSL encrypted?

I'm a voter for this bug, but if I were a developer reading the last few comments, I probably would tell you to go f??? yourself... I don't see how being rude to someone is more likely to get them to do what you need...

It would, in fact, be very nice to have IMAP filtering. I haven't a clue how hard it is. I'm not a developer. If I were, maybe I'd write it myself.

Cheers.

Whats interesting is that I'm a voter on this  as well, but as a work around I 
switched to using the disconnected imap which seems to work quite well with 
filtering.  I don't really have a need for directly connected imap anymore.

Jason

On Monday 12 July 2004 10:40 am, Daniel Berdine wrote:
------- You are receiving this mail because: -------
You are a voter for the bug, or are watching someone who is.

http://bugs.kde.org/show_bug.cgi?id=50997




------- Additional Comments From berdine pas rochester edu  2004-07-12 17:40
 ------- I'm a voter for this bug, but if I were a developer reading the last
 few comments, I probably would tell you to go f??? yourself... I don't see
 how being rude to someone is more likely to get them to do what you need...

It would, in fact, be very nice to have IMAP filtering. I haven't a clue how
 hard it is. I'm not a developer. If I were, maybe I'd write it myself.

Cheers.

The only sieve interface I have ever found is a cgi called websieve.cgi.  I seriously doubt most implementations of cyrus imap servers allow direct access to sieve filtering, most of them probably turn it off.  It would be oh so wonderful to have it in kmail, then it wouldn't matter what imap server was being used.  I personally have even offered to pitch in money to have this feature developed...

--- I don't see how being rude to someone is more likely to get them to do what you need...  ---

Waiting patiently and asking politely haven't helped..  Check the create date on this bug..

If search (including search folders) can do IMAP header searches and I can ctrl-a and manually move the results around from the search display, how much effort is it to apply that to filters?  The capability is there, it's just nobody seems to think it's important, or worse, that it's not The Right Thing To Do.

I want to use a fully KDE/Qt solution because of Baghira, all the chrome, DCOP (lineakd' to my net keyboard), kwallet, etc.  I really don't want to use Ffox/Tbird: they're largely _ugly_ in KDE, and the projects had issues with sharing preferences and pref access collision.  However, I've already emerged the latest thunderbird-bin because I really see no progress on this front and it's the kind of feature that leads me to not recommend KMail to companies I work/consult for.  It plays nicely with Ffox now, and looks pretty good with GTK-Qt.

The point is, assuming everyone is working on bugfixes which are more important of course, when there's spare cycles, one would assume the priority of new features would be dictated by the number of votes in the user community.  Consult that list.  Tell me which one is number 1, and has been so for how many months.

Do I need to buy a KDE distro just so I can put a bug against THEM and maybe they'll code this?  What's the CVS_ROOT for just KMail?  Is there a project file for KDevelop for KMail?  I'm no coder but I used to be years and years ago...

I have implemented courier imap servers, and so would also like maildrop filters (as well as sieve (cyrus) flexibility is important in opensource).  Maybe it is better to use the search/filtering of the client to move the files, just like pop.....even if it's just an option?  That way you can deal with any imap server, and always get the right result.  

The only problem I see is when users read their mail remotely, say through the company webmail program, their mail will not be filtered.....but maybe this is not a problem.

I have implemented squirrel mail where I work, and they have a number of plugins to create filters for both cyrus and courier imap - the code is their guys, and opensource! :)  The biggest problem they face with a webmail is that the filter file has to have "-rw------- username username" (with courier/maildrop anyways) permissions set, when the webserver is running as someone else, so ofcourse it is quite difficult - but when a clientside program is writing the rule file, that is easy!

So here's the sieve rule page: http://www.cyrusoft.com/sieve/
Here's a squirrelmail sieve plugin: http://www.squirrelmail.org/plugin_view.php?id=73
And a procmail/maildrop filter for most other imap servers: http://www.squirrelmail.org/plugin_view.php?id=210

now the one thing I have not seen in any of there implementations - and would really blow the competition away - is something that, once a rule has been written, it checks it work with a test email. 
 So may times I have set up these rules, and something screws up the file (most people use cut&paste to get the addresses or whatever they want to filter on and sometimes have odd characters like new line chars that screw the filter)....but anyways, I have to get back to a room full of bloody outlook - please help :)

On Tuesday 13 July 2004 03:26, Mike Green wrote:

> ------- Additional Comments From kdebugs linuxwiz org  2004-07-12
> The only sieve interface I have ever found is a cgi 
> called websieve.cgi.  I seriously doubt most implementations of
> cyrus imap servers allow direct access to sieve filtering, most of
> them probably turn it off.  It would be oh so wonderful to have it
> in kmail, then it wouldn't matter what imap server was being used. 
> I personally have even offered to pitch in money to have this
> feature developed...

Mike thanks very much for your help in funding this feature. As a 
contributor to the improvement system you're on my high priority 
support list so I've provided a status report below for your 
convenience.

KMail is part of the KDEPIM package, this package entered 'feature 
freeze' on May 24th. While KMail remains in this feature freeze state 
no new features such as online IMAP filters (50997) should be 
committed to KDE CVS.

KMail will remain in feature freeze until KDEPIM 3.3 is released. 
KDEPIM is scheduled to be released at the same time as KDE 3.3 which
currently is Wednesday August 18th, 2004. After KDEPIM 3.3 is released 
the feature freeze will be over and new features like online IMAP 
filters (50997) can be worked on in KDE CVS again.

( The KDE 3.3 release schedule is here:
http://developer.kde.org/development-versions/kde-3.3-release-plan.html
the KDEPIM 3.3 release schedule is here:
http://developer.kde.org/development-versions/kdepim-3.3-release-plan.html )


Regarding the status of online imap filtering (50997). I have a 
prototype patch finished, and much of that has been committed to KDE 
CVS already. But I ran out of time and could not finish the patch 
before the May 24th KMail feature freeze was reached.

The problem is part of the goal of the patch is to support filtering 
between IMAP folders (bug 59685). Previously KMail only supported 
filtering to local folders only. Supporting IMAP folders as targets 
for the move filter action will require making the filtering system 
asynchronous at a lower level. This has made it necessary to rewrite 
significant parts of the filtering system, which is a time consuming 
task.

But much of this work has already been completed and it's possible 
I'll have a fully operational patch for online IMAP filtering not too 
long after KDEPIM 3.3. is released. (I've estimated 6 weeks is 
required to complete this feature).

Unfortunately this means online IMAP filtering won't be available as 
part of a KDEPIM release until the next release after KDEPIM 3.3, I 
expect this won't be made until some time in 2005.

But as an improvement system customer if you provide me with the name 
and version of the distribution you are using (as normally found in 
the /etc/issue file) then I will try to make a binary (and/or source) 
package available for you. My ability to provide such a package will 
be limited by my need to have a copy of the distribution you are 
using myself.

That's an open offer, if anyone wants to avoid waiting for online IMAP 
filtering and get a binary rpm for their system as soon as the online 
IMAP is ready for distribution please email me.

For the impatient another alternative would be to try a disconnected 
imap account (but please note this is marked as experimental).

Don.
http://kontact.org/shopping/

On Tuesday 13 July 2004 12:04 am, Don Sanders spammed:

> Mike thanks very much for your help in funding this feature. As a
> contributor to the improvement system you're on my high priority
> support list so I've provided a status report below for your
> convenience.

Thanks for the update Don, and for the damn good work!

Hi,

Does anyone object to updating the filter dialog to support multiple 
selection support? Specifically I mean changing kmfilterdlg.*, so 
that the selection mode of the mListBox is set to Extended. And 
making sure the rest of the code is updated to work after that 
change.

I would like to do make this change because it could improve the 
quality of the filter dialog, and because it seems to be a nice basis 
for the GUI work on client side IMAP filtering improvement I'm 
currently working on.

This change could improve the quality of the filter dialog for two 
main reasons. Firstly it would allow multiple filters to be 
moved/deleted simultaneously. (I recall requests for this but don't 
remember a bug number). Secondly while the Filter Criteria and Filter 
Actions groupboxes would have to be disabled when multiple filters 
are selected all of the controls in the advanced group box could 
still be used. This last point requires further explanation.

I'm not sure there's an equivalent in KDE, but on OS X I quite like 
the font dialog in the Apple TextEdit application. This dialog uses 
tristate checkboxes, so for instance if I select a paragraph 
containing bold and normal text, the bold checkbox will be shown in 
the third blank/unknown/ambiguous state. Then the boldness of the 
entire selected paragraph can be changed by clicking on the bold 
checkbox.

I think this is actually quite an elegant/nice/simple UI idiom and can 
be applied to all of the advanced options in the filter dialog. That 
is all the apply this filter, stop processing here, and add filter to 
the menu checkboxes could be applied to multiple (selected) filters. 
Perhaps this is also true of the 'Icon for this filter' button.

Finally and most importantly, for client side IMAP filtering I would 
also like to add a 'for this account' combobox in the advanced 
options section immediately below the 'apply this filter' checkboxes. 
This is so that the apply this filter checkboxes (incoming, outgoing, 
manual) can be specified on a per account basis, (and so that I don't 
need to force a change of behavior for existing users when adding 
client side IMAP filtering).

Now having multiple selection support in the filter dialog will be 
useful as it will allow users who want to enable client side IMAP 
filtering to select their IMAP account in the proposed new 'for this 
account' combobox then select all their filters and with a single 
checkbox toggle all their filters on (or off) for their selected IMAP 
account (for incoming/outgoing/manual filtering).

I hope that makes sense. I discussed it with Till on IRC and he is 
supportive of the idea. I can attach a screenshot if it would help. 
Previously I've added/improved multiple selection support to the list 
of headers widget and the search dialog, I think that went ok.

If there are no objections (nice) then I'll work on a patch to send to 
the list.

Don.

An example for this 'tristate' is the 'bold'-button in OpenOffice. If you select text that is both bold and normal, the bold-button becomes 'grayed'. If you click on the button, the whole selected text is made bold, if you click again it is made normal. KWord 1.3.2 allows the same functionality, but does not give visual feedback.

Another word on Sieve server side filtering. We do have this now with out SUSE Linux Open Exchange (SLOX) server. The problem is, that my mailreader (Mozilla) does not know if messages have been sorted into subfolders. Therefore the subfolders are displayed as 'containing 0 new messaged'. As I have 20+ folders, messages can get overlooked. Selecting the folder displays the number of new messages, but takes some time, so I guess this is the reason why not every folder is checked by default.

On Friday 27 August 2004 03:28, Ingo Kl

On Tuesday 31 August 2004 07:28, Ingo Kl

My 2c worth...

>2) Operating on multiple filters simultaneously.
>
>I see how this advanced tab design gives an 'overview about the status 
>of all accounts' for a particular filter. But there is still the 
>problem of existing IMAP users who (currently manually apply filters 
>but) want to use filters on incoming mail. It would be preferable if 
>they didn't have to iterate through each filter toggling on the 
>'Apply for all accounts' checkbox.
>
Don, I like this whole proposal, and I really don't think that the above 
should be of real concern.  Some with so many filters that scrolling 
through each one and having to click apply, should have the technical 
know how to make this happen while understanding why.

Most users will not fit into the above category, and as long as you 
cater for them, and you have, then viola, we're off.

What would happen if they removed the IMAP account and added it again, 
would that not then take the defaults into account as suggested by Ingo 
removing the need to click apply on each filter.  The user would then 
only need to remove any filter they didn't want applied.

Thanks for you effort and enthusiasm in working on this feature.

I've implemented the visual filter dialog changes discussed with Ingo, 
and created two tabs, a general tab 
( http://invertedlogic.com/generaltab.png ) and an advanced tab 
( http://invertedlogic.com/advancedtab2.png ) 

The next step is to change KMFilter to allow filters to be enabled for 
selected accounts only, check/ensure that the asynchronous filtering 
system (action scheduler) that was designed last year is still 
working, and then migrate at least the imap account type to use the 
action scheduler.

Don.

Created attachment 7866 [details]
client side imap filtering gui and KMFilter changes

Implementation of the discussed changes to the filter dialog to support client
side imap filtering and also an update to KMFilter to support optionally
applying filters to a set of specified accounts.

The client side IMAP filtering GUI is operational now. I have provided a diff against the 3.3 release. This is just the GUI (and some changes to the basic filter class) there's still quite some work on the non GUI code required.

I'll move on to checking that the action scheduler is still working ok now, and if so modify it to respect optionally applying filters to a set of specified accounts. 

I spent some time looking at exactly where to add the code for scheduling new IMAP messages for filtering. I think I will need to add some additional code for recording/determining which messages have already been filtered also, this will take a bit of extra time.

There is also an efficiency concern with trying to make very simple filters like "if From = Don Sanders then move to some imap folder on the same server" work without downloading the message completely, modifying it and uploading again. But their is a counter-concern with ensuring that we permanently mark messages that have been filtered to ensure that they are not re-filtered.

Basically there are only a couple of relatively small steps to getting client side IMAP filtering working now, and then a larger step to get things working efficiently (I guess that's a never ending problem/goal).

Don.

*** Bug 78931 has been marked as a duplicate of this bug. ***

Is the ability to select an IMAP folder in a filter currently ony available as a patch? I've seen talk of different kind of filters here, so I'm not sure what's been implemented yet and not.

When I send out emails I store them into different folders depending on the recipient and those folders happen to reside on an IMAP account. Right now how I go about it is I select a miscellaneous folder to hold sent mail in my Identity settings (this misc. folder resides on IMAP too). I have to manually go into this sent mail folder to move them into their final destination folders. It'd be nice if this can be automated using filters that apply on outgoing emails.

Is this in CVS already?

*** Bug 93531 has been marked as a duplicate of this bug. ***

As long as Kmail (beside this very feature the best mail-clinet out there)
can't filter incoming IMAP-mails to imap folders, and have trash,sent,draft
and other system folders on the imap server, I would't use it as a
mail client.
Sorry for this, but no one peace of mail is on a local machine in my network.

*** Bug 94387 has been marked as a duplicate of this bug. ***

Hi,

Here is a status report concerning the work we are doing, on making the 
filters work with imap folders so that mail may be filtered into imap folders 
59685. So that new mail in imap folders is automatically filtered 50997, and 
in making filtering mails into external tools like spam assassin 
non-blocking, a.k.a, asynchronous 41514.

Firstly I want to say that I'll close 41514 once I'm satisfied that filtering 
mail into an external tool like spam assassin no longer blocks for lengthy 
periods. Could people who have some other blocking problem please open up a 
new bug rather than comment on 41514 as commenting on 41514 will result in 
their feedback being forgotten when 41514 is closed.

What has been achieved in the last month? There is now a functioning prototype 
of KMail enhanced to support all the improvements listed above working in my 
laboratory at home where I work on a machine disconnected from the internet. 
I have also started using this system on my production machine at work. 
However I'm encountering problems just as fast as I can fix them so I 
anticipate it will take some time before things stabilize enough to commit 
the code to cvs. I apologize for this delay.

There is also a complication in that only mail manually filtered with 'Apply 
Filters', and incoming IMAP mails are currently being filtered using the new 
filtering system. Local, POP, and Sent mails (and disconnected) are still 
being filtered with the old filtering system. This means that mails filtered 
from these kinds of accounts can't be filtered/moved into imap folders (even 
though that appears as an option in the filter dialog).

I've promised not to address Local, POP, and Sent mails, before completing a 
'Quick insertion of common phrases' feature. Thus it seems necessary for me 
to now work on asynchronous filtering, imap folder targets, automatic 
incoming imap mail filtering, and quick insertion of common phrases 
simultaneously. Which is what I plan to do in the following weeks.


If people are interested in the details of the currently known bugs they are:

POP, Local, Sent (and disconnected?) accounts need to be ported over to using 
the action scheduler.

Need to automatically check imap inboxes when new mail is discovered in them. 
Currently need to manually kick of filtering by changing into an imap inbox.

I don't have any logic for handling crashes. Mail shouldn't be lost but in the 
event of a crash some mail may not be filtered.

Somewhere in the networking stack below KMail (KIO or the IMAP server I'm 
using in my lab) there is a problem that is causing a lot of timeouts and this 
means some mails are not being filtered.

I'm only about half way through my tests, I'm currently still in the stage of 
encountering new problems just as fast as I fix the old ones.


Finally I'd like to say thank you again to all the people who are funding the 
development of these features for all KMail users, through the commercial 
improvement system. We really appreciate your support.

Don Sanders
http://kontact.org/shopping/sanders.php

Thank you for all of your work... This is the last feature (I think) that is keeping any part of GNOME (I'm running Evolutoin until this is fixed) on my Gentoo box.  Thanks again

Is this feature going to be included in kde 3.4? Didn't find it in 3.4 beta2, so I guess not then, or?

Thanks for good work though.

Thanks for kmail, it's a great mailclient, but the lack of IMAP filtering is making me want to change several times a day. If it wasn't for the rest og the kontact suite, I would probably skip it.

It has been three years and several major releases of KDE now, I have offered to PAY for this feature, and it STILL appears to have no traction.  Why?  This is a very basic feature that should be present for compatibility with imap servers.  Particularly since kolab is based on the cyrus imap server, which supports sieve.  I have noticed that sieve is supported when using disconnected imap, why can't it be used on live imap?

This really is a mission critical feature as far as I am concerned.  Can someone who is a kmail developer please explain why this wish is not high on any priority list, and if it will ever be bumped up on the priority list?  Is it because there are so few requests?  As previously mentioned, I would be willing to test or help in any way possible.

On a positive note, I see that kmail 1.8 (released with KDE 3.4.0) supports moving/expiring folders between imap servers, that is a very welcome improvement!  I will no longer have to use outlook express to archive my email between imap servers!

Hi Mike.

Please let me try to address some of your concerns. This improvement is the highest priority KDE feature I'm currently working on. Let me assure you of that.

I did try hard to get this feature into the 3.4 release, but was unable to. The reasons for this were twofold firstly I only have a prototype working for IMAP, but not for POP yet. This means that in the filter dialog IMAP folders can be selected as the destination for 'File into Folder' actions, but this will only work for mail collected from IMAP folders and not for mail collected from POP folders. For this reason the patch was considered by the other developers to be too immature and not ready for committing to KDE CVS. Secondly the prototype is buggy, I'm finding bugs in it just as quickly as I can fix them, especially over the Christmas period I tried hard to get the patch ready for committing but was simply unable to get it reliable enough.

Additionally I suffered a hard drive crash about a month ago, lost some data, and this cost me a few weeks. Also I've noticed some severe regressions in the search folder code, and I am trying to fix these bugs before I commit a new feature.

Finally I really appreciate the pledges I've received from everyone, including yours. However it's my day job that pays my bills and keeps my wife and I clothed, feed, and sheltered, I have to give priority to my day job and this only leaves me with maybe two days a week to work on KDE. A lot of this time is taken up just by reading mailing lists and bug reports.

What I can do is spend some large blocks of time this weekend working on, including taking time away from my wife and coming into work during the extended weekend due to the holidays, (and I really need large blocks of time to make progress as I can't see any easy way to understand the bugs I'm finding and fix them, it simply requires a lot of time and effort), and give a report on my progress next Thursday, so that's on the 31st March.

Even though I missed the 3.4.0 release for this feature, maybe this isn't that terrible as I'm hoping to get it finished, backport it into the (more stable) 3.4.1 release (when that is made) and release that on kontact.org/donsanders.org.

Don Sanders

"I have noticed that sieve is supported when using disconnected imap, why can't it be used on live imap?"

I'm not sure how to respond to this, I don't understand what is meant. I know several people who use sieve scripts with live/normal KMail IMAP. The sieve filtering happens on the server.

In reply to Don Sanders,

I, for one, would simply like to express my gratitude to know that someone is working on this bug throughout other problems.  As I am unable (due to lack of knowledge and skill) to work on a project or feature like this, I can only thank the one(s) that does(do) and spends the time.

No doubt there are many of us out here (the votes tell me there are) and no doubt all of us appreciate your efforts - just silently.  ;)

Have a great day, and good luck!

>I'm not sure how to respond to this, I don't understand what is meant. I know
>several people who use sieve scripts with live/normal KMail IMAP. The sieve
>filtering happens on the server.

Obviously I am confused about this.  Reading through the help documentation for the latest kmail, it states that sieve is supported only for kolab.  I assume this means you can create/edit filters via kmail and save them to the kolab server via timsieved as sieve scripts.  Which of course, filters on the server.  I don't have access to a kolab server (which appears to be a nightmare to set up), so I can't really check out why there is a sieve option in the server configuration.  To tell you the truth, I would be happy having filtering via either route, native to kmail applied to live imap, or via sieve scripts on the imap server.  Right now it is not possible _either_ way.

>Finally I really appreciate the pledges I've received from everyone, including
>yours. However it's my day job that pays my bills and keeps my wife and I
>clothed, feed, and sheltered, I have to give priority to my day job and this
>only leaves me with maybe two days a week to work on KDE. A lot of this time
>is taken up just by reading mailing lists and bug reports.

Don, please forgive me for the frustrated tone of my comment.  I took it for granted that there was more than one developer working on this, and that you guys are all super human and independently wealthy :)  

Don't get me wrong, I love kmail and get excited at every new release, it keeps getting better and better.  This feature, however, is the one I have wanted the most.  I hate manually filing emails into folders and having to set up things like websieve to get sieve filtering working.  Being able to configure filtering (on server or client via kmail) will be the icing on the cake.  Even without the filtering, I highly appreciate kmail and the work you have put into it...

(About sieve, I don't think you can create/edit sieve scripts with 
KMail yet.)

> Don, please forgive me for the frustrated tone of my comment.


No worries, to be honest it's nice to know people care. It's 
motivating.

> I 
> took it for granted that there was more than one developer working
> on this, and that you guys are all super human and independently
> wealthy :)


I guess it's just me working on it at the moment, once/if I get it 
ready to commit others might well pitch in and help. But getting it 
ready to commit is not a small job.

I'm working on it.

Don.

may i suggest a workaround in the meantime before don (lame user be genuflect 
before him) manages to come up with a real solution.

what i do usually is download all imap mail from one imap server to a local 
folder with the imap server's name. then i select all messages in there and 
press ctrl-j and voilà, all my mails are filtered. the remaining ones in the 
folder are a bunch of unfiltered messages like another "saved-messages" 
folder in pine.

so if we could have an option to do this as a temporary workaround, then, don 
(lame user be genuflect before you) that would be very nice ...

a kde lame user

>...firstly I only have a prototype working for IMAP, 
> but not for POP yet. This means that in the filter 
> dialog IMAP folders can be selected as the destination 
> for 'File into Folder' actions, but this will only work 
> for mail collected from IMAP folders and not for mail 
> collected from POP folders. For this reason the patch 
> was considered by the other developers to be too immature 
> and not ready for committing to KDE CVS.

While folders on a mail server that supports IMAP are fairly common, POP folders are somewhat less common.  Further, while standard IMAP supports uploading messages to folders, (and may support moving the message directly), I don't know that all POP implementations do.

Typically, if mail is being downloaded on the server and stored locally, POP is the protocol of choice, and is more efficient.  IMAP, on the other hand, is the protocol more likely to be chosen when messages are being stored, filed, and manipulated on the server.  

I know that the ability to "filter" messages and move them to other folders on the server exists within the kmail code - I can do it manually.  I can select all the messages, for instance, with "[Bug 50997]" in my inbox, and move them to the "Software/KDE" folder on the same server.

This is what I am asking for in participating in this bug report - let me define a filter that instead of delivering the messages to my local folders delivers them to a folder on the server.  All of the functions are there in the current distributed versions (1.7.1 for Mandrake) - all they need is to be put together.

> what i do usually is download all imap mail from one imap server to
> a local folder with the imap server's name. then i select all
> messages in there and press ctrl-j

...
> so if we could have an option to do this as a temporary workaround,


I think having an option to do that automatically as a workaround 
should be possible, I guess. (This won't support using filters to 
file mails into IMAP folders however, and it won't support more 
asynchronous filtering so spam filtering will still block badly).

Really it depends a lot on what Mike Green and the other commercial 
improvement system clients want. I'm pretty much working for them on 
this one.

Created attachment 10423 [details]
Simple client side imap filtering for KMail

Good news. I just uploaded a patch for simple client side imap filtering for KMail.

I didn't get a response from any clients about working on a temporary workaround to just automate applying filters using the existing filter system to incoming messages (as quoted in the previous comment #67). But I decided to work on it for a few reasons

(1) Creating such a patch required that I look at IMAP mail checking to work out how "to automatically check imap inboxes when new mail is discovered in them" without needing to "manually kick of filtering by changing into an imap inbox". As I wrote in comment #55 I need to do this anyway.

(2) It wasn't that much work (well took me all of Friday, Saturday, some of Sunday, to read the code and all of Tuesday and some of today) to get a basic client side imap filtering patch working.

(3) I guess basic client side imap filtering would be useful to a lot of people, so it seemed like a nice thing to do.

So attached is a relatively simple patch that automatically applies filters to messages that are delivered to the inbox of imap accounts. I think this is what a lot of people want when they ask for imap filtering.

This patch has some quirks. Hopefully these quirks are relatively minor. If anybody does test the patch and finds it doesn't work for them (crashes or whatever), then I'm interested in knowing about it since, any bugs in this simple patch could affect the larger patch I'm still working on. Anyway the known quirks are:

(1) All messages that arrive in imap system folders are filtered, instead of just the inbox. Personally for my imap folders this is just the inbox, so it doesn't matter. I do need to find a better way of checking for the inbox.

(2) Only filters that are marked as being "apply this filter on manual filtering" will be applied, it would be more correct if only filters that are marked as being "apply to incoming messages" should be used. But currently when a user selects messages in an imap folder and applies filters manually only filters that are applied on explicit filtering are applied, so this patch is consistent with automating what happens when a user selects messages in an imap folder and manually applies filters. (So again this shouldn't matter).

(3) Only existing filters are supported, so filing into imap folders is not supported by this patch.

(4) Filtering has not been made more asynchronous. So spam filters will still block badly. (I could make it more asynchronous but this would mean I would end up duplicating work I'm doing for the other big patch).

(5) There are still some questions in my mind about the graphical interface. Some people will only want to apply filters to some imap accounts, and some people will want to only apply certain filters to certain accounts. I designed a GUI for this but it's not included in this patch. There's no new options in this patch, graphical or otherwise, the option is whether you apply the patch or not.

So as I said this patch could met the needs of people who have been waiting for imap filtering in KMail. However to me this is just a milestone on the way to upgrading KMail's filtering capabilities to make them more asynchronous (don't block as badly while filtering spam), supporting imap folders as the destination of filters and allowing per account filters. (So addressing issues 3, 4, and 5 in the previous list).

I've still got to merge this patch with my bigger patch, and work my way through the issues outlined in comment #55. (But at least one of those issues is now resolved, however I think there may be additional regressions in HEAD that require looking into, I've been working with 3_3_BRANCH recently). 

Unfortunately there is one problem. I can't give this improvement my undiluted attention. There are some regressions in KMail (search folders are broken they aren't updating properly and system folders aren't being compacted correctly) that I think need some attention. I need to look at these issues.

That said I hope to merge this patch in with my bigger patch, keep on working on the bugs and give another update next month ... hopefully with a patch ready for committing to CVS. (wish me luck!)

Don Sanders 
http://kontact.org/shopping/sanders.php 

Created attachment 10427 [details]
Simple client side imap filtering for KMail

Fix bug.

Minor update.

> I've been working with 3_3_BRANCH recently

Sorry I meant to say 3_4_BRANCH.

Also I found and fixed a bug in the simple (automatic) client side imap filtering for KMail patch, and uploaded it, current version is 10427.

> (1) All messages that arrive in imap system folders are filtered, instead
> of just the inbox. Personally for my imap folders this is just the inbox,
> so it doesn't matter. I do need to find a better way of checking for the
> inbox.


Simply check for the imapPath: it's /INBOX/

On Wednesday 30 March 2005 02:15 am, Don Sanders wrote:

> (1) All messages that arrive in imap system folders are filtered, instead
> of just the inbox. Personally for my imap folders this is just the inbox,
> so it doesn't matter. 


I don't know if it would save any time for that to be configurable, ie be able 
to set which folders to check for filtering.  I think for most people either 
mail is all delivered to the inbox, which is why we need client-side 
filtering, or we have server-side filtering to sort it into seperate boxes, 
and don't need the client side as much.

> (2) Only filters that are marked as being "apply this filter on manual
> filtering" will be applied, it would be more correct if only filters that
> are marked as being "apply to incoming messages" should be used.


If I understand the rational behind "apply on manual filtering", the idea is 
to be able to look at all messages prior to filtering, then manually apply 
the filters.  So yes, there is a purpose for "manual" vs. "automatic" 
filters.

Thanks for the work.  Hope you can look at making IMAP folders destinations 
for filters soon.

Created attachment 10464 [details]
Simple automatic client side imap filtering for KMail

> Simply check for the imapPath: it's /INBOX/

Cool. I created a new version of patch using this info. Here it is, it applies
to the 3_4_BRANCH.

Created attachment 10588 [details]
Simple automatic client side imap filtering for KMail with GUI

Here's an updated patch. I've updated and merged the "client side imap
filtering gui and KMFilter changes" and "Simple automatic client side imap
filtering for KMail" patches.

This means when this patch is applied the user can specify which accounts a
filter is applied to for each filter. e.g. the user can specify that a filter
is applied only to incoming mail from one IMAP account, or all accounts, or
some subset of accounts. This can be done in the (new) advanced tab of the
filter dialog.

The code to filter incoming messages for IMAP accounts is still working. I've
tested it for the last 4 days without encountering a problem. (Last week I
detected and fixed an infinite loop that occurred when trying to filter a
message on an IMAP account while it was selected and being shown in the KMail
GUI, but since then I've detected no new errors/limitations).

There is still a problem in that if you quit KMail some IMAP messages may not
be filtered ( I still have to save the list discovered but unfiltered messages,
when KMail quits. Not hard it'll just take a bit of time to test).

Also I haven't handled, disconnected IMAP. Actually I haven't tried to, maybe
they will work ok with this patch... except I don't show them in the advanced
tab of the filter dialog.

Additional I haven't merged in the code/patch that allows IMAP folders to be
selected as the target of File/Move into folder filter actions (and is less
blocking). That code still needs more work/testing.

Now the good thing about this patch is that, I think it's basically good enough
for committing to CVS. I'll spend some more time checking it, but basically I
think this patch solves this bug (50997), and bug 73758.

So I'll test it some more, and then send it to the KMail dev list for review
(within a week?).

Afterwards I'll get back to working on the code that makes filtering less
blocking and allows IMAP folders to be destinations of filters, (e.g. bugs
41514 adn 59685).

*** Bug 106027 has been marked as a duplicate of this bug. ***

Don, is this done with svn commit 408531?

Hi Ben, fancy meeting you here. No unfortunately it's not completely done with 408531. I decided that I didn't want to close this bug until imap folders as destinations of filters are supported. I have this working (well still some debugging to go possibly) locally for online imap accounts and on manual application of filters (as well as more async filtering esp for spam filtering using till's work). 

But I don't have it working for all account types yet (e.g. pop, local and disconnected imap) and in the result of a crash some messages may be unfiltered (not lost just not filtered) thus there was an objection about committing before these things are done. I intend to work on this once I get a decent block of time. I'm 70% sure I'll have a patch ready by July 21.

It's not so much the amount of code required it should just be a few lines in the action scheduler and a few lines for each of the unfinished account types. But I will need to spend quite a lot of time testing.

Created attachment 11840 [details]
Imap folders as targets and more asynchronous filtering

I've got good news and bad news.

The good news is that I've attached a patch for KMail that implements support
for imap folders as targets of filters and makes filtering more asynchronous
for incoming mails for imap accounts and on manual application of filters.
(Which I believe is exactly what you guys want).

The bad news is in two parts. Firstly this patch activates the action scheduler
which is a new filtering engine for KMail, and this engine isn't full debugged
yet. It is still stalling from time to time and it's necessary to restart KMail
to get it working again. I'm working on fixing that, right now.

The second part of the bad news is that I haven't activated the action
scheduler for other types of filtering, namely sent mail filtering and pop,
local, and disconnected imap account incoming mail filtering. Though once
things are working well for online imap only about half a dozen lines of code
should be required to get the action scheduler working for each account type.
(It just takes a considerable amount of time to test everthing).

This second issue is especially important as there have been objections to
committing this work before the action scheduler is working for all account
types. (I wonder if any core developers have time to help out with this).

Actually another issue is that the new copy filter action and filter logging
code may not work with the action scheduler (I would like to discuss this
further with Andreas Gungl).

On the positive side by using the action scheduler in the Delete, Move and Copy
commands it should be possible to make these actions much more asynchronous
which is something I believe Staikos Computing (Hi George) should be interested
in (although gettting the Copy command to use the action scheduler will require
a small enhancement to the action scheduler).

So in summary I've pretty much ended up in the 30% slot of not getting
everything working by July 21 (although I hope to at least get imap folders as
targets for incoming online imap accounts working before akademy as I want to
use them myself to read my mail there).

Finally I just like to say thank you once again to all those who have supported
the commercial improvement system, your support is greatly appreciated.

Don Sanders http://donsanders.org

Actually, this is a question regarding the handling of imap connections, which 
you might not have anything to do with but are at least aware of.

The idea of a persistent imap conncetion on a local network is something that 
I don't think I've encountered in other imap mail clients.  Generally, if a 
session persists beyond a few transactions, that's a bit unusuall.  

I have encountered a few problems with an otherwise persistent connection over 
the internet.  It's not a dial-up connection, but I need to be able to push 
messages back up to the spam folder on the remote server so that they can be 
processed by spamassassin's learn feature.  Imap's the only way I know to do 
that.  

The main problem seems to be that if the connection is dropped for inactivity 
by the remote server, as it should be, KMail doesn't simply fall back to 
re-establishing the connection but fails with an error and doesn't proceede.

Created attachment 11962 [details]
Imap folders as targets and more asynchronous filtering

I've attached a patch that conditionally enables the action scheduler 
for filtering of incoming mails for online imap accounts and for 
manual filtering.

To try out the action scheduler apply the patch and add an 
action-scheduler=true entry to the [General] section of your kmailrc 
file.

The scheduler is still stalling after a day or so, I'll try to look into this
next week. Apart from that it seems to be working pretty well.

I am not quite sure whether this is related. I am looking for a way to abuse IMAP accounts like POP mailboxes. The idea is to query different e-mail accounts (some POP, some IMAP) and to collect all mails in a single local folder mailbox (inbox). Having all mails in different inboxes is quite unpractical in my case.

Would filters be the way to do this?

Filters got nothing to do here since you have to speficy various logins, passwords and server names for each account. Mozzila does not support (and doubt it will). You can create as many accounts as you want and for each of them set simple filter to move any message to one target account you want.

On Sat, 20 Aug 2005 22:10:57 +0200, "Ingo Kl

I've now submitted patches to support filtering of incoming mail for online IMAP accounts and so that online IMAP folders can be used when filtering into folders, so I'm finally closing this bug.

There are still a wide spectrum of potential issues, too many for me to enumerate here, but (excepting unforseen problems) the online IMAP filtering functionality should be fully operational in the KDE 3.5 release!

As an additional benefit, online IMAP account users should benefit from more asynchronous filtering including non-blocking spam filtering (thanks to Till Adam's work). I hope to port other accounts types over to this new filtering system to address bug#41514 but currently more asynchronous filtering is only supported for online IMAP accounts (and manual filtering).

If bugs are found in the online IMAP filtering instead of re-opening this bug instead please open up a new bug report with a specific description of the problem and if at all possible a method for (someone other than yourself) to reproduce the problem.

Finally I'd like to once again thank the supporters of the commercial improvement system, in many ways it is their encouragement that has helped make the implementation for this improvement possible. Thank you!

Don Sanders http://donsanders.org