Bug 282891 - searching is severely broken (contains multiple reports)
Summary: searching is severely broken (contains multiple reports)
Status: RESOLVED UNMAINTAINED
Alias: None
Product: kmail2
Classification: Applications
Component: general (show other bugs)
Version: 4.7
Platform: Ubuntu Linux
: NOR normal
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-09-27 07:35 UTC by Wolfgang Rohdewald
Modified: 2015-09-09 20:37 UTC (History)
18 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Wolfgang Rohdewald 2011-09-27 07:35:34 UTC
Version:           4.7 (using KDE 4.7.1) 
OS:                Linux

There are so many problems - I am just listing all I had so far here. Please tell me which parts you want me to open a separate bug report for.

I have several thousands of mails with a subject starting with SF##### where ##### is a number with between one and five digits. 

when searching for subjects containing SF, I only get mails where SF is NOT followed by a digit - expected are ALL mails containing SF. I also get mails with sf in the subject (which I think is wrong).

When searching for a specific string like SF18013 I get correct results.

When searching with regular expression SF, I get lots of mails where nothing like SF or sf is in the header. I would have expected (assuming this regex is not anchored) to get all mails containing SF anywhere in the header. Same for regex ^SF.* - I also get tons of unrelated mails - I would have expected only those starting with SF

I just sent myself a mail with subject ABC567 - whatever I try I cannot define a search returning this mail. But some time later (meanwhile I restarted kmail) it appears twice for the sent message and twice for the received message. Should only appear once for sent and once for received

Searching by age in up to 15 days returns thousands of year-old mails
Searching by age greater or equal 15 days returns nothing - such mails do exist however.

there seems to be no way to get the "Stop" button enabled - searching cannot be stopped. There is no way to say "you already found enough - I will analyse this, you do not have to search further" - this was possible before kmail2

when changing the search term during an ongoing big search, the mails not matching anymore are only partially removed from the mail list in the search dialog. Right now the message count of the search folder says 20 messages while 
actually holding lots and lots of messages - the message list is constantly shuffling in strange ways. At the same time the search dialog says "0 matches Searching in Last Search" (shouldn't "Searching" be lowercase?) while the mail list in the search dialog shows lots of mails not matching at all. After closing the search dialog, I try to delete the search folder - it disappears only after shuffling mails for about two minutes.

after having deleted the search folder the next search tells me "Can not get search result. Unknown error. (No such collection). The field for entering the Search folder name is disabled - shouldn't it be enabled? At this point searching is not possible anymore. When now pressing "Open search folder" I get an empty folder in the main view - while the folder list in the main view shows no search folder at all. When no search folder exists, that button should be disabled.

After that I tried to search again. Starting the search window, I first entered a new search folder name (now it was enabled) only to get the message "There was a problem renaming your search folder. A common reason for this is that another search folder with the same name already exists." (which is not the case since there is no search folder at all). Also, why does this appear while I type in the name? This makes it impossible to define a search folder "X5" when a search folder "X" already exists.

often when modifying a search I get "Can not get search result. Unknown error. (Only resources can modify remote identifiers). Presenting such a message to a stupid end user like me is a severe usability bug IMHO - how should I know what this means and what I am supposed to do?

when removing the only search folder I have, the next search will tell me "Can not get search result. Unknown error. (No such collection). Instead it should allocate the search folder. If I close kontact and restart it, there still is no search folder but if I search now, it will correctly be allocated.

Reproducible: Always

Steps to Reproduce:
see above

Actual Results:  
see above

Expected Results:  
see above

OS: Linux (x86_64) release 3.0.0-12-generic
Compiler: gcc
Comment 1 kazamatzuri 2011-10-23 10:41:10 UTC
I can confirm the problems with search. 

additionally the search does not search in imap accounts/folders. Is this intentional?
Comment 2 Patrick Scharrenberg 2011-10-29 12:11:01 UTC
I can confirm the problems with search. 

additionally searching for sender or receiver does not any results
Comment 3 bugs.kde.org-1 2011-12-23 19:18:27 UTC
*** This bug has been confirmed by popular vote. ***
Comment 4 Hussam Al-Tayeb 2011-12-29 15:32:16 UTC
with kde 4.7.95, under some setups, kmail crashes when it finds emails in search.
Under most other setups on my machines, it refuses to find search results.
Comment 5 Ralph Moenchmeyer 2012-02-04 14:41:09 UTC
I have updated to KDE 4.8 on an Opensuse 12.1 system. In parallell I installed Opensuse 12.1 and KDE 4.8 from scratch on a deifferent machine. I had hoped that the search dialog and its functionality now would have been corrected. 

However, the search dialog still does not offer any option for choosing the mail folder to search in. 

A search with conditions via the search dialog is not working at all any more. It always gives an empty result list.  

Unfortunately I must say, that bugs regarding email search are present since the change to the akonadi based Kmail2. Right now the search dialog and its functionalities are completely useless. 

The only search process that works at all is a search via the search input field in the normal Kmail window. This kind of search, however, is a kind of conditionless full-text search. This is not enough for professional usage and a severe deficit/defect of Kmail.
Comment 6 Andrew Gaydenko 2012-03-08 04:30:18 UTC
4.8.1 has added folder selection but the result is the same - nothing found. The "Only resources can modify remote identifiers" error (on the second search start) still takes place.
Comment 7 Mark Fraser 2012-05-06 13:42:12 UTC
I've upgraded to Kubuntu 12.04 with KDE 4.8.3 and I'm still getting this error.
Comment 8 Graeme Hewson 2012-05-23 17:42:54 UTC
I am also seeing multiple errors with Kubuntu 12.04 and KDE 4.8.3. For instance:

Searching on "From" "contains" matches only on the email address itself and not the preceding name. E.g., From: header is "Fred Smith <fs@example.com". Searching for "smith" returns no results, but "fs" does return results.

Each of the preceding matches is duplicated. E.g., "Find Messages" dialog box says "16 matches Searching in Last Search", but the main body of the box contains 2x16 results.

However, this isn't consistent. E.g., searching on "From" "contains" "john" might not return any matches, even if there are messages from "John Smith <john@example.com>"

Search only in: akonadi-local/sent-mail returns matches from all folders, not only the requested folder. But sometimes no matches are found, even though they should be.
Comment 9 Thomas Arend 2012-10-23 19:32:56 UTC
I can confirm this bug for 4.9.2 . The search for "diese Gelegenheit" returned only 2 messages. But there were dozens of spam mails in my folder, which contain these words.
Comment 10 Hussam Al-Tayeb 2013-04-06 11:07:21 UTC
Still happening in kde 4.10.1
Comment 11 Andrew Gaydenko 2013-04-06 11:13:33 UTC
(In reply to comment #10)
> Still happening in kde 4.10.1

Do you mean 4.10.2? Last one has many fixes related to kmail.
Comment 12 Hussam Al-Tayeb 2013-04-06 11:22:12 UTC
(In reply to comment #11)
> Do you mean 4.10.2? Last one has many fixes related to kmail.

yes. sorry. help -> about kmail says:

KMail
Version 4.10.2
Using KDE Development Platform 4.10.2
Comment 13 Hussam Al-Tayeb 2013-04-06 11:27:13 UTC
it's mainly "Can not get search result. Only resources can modify remote identifiers" that feels like a scary message to users :)
restarting kmail makes search work again.
Comment 14 Andrew Gaydenko 2013-04-06 12:28:53 UTC
(In reply to comment #13)
> it's mainly "Can not get search result. Only resources can modify remote
> identifiers" that feels like a scary message to users :)
> restarting kmail makes search work again.

Thanks for sharing the experience! As far as thunderbird has become "too free" now, it is a nice opportunity for kmail developers to bring old users back :) Of course, must-have features is expected to be perfect, and searching is one of the main such features, I guess..
Comment 15 Hussam Al-Tayeb 2013-04-06 12:36:04 UTC
This is how I can reproduce this:
1) run a search but don't close the "find messages" window.
2) run another search in the same "find messages" window and notice the warning message instead of search results.
Workarounds:
1) restart kmail.
or:
2) close the 'fine messages' window, view the "last search" folder and now searching works again.
Comment 16 Graeme Hewson 2013-04-06 14:51:07 UTC
Should "Body of Message" "contains" be working? Nothing happens when I press the Search button, nor has anything ever happened since I first installed KDE4.

I deleted $KDEHOME/share/apps/nepomuk and let nepomuk recreate it as mentioned in http://userbase.kde.org/Nepomuk. Still search doesn't work for me.

In fact, now the default "Subject" "contains" search doesn't work, though it used to: again, nothing happens when I press the Search button.
Comment 17 piedro 2013-07-01 10:54:01 UTC
KDE 4.10.4 on Arch: 

Search for IMAP folders seems to work for me but searching in local folders (yes, they have been indexed!) doesn't bring up ANY results ... restart doesn't help.

Well, it's not an important feature that is broken since kmail2 was published ... nobody really needs to search mail reliably ... it's just a hickup - so please fix important stuff first! 

thx for reading, 
p.
Comment 18 davidblunkett 2013-07-04 08:44:00 UTC
This is so broken - please give up on nepomuk and associated tools and just provide a search tool? preferably based on grepmail which is the bomb.
Comment 19 davidblunkett 2013-07-04 09:24:47 UTC
I suggest to all that have a problem with kmail searching that they try mairix, it is way faster than kmail's seaching in any event and dead easy to use - you do need to to configure it first:

(1) install
(2) make the config file ~/.mairix here is mine:

base=/home/XXXX/Mail
database=~/.mairixdatabase
maildir=*
omit=searches*
mfolder=searches

(3) index the files (doesn't take long) by just running mairix
(4) search from the command line eg
mairix b:mysearchterm
(5) find the results in the searches folder in kmail
(6) mairix to reindex files (or mairix -F to do this faster but plain mairix is instant for me).

It doesn't do regexps unfortunately but it seems to work reasonably well and is more reliable at finding mails than kmail's search (when it worked).
Comment 20 Laurent Montel 2013-08-16 19:20:54 UTC
Git commit 29029f06bd3e1e750e25357b6d785e6ce1308ac6 by Montel Laurent.
Committed on 16/08/2013 at 19:19.
Pushed by mlaurent into branch 'master'.

We need a debug dialog for search. We have a lot of bug report about it.

=> i create a widget for debugging it.

M  +1    -2    CMakeLists.txt
M  +3    -0    kmail/CMakeLists.txt
A  +103  -0    kmail/searchdialog/searchdebugdialog.cpp     [License: GPL (v2)]
A  +43   -0    kmail/searchdialog/searchdebugdialog.h     [License: GPL (v2)]
A  +201  -0    kmail/searchdialog/searchdebugwidget.cpp     [License: GPL (v2)]
A  +60   -0    kmail/searchdialog/searchdebugwidget.h     [License: GPL (v2)]
M  +16   -6    kmail/searchdialog/searchwindow.cpp
M  +3    -0    kmail/searchdialog/searchwindow.h
A  +99   -0    kmail/searchdialog/sparqlsyntaxhighlighter.cpp     [License: GPL (v3+)]
A  +49   -0    kmail/searchdialog/sparqlsyntaxhighlighter.h     [License: GPL (v3+)]

http://commits.kde.org/kdepim/29029f06bd3e1e750e25357b6d785e6ce1308ac6
Comment 21 Laurent Montel 2013-08-16 21:41:40 UTC
Git commit 9f1686fefc6cfce67243e165b3c9100fecc089f7 by Montel Laurent.
Committed on 16/08/2013 at 21:41.
Pushed by mlaurent into branch 'master'.

Return correct error now. User is informed about errors now

M  +12   -7    kmail/searchdialog/searchwindow.cpp
M  +6    -10   mailcommon/search/searchpattern.cpp
M  +9    -1    mailcommon/search/searchpattern.h

http://commits.kde.org/kdepim/9f1686fefc6cfce67243e165b3c9100fecc089f7
Comment 22 Laurent Montel 2013-08-16 21:42:35 UTC
In 4.12 we have messagebox to inform user why we can't search.
in 4.11.1 I can't add messagebox+i18n so I use qDebug to inform users.
Comment 23 Blackpaw 2013-08-16 23:19:28 UTC
Thanks Laurent. 

The search debug dialog is for running  sparsql queries?
Comment 24 Laurent Montel 2013-08-17 06:44:36 UTC
Yes you have current query and you can modify it and re-search
so you can debug it if necessary.
Regards
Comment 25 Laurent Montel 2013-09-12 12:14:51 UTC
I fixed a lot of bugs in 4.12 and backported them to 4.11.2
but I know that there is still a lot of bug.
Will continue to investigate them
Comment 26 Andrew Gaydenko 2013-09-12 17:42:45 UTC
(In reply to comment #25)
> I fixed a lot of bugs in 4.12 and backported them to 4.11.2
> but I know that there is still a lot of bug.
> Will continue to investigate them

Laurent, your current and upcoming efforts give us a hope to return to favourite email Klient. Thanks!!
Comment 27 Laurent Montel 2013-09-21 14:40:58 UTC
I continue to investigate pb with search.
In 4.12 I cleaned search dialog and now each search works, so I hidden search condition which doesn't work for example:
- size it's not supported by nepomuk feeder
- tag search has just condition "contains"/"not contains" and not begin with etc.
- To/CC/Bcc can't search in addessbook
- etc.

If I can reactivate it (need to code in nepomuk) I will reactivate but until it will do I disabled theses conditions. So in 4.12 it will clean (I hope)
Comment 28 Andrew Gaydenko 2013-09-21 14:57:07 UTC
Laurent, 

Just for information. If I remember correctly, with old code base attempts to search with "contains" were "more lucky" when search context was a whole word. So, if a message contains "abc" word, it will (probably) be found with context "abc",  but not with context "ab".
Comment 29 Laurent Montel 2013-09-21 15:05:48 UTC
I was fixed.
It's right before "contains" searched exact word and not "ab*"
now it's fixed.
But a nepomuk (virtuoso) limitation is word must have 4 characters
So you can search "welc" it will find "welcome" with contains condition but will not work when you will search "wel". For this case I can't do anything. I added a warning about it.
Comment 30 Blackpaw 2013-09-21 15:13:51 UTC
(In reply to comment #27)
> I continue to investigate pb with search.
> In 4.12 I cleaned search dialog and now each search works, so I hidden
> search condition which doesn't work for example:
> - size it's not supported by nepomuk feeder

Well bugger. I had cause to search for messages by size recently, no wonder it didn't work :(
Comment 31 Laurent Montel 2013-09-21 15:46:53 UTC
(In reply to comment #30)
> (In reply to comment #27)
> > I continue to investigate pb with search.
> > In 4.12 I cleaned search dialog and now each search works, so I hidden
> > search condition which doesn't work for example:
> > - size it's not supported by nepomuk feeder
> 
> Well bugger. I had cause to search for messages by size recently, no wonder
> it didn't work :(

yes I know it's not good :(
Comment 32 Blackpaw 2013-09-21 21:16:03 UTC
(In reply to comment #31)
> (In reply to comment #30)
> > (In reply to comment #27)
> > > I continue to investigate pb with search.
> > > In 4.12 I cleaned search dialog and now each search works, so I hidden
> > > search condition which doesn't work for example:
> > > - size it's not supported by nepomuk feeder
> > 
> > Well bugger. I had cause to search for messages by size recently, no wonder
> > it didn't work :(
> 
> yes I know it's not good :(

The capability  appears to have been removed deliberately:

https://projects.kde.org/projects/kde/kdepim-runtime/repository/revisions/3435bc0e9c2d63e958227fb8bd6badce4166d97f

"Nepomuk Email Indexing: Do not store the nie:byteSize. No one cares about the byteSize of the email"

BTW, I would disagree with that.
Comment 33 Laurent Montel 2013-09-22 05:34:04 UTC
(In reply to comment #32)
> (In reply to comment #31)
> > (In reply to comment #30)
> > > (In reply to comment #27)
> > > > I continue to investigate pb with search.
> > > > In 4.12 I cleaned search dialog and now each search works, so I hidden
> > > > search condition which doesn't work for example:
> > > > - size it's not supported by nepomuk feeder
> > > 
> > > Well bugger. I had cause to search for messages by size recently, no wonder
> > > it didn't work :(
> > 
> > yes I know it's not good :(
> 
> The capability  appears to have been removed deliberately:
> 
> https://projects.kde.org/projects/kde/kdepim-runtime/repository/revisions/
> 3435bc0e9c2d63e958227fb8bd6badce4166d97f
> 
> "Nepomuk Email Indexing: Do not store the nie:byteSize. No one cares about
> the byteSize of the email"
> 
> BTW, I would disagree with that.

I know I spoke with vishesh 3 weeks ago and he told me that a fix will be more expensive
So we decided to disable it.
Comment 34 Blackpaw 2013-09-22 05:36:35 UTC
What was wrong with it in the first place?
Comment 35 Andrew Gaydenko 2013-09-22 09:12:50 UTC
Four char to search is too restrictive, I guess. How to search at case a complete word has, say, two chars?
Comment 36 Laurent Montel 2013-09-23 06:34:03 UTC
(In reply to comment #35)
> Four char to search is too restrictive, I guess. How to search at case a
> complete word has, say, two chars?

You can't use "contains" .
Use "is"
Comment 37 piedro 2014-01-17 17:22:58 UTC
I don't want to start a new thread, so I put my problems in here. 

My Kde on Arch (fully upgraded) is the very new version 4.12.1. Still searching emails has so many problems for me, that it is completely unusable, which is a shame and I feel there are some regressions even. I will give a list of the issues that won't work for me (anymore): 

- first problem is that any search I saved creates a kind of virtual folder, which is exactly as intended I guess, but after this folder is created it doesn't offer any possibility to modify it (say refine my search folder afterwards by adding more search conditions) - this behaviour makes the idea of search folders very clumsy to use and I can't imagine this is intended ... the same problem applies when searching for one criteria and then adding another one after the search has finished ("only ressource modifiers can change..." - error message) 

- the search folder from kmail also shows up in kaddressbook and korganizer which doesn't make sense at all (I haven't tested what happens to searches in kaddressbook but I guess those show up then in kmail also ...) - I think this is very annoying and quite a show stopper for using search folders at all... 
I think the search should also save the kind of object it' looking for and a filter should make sure that kmail only shows searches with email objects as kaddressbook only shows searches containing address objects and so forth ... 

- the dialog for searching mails in selected folders should also offer a tick box saying "search recursivly" or "search in subfolders of selected folders" as the dialog for selecting only one folder offers already. 
It is very unconvenient to select all the subfolders tick by tick when you are using many postboxes with many email addresses (and some subfolders hierarchies). Also a "select all" or " unselect all" possibilty would be a great enhancement here ... 

- sometimes a search folder simply refuses to be deleted (I have no idea under what condition this happens) - using "delete folder" doesn't give any error message but leaves the folder in place ... after restarting kmail it sometimes can be deleted again or it can't in other cases ... very confusing ... 

-  searching for "peter.pan@" to catch all email of Peter Pan should give the same results when searching in "message header", "full message" or using the three conditions "from", "to" or "copy to" (given that the search term actually only appears in the "to", "from" or "copy to" lines of the messages in question). For me this is not the case: I might get less hits with a full message search than with a header search. the system settings show that email indexing is done the full 100%. If this is due to uncomplete indexing of any kind there should be a warning messgae saying: plz wait for indexing to complete to get accurate results. In the current state it's irritating because you can't trust any search. On the other hand we all know: you simply can't easily catch Peter Pan... 

Sorry for long text but I think these search issues are still very important for using kmail productively, 
thx for reading and all your efforts, kmail2 has come a long way since the somewhat messy start ... 
 
piedro
Comment 38 Andrew Gaydenko 2014-01-17 18:28:16 UTC
@piedro, yes, as far as nobody from KDE team use kmail, we'll still be using thunderbird. I try kmail every KDE release :) With last one kmail has found 70 messages while thunderbird has found 700 messages from the same archive..
Comment 39 piedro 2014-01-17 19:03:00 UTC
@andrew: 
this is a bit frustrating ... thunderbird on the other hand is not integrated very well ... and sucks at other tasks imho (rewriting imapfolder names on servers e.g.) ... but you are right on virtual folders and searching: that is done really nicely and very reliable ... maybe it would be a good idea to take a look at the concepts behind that - it's open source after all, Kde team could learn something here ... but I guess most people want to create things their own way for the fun of it (and they have every right to do so) ... I guess I simply have to accept that there is still no good PIM solution in KDE (same applies for gnome and the other environments ... maybe be the elementary concept of well integrated, lightweight single-task applications that interconnect in an intelligent way is the most promising at the moment ...) 

Thing is: I like the idea behind Kontact! But the Kontact shell is kind of decaying: 
- ktimetracker (such a nice application to integrate with kjournal and korganizer!) had to be removed ... 
- knotes and kjots seem to not improve anymore and are less powerful than any other stand alone application out there 
- korganizer seems to be quite good but without a good way to browse through past events like all appointments with Peter Pan for the last two years it's hard to use it productively 
- same for kjournal 

...and so forth: Quo vadis KPIM? 

cya, piedro 


  

still it is so sad :-( 

btw: I wrote korganizer shows search folders - I meant kjots (which I doubt very much to be a reasonable candidate for the default note taking ... why not work on basket, a tomboy or zim port (there are many more promising projects) instead of holding onto kjots which isn't actively developed afaik ... 

just my feew cents, 
piedro
Comment 40 piedro 2014-01-17 19:05:25 UTC
I have to apologize: I just realized that I am writing all these general concern in a specific bug report. My above thoughts do not belong here. Since I cannot delete my comment, I apologize. 

sry, piedro
Comment 41 Stephan Diestelhorst 2014-02-10 21:56:09 UTC
Happened to me just now, too: Kmail 4.12.1 ;-( (Kubuntu Packages)
Comment 42 Martin Steigerwald 2015-09-09 20:26:05 UTC
Wolfgang and all the commenters, thank you for your reports. As far as I see they are related to a version of KMail which uses Nepomuk and is unmaintaining. Thus closing. If you still see issues with searching, please open new bug reports. But please follow the following guide lines:

- Open exactly one report for *one* issue. Do not mix several issues into one report cause it basically makes a report almost unactionable.

- Research whether someone else reported the issue already.

- Use KDEPIM 4.14. Or if you are daring KDEPIM from master.

- Note that in KDEPIM 15.08 Tools/Find Messages currently crashes. This is known, has been fixed in master already. Thus please do not report.

Thank you and greetings from KDE Randa Meetings,
Martin