Bug 312460

And how to reproduce it ?

Start kmail, and use it normally. In may case, sooner or later - sometime during the day, some folder will have this problem. Not the inbox, a folder where mails are moved to by a filter, in my case those are mailing list folders, in an IMAP account. I never saw more than one folder with wrong count at a time, and I have seen a miscount one or two. The issue is new since running KDE 4.10 RC1.

Created attachment 76132 [details]
wrong message count in kde-pim folder: all mails appears to be read

"go to next unread mail" will go to next unread in a folder containing unread mail. ONLY the count in the folder list is wrong, and the systray icon also shows.

Now two folders.

So, restarting kmail:
* Ctrl + Q
* wait 10 sec
* issue 'kmail' in krunner
* crash
* issue 'kmail' in krunner
* bouncing cursor for loooooooong...
* krunner won't show, so issuing "killall kmail" in a konsole window
* issue 'kmail' in krunner, that now can show again
* kmail starts

Now, the missing mails show - so deeper down, the problem is that the folders do not know about all the mails that the folder list count does... :$

I don't think it is showing the wrong number of unread emails, KMail just isn't allowing you to read those unread emails. Try quitting and then reloading KMail, you'll see that where the count was wrong there are now unread emails to read.

Yes, I have come to the same conslusion. Restarting kmail makes the missing mails show up.

*** Bug 312452 has been marked as a duplicate of this bug. ***

I see exactly the same issue. However with me this can happen at the same time with multiple folders. 

I hope that this gets fixed soon, because this is really becoming annoying. Almost every mail check delivers this situation.

This seems to be a bug in Kmail itself, as that on akonadi level the emails are available. I will attach a screenshot where I have the situation that Kmail indicates unread emails, but doesn't show them, but akonadiconsole (which was opened at the same time as Kmail) clear shows those emails. 

I have setup the akonadi resouces to stay online all the time and when I restart Kmail, the missing emails are then shown correctly. This clearly points that somewhere Kmail looses contact with Akonadi regarding the email headers.

Created attachment 77288 [details]
Kmail and Akonadiconsole showing the emails, which clearly shows the fault in kmail

Not sure whether it's the same bug, but it has the same result.

1. Open kmail and a composer window
2. Quit kmail and leave the composer window open
3. Start kmail and leave the composer window open

Result: The number of unread emails shown in the folder tree increases, yet they are not visible in the message view until one quits all of kmail, i.e. including the composer window, and restarts it.

Also still present in version 4.10.1.

Debugging suggestions from http://lists.kde.org/?l=kde-pim&m=136544564201632&w=2:

"""
- check akonadiconsole debugger and debug output for retrieval errors

- unread count updates, so we can assume monitor signals arrive from the 
server, might still be worth checking if they make it out of Akonadi::Monitor 
or if they are filtered out (gdb or debug output in Monitor or maybe even 
better in ETM)

- use GammaRay to check if the items don't exist in ETM or if they are dropped 
somewhere above that in the proxy stack
"""

(In reply to comment #13)
> Debugging suggestions from
> http://lists.kde.org/?l=kde-pim&m=136544564201632&w=2:
> 
>.....
> - use GammaRay to check if the items don't exist in ETM or if they are
> dropped 
> somewhere above that in the proxy stack

I'm not able to reliably reproduce this issue, but it happens from time to time. GammaRay shows that the item is /not/ in the ETM, so it's probably a Monitor issue.

> """

This might be related to the message storage sort order settings.

I have what looks like this problem when the message sort order is set to ' none (storage order)' (to reproduce right click on a column header in the message list --> sorting --> check 'none (storage order)'. When there are new mails in a folder setting to 'storage order' makes them disappear from the message list. By a new mail is one which has arrived since kmail/kontact was last restarted. If some of the messages are unread the number of unread mails still shows up in the folder list.  

If I set the storage order to 'by date/time', the new messages (read and unread) show up in the message list.  

This seems to happen in all folders, the inbox and ones where message are moved to be filters.

This might be the same https://bugs.kde.org/show_bug.cgi?id=312484 .

FWIW, I have my sorting order set to "By Date/Time" "Most recent on top" and I have this issue.

Created attachment 78843 [details]
debug logs from akonadiconsole

Akonadi console output while reproducing the problem.

gdb tells me:
Akonadi::MonitorPrivate::emitItemNotification get's called as it should with all the right parameters and "emit q_ptr->itemAdded( it, col );" gets called.

The etm receives collectionChanged signals (for for the appropriate monitor notification), but never the itemAdded signal.

I placed breakpoints in the monitor and the ETM monitoredItemAdded and monitoredCollectionChanged slots.

Apparently there are two monitors active, where only one is correctly connected to the ETM:

Breakpoint 6, Akonadi::MonitorPrivate::emitNotification (this=0x274ed90, msg=...) at /usr/src/debug/kdepimlibs-4.10.2/akonadi/monitor_p.cpp:284
284     {
(gdb) print msg.toString()
$41 = "Item (232497, 20) in collection 403 added"
(gdb) continue
Continuing.

Breakpoint 7, Akonadi::MonitorPrivate::emitItemNotification (this=this@entry=0x274ed90, msg=..., item=..., collection=..., collectionDest=...)
    at /usr/src/debug/kdepimlibs-4.10.2/akonadi/monitor_p.cpp:484
484     {
(gdb) continue
Continuing.

Breakpoint 6, Akonadi::MonitorPrivate::emitNotification (this=0x274ed90, msg=...) at /usr/src/debug/kdepimlibs-4.10.2/akonadi/monitor_p.cpp:284
284     {
(gdb) print msg.toString()
$42 = "Collection (403, /INBOX) in collection 280 modified parts (uidnext)"
(gdb) continue
Continuing.

Breakpoint 5, Akonadi::EntityTreeModelPrivate::monitoredCollectionChanged (this=0x2818620, collection=...) at /usr/src/debug/kdepimlibs-4.10.2/akonadi/entitytreemodel_p.cpp:981
981     {
(gdb) print m_monitor
$43 = (Akonadi::ChangeRecorder *) 0x2868280

So it seems the ETM which receives the monitoredCollectionChanged, is connected to a different Monitor/ChangeRecorder than the one which receives the ItemAdded notification, and m_monitor is again different from the monitor which actually fired the signal which seems to say that m_monitor is somehow being reset and apparently the monitor which receives the ItemAdded notifications fires into nowhere (although it clearly has a connection to some object, I just don't know which).

Enough weirdness for the moment, I have to figure out how I can debug the signals-slots connections.

I was comparing the pointer of the monitorprivate to the pointer of the monitor which are obviously different. Here's how it should looks:

Breakpoint 2, Akonadi::EntityTreeModelPrivate::monitoredItemAdded (this=0x292c2a0, item=..., collection=...)
    at /usr/src/debug/kdepimlibs-4.10.2/akonadi/entitytreemodel_p.cpp:1037
1037    {
(gdb) continue
Continuing.
[New Thread 0x7f6f79c85700 (LWP 4444)]

Breakpoint 3, Akonadi::MonitorPrivate::emitItemNotification (this=this@entry=0x26656a0, msg=..., item=..., collection=..., collectionDest=...)
    at /usr/src/debug/kdepimlibs-4.10.2/akonadi/monitor_p.cpp:484
484     {
(gdb) print q_ptr
$6 = (Akonadi::Monitor *) 0x2777fc0
(gdb) print msg.toString()
$7 = "Item (232625, 4777) in collection 43 added"
(gdb) continue
Continuing.
[Thread 0x7f6f79c85700 (LWP 4444) exited]

Breakpoint 2, Akonadi::EntityTreeModelPrivate::monitoredItemAdded (this=0x292c2a0, item=..., collection=...)
    at /usr/src/debug/kdepimlibs-4.10.2/akonadi/entitytreemodel_p.cpp:1037
1037    {
(gdb) print m_monitor
$8 = (Akonadi::ChangeRecorder *) 0x2777fc0
(gdb) print item.id()
$9 = 232625
(gdb) continue
Continuing.

Now I have to wait until I run into the problem again to be able to check again what's happening when it's going wrong.

Figured it out at last, here's what's happening:
* To reproduce:
** open kmail
** open a mail in a separate window
** close the mainwindow only (leave the editor/viewer running)
** open kmail again => voila you're in the broken state

KMMainWidget::destruct calls
"disconnect( kmkernel->folderCollectionMonitor(), SIGNAL(itemAdded(Akonadi::Item,Akonadi::Collection)), 0, 0)"
disconnecting everything from the monitor instead of only itself.

Because the open editor keeps the kmkernel running, and the kmkernel holds the monitor AND the etm (which are now disconnected). Starting kmail again results in it attaching to a model without connected monitor.

Really a good example on why singletons are evil.

A fix will be up shortly.

Git commit c7acedfeebbb1edd0912b9cbfda9ca707d2e8c83 by Christian Mollekopf.
Committed on 07/05/2013 at 17:42.
Pushed by cmollekopf into branch 'KDE/4.10'.

Don't disconnect the ETM from the Monitor when closing the mainwindow.

Otherwise we'll end up with an ETM that doesn't update anymore.

FIXED-IN: 4.10.4

M  +5    -5    kmail/kmmainwidget.cpp

http://commits.kde.org/kdepim/c7acedfeebbb1edd0912b9cbfda9ca707d2e8c83

*** Bug 312484 has been marked as a duplicate of this bug. ***

Christian,

Maybe you have resolved the issue for that particular use case, but I just had this issue occurring again and I am running kmail git version from last night. 

I don't do anything strange with kmail. Kmail is sitting in the systray and the main window is minimized. I had it active for about 3,5 hours and everything worked fine. Now I see that I have an unread email, but kmail doesn't show that unread email. Only the unread count is shown.

(In reply to comment #24)
> Christian,
> 
> Maybe you have resolved the issue for that particular use case, but I just
> had this issue occurring again and I am running kmail git version from last
> night. 
> 
> I don't do anything strange with kmail. Kmail is sitting in the systray and
> the main window is minimized. I had it active for about 3,5 hours and
> everything worked fine. Now I see that I have an unread email, but kmail
> doesn't show that unread email. Only the unread count is shown.

If you're sure that you have that commit, I guess we have another culprit. If you find a way to reproduce let me know, it would help a lot. I'm pretty sure it's the same issue, which you should notice instantly. It's not some queue getting stuck or alike, just try something send an email to one of your accounts, and check if you can see it.

(In reply to comment #21)
> Figured it out at last, here's what's happening:
> * To reproduce:
> ** open kmail
> ** open a mail in a separate window
> ** close the mainwindow only (leave the editor/viewer running)
> ** open kmail again => voila you're in the broken state

That's bug 311981.

The result of this bug is that I can no longer keep KMail loaded all the time and have to keep closing it in order that I will be able to actually see any new emails that arrive.

Apparently there's a mix of issues in this report.
Bug 311981 has been fixed, now I need to know what's still broken.

Is it (potentially) filter related?
Are all folders affected or only one particular?
When you get into the error state, are you seeing any new mail arrive at all (is only a subset of the received messages invisible)?
Does it only have to do with the unread mails count, and not with the messagelist itself?
What exact version are you using?

Please summarize again the exact symptoms (and a way to reproduce would be awesome of course).
Thanks

Reproduced with master from this morning

Symptoms: after a while, the folder list indicates there are unread message in a given collection, but once the folder is selected, the message list doesn't show the new items.

Once the issue starts to appear for a given folder, any new message will be hidden

The issue then propagates to the other collections.

Filtering is not involved (filters are run on the server side on my case), neither is the composer window or a message opened in a new window.

I'm still unable to find a way to reproduce the issue, even akonadiconsole doesn't give any clue

Ok that sounds a lot like Bug 311981. It assume the monitor get's disconnected from the model in another place then (damn global variables).

(In reply to comment #29)
> Reproduced with master from this morning
> 
> Symptoms: after a while, the folder list indicates there are unread message
> in a given collection, but once the folder is selected, the message list
> doesn't show the new items.
> 
> Once the issue starts to appear for a given folder, any new message will be
> hidden
> 
> The issue then propagates to the other collections.
> 
> Filtering is not involved (filters are run on the server side on my case),
> neither is the composer window or a message opened in a new window.
> 
> I'm still unable to find a way to reproduce the issue, even akonadiconsole
> doesn't give any clue

Can you confirm though that you cannot reproduce it anymore as described above in comment 21?

You can check if the monitor is indeed disconnected from the ETM by setting breakpoints in: Akonadi::MonitorPrivate::emitItemNotification and  Akonadi::EntityTreeModelPrivate::monitoredItemAdded.

if the ETM slot isn't called after emitItemNotification we have the same problem again as I fixed already in one place.

Another option would be something calling EntityTreeModel::setItemPopulationStrateg( NoItemPopulation ) on the etm in the kmkernel, which would disconnect the signals as well.

Here is my observation (still based on 4.10.2):

> Is it (potentially) filter related?

No. I have no client side filtering, all filtering happens on server via sieve. 


> Are all folders affected or only one particular?

It never happens for all folders simultaneously. It starts with one, others still working normally, and then propagates. Once it has started for a folder that folder will never recover, though.


> When you get into the error state, are you seeing any new mail arrive at all (is only a subset of the received messages invisible)?

I only have an invisible subset.

When the effect appears it seems like all new messages are invisible. 

But if I leave Kmail running I'll occasionally see another message coming in.  No pattern to it, though. Some visible, some not. What was invisible remains invisible. What was visible remains visible.


> Does it only have to do with the unread mails count, and not with the messagelist itself?

The unread count is correct as far as I can tell. I just don't see the messages in the list.


> What exact version are you using?

4.10.2 on Fedora from Red Hat KDE repo.

I made a couple tests (I'll try again tomorrow)

My KMail setup : 4 dimap accounts

Test #1: Launch KMail, select different collections from a given account without selecting new messages or doing anything else than selecting the folder -> no issue for a couple hours

Test #2: Still on the given account, tried a couple RMB/mark all as read and read some messages → the issue appeared 5/10 minutes later

I'm still unsure but I think the issue always appear in the same folder (nothing particular, it only contains some kde-commits@ messages). It then affects the other folders

Just had a slightly different version of this bug. Folder list was showing 1 unread message for a certain folder which I couldn't read and a few moments later some new messages were added to that folder which I could read, but I still can't see that 1 unread message from before.

I have the same bug. I also experience sometimes the issue that I can't move messages from the imap inbox to another folder. I started kontact in a terminal and I get those messages. May be a hint: 
kontact(11188)/kdeui (kdelibs) KXMLGUIClient::~KXMLGUIClient: 0xf17468 deleted without having been removed from the factory first. This will leak standalone popupmenus and could lead to crashes. 
kontact(11188)/kdeui (kdelibs) KXMLGUIClient::~KXMLGUIClient: 0xe0b4a8 deleted without having been removed from the factory first. This will leak standalone popupmenus and could lead to crashes. 
kontact(11188)/kdeui (kdelibs) KXMLGUIClient::~KXMLGUIClient: 0xefd620 deleted without having been removed from the factory first. This will leak standalone popupmenus and could lead to crashes.

Just confirmed this issue is still present in 4.10.4.

Still present in 4.10.4 for me, too.

*** Bug 319673 has been marked as a duplicate of this bug. ***

It seems that I found the reason at least for postgresql backend.

There are a lot of errors in akonadiserver.error log like

cannot execute "SELECT id, name FROM FlagTable WHERE ( name = ( :0 ) )" query
where :0 is '\SEEN'

Indeed
akonadi=> SELECT id, name FROM FlagTable WHERE ( name = ( '\SEEN' ) );
returns 
bytea
ERROR:  incorrect syntax for bytea type.

However '\\SEEN' works fine
akonadi=> SELECT id, name FROM FlagTable WHERE ( name = ( '\\SEEN' ) );
 id |     name     
----+--------------
  7 | \x5c5345454e

Attribute 'standard_conforming_strings' was set 'on' in postgresql.conf


I found a hack in changing type of FlagTable.name column to text type.

So there are tow way to fix the bug:
1. change type of the FlagTable.name column
2. Escape all '\' characters by additional backslash in queries.

Created attachment 81027 [details]
Correct fix.

Correct fix is point 2.

Still present in 4.11.0, mysql backend. Manually restarting akonadi also solves the problem.

(In reply to comment #41)
> Created attachment 81027 [details]
> Correct fix.
> 
> Correct fix is point 2.

This causes an additional problem for me, at least with the MySQL backend and with akonadi-1.10.2: Unread count in the folder list does not reflect actual number of unread messages, but instead shows the total number of messages in the folder.

BTW, there is also a similar query (i.e. itemWithFlagCount on SEEN) in ./server/src/handlerhelper.cpp that the patch does not change. Should it?

I have same problem, KDE 4.11.1, openSUSE 12.3.

In IMAP resource sometimes, KMail shows, lets say, 3 new e-mails in some folder (some folders seem to have this bug more often), but when I go to that folder, I don't see new e-mails. Akonadi is IMHO working, because I see notifications about new e-mails. When using another IMAP client or webmail, mails are there. I have to restart KMail to be able to view those e-mails.

I also experience this issue https://bugs.kde.org/show_bug.cgi?id=324831

*** Bug 324831 has been marked as a duplicate of this bug. ***

Just to be sure, is anybody seeing a wrong messagecount, or is the only bug indeed that the message count is correct but the new mails are not visible in the message list?

This would mean emails arrive correctly in the akonadi server (and are thus visible from akonadiconsole), but are lost somewhere on the way to the kmail messagelist.

So please speak up if you have problems with the messagecount, otherwise I'm going to assume this is only about invisible messages.

To try to isolate the problem a bit (since I can't reproduce it) it would be great if someone could:
* Wait for the bug to appear in a folder
* Switch to another folder and back => Bug still present?
* Create a new tab (ctrl+shift+t) => Is the Bug also present in the new tab?

If a new tab fixes the problem the bug is somewhere above the ETM in the modelstack, otherwise below.

(In reply to comment #40)
> It seems that I found the reason at least for postgresql backend.
> 
> There are a lot of errors in akonadiserver.error log like
> 
> cannot execute "SELECT id, name FROM FlagTable WHERE ( name = ( :0 ) )" query
> where :0 is '\SEEN'
> 
> Indeed
> akonadi=> SELECT id, name FROM FlagTable WHERE ( name = ( '\SEEN' ) );
> returns 
> bytea
> ERROR:  incorrect syntax for bytea type.
> 
> However '\\SEEN' works fine
> akonadi=> SELECT id, name FROM FlagTable WHERE ( name = ( '\\SEEN' ) );
>  id |     name     
> ----+--------------
>   7 | \x5c5345454e
> 
> Attribute 'standard_conforming_strings' was set 'on' in postgresql.conf
> 
> 
> I found a hack in changing type of FlagTable.name column to text type.
> 
> So there are tow way to fix the bug:
> 1. change type of the FlagTable.name column
> 2. Escape all '\' characters by additional backslash in queries.

I don't think this problem is related but it sounds like a genuine bug. Could you please open a separate bugreport including your patch?

Thanks

In reply to comment #46
Message count is correct, new unread mails sometimes don't show up in the message list.

I've never seen the missing messages suddenly appear in an already selected folder, only when entering a folder.

Creating a new tab (and switching folders in it) doesn't seem to have any impact.

Hi Christian,

Message count is indeed correct, as checking in akonadiconsole the "new" message is indeed there. Switching to another folder and back doesn't have any effect, neither does the New Tab have any effect on this. 

So far the only real solution is to restart kmail or to restart the akonadiserver.

Same as above. The message count is correct. new messages just don't show up in the message list.

Ok, thanks for the info.
In that case the bug must be somewhere in Akonadi::Monitor or Akonadi::EntityTreeModel. I'll try to figure out what could go wrong in there.

In order to provide input:

* Wait for the bug to appear in a folder

-> Done.

* Switch to another folder and back => Bug still present?

-> Yes.

* Create a new tab (ctrl+shift+t) => Is the Bug also present in the new tab?

-> Yes.

(all of this on 4.11.1)

input for #324831 (marked as possible duplicate):

* Switch to another folder and back => Bug still present? -> Yes.
* Create a new tab (ctrl+shift+t) => Is the Bug also present in the new tab? -> Yes.
* akonadiconsole shows all the messages

I did the following:
* checked and downloaded emails (inbox)
* mark some new emails as read and move to a sub-folder of the inbox
* some time later the (imho unnecessary) notification with new emails in this folder shows up
* switching to this folder only shows the emails I previously moved there

I have CRM114 installed and do filtering on my imap account. This goes into a spam folder. While other folders are somewhat reliable, this one regularly makes trouble. Today again, I had it show 9 new messages in the left folder tree while not showing any message. I started akonadiconsole and akonadiconsole showed one message for that folder but told me it has 12 messages for 3Mb. So IMHO it affects not only kmail2, but also akonadi

Thanks for the responses so far.

Here's another test I'd need the results from:
* wait for the bug to appear
* close or at least don't use other akonadi apps than kmail
* fire up akonadiconsole
** go to "Notification Monitor", rightclick and check "Enabled"
** go to "Debugger" and check "Enable Debugger" on the top left
* If possible, use webmail to send an email to your account so it will end up in the affected folder. Avoid using KMail or any other akonadi apps to not clutter the logs.
** If this is not possible, move an email using webmail (preferred) or kmail to the affected folder.
* copy the content of the Debugger "KMail Kernel ETM" tab, and make a screenshot of the "Notification Monitor" with all items expanded.
* Double check that the email was actually affected by the bug and note the item id (by checking the item in akonadiconsole).

We're looking for:
* An "Add" notification in the notification monitor signaling that the sent email has been added to akonadi.
** This ensures the akonadi server properly notified about the email being added.
* A "UID FETCH $ITEMID" in the debugger tab
** If this is present, the Akonadi::Monitor properly received the notification and fetched the payload (narrowing the error down to notification delivery from monitor to etm or ETM notification processing)

After that I'll have to start posting patches with debug messages I think.

Thanks for your help!

(In reply to comment #54)
> I have CRM114 installed and do filtering on my imap account. This goes into
> a spam folder. While other folders are somewhat reliable, this one regularly
> makes trouble. Today again, I had it show 9 new messages in the left folder
> tree while not showing any message. I started akonadiconsole and
> akonadiconsole showed one message for that folder but told me it has 12
> messages for 3Mb. So IMHO it affects not only kmail2, but also akonadi

So you're saying:
* new mail count is 9 in kmail in the folder tree
* the messagelist is empty in kmail
* akonadiconsole shows 12 as "Total" count in the collection browser
* akonadiconsole only displays one message in the list

I suspect this is a different bug as it seems akonadi is unable to fetch the messages (and that may or may not be related to the filtering).
Please check what happens if you enable the debugger and then reselect the affected folder (switch to another one and back). Does the debugger show any errors?

Hi Christian, 

see
http://paste.kde.org/pc2cba917/

this time, the incomplete list of emails was shown in akonadiconsole. 
Note that to get clear debugger messages, I had to turn off akonadi-
nepomuk-feeder. Then I moved to inbox, cleared, switched to the other 
folder again and took the snapshot of messages into the paste.bin. 

Regards
 Rigo Wenning

On Monday 30 September 2013 17:38:36 Christian Mollekopf wrote:
> https://bugs.kde.org/show_bug.cgi?id=312460
> 
> --- Comment #56 from Christian Mollekopf <mollekopf@kolabsys.com> ---
> (In reply to comment #54)
> 
> > I have CRM114 installed and do filtering on my imap account. This
> > goes into a spam folder. While other folders are somewhat reliable,
> > this one regularly makes trouble. Today again, I had it show 9 new
> > messages in the left folder tree while not showing any message. I
> > started akonadiconsole and akonadiconsole showed one message for
> > that folder but told me it has 12 messages for 3Mb. So IMHO it
> > affects not only kmail2, but also akonadi
> So you're saying:
> * new mail count is 9 in kmail in the folder tree
> * the messagelist is empty in kmail
> * akonadiconsole shows 12 as "Total" count in the collection browser
> * akonadiconsole only displays one message in the list
> 
> I suspect this is a different bug as it seems akonadi is unable to
> fetch the messages (and that may or may not be related to the
> filtering). Please check what happens if you enable the debugger and
> then reselect the affected folder (switch to another one and back).
> Does the debugger show any errors?

I already have one report claiming that no notifications are visible in the "Notification Monitor" at all, which either means that akonadiconsole has a bug, or that the server emits no notifications for the new email.

It would be great if someone could confirm this, and make sure that notifications are generally visible in the "Notification Monitor" (i.e. when checking mail or moving stuff around).

Also, don't forget to note the item id (and possibly parent collection) if you have debug output (you can also send it directly to me for privacy reasons).

(In reply to comment #58)
> I already have one report claiming that no notifications are visible in the
> "Notification Monitor" at all, which either means that akonadiconsole has a
> bug, or that the server emits no notifications for the new email.
> 
Apparently akonadiconsole doesn't display any notifications in that case, so there seems to be an additional bug there.

Notifications Monitor in AC is indeed  broken due to my batch-notifications change in 4.11. It has been fixed in master, I'll see whether the fix can be backported.

Just confirmed the issue once more in 4.11.3 on Fedora.

(In reply to comment #61)
> Just confirmed the issue once more in 4.11.3 on Fedora.

That was to be expected.
Since the notification monitor should now be fixed, could you repeat the test in comment 55 and let me know what notifications you get (i.e. with a screenshot)?

Thanks

I have a bug that I can reproduce and I wonder if this is related to this bug. I use imap. If I move email from my inbox folder to some other (topic) folder, that already contains lots of messages, kmail only displays the messages that were freshly moved there (e.g. showing 5 messages in the list instead of 500). Akonadiconsole shows the correct number of emails and also the correct body of messages (sorting on date or subject in the browser is difficult and not very usable though...

We have some new debug info:
* It was observed once that the bug seems to only reproduce if the affected folder is not currently selected in kmail. As this was only observed once it would need to be backed up by some further tests.
* When reproduced:
** The server emits the "Add" notification
** The KMail Kernel ETM does not fetch the message
** At the same time the Mail Filter or NewMailNotifier fetch the message.

This indicates that the Kernel ETM monitor either doesn't receive the notification, or for some reason doesn't fetch the message after it received the notification.

(In reply to comment #64)
> We have some new debug info:
> * It was observed once that the bug seems to only reproduce if the affected
> folder is not currently selected in kmail. As this was only observed once it
> would need to be backed up by some further tests.

I have same experience. I don't have any rigorous debug info, only my experience.
And some folders are affected more, than others, although I don't see any pattern in it. It affects IMAP folders and also local spam folder, where e-mails marked as spam are automatically moved by filters.

(In reply to comment #64)
> We have some new debug info:
> * It was observed once that the bug seems to only reproduce if the affected
> folder is not currently selected in kmail. As this was only observed once it
> would need to be backed up by some further tests.
> * When reproduced:
> ** The server emits the "Add" notification
> ** The KMail Kernel ETM does not fetch the message
> ** At the same time the Mail Filter or NewMailNotifier fetch the message.
> 
Actually,  the "MailFilter Kernel ETM" doesn't fetch the message, just like the "KMail Kernel ETM", but "akonadi_mailfiter_agent" does. The statistics are updated though (STATUS).

> This indicates that the Kernel ETM monitor either doesn't receive the
> notification, or for some reason doesn't fetch the message after it received
> the notification.

I have the same problem mentioned in comment #63 as well. Is that related to this bug? If not, is there a bug report for that problem already or should we open a new one?

(In reply to comment #67)
> I have the same problem mentioned in comment #63 as well. Is that related to
> this bug? If not, is there a bug report for that problem already or should
> we open a new one?

Yes we think it is the same bug. https://bugs.kde.org/show_bug.cgi?id=324831 is already a duplicate of this.

Quick update:
I'm pretty sure the bug is related to the ref counting and buffering in MonitorPrivate:
* The bug surfaced shortly after Milian fixed the buffering (778c6a1c933). His commit looks correct, but since the buffering never worked before the bug may have never been exposed.
* The bug seems to affect only monitors that are used together with an ETM that uses LazyFetching.

I'm now trying to understand what this code is supposed to do and what is going wrong, but I think I'm getting closer...

Patches are pending here https://git.reviewboard.kde.org/r/113680/.

While I already once fixed the wrong when I though it was this one, I'm pretty sure I got it right this time as I could finally reproduce the symptoms.

The problem with reproducing was that the bug only appeared after a folder stops being cached, which happens after clicking 10 other folders (hence also the weird "infectious" behaviour). Since I'm apparently not doing that much, I never noticed.

I'm not sure if https://bugs.kde.org/show_bug.cgi?id=324831 is indeed a duplicate, but the problem might be fixed anyways by my patches, we'll have to see.

Hi Christian,

I already updated kdepimlibs with your patch from the reviewboard yesterday and have been running with it during the whole day. Normally the issue occurs multiple times a day on my system, but this time all the unread emails could be seen in the folders. 

So I guess you hit the jackpot with your patch :) 

Thanks for fixing. 

Raymond

Congrats Christian, now waiting for the fix to come down the repositories

I tested this on Fedora 19 and it solves another obviously related bug for me.
As I deleted mails from my inbox and came to last 3-4 suddenly kmail would show
no mails in the mail list. Clicking on to another mail folder and coming back to inbox
would show the remaining mails again. This would go on until the last mail is deleted.
I just tested it and everything is fine with the above patch.

Tested with the new kdepim packages for Fedora 20 and it seems fine now with your patches, thanks for fixing this.

Git commit af4cb1739873d7b95c1b44a23bb06440d0ecb96c by Christian Mollekopf.
Committed on 06/11/2013 at 11:30.
Pushed by cmollekopf into branch 'KDE/4.11'.

Fixed refcounting in ETM and Monitor

This is a combination of several commits in order to fix the buffering
in the ETM.

In particular it incorporates the following changes:

1.

Fixed fetching of items that exited the buffer after being referenced.

After a collection exits the buffer after being referenced,
the monitor no longer emits updates for this collection.
It is therefore necessary for the ETM to refetch the items to get missing updates.

2. (regarding the removal of the MAXITEMS limit)

Don't keep outdated copies of items.

A collection is purged if reference counting is used and a collection
exits the buffer after being referenced. By not purging the items, it becomes
possile that we miss updates, and when refetching the collection because it's
referenced again, we don't emit change notifications because the items were in the model already.

Since we anyways have to fetch all items, we can as well purge all items.

Alternatives:
* compare revisions and emit change notifications if necessary in itemsFetched
* Still emit notifications in the monitor for modifications only

3.

Only buffer a collection after the refcount reaches zero.

Before, a collection that was dereffe'd at least once would
already be buffered (although the refcount is still >0), resulting in
the buffer being occupied by reffe'd collections (which is pointless).
REVIEW: 113680
FIXED-IN: 4.11.4

M  +8    -14   akonadi/entitytreemodel_p.cpp
M  +22   -7    akonadi/monitor_p.cpp
M  +13   -0    akonadi/monitor_p.h
M  +1    -0    akonadi/tests/CMakeLists.txt
A  +188  -0    akonadi/tests/lazypopulationtest.cpp     [License: LGPL (v2+)]

http://commits.kde.org/kdepimlibs/af4cb1739873d7b95c1b44a23bb06440d0ecb96c

On Fedora 19 I just experiences mail from inbox being incorrectly sent to Trash. I had 39
emails in my inbox and I started deleting them one by one. As I was doing this a few times
the subsequent emails also went to Trash. I think this was working with the previous version
of the fix.

(In reply to comment #74)
It is not fixed totally, it is only less of an issue. If you are on imap and move messages from inbox to a folder and you go directly to that folder it only shows the messages you just moved. This is KDE 4.12.2 from KR packages on opensuse. Now click on the neighboring folder, wait 2 seconds and click back on the folder with the moved messages. All messages are displayed. Beforehand, one had to restart kmail2, now juggling folders is sufficient. But this isn't a fix.

(In reply to comment #77)

I can confirm that. KMail 4.12.2. But no IMAP here, only POP3. After manually moving mails from inbox into another folder, that other folder will sometimes only show those new mails. Fix is akonadictl stop / start. Switching to some other folder did not help me, but maybe I should test that again next time more thoroughly.

Same here, with my version of KMail.

What you're reporting is 324831 which apparently indeed is not yet fixed (I reopened it now).
I'm closing this one though because I'm pretty sure this is in fact fixed.