Bug 324925

Hi!

My old e-mail messages in the maildir folders do not get indexed.  The new ones are, as they arrive.
I think akonadi_nepomuk_feeder does not send to nepomuk all the due "notifications"  (sorry for this improper term).

If I remove and add back the maildir resource, the mails are still NOT indexed.
If I force a reindex via the "Folder Properties"  window, the mails are still NOT indexed.
If I create a NEW maildir resource and copy the old messages to the new one, MOST of them get indexed, others don't.
After two months playing around with this, I realised that it is just necessary to TOUCH the files within the maildir folders to get akonadi_nepomuk_feeder send "notifications" to nepomuk, which actually starts indexing, without the need to create a new maildir resource to move the old messages to.
BUT: Still, not all the messages are getting indexed. I have 80,000 mails and only the half of them gets indexed.

In the console terminal (and in ~/.xsession-errors) I can read a lot of these messages:
akonadi_nepomuk_feeder(1104) ItemQueue::fetchJobResult: Not all items were fetched:  2 100

So, I tried to touch less mails at once, splittng the nested maildir folders in smaller sub-sets, and it works better. In the end, I strongly assume that the touch-workaround works, but the huge number of updates cannot be handled by akonadi all at once.
It might also be related to the system limits, but I'm not sure how to change them.

I managed to find a solution by touching the files in one folder at the time, waiting for nepomuk to finish indexing, and then touching the ones in the next folder. That could be done with a script (attached below), querying the NepomukFeeder agent via qdbus.

Hope this helps!

NOTES:
1. You have to touch the FILES in the folders, not the folders themselves.
2. Several other people have been repeatedly reporting similar issues in the past, since 2010, but I think I've come closer to the problem with this version of KDE (4.11).


Reproducible: Always

Steps to Reproduce:
1. Add an already-existing maildir resource to akonadi

and/or:

1. right-click on the maildir resource in Kmail,
2. choose "Folder Properties"
3. go to the "Maintenance Tab" 
4. click "Force reindexing"

Actual Results:  
No e-mails in the maildir resource are indexed by nepomuk.

Expected Results:  
The e-mails in the maildir resource should be indexed by nepomuk.

* my ulimit configuration:
ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 30
file size               (blocks, -f) unlimited
pending signals                 (-i) 96113
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 99
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 96113
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

* my kernel:
Linux luna 3.10.10-1-CHAKRA #1 SMP PREEMPT Thu Aug 29 21:57:49 UTC 2013 x86_64 GNU/Linux

* I have 12 GB RAM and a quad-core i7 @3 GHz.

* The script I used to force reindexing:
#!/bin/bash

LISTA=(
# full path of the "cur" folders in the several maildir resources
# one folder per line, within double quotes
# You can get a list of folders with something like:
# find ~/Mail -type d -name cur -exec echo \"{}\" \;
"~/Mail/Friends/cur"
"~/Mail/.Friends.directory/closest ones/cur"
)

function wait_for_start_of_indexing()
{
   ciccio=0
   while [[ "`qdbus org.freedesktop.Akonadi.Agent.akonadi_nepomuk_feeder  / org.freedesktop.Akonadi.NepomukFeeder.isIndexing`" == "false" ]]; do
      (( ciccio > 2 )) && break # wait max 120 seconds = 2 minutes!
      sleep 60
      (( ciccio++ ))
   done
}

function wait_for_end_of_indexing()
{
   ciccio=0
   while [[ "`qdbus org.freedesktop.Akonadi.Agent.akonadi_nepomuk_feeder  / org.freedesktop.Akonadi.NepomukFeeder.isIndexing`" == "true" ]]; do
      (( ciccio > 10 )) && break # wait max 600 seconds = 10 minutes!
      sleep 5
      (( ciccio++ ))
   done
}

# Touch all files in a maildir folder,
# wait for nepomuk to start indexing
# wait for nepomuk to stop indexing
# and then move to the next folder

for ((i=0; i<${#LISTA[@]}; i++)); do
   echo  ${LISTA[i]}
   echo touch \"${LISTA[i]}/\"\* | bash
   wait_for_start_of_indexing
   wait_for_end_of_indexing
done

########### end of script ###########