I recently tried to migrate a large maildir structure from one machine to another, by creating an archive first on one machine and then importing the archive on the other. The import, into a temporary "imported" folder, seemed to have worked alright. I then tried to move the imported folders to their new destination. This worked fine for the first two folders, so I got a bit carefree and drag-n-dropped several folders to their new locations before the old moving operation had finished. A few hours later I checked back in and found that the process seemed to have finished, but some of the subfolders (at their new locations) were completely empty, the messages were gone, seemingly both from the database (not showing up in KMail) and from the file system (the respective maildir folders were empty). Not sure if this is related to bug #322697. KMail: 5.6.3 Qt: 5.9.2 KDE Frameworks: 5.37.0 (all installed as 17.08.3 on Debian testing)
A small addition: I was able to reproduce this by moving a single folder (which contains subfolders) to a different location in the same maildir structure: -+-A | +-B +-C +-D Here I was trying to move C into A so that the structure would look like this: -+-A | +-C | +-D | +-B In the folder overview D looked as if it had still a large number of unread mails, but when I entered the folder, the number of unread mails was reset to zero and the folder was empty.
OK, I think I can fully reproduce the problem now and have found its reason: Steps to reproduce ------------------ 1) Create a KMail archive, it should contain a folder ("somedir") and some subfolders and sufficiently many emails (in my case, a total of 700 emails was enough). 2) Create a target maildir directory, e.g. "target" 3) Import the archive into a folder, e.g. "imported" 4) Shortly after the import has finished, move the imported folders, e.g. "imported/somedir" to "target". Expected result --------------- 1) The moved folders should have an identical content compared to the archive's source. Actual result ------------- 1) Some, or all, of the emails are missing. Workaround ---------- 1) Wait until the folder structure on the file system exists and has been populated. This can be checked e.g. by watching the output of: find ~/.local/share/akonadi_maildir_resource_0/.imported.directory/ -type f | wc -l 2) Only when the number looks reasonable and does not change any more, start moving the folder. Notes ----- Since in this case, the bug only happens after importing mails, maybe the severity can be downgraded, because no data is lost as long as the archive is still available and the problem has been noticed soon enough.
I think that may be related to [Akonadi] [Bug 364114] New: moving a folder within one maildir resource is extremely slow and inefficient and thus [Akonadi] Foreign payload https://phabricator.kde.org/T630
Ah, I missed to mention my current work around: I move the folder in the filesystem and let Akonadi update the database. That still takes some time, but on my system it is much, much faster than doing it within KMail and thus Akonadi itself. More detail about this in the other bug report I mentioned.
Hmmm, okay, here it is about moving the folder before the import in the background has finished. But yet also there, as a work-around, just moving it in the filesystem should be faster. And it should safe, as it circumvents that Akonadi first stuffes mail into the database and then puts them into the destination location. It may confuse Akonadi at first, but I bet it will pick up the move in the filesystem as it uses inotify to watch the maildir folders.