Hi! I had 1-byte mail file size problem when using CRM114 filtering (see [Bug 319226] New: produces 1-byte-sized files on failed move attempts while filtering). I have been copying my main accounts mail and recreated a fresh POP3 resource and imported all my filter rules and added the CRM114 rules with just a minor modification and I wasn´t able to reproduce the issue, filtering worked slow, but fine. (I used different POP3 account which got gets the same mail as my main POP3 account to ensure mails are safe.) Reproducible: Always Steps to Reproduce: So I thought it might be best to reproduce with a clean 100% copy of my original setup. I did 1141 rsync -aAHXSP --del martin/.local/share/akonadi martin2/.local/share 1142 rsync -aAHXSP --del martin/.config/akonadi martin2/.config 1143 rsync -aAHXSP martin/.kde/share/config/akonadi* martin2/.kde/share/config 1144 rsync -aAHXSP --del martin/.local/share/local-mail martin2/.local/share/ […] 1147 rsync -aAHXSP martin/.kde/share/config/kmail* martin2/.kde/share/config 1148 rsync -aAHXSP martin/.kde/share/config/email* martin2/.kde/share/config […] 1150 chown -R martin2:martin2 martin2 while neither the Akonadi of martin nor the Akonadi of martin2 was running. Actual Results: Still I get the Local Folders resources duplicated in the martin2 account and apparently mails duplicated as well. I do have the exact same database, the exact same maildir, the exact same config – why on earth do I see two Local Folders resources? And why does Akonadi seem to copy all the mails from one of the Local Folders resources to the other (and filling the remainder of my /home with it)? After aborting this scenario I got 11GB for ~/.local/share/local-mail on both users, so there nothing seems to be duplicated, but I see: merkaba:/home> du -sh martin/.local/share/akonadi/db_data 1,5G martin/.local/share/akonadi/db_data merkaba:/home> du -sh martin2/.local/share/akonadi/db_data 1,8G martin2/.local/share/akonadi/db_data merkaba:/home> du -sh martin2/.local/share/akonadi/file_db_data 204M martin2/.local/share/akonadi/file_db_data 300 MiB additional stuff in database and 200 MiB in file_db_data when I just told Akonadi to use an existing and I thought *complete* configuration? I sure hope that Akonadi doesn´t store any absolute directory paths in any configuration or database! Well it does: AkonadiAgentServer(7224): akonadi_ical_resource: Can't find incidence with uid "libkcal-224511270.1044" ; item.id() = 1 ItemRetrieverException : Unable to retrieve item from resource: <html>Ungültigen Eintrag erhalten</html> akonadi_nepomuk_feeder(7228) FeederPluginloader::feederPluginsForMimeType: No feeder for type "text/calendar" found AkonadiAgentServer(7224): akonadi_ical_resource: Can't find incidence with uid "libkcal-224511270.1044" ; item.id() = 1 ItemRetrieverException : Unable to retrieve item from resource: <html>Ungültigen Eintrag erhalten</html> akonadi_maildispatcher_agent(7226)/libakonadi Akonadi::GetLockJob::Private::timeout: Timeout trying to get lock. Check who has acquired the name "org.kde.pim.SpecialCollections" on DBus, using qdbus or qdbusviewer. akonadi_maildispatcher_agent(7226)/libakonadi Akonadi::SpecialCollectionsRequestJobPrivate::lockResult: Failed to get lock: "Zeitüberschreitung beim Erhalten der Sperre." akonadi_maildispatcher_agent(7226) OutboxQueue::Private::localFoldersRequestResult: Failed to get outbox folder. Retrying in: 5000 AkonadiAgentServer(7225) MaildirResource::maildirForCollection: RID mismatch, is "/home/martin/.local/share/local-mail" expected "/home/martin2/.local/share/local-mail" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ akonadi_maildispatcher_agent(7226)/libakonadi Akonadi::ResourceScanJob::Private::fetchResult: Resource has more than one root collection. I don't know what to do. akonadi_nepomuk_feeder(7228) Akonadi::NepomukFeederAgent::foundUnindexedItems: FindUnindexedItemsJob failed So Akonadi maildir resource will break whenever a user name changes. Akonadi stores the path to the maildir resource in absolute form: martin@merkaba:~/.local/share/akonadi/db_data/akonadi> LANG=C grep /home/martin/.local/share/local-mail * Binary file collectiontable.ibd matches Binary file parttable.ibd matches I think thats a bug. Anything related to Akonadi except socket file path is relative to $HOME, this is not. Why? How can I change that RID path? Expected Results: Akonadi just used copied configuration without insisting to duplicate Local Folders resource. Akonadi stores relative path to Local Folders in database by default. I really want a 100% copy of my original account so I can test out CRM114 filter rules without risking any of my precious mails one again. I do not feel comfortable with testing with my main setup. I still use my I delete spam mails manually (!) work-around there. This still is with KDE SC 4.10, but I intend to install KDE SC 4.11 + newer Akonadi from Debian experimental soonish. I cannot easily have the same absolute path for testing, so I chose a different user name.
At this moment, unfortunately, the Akonadi database is *not* movable - except for your problem with RIDs, there are also absolute paths stored in the PartTable, so Akonadi will still try to load data from "wrong" ~/.local/share/akonadi/file_db_data folder. This is partially solved in current Akonadi master, which stores relative paths, but only for *newly* created data file (old ones are not migrated).
Thanks. Hmmm, I see. Well since I told Akonadi POP3 to leave mails on the server for 14 days or so, I may be testing this way. I could restore mails that may get broken by downloading Maildir from server and insert any with not found Message-Id manually into Akonadi.
Daniel, in order to progress further with [Bug 319226] New: produces 1-byte-sized files on failed move attempts while filtering I need to be able to do this. I tested the CRM114 filter rules that worked okay in my one POP3 resource test setup with my production setup and again got botched mails. I need to be able to test this safely with a second completely independent local UNIX user and also duplicated POP3 accounts for my main mail accounts which are on my own server. Otherwise I am stuck here as then I could just try to setup everything from scratch and hope that this time it wouldn´t trash mails.
While Dan is obviously addressing Akonadi server internals, this is most likely an issue with most file based resources. I guess the RID of the top level collection would not only need a relative path but also information what it is relative to. Like KConfigGroup replacing the homedir path with $HOME. The question is what is the most expected behavior if the path is relative to a "sub variable". I.e. the default maildir resource points to $HOME/.local/share/local-mail but that in fact is $XDG_DATA_HOME/local-mail and $XDG_DATA_HOME defaults to $HOME/.local/share if unset. what should we use as a base then?
Well, for current Akonadi setups I think the base of the relative path that makes the most sense is $HOME as everything is store there. But for configurability, i.e. in order to be able to separate Akonadi data from $HOME to place them somewhere else than NFS based home directories or whatever reason, it may make sense to reference some other environment variable. While I can deal with my current work-around not to use the CRM114 spam filter rules and well delete the spam that policyd-weight lets through manually, for further attempts to get to the cause of that severe data loss bug, I need a test enviroment that is 100% identical to my current production setup. Thats all I wanted to point out. I am not willing to test this on my real production setup once again, I am not willing to risk loss of mails in order to reproduce that bug once again, despite in a test setup. As a fresh from start setup didn´t trigger the issue, I need to copy the current setup as is. Only other way forward would running a virtual machine where I can have another /home/martin (instead of /home/martin2), but currently I do not have enough space on this SSD to do that.
The configuration I am concerned about is $HOME and $XDG_DATA_HOME overlapping but $XDG_DATA_HOME being set. If $XDG_DATA_HOME is $HOME/data, then the default locations for a couple of resources would be in $HOME/data, e.g. $HOME/data/local-mails. On that system/account using data/local-mail relative based on $HOME would be OK, but is that what the user would assume? Or would the user assume that the relative paths are based on $XDG_DATA_HOME? As for test setup: I use identical configurations, just not a copy of Akonadi's data directory.
This bug has only been reported for versions older than KDEPIM 4.14 (at most akonadi-1.3). Can anyone tell if this bug still present? If noone confirms this bug for a recent version of akonadi (part of KDE Applications 15.08 or later), it gets closed in about three months.
Just as announced in my last comment, I close this bug. If you encounter it again in a recent version (at least 5.0 aka 15.08), please open a new one unless it already exists. Thank you for all your input.
Thank you Denis for taking care to close old bugs. However this one still happens. As this is still quite concise I do not see much sense in opening another one about the same issue. I tested it by starting KMail 5.2.3 with empty home directory. This is what I get: martin2@merkaba ~/.local/share/akonadi/socket-merkaba % mysql --socket mysql.socket Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 60 Server version: 10.0.28-MariaDB-3 Debian buildd-unstable Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> use akonadi; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed MariaDB [akonadi]> select id, remoteId from collectiontable; +----+-------------------------------------------------------+ | id | remoteId | +----+-------------------------------------------------------+ | 1 | NULL | | 2 | akonadi_birthdays_resource | | 3 | /home/martin2/.local/share/contacts/ | | 5 | /home/martin2/.local/share/akonadi_maildir_resource_0 | | 6 | outbox | | 7 | sent-mail | | 8 | /home/martin2/.local/share/apps/korganizer/std.ics | | 10 | /home/martin2/.local/share/notes/ | | 11 | inbox | | 12 | drafts | | 13 | trash | | 14 | templates | | 15 | Test | | 16 | Test2 | +----+-------------------------------------------------------+ 14 rows in set (0.00 sec) MariaDB [akonadi]> % ls -l akonadi_maildir_resource_0 insgesamt 0 drwxr-xr-x 1 martin2 martin2 18 Jan 8 12:33 drafts drwxr-xr-x 1 martin2 martin2 18 Jan 8 12:33 inbox drwxr-xr-x 1 martin2 martin2 18 Jan 8 12:33 outbox drwxr-xr-x 1 martin2 martin2 18 Jan 8 12:33 sent-mail drwxr-xr-x 1 martin2 martin2 18 Jan 8 12:33 templates drwxr-xr-x 1 martin2 martin2 18 Jan 8 12:33 Test drwxr-xr-x 1 martin2 martin2 18 Jan 8 12:33 Test2 drwxr-xr-x 1 martin2 martin2 18 Jan 8 12:33 trash while ~/.local/share/local-mail appears to be empty, except cur, new and tmp directories. So while there is a change as it uses the name of the maildir resource and not "local-mail" anymore, it still has that reference to an absolute path instead of using something relative to $HOME or the suitable XDG directory. At least it seems to store an absolute path just once for each resource and not once for every folder or item as I verified with PimItemTable.