| Summary: | akoandi does not delete mails from .local/share/akonadi/file_db_data | ||
|---|---|---|---|
| Product: | [Frameworks and Libraries] Akonadi | Reporter: | Christian Trippe <christiandehne> |
| Component: | Maildir Resource | Assignee: | kdepim bugs <kdepim-bugs> |
| Status: | CONFIRMED --- | ||
| Severity: | normal | CC: | ab4bd, amantia, auxsvr, cfeck, dennis.schridde, ht990332, juha.heljoranta, kde-bugs, kde, luisfe, marcinkocur, ml, post, prozac, sven.burmeister, t.zell, Usenetmail.Christine, wstephenson |
| Priority: | NOR | ||
| Version: | 4.7 | ||
| Target Milestone: | --- | ||
| Platform: | openSUSE | ||
| OS: | Linux | ||
| Latest Commit: | Version Fixed In: | ||
| Sentry Crash Report: | |||
|
Description
Christian Trippe
2011-09-16 19:12:32 UTC
*** This bug has been confirmed by popular vote. *** Please clarify if you are using maildir or mixedmaildir. To do that turn akonadiconsole, on the Agents page select Local Folders and look at the Identifier and Type in the lower part of the window. It is a maildir resource. As for me, the situation looks like this My "Local Folders" are: Identifier: akonadi_maildir_resource_3 Type: Maildir (~/.local/share/local-mail) But some older mail is stored in "KMail Folders" which have different characteristics: Identifier: akonadi_mixedmaildir_resource_2 Type: KMail Maildir (~/.kde4/share/apps/kmail/mail) Since a few days I use akonadi-1.6.2. The good news is that new e-mails from POP3-servers are only saved for a short moment in this directory, depending on the settings (1) these new e-mails will be deleted when they're not needed anymore. The new "problem" is, what to do with files that are saved in this directory before the update to akonadi-1.6.2. Can I delete these files without losing e-mails and also not lose the flags of these e-mails (read/unread/important/action)? (1): settings are described here: https://bugs.kde.org/show_bug.cgi?id=271231#c10 > The new "problem" is, what to do with files that are saved in this directory
> before the update to akonadi-1.6.2. Can I delete these files without losing
> e-mails and also not lose the flags of these e-mails
> (read/unread/important/action)?
>
Tried this. Emails bodies got lost. Fortunately had a backup. What should I do witch 11233 files?
I've found a solution This is what I did: 1) Quit KMail/Kontact; 2) Stop akonadi (akonadictl stop); 3) Rename file_db_data into file_db_data_old; 4) Start akonadi (akonadictl start); 5) Start KMail/Kontact; 6) Move e-mails where the body is lost to a different folder and back (example: inbox -> draft -> inbox); 7) Wait a while (I was away for a few hours); 8) The e-mails in step #6 should have their body back. The items inside the file_db_data directory are not all e-mails, there are also contacts and maybe also other items. Check other akonadi data after doing this, or maybe you can use grep to find non e-mail items inside the old directory (file_db_data_old). If it doesn't work for you, do step 1 and 2, rename the old file_db_data_old to file_db_data, and then do step 4 and 5. For #6: email bodies are only visually lost, not for real. As you discovered just removing the files are not enough, the database must also be cleaned up, so akonadi knows the files are not in cache and they need to be retrieved again if needed. If you know sql, you can do the cleanup in akonadiconsole. You need to remove (set to null) the data and datasize columns for the respective items in parttable. I deliberately don't give exact command here, so people will not try it and in case of problems blame me... Btw, with the new akonadi server, you might try akonadictl fsck to discover the problematic items. @Andras 4.7.2, akonadi 1.6.2. I only have the mandatory Local Folders maildir resource which is only used for outbound mail, the rest of my mail is in 4 imap resources, both online and disconnected types, with fetch message bodies on demand, keep for 60 minutes. I also have 93000 files totalling 1.2G in file_db_data, some of which is from today. At least some of these files must be headers of the normal disconnected imap cache, correct? Is Christian's summary of this bug at https://bugzilla.novell.com/show_bug.cgi?id=726206 as "This is supposed to be a cache but did not work." accurate? If so, how would I detect any stale files? Disconnected imap mail bodies are kept in the cache, that's correct. This is needed due to the disconnected behavior. A problem would be if you have mails in the cache for maildir or online IMAP that are not deleted after some time. The time you can set in the properties of each folder, start with the top level one as that is inherited by the others. Look at Properties->Retrieval. As for checking for stale items: akonadictl fsck. The result of the check appears in the same place where akonadiserver debug outputs, which is usually .xsession-errors or the console from where akonadiserver (akonadictl start) was started. As for the database (not file_db_data), yes mail headers are stored there for all your mails, sorry there is no way around this. KMail1 also had indexes (.index files), it also had a local cache for disconnected IMAP (and for headers for online imap). I'm not sure how big is the difference between the size of the ~/.local/share/akonadi and the total size of what KMail1 used for indexing and cache, if one does a test, I'm happy to see it. What *could* eat space is MySql not cleaning the db after you remove items, just marking as deleted (usual behavior for databases). You could use from time to time akonadictl vacuum, but this is slow and requires at least once more the size of the database as free space. So, we know all what this bug is about. Lets assume I don't know about it's existance, so I'm a typical user who has stale copies in cache. Are they going to be automatically removed by upstream fix? They should be... (and actually every workaround proposed here is not good enough for me as I don't know sql and can't get on with aconadiconsole... I don't want to break anything; moving messages to other folder is a manual work which even could not be completed due to user's omission). I optimized the tables using mysql (apparently I don't have the "new akonadi server" ... ackonadictl does not have a "vacuum" command, only start, stop, restart, and status, I'm running KDE 4.7.3). But this was to no avail! The ~/.local/share/akonadi/ directory still contains as much data as the ~/.local/share/.local-mail.directory (roughly 2GB), which means that all my messages are stored twice. The snapshot version of 2011-dec-03 has the options "vacuum" and "fsck". You have to compile and install the latest snapshot version or wait until the next release version of Akonadi (current release version is 1.6.2). (In reply to comment #13) > The snapshot version of 2011-dec-03 has the options "vacuum" and "fsck". You > have to compile and install the latest snapshot version or wait until the next > release version of Akonadi (current release version is 1.6.2). Do I need to get a snapshot from git master or are there snapshot tarballs? You can find tarballs here: http://quickgit.kde.org/?p=akonadi.git&a=summary Just make it clear: vacuum compresses the mysql database (not related to files_db_data). fsck find the files from that directory that are not referenced anymore from the database, and moves it to the lost&found folder. @András Manţia So we remove file_db_data, do fsck and then still need to run akonadiconsole and remove items manually? Do not remove the files_db_data manually! Just run fsck for now. After upgrading to KDE 4.8.0, I ran both akonadictl fsck & vacuum but the amount of data in ~/.local/share/akonadi/ did not change at all. The SQL query from the openSUSE bugreport referenced above gives: +--------+-----------------------------+--------------+-----------------------+--------------+-----------------+ | res_id | resourcename | name | count(parttable.name) | datasize_avg | datasize_sum_MB | +--------+-----------------------------+--------------+-----------------------+--------------+-----------------+ | 5 | akonadi_maildir_resource_5 | PLD:ENVELOPE | 29264 | 376 | 10 | | 6 | akonadi_imap_resource_1 | PLD:ENVELOPE | 118 | 426 | 0 | | 5 | akonadi_maildir_resource_5 | PLD:HEAD | 29264 | 1582 | 44 | | 6 | akonadi_imap_resource_1 | PLD:HEAD | 118 | 1302 | 0 | | 5 | akonadi_maildir_resource_5 | PLD:RFC822 | 29264 | 57817 | 1614 | | 6 | akonadi_imap_resource_1 | PLD:RFC822 | 45 | 30357 | 1 | | 7 | akonadi_contacts_resource_0 | PLD:RFC822 | 174 | 159 | 0 | +--------+-----------------------------+--------------+-----------------------+--------------+-----------------+ [mk@linux ~]$ akonadictl fsck [mk@linux ~]$ "Looking for collections not belonging to a valid resource..." "Checking collection tree consistency..." "Looking for items not belonging to a valid collection..." "Looking for item parts not belonging to a valid item..." "Looking for overlapping external parts..." "Found overlapping part data: /home/mk/.local/share/akonadi/file_db_data/18216_r0" "Found 1 overlapping parts - bad." "Verifying external parts..." "Found 11198 external files." "Found 11198 external parts." "Found no unreferenced external files." "Looking for dirty objects..." "Collection "Search" (id: 1) has no RID." "Found 1 collections without RID." "Item "18498" has no RID." "Item "18606" has no RID." [and so on] "Item "1" has RID and is dirty." "Item "2" has RID and is dirty." "Item "3" has RID and is dirty." "Item "4" has RID and is dirty." [and so on] It created a ~/.local/share/akonadi/file_lost+found/ folder which contains only one item. file_db_data has now 1 element less (but it is still ~800MB!). What can I do? Thomas Zell: what is the cache policy for the maildir resource? Right click in KMail on the toplevel maildir folder (e.g Local folders or whatever name you have), Folder properties, Retrieval tab and the "Keep message bodies locally" setting is interesting. mkkot: you should also run the same query (see below), so we can see what is using the space. You can run i t in akonadiconsole'd DB console tab or in a mysql command line (in the latter case you can copy/paste it). select resourcetable.id as res_id, resourcetable.name as resourcename , parttable.name, count(parttable.name), round(avg(parttable.datasize)) as datasize_avg, round(sum(parttable.datasize)/1024/1024) as datasize_sum_MB from parttable,pimitemtable, collectiontable, resourcetable where pimitemtable.collectionid=collectiontable.id and parttable.pimitemid=pimitemtable.id and parttable.name like 'PLD%' and collectiontable.resourceId = resourcetable.id group by parttable.name, collectiontable.resourceId; Thanks András. I'm not actually a database geek so here's a screenshot ;) http://pliki.netau.net/images/screen134.png akonadi_maildir_resource_5 takes 831MB. Volker (main Akonadi server developer) says that if the database was created priort to 4.7.4/4.8.0 the bug will persist, as there is no mechanism yet to do an automated cleanup of the problematic items (most probably those Item "X" has RID and is dirty items). A possible way to solve this would be: - recreate the database (see http://userbase.kde.org/KMail/FAQs_Hints_and_Tips#Clean_start_after_a_failed_migration). A radical way, but will get the cleanest setup. - add a new maildir resource in KDE 4.8.0 and after that remove the old one. You still need to do the same fixes for filters, favourite folders as in the first case. Removing the old one might take some time until the db is cleaned up. An akonadictl vaacuum is recommended after the removal is finished. I'd like to know if after doing the above the problem persist (the cache keeps growing for maildir) or not. Eg. for I have 76542 items that use only 2MB, or in other words the file cache is almost empty for the maildir mails. Thank you very much András for your answers! BUT: Before I delete the whole Akonadi database (recreating it will render my computer unusable for many, many hours ... I already did that twice with versions prior to 4.7.4 and that were horrible experiences I would want to avoid going through again): Is there any way to manually empty the mail cache without deleting everything else? From a naive point of view, it should be possible to delete all mail bodies stored in the database with a simple SQL query? I have no problem fiddling with SQL queries but I am completely unfamiliar with the Akonadi database structure and I would be glad about some pointers ... > I'd like to know if after doing the above the problem persist (the cache keeps
> growing for maildir) or not.
Thanks, after cleaning up database I have 4 files which take 97kb. Anyway, I think you still should do something for the people who aren't even aware of this bug.
Ok, after deleting the "Local folders" resource (which then seems to be readded automatically by akonadi), running akonadictl fsck and vacuum, the db size dropped to 11MB. The whole .local/share/akonadi directory still takes up more than 200MB which I find quite a lot (after deleting the "lost+found" directory created during the previous step). Now, I've imported some more mails using kmailcvt. The contents of file_db_data jumped up to 87MB. There clearly is something wrong with this. btw: During the whole procedure all information on attachment and encryption status that is normally shown in the message list was lost. It doesn't even reappear when clicking on a message. I can reproduce this problem, also with kmailcvt. I am using akonadi 1.7.2 provided by Debian testing/sid with postgresql backend. If I import a maildir from kmail1 using kmailCVT 4.8.4 then: - The new mails are physically stored in file_db_data. But they are supposed to be only cache? - No new mail is stored in .local/share/local-mail, the unique new dir created is one named KMail-Import but it does not contain the imported folders. - akonadictl fsck reports thousands of items without RID. Can anyone reproduce the problem with KDE 4.9.x and a new akonadi database? newly created user akonadi 1.8.51 compiled from sources, mysql backup kde 4.9.3 kmailcvt 4.9.3 kmail 4.9.3 A directory imported - The new mails are physically stored in file_db_data. But they are supposed to be only cache? - No new mail is stored in .local/share/local-mail, the unique new dir created is one named KMail-Import but it does not contain the imported folders. - akonadictl fsck or akonadictl vacuum do not seem to affect the result. I do not get the RID warnings though (In reply to comment #30) > - No new mail is stored in .local/share/local-mail, the unique new dir > created is one named KMail-Import but it does not contain the imported > folders. Your mails are probably in .local/share/.local-mail.directory and/or in .KMail-Import.directory (note the dot - these are "hidden" directories") (In reply to comment #31) > Your mails are probably in .local/share/.local-mail.directory and/or in > .KMail-Import.directory (note the dot - these are "hidden" directories") No, they are not kdec@mychabol:~/.local/share$ du -h local-mail/ 4,0K local-mail/sent-mail/tmp 4,0K local-mail/sent-mail/cur 4,0K local-mail/sent-mail/new 16K local-mail/sent-mail 4,0K local-mail/.inbox.directory/KMail-Import/tmp 4,0K local-mail/.inbox.directory/KMail-Import/cur 4,0K local-mail/.inbox.directory/KMail-Import/new 16K local-mail/.inbox.directory/KMail-Import 4,0K local-mail/.inbox.directory/2012/tmp 4,0K local-mail/.inbox.directory/2012/cur 4,0K local-mail/.inbox.directory/2012/new 16K local-mail/.inbox.directory/2012 36K local-mail/.inbox.directory 4,0K local-mail/outbox/tmp 4,0K local-mail/outbox/cur 4,0K local-mail/outbox/new 16K local-mail/outbox 4,0K local-mail/inbox/tmp 4,0K local-mail/inbox/cur 4,0K local-mail/inbox/new 16K local-mail/inbox 88K local-mail/ kdec@mychabol:~/.local/share$ du -h akonadi/ 780M akonadi/file_db_data 4,0K akonadi/db_misc 212K akonadi/db_data/performance_schema 4,0K akonadi/db_data/test 1016K akonadi/db_data/mysql 27M akonadi/db_data/akonadi 174M akonadi/db_data 954M akonadi/ And I confirm that the files in akonadi/file_db_data are the imported mails. (In reply to comment #32) > (In reply to comment #31) > > Your mails are probably in .local/share/.local-mail.directory and/or in > > .KMail-Import.directory (note the dot - these are "hidden" directories") > > No, they are not > > kdec@mychabol:~/.local/share$ du -h local-mail/ You checked the wrong directory. Try (as I wrote above) cd ~/.local/share/.local-mail.directory and check what you find there. I have no .local-mail.directory on this new user account. kdec@mychabol:~$ du -h .local 4,0K .local/share/Trash/info 4,0K .local/share/Trash/files 12K .local/share/Trash 4,0K .local/share/local-mail/sent-mail/tmp 4,0K .local/share/local-mail/sent-mail/cur 4,0K .local/share/local-mail/sent-mail/new 16K .local/share/local-mail/sent-mail 4,0K .local/share/local-mail/.inbox.directory/KMail-Import/tmp 4,0K .local/share/local-mail/.inbox.directory/KMail-Import/cur 4,0K .local/share/local-mail/.inbox.directory/KMail-Import/new 16K .local/share/local-mail/.inbox.directory/KMail-Import 4,0K .local/share/local-mail/.inbox.directory/2012/tmp 4,0K .local/share/local-mail/.inbox.directory/2012/cur 4,0K .local/share/local-mail/.inbox.directory/2012/new 16K .local/share/local-mail/.inbox.directory/2012 36K .local/share/local-mail/.inbox.directory 4,0K .local/share/local-mail/outbox/tmp 4,0K .local/share/local-mail/outbox/cur 4,0K .local/share/local-mail/outbox/new 16K .local/share/local-mail/outbox 4,0K .local/share/local-mail/inbox/tmp 4,0K .local/share/local-mail/inbox/cur 4,0K .local/share/local-mail/inbox/new 16K .local/share/local-mail/inbox 88K .local/share/local-mail 780M .local/share/akonadi/file_db_data 4,0K .local/share/akonadi/db_misc 212K .local/share/akonadi/db_data/performance_schema 4,0K .local/share/akonadi/db_data/test 1016K .local/share/akonadi/db_data/mysql 27M .local/share/akonadi/db_data/akonadi 174M .local/share/akonadi/db_data 954M .local/share/akonadi 954M .local/share 954M .local I have tried a couple of times more and now I cannot reproduce the problem. I erased everything in .local .kde and .config and start over and this time the import is correct. I will try to see if I can reproduce it again and how. I created a complete new setup with KDE 4.9.2 or 4.9.3 and have collected a few entries but cannot figure out a pattern. Using only pop3 accounts a maildir resource christian@asterix:~> ls -alh .local/share/akonadi/file_db_data/ insgesamt 1,5M drwxr-xr-x 2 christian users 312K 15. Feb 21:38 . drwxr-xr-x 6 christian users 4,0K 15. Feb 20:59 .. -rw-r--r-- 1 christian users 11K 18. Nov 22:31 1117_r1 -rw-r--r-- 1 christian users 5,6K 19. Nov 21:53 158908_r0 -rw-r--r-- 1 christian users 5,3K 20. Nov 19:25 158919_r0 -rw-r--r-- 1 christian users 5,8K 20. Nov 20:21 159353_r0 -rw-r--r-- 1 christian users 5,3K 21. Nov 06:49 159527_r0 -rw-r--r-- 1 christian users 7,4K 21. Nov 19:26 159852_r0 If there is any additional information or debugging I could try feel free to tell. The only guess I have so far is that this might be related to filtering as I think I have only seen this for mails which were run through a filter when downloading. I see that still a lot with kde 4.14.2 akonadictl fsck keeps putting a lot of emails in file_lost+found I found something: Most of the emails that get stuck in "/home/hussam/.local/share/akonadi/file_db_data/" are emails that were junk filtered into trash folder and permanently deleted later but are somehow still in "/home/hussam/.local/share/akonadi/file_db_data/". Perhaps force-remove an email from file_db_data when kmail deletes it from trash as part of "Empty local trash folder on program exit" as a dirty fix? baloosearch -e some_keyword_in_deleted_email returns no results. The deleted email is still in /home/hussam/.local/share/akonadi/file_db_data/ and if I run akonadictl fsck, it gets moved to the lost+found folder. so the email was correctly unreferenced but not purged from cache. |
