Bug 298275 - After collection is scanned, Amarok loses collection and stats
Summary: After collection is scanned, Amarok loses collection and stats
Status: RESOLVED FIXED
Alias: None
Product: amarok
Classification: Applications
Component: Collections/Local (show other bugs)
Version: 2.5.0
Platform: Fedora RPMs Linux
: NOR major
Target Milestone: 2.6
Assignee: Amarok Developers
URL: https://git.reviewboard.kde.org/r/105...
Keywords: release_blocker
: 273278 (view as bug list)
Depends on:
Blocks:
 
Reported: 2012-04-17 04:44 UTC by Eric Mesa
Modified: 2012-07-26 18:08 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In: 2.6
Sentry Crash Report:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Eric Mesa 2012-04-17 04:44:13 UTC
After upgrading to 2.5 every time amarok is started it will scan my collection and then erase everything but one song.  Then I go into my last good backup of the mysqle database and copy that to ~/.kde/share/apps/amarok/mysqle/amarok and restart it.  Once again, it erases my collection.

Reproducible: Always

Steps to Reproduce:
1. copy good mysqle files to directory
2. start amarok
3. collection erased
Actual Results:  
collection gone

Expected Results:  
collection should remain

output from amarok --debug:

a bunch of lines like this: 

amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://29fbb5717859f0b4bbe125ac7b7df571" url id 25382 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://ff52721d7ab570c866409f461c28d347" url id 25383 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://2bfc6a7ec02cc6480ce3ede5f5099599" url id 25384 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://3697d3fcfa1d584f80f407ba1eb57eff" url id 25385 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://f615b932cf671484ef5f2fb430e217ad" url id 25386 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://1c0063f6f0087d43622913ab09b9a7a0" url id 25387 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://a823ff760f9e60d149e869be6ff2de28" url id 25388 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://ba33bd4bb4cfc17fad3b391e517eb90c" url id 25389 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://fbf48fd9c803bc9b78b49743c018c8ad" url id 25390 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://9a0f8a9ab8deee74a67be341bcc93e22" url id 25391 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://79f5315354ccd86a523a4d1999596c90" url id 25394 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://b6ba3967113f09b7e3c08afe31f15b84" url id 25393 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://78f50f296c0e3f7fd24ea3634315ee6b" url id 25392 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://ccb83cc63bd22e40be7dcc6d8d5bf380" url id 25395 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://9c1dda60a5816c3946b04516462e4884" url id 25396 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://1177b022d9bf701aeb49de86f1270022" url id 24898 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://c6ebe96e0742c5ab80b5075aa36cd90f" url id 25555 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://2db999a64e9583c6d73cb8aa7c4efe3b" url id 24788 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://f1e92fff677a2c507e2cf1653d474a75" url id 25136 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://a4ce9f69bfa99e5763fb5159ca30f356" url id 23541 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://210473c855d88b902e009f54c1285344" url id 23542 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://1071c40d7f75aa33e18e1b4e23d28be0" url id 23543 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://03976f3ee6f769b3ba9543312173c430" url id 23544 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://c39b15200b648be20ea3ca44b2f95e6f" url id 23545 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://6fc8b6de16cf2577d52433d0ff4a9639" url id 23546 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://e4f5a6ebc38832650c9b49a2443f3d69" url id 23547 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://127c589283360a035abd67faa214f60e" url id 23548 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://cf9f13e19dc0daf91d3ad29c51f8a45b" url id 23549 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://2858ded6c4d06bdeec06ddb54194487f" url id 13952 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://8f46243f7c20a3ca8067f9f394214782" url id 22864 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://8aa9b927c4266176e4db505fffe407ab" url id 22865 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://882a4b931055df638ad225789e414f96" url id 22866 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://3116620ea5cbdb162fa96af91a12177e" url id 22867 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://cf2cfc97b9ce7d033b192f40223da645" url id 22868 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://14c2983819301fb362856c9d7aa825cb" url id 22869 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://8e357a5cfdbed40fb5e2eb22856f0204" url id 22870 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://a04576a2b32b143836e65ad70cdc9319" url id 22871 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://55686db84b69b0815aeb158ce85403de" url id 22872 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://43f85d4634f2994902420e4f88682279" url id 22874 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://eed2259d75546ee3dda332474f6df054" url id 22873 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://e6fd0144dd0116f5cb619ae5c4d63efd" url id 22875 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://73a020cb29788aa6a93ba59a1daf9b39" url id 13181 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://2d5c4e81d84427213074458482388962" url id 13185 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://1ed61a704aa1f268a6346c3ed84fd6f4" url id 13187 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://f025351e4072cb2f68090942baca2bae" url id 13192 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://8b275dddc8a517e639117d9522e1a5c1" url id 12969 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://8cc03d44500088a976295e6a4a04d56d" url id 21964 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://1b6a0e3071bd5cbed124943b834b3cc3" url id 25514 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://b25e0862486d87801b36cc196aec3cd6" url id 19520 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://9035749305c0ff3f0e2f42cb2357d00e" url id 19521 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://c6bd44653e2a3ca6130d1c1764aeb51f" url id 19522 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://fe7fe8f77f2c7636acc4c606155d318b" url id 19523 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://bdbe73dbd39651c884bea5fd9efcb4d6" url id 19524 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://47081992f98a3c079742bb3acc211d59" url id 19525 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://cd9448bb2e523667f5cac83affdcb142" url id 19526 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://64c385f0f8a3e9b1b076b61166dba27d" url id 19527 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://76390597e03ca8c5a20da732a9935991" url id 19528 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://bdfdb5ea33dded6a0780b220eda396dc" url id 19529 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://5107822f54b5cabaf7e16574e24fe379" url id 9302 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://ac54ed65cb1dca92a78592bc9e826891" url id 9305 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://e8fbc40214289f324fc920203398177e" url id 9306 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://6160ed75cb8a09de9c3e275e0ca572b2" url id 9312 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://02d2fd270cea94c094dde0accce9b1ee" url id 24789 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://51153604c1ead8f444832d1cc51040b6" url id 24790 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://279c414fbfae43308bddab03ebf5b113" url id 24791 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://91679fc3c22c62d6332a05467368231c" url id 24792 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://43120f97399ea641c83d14894409f6c6" url id 24793 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://c7b4215746292c21c585c6104f4075b7" url id 24794 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://6e741bef679d44f90924b9c498e34e36" url id 24795 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://8e8a2efa65d5cf3376a1e998cff26a98" url id 24796 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://5337c27beec21adb1a2f0f9c59a7c6da" url id 24797 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://72e5e587dd0d621d53c8b9c228967cfa" url id 24798 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://94f5e7a6d08125acd23f4572efed4700" url id 24799 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://06c49903a9d9778c94447c7a5cff3cf4" url id 24800 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://b4bd1785c510d173034ac68a4a691a48" url id 18605 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://fc01646c3ad3d464c3eba989d06df9ef" url id 19130 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://1ad2932cafde1296bd2a8d3af6fab529" url id 19508 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://4a70899e4a5b4bad9faf57f66626520c" url id 19509 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://9396a7d72ecdd5af6931542955593435" url id 19510 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://6ffd14b521f07610385147d80e0833b2" url id 19511 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://dcbe5acd97aa83f4eb85b17343f7e60c" url id 19512 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://3ec08d2f843c016866c0a50032129643" url id 20542 
amarok:       [SqlScanResultProcessor] deleteTrack "amarok-sqltrackuid://cd23ee30fcf92bb90970537844ea9179" url id 22379
Comment 1 Myriam Schweingruber 2012-04-17 16:32:53 UTC
Do you have the option "watch folders for change" enabled? Then Amarok will scan the collection on startup and update it, that is a normal behavior, not a bug.
Comment 2 Myriam Schweingruber 2012-04-17 17:05:58 UTC
Sorry, I completely misread your first sentence and read "I will scan and erase all files but one". Please disregard my comment.
Comment 3 Eric Mesa 2012-04-21 21:48:02 UTC
It just lost my collection again.  this is very annoying.  Let me know what I need to provide to fix this.
Comment 4 Myriam Schweingruber 2012-04-22 09:08:25 UTC
Where is your collection located? Please make sure the location is mounted and accessible when you start Amarok.

FWIW, I can't reproduce this on Amarok 2.5-git of today
Comment 5 Eric Mesa 2012-04-22 19:55:53 UTC
Music is on my /home partition so it's definition mounted.  Is this a known issue if Amarok crashes in 2.5?  That's one thing that happened before the second time the music disappeared.  The next time I started it all the music was gone after the collection scan as described above
Comment 6 Myriam Schweingruber 2012-04-22 21:41:57 UTC
Did you try erasing the amarok* configuration files in $HOME/.kde/share/config/  (might be in $HOME/.kde4/share/config/ depending on the distribution)?

A known issue? That really depends on the backtrace, else this is rather a wild guess. You didn't talk about a crash in this report before, is the db loss happening after a crash? A crash in the database can of course lead to a corrupted table, but you would also get a message on the console. Could you please provide a backtrace?
Comment 7 Eric Mesa 2012-06-28 14:17:21 UTC
Just happened again yesterday.  I was playing music on Amarok.  Then I noticed it kept playing the same song over and over.  Then I noticed that all the music in the playlist had lost its album cover.  Exited Amarok.  Started it again and my collection was gone.  

Here's a neat little fact that might help pinpoint what is happening.  All of my collection was gone EXCEPT the songs I'd added this week.  You'd think it'd be all or nothing, right?
Comment 8 Myriam Schweingruber 2012-06-28 18:49:42 UTC
Weird. I mark it as a release blocker, as this needs to be sorted out before we release 2.6. Thank you for your feedback.

Ralf, Matej: any ideas?
Comment 9 Matěj Laitl 2012-06-29 14:12:23 UTC
Eric, I have an idea what causes it. I've found out that Amarok's collection doesn't survive when you toggle "Local Files and USB Mass Storage Backend" in Configuration -> Plugins. Can you confirm this is the cause? Please do the following:

A1. Enable the plugin, quit Amarok.
2. Restore Amarok db from backup
3. Poke around, trigger "Update Collection" if everything okay do even a Full Rescan. Tracks _and_ statistics (e.g. play counts) should not disappear.

And they try with B1. Disable the plugin, quit Amaork; rest of the steps is the same.
Comment 10 Matěj Laitl 2012-07-09 00:42:13 UTC
Reporters, please apply the patch from https://git.reviewboard.kde.org/r/105488/ on top of the current git and report back. It's worth to read the review request comments. Alternatively, the code is available in my personal repo clone branch: http://quickgit.kde.org/index.php?p=clones/amarok/laitl/amarok.git&a=shortlog&h=refs/heads/scanresultprocessor-bugfix
Comment 11 Matěj Laitl 2012-07-10 16:07:02 UTC
*** Bug 273278 has been marked as a duplicate of this bug. ***
Comment 12 Eric Mesa 2012-07-10 19:06:28 UTC
I haven't messed with that plugin at all.  Usually the songs just disappear after something has happened (don't know what).  It's also often accompanied by some crashes.  But I just read your blog post about your SoC and it appears you've tackled a bunch of collection data loss issues so maybe you've fixed the issue?  I could toggle that plugin to test what you've done, but I never toggled it before so I doubt that's what was going on with me.
Comment 13 Matěj Laitl 2012-07-10 19:17:32 UTC
(In reply to comment #12)
> I haven't messed with that plugin at all.

Eric, just please do what I instructed you in comment #9, note whether bug is reproducible, then apply path from comment #10, redo testing from comment #9, report back whether the bug is fixed by patch.
Comment 14 auxsvr 2012-07-10 19:33:02 UTC
I had the same problem, described in bug 290344. I fixed it by opening the mysql database directly and editing the path and device data stored in it.
Comment 15 Matěj Laitl 2012-07-25 00:21:50 UTC
Git commit 5a0d3d39f6a0f588b6419c6c27e0227b512712e6 by Matěj Laitl.
Committed on 09/07/2012 at 01:41.
Pushed by laitl into branch 'master'.

SqlScanResultProcessor: don't accidentally delete tracks, defensive rewrite

This fixes data-loss that I can trigger every time by toggling "Local
Files & USB Mass Storage Backend" in Config -> Plugins, restarting
Amarok and triggering collection update / rescan.

In theory, more things such as cloning changing disk could trigger this
problem, from SqlScanResultProcessor::deleteDeletedDirectories() comment:

We need to match directories by their (absolute) path, otherwise following
scenario triggers statistics loss (bug 298275):

1. user relocates collection to different filesystem, but clones path structure
   or toggles MassStorageDeviceHandler enabled in Config -> plugins.
2. collectionscanner knows nothings about directory ids, so it doesn't detect
   any track changes and emits a bunch of skipped (unchanged) dirs with no
   tracks.
3. SqlRegistry::getDirectory() called there from returns different directory id
   then in past.
4. deleteDeletedDirectories() is called, and if it operates on directory ids,
   it happily removes _all_ directories, taking tracks with it.
5. Tracks disappear from the UI until full rescan, stats, lyrics, labels are
   lost forever.

Also add a handful of asserts, ScanResultProcessor is very complicated
and small error or corner-case in logic may result in horrible data
losses.

Reporters of linked bugs, please try to reproduce your data-loss with
this patch applied and report back in both cases. In negative case,
please reopen and attach full updated amarok --debug log.

After this patch, only (statistics, lyrics and labels)-loss operations
should be:
 a) moving track out of mounted collection folders [by design]
 b) changing both metadata and url from outside of a track not tagged
    by amarok_afttagger [we can do nothing about this]
FIXED-IN: 2.6

M  +4    -0    ChangeLog
M  +199  -75   src/core-impl/collections/db/sql/SqlScanResultProcessor.cpp
M  +36   -16   src/core-impl/collections/db/sql/SqlScanResultProcessor.h

http://commits.kde.org/amarok/5a0d3d39f6a0f588b6419c6c27e0227b512712e6
Comment 16 Eric Mesa 2012-07-26 18:08:20 UTC
apologies, but I can't do the testing.  I've tried for the past two weeks but haven't been able to find time.  Your patches sound pretty all-encompassing.  If it still happens post-2.6 I'll reopen a ticket and hopefully will be able to find time for testing patches at that time.

Thanks for your awesome help in this important issue