Bug 209822 - amarok albums missing in collection after upgrade to 2.2.0
Summary: amarok albums missing in collection after upgrade to 2.2.0
Status: RESOLVED FIXED
Alias: None
Product: amarok
Classification: Unclassified
Component: general (show other bugs)
Version: 2.2.0
Platform: unspecified Linux
: NOR normal (vote)
Target Milestone: ---
Assignee: Amarok Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-10-08 00:56 UTC by Martin Schwinzerl
Modified: 2009-10-26 19:13 UTC (History)
3 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
Screenshot describing the first mentioned problem (433.26 KB, image/png)
2009-10-08 00:59 UTC, Martin Schwinzerl
Details
Screenshot describing the second mentioned problem (370.88 KB, image/png)
2009-10-08 01:01 UTC, Martin Schwinzerl
Details
handling for default tag "[mb track uuid]" (1.26 KB, patch)
2009-10-16 21:35 UTC, Egbert König
Details
simplify some SQL queries during collection scanning (4.92 KB, patch)
2009-10-16 21:38 UTC, Egbert König
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Schwinzerl 2009-10-08 00:56:12 UTC
Version:           2.2.0 (using 4.3.1 (KDE 4.3.1), Arch Linux)
Compiler:          gcc
OS:                Linux (i686) release 2.6.31-ARCH

Problem description:
------------------------------------------------------------------------------

Yesterday, I did an system upgrade of my Arch Linux computer, including an update to amarok 2.2.0 from 2.1-1. Upon the first start of amarok after the 
update, the configuration screen poped up, the directories for my local collection was selected properly. After confirming the dialog, the application (seemed) to rescan the directories in question.

After it was finished, several inconsistencies in the collection appeared:

- Tags for several albums seem to have been mangled/mixed up in the process 
  of updating/rescanning the collection.
  Example: The albums for one artist are present, yet contain no files.
  Adding the files by hand via the file interface of amarok shows that
  amarok believes to belong to a different artist, yet neither the album
  nor the files turn up in the list for the (wrong) artist. The tags of the
  files seem to be correct, though (at least according to kid3).
  The albums and the files were present and associated with the correct     
  artists in the collection prior to the update.

- Some albums turn up multiple times, even though the files in question are      
  only present in location. 

I will attach screenshots depicting these problems below, if necessary I can provide further information (e.g. a dump of the local database, although instructions for doing this would be appreciated). 

Restarting the application (even after an unrelated reboot) did not solve the problem, neither did several attempts to rescan the collection. 


Many thanks for your efforts, please let me know if you need more information.
Best regards

Martin
Comment 1 Martin Schwinzerl 2009-10-08 00:59:24 UTC
Created attachment 37439 [details]
Screenshot describing the first mentioned problem

depicting problems
 - "Empty" albums with files attributed to wrong artists
 - The album does not show up as belonging to the wrongly identified album   
   either

Note that the information given by kid3 is stored within tag2, tag1 seems to be empty
Comment 2 Martin Schwinzerl 2009-10-08 01:01:39 UTC
Created attachment 37440 [details]
Screenshot describing the second mentioned problem

Problem depicted:
- Search result does not contain the "correct" files, even though they are present on the system, part of the directories that got scanned and the tags are correct (According to kid3).
Comment 3 Myriam Schweingruber 2009-10-08 01:14:38 UTC
Did you try a rescan, then restart Amarok?
Comment 4 Martin Schwinzerl 2009-10-08 07:52:59 UTC
@Myriam Schweingruber

Yes, I did so, with no effect (should have formulated this more clearly in my post, sorry for that). My next step is to remove/rename

~/.kde4/share/apps/amarok directory
~/.kde4/share/config/amarok_homerc
~/.kde4/share/config/amarokrc

and start all over again from scratch. Would that be Ok, or do you need further information from these directories/files prior to that?

Best regards
Martin
Comment 5 Myriam Schweingruber 2009-10-08 11:01:00 UTC
Try touching the folder with the missing album first to change it's time stamp.
Comment 6 Martin Schwinzerl 2009-10-09 20:54:32 UTC
Touching (both the parent folders as well as the ogg files in question) had no effect, regardless of whether I initialised a rescan manually via the "rescan collection" button from the "Configure Amarok" dialog or via Tools -> Update Collection.

But what did the trick was 

a.) actually moving the directory with one of the "wrong" albums outside the directory tree that is searched by amarok
b.) rescan (via Settings -> Configure Amarok -> Collection -> Rescan Collection)
c.) move the directory back to it's initial position once the rescan was finished and finally
d.) rescan again, repeating the steps from b.)

Since a lot of albums are affected (Last count was > 20, not counting individual files that seem to have gone missing in the collection here and there), it is probably more efficient to start from scratch and save the content of the amarok folder for future reference.

I can live with this workaround but would still be interested to learn what caused this hickup in the database, hence it would be nice if you (or anybody else knowledgeable) could share your thoughts on this problem.

Many thanks for your efforts & best regards
Martin
Comment 7 Jeff Mitchell 2009-10-09 21:27:15 UTC
Martin,

I'll try to think about what could have caused this. It might be related to a regression (which will be fixed in 2.2.1) where some dirs, if they weren't picked up in an initial full rescan, would not be picked up in subsequent incremental scans, even if touched.

As for tags showing up messed up -- did closing Amarok and re-opening help? In 2.2 those are probably collection browser issues (which closing and opening refreshes). A full rescan, which should have been kicked off upon first upgrade, should have fixed things in the database itself (other than that regression, which may have caused your problem).
Comment 8 Martin Schwinzerl 2009-10-10 08:53:17 UTC
@Jeff Mitchell

Thank you very much for your quick response.
Restarting amarok did not help, neither did - unsuprisingly, I know - rebooting the computer or logging out and in to my KDE session.

Just for the sake of completeness: I did a system - upgrade ( including amarok) on my laptop using the very same packages as for my desktop computer (which is the one this bug report is all about) and so far everything seems to be Ok, including no peculiarities in the collection browser. I'm in the process of re-initialising the amarok settings on my desktop back to a clean state and will report back in case any problems occur. 

In case you or one of the other amarok devs are interested, I can provide you with a tarball containing the amarok specfic contents of my ~/.kde4 directory prior to this restart operation.

Regards
Comment 9 Jeff Mitchell 2009-10-10 15:51:07 UTC
Martin,

That would actually be quite nice, as I can dump that out to a SQL file and load it into my MySQL server for examination. I'm interested in knowing why this would happen, since results for the vast majority of people are that 2.2 finds albums that 2.1 had missing for various reasons, not the other way around  :-)

However, the data might be somewhat meaningless to me without enough context -- are you ever on IRC or IM? If I can catch you there at some point then that would probably help quite a bit. Let me know (email me your details if you don't want them on this public bug report  :-)  ).
Comment 10 Egbert König 2009-10-16 21:33:37 UTC
I had too issues with the collection after upgrading to Amarok 2.2. Songs were missing in the collection that were - according to easytag - tagged correctly. I have debugged the database and Amarok. I have found out, that those songs have been false positive recognized as duplicates. I will add a patch to this bug which solves this problem. In addition I will add a patch which simplifies some SQL queries.
Comment 11 Egbert König 2009-10-16 21:35:30 UTC
Created attachment 37625 [details]
handling for default tag "[mb track uuid]"

This patch prevents false positive duplicate detection for MP3 files with the default tag "[mb track uuid]".
Comment 12 Egbert König 2009-10-16 21:38:57 UTC
Created attachment 37626 [details]
simplify some SQL queries during collection scanning

This patch simplifies some SQL queries during collection scanning. Concrete it avoids selecting ID's into QStringList to convert them to (potentially very) long WHERE clauses. It uses instead SQL subselects directly in the WHERE clause. So a lot off data need not to be transferred between the SQL server and the application. Also there might be limitations to the length of SQL statements or WHERE clauses which could cause failures during collection scanning.
Comment 13 Jeff Mitchell 2009-10-17 03:29:43 UTC
Re: comment #11, thanks!

I didn't know that MusicBrainz had default values -- I'd only seen tracks that simply had empty values. Patch applied in 15a3865.
Comment 14 Jeff Mitchell 2009-10-17 03:33:30 UTC
Egbert,

Regarding comment #12 -- most of the patch is great, thanks -- I'd seen those queries in passing (didn't write them) and figured they could be sped up, but I'm not really a SQL guy and although I've used subqueries in the past I rather forgot about them. I can see this really being a nice speedup.

However -- again, not being a SQL guy -- can you explain to me exactly what is being done in the last two parts of the diff? I trust that it's correct, but I'd like to have an idea what it's doing before I apply it.

Thanks!
Comment 15 Martin Schwinzerl 2009-10-17 07:16:58 UTC
@Jeff Mitchell

Please note the mail I wrote to your mitchell@kde.org address which contains
a link to the dump of the contents of my home directory. If you need additional information and/or somebody else needs the dump too, please don't hesitate to drop me an email or comment here.

Thanks in advance
Comment 16 Egbert König 2009-10-17 18:57:09 UTC
Hello Jeff,

the last 2 changes were to get debug prints also for the temporary tables that 
are being used during the collection scan.

First off both:

the temporary tables are created with the same columns as the global tables. 
So it is possible to select the columns for these tables using the names of 
the global tables. This must be done, as the temporary tables are obviously 
not recorded in INFORMATION_SCHEMA. This was the reason that debug prints did 
not appear for the contents of the temporary tables.

Second off both:

the temporary tables are not part of schema "amarok" but they are part of the 
schema "session". The easiest way to solve this problem was not to qualify the 
schema while generating debug prints. This should be O.K. as normally there 
will not be different schemes with the same tables in the amarok database.

By the way, I like amarok 2 much, especially the wikipedia and the lyrics 
plugin and also the speedup in collection scanning. While amarok 1 needed 5-6 
hours for my collection, amarok 2 does it in less than 30 minutes.

Best regards,

Egbert


Am Samstag, 17. Oktober 2009 03:33:31 schrieb Jeff Mitchell:
> https://bugs.kde.org/show_bug.cgi?id=209822
> 
> 
> 
> 
> 
> --- Comment #14 from Jeff Mitchell <mitchell kde org>  2009-10-17 03:33:30
>  --- Egbert,
> 
> Regarding comment #12 -- most of the patch is great, thanks -- I'd seen
>  those queries in passing (didn't write them) and figured they could be
>  sped up, but I'm not really a SQL guy and although I've used subqueries in
>  the past I rather forgot about them. I can see this really being a nice
>  speedup.
> 
> However -- again, not being a SQL guy -- can you explain to me exactly what
>  is being done in the last two parts of the diff? I trust that it's
>  correct, but I'd like to have an idea what it's doing before I apply it.
> 
> Thanks!
>
Comment 17 Jeff Mitchell 2009-10-18 02:39:46 UTC
Egbert,

Thanks for clarifying. Committed in ed7fcde.

I'm glad you see such large speedups in scanning with Amarok 2. Hopefully everything is still actually being scanned  :-)

Most people have seen huge speed increases with Amarok 2.2 compared to 2.1 but for some reason some people have seen large slowdowns. If you're a MySQL guy and are willing to help out, take a look at bug 210201.  :-)
Comment 18 Jeff Mitchell 2009-10-26 19:13:12 UTC
Since I believe that this has been fixed, I'm closing it. Please re-open after 2.2.1 comes out if it's still a problem.