Bug 258555

Sorry for the crippled sentence.
Actual Results:  
Only one of the two albums shows up.

Amarok might have been saved the Amarok UID in the tags. It used to do that and it might still be doing it in special cases.

The same problem appears when you have tracks without a stored UID which by chance have the same album/artist/title, e.g. tracks for a life recording and the actual album.

In my opinion Amarok should give a new random UID for such files.

The problem is restricted to mp3 files, i was not able to reproduce it with ogg or wmas.

I am using amarok 2.3.1 and have found that it does not allow more than one track with the same title+album+artist, even if other tags are different and the files themselves are different.  This is a limitation for some collections e.g. multi-CD live box sets with multiple versions of the same song (thinking Live Trane by John Coltrane).  My workaround is to add some "meta-data" to the song title field, e.g. "Impressions - version A", but this is a hack as "version A" is not really part of the song title.

@davidstuartbruce: this is a report about Amarok 2.4-git which is the developer version ahead of yours.

I agree.
Since Amarok exists it's computing an unique ID for every track.

The unique ID is fetched from the track (e.g. a MusicBrainz id or a stored Amarok ID) or computed from the title/artist/album tags.

When importing tracks duplicates are rejected.

Now there might be a reason for this, but the result is the confusing Amarok behavior.

I plan to change this for the next release (probably 2.5).
Please vote on this bug so that I have more reason to push this change through.

This is a very serious bug.  Here is why.

1.  I had some low-bitrate MP3 versions of certain tracks.
2.  I got higher-bitrate MP3 versions of the same tracks.
3.  I used Amarok to copy them to my collection.
4.  The files were organized properly, but only the low-bitrate versions appeared in the collection.
5.  After much hair-pulling, I deleted the low-bitrate versions, expecting the higher-bitrate versions to appear.  But they did not.
6.  I Updated the collection over and over.  No change.
7.  I rescanned the entire collection 3 times.  No change.
8.  As a result, I cannot see 9 of the 20 tracks in the album in Amarok.  The files are in the directory, but Amarok just won't show them in the collection at all.

This is a HUGE REGRESSION and it makes Amarok nearly USELESS.  What's the point of using a music player app that won't show my music?!  

And if rescanning the collection from scratch won't do it, what will?  I was going to dig into the MySQL files to try to remove whatever is stopping Amarok from adding the tracks to the collection, but I can't find a simple way to do that, and I have little time for it.

So what can I do now?  Throw out my collection db and lose all my ratings and tags?  That's not a solution at all.

This bug should have #1 priority for the next release of Amarok, and it should be patched in distros that have older versions.  As it is I guess I'll have to stop using Amarok if I want to listen to my music.

By the way, this is NOT a "normal" bug--this is a serious regression, an
apparent-dataloss bug, and major-broken-functionality bug.

Hi Adam,
sorry that you are frustrated.

I hope that we can change the current Amarok behaviour of rejecting duplicate tracks if just enough people speak out against it.
So don't forget to vote for this bug.

Regarding your problem: There is a "full rescan" button in the settings menu. Did you try this?
Or, if you like, you can try to "touch" (update the modification date) of the directory with the high-quality tracks and then do a "normal" update.

Hi Ralf,

Thanks for your kind reply.  As I wrote, I did try the full rescan several times, but apparently Amarok didn't remove the old tracks from the collection db even after I deleted the files, so I suppose it still thought the newer tracks were duplicates.

I managed to get them into the collection, but I'm still not sure exactly what did it.  I basically removed ALL of the album's tracks from my collection directory on-disk, did a full-rescan, restarted Amarok, and used Amarok to Copy-to-Collection all of the tracks again.  I think I may have tried changing the album tags too, or something.  Eventually it worked, but it was a major hassle, and 99% of users will just quit using Amarok and never try it again, I'm afraid.

I think Amarok should let the user decide what to do with duplicate tracks--he might have a good reason for them.  Amarok shouldn't silently disappear tracks just because Amarok thinks they are duplicates, and because Amarok thinks the user won't want to see them.  I had just bought higher-bitrate versions of the tracks in that album, and I REALLY WANTED the "duplicates" to be in my collection!

Bottom line: the user's data seems to disappear without explanation, and most users will never touch Amarok again.  :(

1. Tracks not being deleted when they are on the file system is another problem.
2. Tracks not being imported because Amarok thinks they are the same is surprising for the user.

Also it's kind of unfair. I have many tracks that are the same but, because they are in different compilations, they are imported twice.

Ralf,

As we've talked about several times on IRC, do not change this behavior.

Most users that we have been in contact with over the years with duplicate tracks do not *want* them to show up, because the collection is not a collection of files, it's a collection of tracks. What you're talking about is turning it into a collection of files -- but we already have the file browser for that. The fact that some users might vote for this bug doesn't trump the many, many users that we've been in contact with over the years that won't be getting on this random bug report.

What you have here is a corner case where the user has multiple bitrate versions of the same file and actually wants both versions to appear in the collection.

The proper solution in this case is to run amarok_afttagger on the files. Since (unlike MBIDs) it's unique to files, instead of tracks, the identifiers will subsequently be different, and the collection will scan them in. It's an easy fix already built into Amarok for cases just like this.

Again, as we've talked about, the right solution is not to suddenly change the behavior that was agreed upon by developer consensus and via interaction with users over a long period of time. And, if you want to do that, using a random bug report with some votes does not give you the okay to go do whatever you want. You need to start a thread on amarok-devel and get developer consensus on your new proposed scheme.

Hi Jeff,
You probably noticed that I disagree.
I also think that a lot of users get surprised (you probably noticed that word from our discussion) that not every file on the disk is displayed by Amarok.

I also think that there should be a better way to detect duplicate tracks than outputting a warning on the console.

However I won't change this behavior.
That won't keep me from voting on this however.

Again, that comes down to whether you consider the Collection to be a collection of music or of files. Whenever this topic has come up, developer consensus has been to treat it as a collection of music, not of files, because we have the File Browser for browsing files. If you want to change that, put a proposal on amarok-devel.

However, like many things, this doesn't have to be a binary solution. Here's my suggestion:

1. Take note of all duplicate tracks
2. Write out a file in Amarok's data dir showing a mapping of the files: A <=> B
3. At the end of a scan if duplicates were detected, give a notification that duplicates were detected. If they click the notification, show some information about where the file exists, and maybe an option to "do not show again"

FWIW -- it was floated in the past to make this an option of the scan (of course if you have competing embedded identifiers you need to autogenerate an identifier for one or the other track), but Mark and Ian were strenuously against it.

Adam,

I took a look at the code. If your file types are MP3, running amarok_afttagger won't currently help because somebody switched the preference ordering of the embedded UIDs for MP3s. I've switched it back so you'll either have to use Git or wait until 2.4.1 for that to work for you.

Hi,

I'm not sure if this is exactly the same issue Adam ran into, but I
have had enormous problems in the last couple weeks with Amarok not
importing all my music files within my ~/ogg directory.  I'm running
amarok 2.4.0 from Debian Sid.

1.  I first had a problem with different tracks with the same
artist/album/title not showing up, because amarok assumed they were
identical even though they were completely different files (actually
different live versions of the same song).  After some investigation,
I settled on making the title tags different by adding e.g. "version
3" to the song title.  This works OK but is sort of a hack - really
the assumption that "artist/album/title" constitutes a unique
identifier isn't always valid.

The situation is even worse for classical music collections, where the
concept of "album" doesn't fit so well.  For my collection, I use the
name of the composition (e.g. "Symphony No. 9 in D minor, Op. 131") in
the album field, and the "name" (really tempo indicator) of each
movement for each track title.  For "artist" I use the composer - it
doesn't make sense to organize a classical music collection primarily
by the name of the orchestra or soloist.  So, for multiple recordings
of the same composition, they are treated as duplicates unless I stick
some extra "metadata" into the track titles.  As flexible as amarok
is, it may be possible to group "Genre = Classical" recordings by
Composer rather than by Artist, and still group popular recordings by
Artist, but if so, it isn't obvious.  It's just a fact of life that
music collection software is geared overwhelmingly to the needs of
popular music fans, not classical collectors.

2.  However, there is more to the collection scanning problem than the
above.  Generally, everything works fine with albums I've ripped from
CD and let amarok tag during import.  But with mp3 purchases from
Amazon, which come pre-tagged, amarok has failed dramatically.   I
recently purchased three Hendrix albums (Electric Ladyland, Are You
Experienced, and Axis Bold As Love) as mp3 downloads, and amarok
choked on all of them.  The albums themselves appeared in the
collection, but only two or three tracks on each album showed up.  I
tried the "fully rescan entire collection" several times to no avail.
I unchecked my "ogg" directory, did a rescan, and then selected it
again, and when my collection was re-populated, I was missing about
400 out of 1300 albums!  (of course the files were still there).  I
eventually removed the entire ~/.kde/share/apps/amarok directory,
re-imported my entire collection, and all my old albums were back, but
I still was missing most of the tracks on the new Hendrix albums.

So, I concluded that there was something wrong with Amazon's tags, so
I copied the album files, deleted all the tags, and tried letting
amarok guess the tags based on the artist/album/title file directory
layout, and it worked fine.  So now I'm back to having a working
collection.  However, I very nearly game up on amarok.
(Banshee/pulseaudio had no collection issues, but had unacceptable
playback problems, otherwise I'd be a *former* amarok user).

Don't get me wrong, I appreciate the work of Free Software developers,
being one myself, but I think there are enough current issues with
amarok's music collection scanning to make it an unacceptable choice
for non-technical users.   (During all of this, my wife kept taunting
me, saying "why don't you just use iTunes on our Mac - it actually
works!).

David Bruce

David,

Can you comment on whether you ran into these problems with Amarok 2.3?

Thanks,
Jeff

Hi Jeff,

> Can you comment on whether you ran into these problems with Amarok 2.3?

I'm not certain exactly when I went from 2.3 to 2.4 with a debian
aptitude safe-upgrade.

I *think* the first issue (not showing tracks with identical
artist/album/title) was on 2.3.

The latter, more serious issue (mp3 albums downloaded from Amazon only
having a couple of tracks show up, and having lots of albums
"disappear" when I deselected then reselected my music collection
directory) was definitely with 2.4.

I'm willing to do whatever testing would be helpful to pinpoint these
problems.  Should I try to install 2.3 in a separate directory (like
inside $HOME) to try to come up with a smaller, reproducible test
case?

David

Please have a look here:
http://userbase.kde.org/Amarok/Manual/AdvancedFeatures/CollectionScanning

Also for tracks not getting scanned, have a look at the amarok debug output (start amarok with --debug)

You will see it amarok considers tracks to be duplicates.

Hi,

I have been having exactly the same problem as David, so I will not describe it again. It is safe to say that since 2.4 (not 2.3 iirc) I have not seen about 60% of my tracks on the track list.

Is there a way to disable this unique id feature and return it to the previous behavior? Perhaps a command line option, or better yet, a collection configuration. I understand you implemented it because some people might like it, and i respect that. But it is ruining the whole amarok experience for me and possibly some others.

So a on/off configuration would solve things, in my opinion. I, for one, really want to see all the files I have on the track list, not the tracks. It is better to err by duplicating data than by losing data.

Arthur

I really hope they reject this "option" in a newer release. At this moment i have more than 25000 tracks that amarok not add in my list.

The option amarok have since a longer time - To remove duplicate is enough i think.

Just to clarify things: 
Amarok should only reject duplicates (but Amarok might consider a track with the same name, artist and album as duplicate)

If non-duplicate tracks are not included then try the aft-tagger thing.
Looking at the debug output might also help in figuring out why tracks are not detected.

Amarok 2.4.0 (My impression from reading this bug is that no changes were made?)

Thought I'd leave my experience in here if it helps:

I already own a compilation album and yesterday downloaded a album by the same artist with some tracks overlapping obviously. (note: different albums here). I force all my albums to use MusicBrainz tagging - (many of the download stores fail badly at tagging) and after running the tagger, some tracks from the new album weren't showing up.

Much head-scratching later, running Amarok in debug mode told me that it was indeed duplicates. I went back and forced my tagger to retag the old compilation and it still refused to show up. (To be clear, I was running a full-rescan after almost every step here)

Eventually I ran afttagger on that directory and its' now started showing everything up. I'm not sure what Amarok uses to generate these ids, so its' possible that the issue is with MusicBrainz tagging? Should Amarok be relying on this?

I saw something above about a collection of files vs tracks, and in this case its an album, so definitely a collection of tracks, and I like to listen to my tracks as an album ;)

Yes,

please vote for this bug so we can change the old design decision.

After posting my comment, I came across this: 

http://amarok.kde.org/wiki/Amarok_File_Tracking

So I guess in my case its the MusicBrainz identifiers. In which case, shouldn't I be able to tell Amarok not to listen to those ids? Or be able to specify a file as in 2 albums, whichever. (Although my neatness makes me favour the first solution)

That's not completely true.

Amarok will use the aft-tag if present.
You can ensure that all files have an up-to-date aft tag by executing the amarok_afttagger.

However, since the aft-tag is a checksum over the track title, artist and album, you will still have the same tags.
Amarok will reliably detect the tracks as duplicates and not import them.

Yes, but I don't want to be running afttagger over all my files, its just an extra step I'd rather not go through. 

I'm perfectly happy with my aft-tag being a checksum over title,artist,album, I'm just not sure I should have to run afttagger to get tracks to show up in multiple albums. It'd be nice in my case if there was a way to turn off AFT or at least the MusicBrainz identifier usage.

*** Bug 279606 has been marked as a duplicate of this bug. ***

I used EasyTag and Picard for renaming all my mp3's. The Problem is: I have some Artists that have lots of Live CD's or some Best of... As an Example: I ripped all my Bon Jovi Albums, Tags with this Programs upside and now i have the Problem that i have some Albums with a tracklist that looks in this way (just an example):
1. bla bla
3.bla bla
8. bla bla
14. bla bla
...

It was nice before they updated Amarok - Before i had all my Songs in my List - And most of all. A full Tracklist on every Album... I think i will change my Music-Player if they change nothing...

Greets Chris

I switched do Clementine months ago and haven't looked back since. It organizes my songs like Amarok used to do and plays them too - that is all I ask for.

If Amarok ever fixes this so-called "feature", then I'll think of switching back. This bug has all the votes I can chip in.

Working on it together with some other changes.

Don't forget to vote for this bug.

Thx for the idea - I will switch to Clementine too - Till this is fixed.

Anyway, it's really annoying.
If there would be a hint how to solve the conflict it would be nice.

If software is shipped that was OK before why not give the user a chance to enjoy?

It seems that the "merged view" button has something to do with this. When I unclick it, it shows the big "local collection" bar and doesn't do this problem, but if I click it again, it gets all clever and combines different tracks with different track numbers into the same.

Hi all,
yes, the merged view will further combine results (which is kind of a reason behind the merged view)

*** Bug 275422 has been marked as a duplicate of this bug. ***

I'm voting for this. I felt that I should add my 2 cents after skimming some of the comments, because I was shocked to see that anyone would actually want this behavior. I don't think that the collection browser should be treated as _just_ a collection of tracks, or _just_ a file browser. It should be neither of these. It should be treated as a searchable database of all of the music in my collection. (I would love to see this expanded upon, but that's a subject for a couple dozen wish list items.) The point is that if I'm looking for songs by Joan Jett, I want to see _all_ songs be Joan Jett, not just unique song titles. There are many reasons why separate tracks may have the same artist/title. They can be different versions, different bitrates, different albums, and so on. Currently, for example, I can only see about half of the songs in "Greatest Hits." So what do I do if I want to add "Greatest Hits" to a playlist? I can't!

For the record, I am using 2.4-git. Please fix this ASAP!

Your right Gareth Bogdanoff - All this reasons you write makes Amarok unusable for big collections... Clementine is now my favorite in music-management. 

I can say for me: I dont care about a fix that needs such a long time. I dont care about amarok now.

Bizarre. SO just having multiple versions of the same track breaks things ?
It seems easy enough that on this error path the UID can be replaced with a new GUID and Amarok can carry on.

Tom,
It can be quite hard to change initial design decisions.

Your solution would work with one small drawback.
If a track is deleted the statistics will be stored together with the UUID.
If the track comes back then the statistics will be found.

If Amarok would use self-generated random internal UUIDs then that would not longer be possible.

But apart from this small hitch it would work.
Do you want to make a patch and submit it to the kde-reviewboard?
The file that you probably want to edit is called SqlScanResultProcessor.cpp

Cheers,
Ralf

The level of C++ is beyond me, sorry.
Maybe a hash of the file would work better than a new random one, then it could be found...

This bug also causes files to become orphaned when doing 'Organize Files.' Just so you know. Is any progress being made on this bug, or are we all just shouting at a brick wall at this point?

This makes amarok absolutely unusable for me. Is anything being done about this?

It even does it to tracks that aren't even really the same track. Lets say I have two tracks: track 1, and track 1 (live version). They both get the same uuid and amarok only shows me the live one... come on.

(In reply to comment #43)
... 
> It even does it to tracks that aren't even really the same track. Lets say I
> have two tracks: track 1, and track 1 (live version). They both get the same
> uuid and amarok only shows me the live one... come on.

Then you should tag your tracks accordingly, if it is different this should be visible to the collection.

The tracks title is different, so it is tagged differently... No other media player has trouble displaying both tracks.

Which version of Amarok are you using? I can't reproduce this here with Amarok 2.5-git of today, I have several versions of Beethoven's 9th and those show all in the collection.

"Maybe a hash of the file would work better than a new random one, then it could
be found..."

Take the first 50kb, hash it. Done. Or use Nepomuk as database....

Amarok is using a hash of the file if there are no tags.

I am just writing the fix to correct the problem.
A little more patience or contact me if you want to provide some help.

Please don't mistake frustration for a bug that is annoying with us not being grateful for the fix :-)

I am currently using Amarok 2.4.3 on Kubuntu 11.10.

I have 2 tracks with 2 copies each - one copy is mp3 and the other is flac. Both copies of both tracks appear in the collections for me.

This bug ist still not fixed in Amarok 2.5.0!!!

I have 28000 files  - only 25000 show up in amarok. It's all up to false duplicates - and, yes, everything is correctly tagged. And by the way: how wants to tag 3000 files manually again after verifying perfect albums with picard when reading the files with a lot of effort from original CDs. 

Using 2.6.0. I think afttagger works somehow but then you have to rerun it on every new album you import. Making sure there's not again two songs the don't show up ... 

I think the design as now simply is a conceptual mistake taken all the individual ways individuals use there music collection. A huge ambitious project like amarok should have a very solid, stable and flexible foundation of collection management. With all the discussion here I think it's sometimes forgotten that people try to use KDE and Amarok in a real effort to work with it. They don't want to change their complete music collection which they have build up over years to another concept and they get pissed when amarok messes up things by regression. 

I always liked amarok but when I tried to really use it on my whole collection things started to mess up. So why not run afttagger on new files automatically (after confirmation by the user) so deliberate duplicates are shown because someone wants them to show up in their "Best Of" or sampler album. I want to play a Yethro Tull album as it was recorded, yes three tracks are also on other of there albums, in another context, serving another musical purpose - don't take that away! 

And I vote for this issue, 
thx for reading, 
piedro

This bug is still present in Amarok 2.6.0!!! it's really annoying. I like amarok, but this bug makes me looking for an alternative.

Interesting snippet from my latest attempt at importing my library into amarok 2.6:

When commiting track /home/brandon/Music/ACDC/1976 - Dirty Deeds Done Dirt Cheap [2003 remaster] (V0)/2 Love at First Feel.mp3 with uid amarok-sqltrackuid://mb-b20a8a3c-c540-49ef-929c-6e18df33b0b2 we detected that the same uid is already commited. This means that you most probably have duplicates in your collection folders. The offending track is /home/brandon/Music/ACDC/1976 - Dirty Deeds Done Dirt Cheap [2003 remaster] (V0)/6 There's Gonna Be Some Rockin'.mp3.


Totally different track, different file size, different file name, different tags, yet still identified as the same track?

(In reply to comment #54)
> Interesting snippet from my latest attempt at importing my library into
> amarok 2.6:

How do you import your library? Normally a full collection rescan, then importing the statistics and ratings from the previous database is the way to do it.

Absolutely no offense to the Amarok crew.  This bug has been around for two years and the target keeps getting pushed out.  Due to this Amarok is unusable for me.

For those suffering from this bug I highly recommend you check out Clementine, the collection scanner works great on my huge collection.

(In reply to comment #55)
> (In reply to comment #54)
> > Interesting snippet from my latest attempt at importing my library into
> > amarok 2.6:
> 
> How do you import your library? Normally a full collection rescan, then
> importing the statistics and ratings from the previous database is the way
> to do it.

That was a full rescan on a fresh install of amarok.

Here is a workaround I found (works for me at least): run the "aft_tagger" to create unique IDs for Amarok. Then rescan and all the files show up. Someone told me aft_tagger has nothing to do with it. Well, I don't care. Works for me. It*s not a fix at all - it's a clumsy workaround: First it takes a long time on a huge collection and you have to run it also on every new batch of files you want to include into your collection ...  and make sure you set the right options to not renew all the IDs created before to speed things up ... as I said: it's clumsy ... 

I tried Clementine now and I have to say I am impressed though Amarok has some nice features that clementine lacks ... but at least it works for the basic things. 

piedro

Sorry for being unprecise:  
"amarok_afttagger -r -v -q" is the command I used ... 

sry

Another workaround is, to remove all UID tags with Kid3. But I don't know how to remove all UID Tags in one step. I'm to lazy to remove the UID tags of 2500 albums seperately. I use Clementine now. I will try the next version of Amarok again.

@semmelrock

Consider using puddletag, it's a very sophisticated masstagger. 

p.

Created attachment 76149 [details]
Amarok 2.7 Beta showing a dialog with duplicate tracks found during collection scan.

Commit http://commits.kde.org/amarok/a44bf6e4 introduced a dialog that apprears upon collection scan where duplicate tracks are found as shown in attachement 76149. The link leads to http://userbase.kde.org/Amarok/Manual/Various/TroubleshootingAndCommonProblems#Duplicate_Tracks

Also note that many problems reported here looking like "Amarok only shows 2500 of my 6000 tracks" were in fact scann result processing errors fixed in 2.6.

Another thing to note is the algorithm used to compute unique id of a track:
1. if an explicitly generated Amarok uid or MusicBrainz uid is found in track, use it
2. else, create hash of a) file size, b) first 16 KB of a file c) *all* metadata stored in file tags (including comment etc.)  and use it.

Above being said, I'm convinced this bug should be considered resolved.

Git commit 23c180b6bc05d54d9c4c40e2f9dd559621a2a1ab by Matěj Laitl.
Committed on 03/01/2013 at 11:22.
Pushed by laitl into branch 'master'.

TagHelper: provide meaningful default implementation of render()

render() is used to calculate unique id of a track that doesn't have it
embedded. Providing default implementation means that track metadata of
more obscure formats (MP4, ASF) will have the metadata contained in
the unique id hash in addition to file size and first 16 KB of a file.

This can change hashes of existing files, but thanks to recent fixes,
Amarok can cope with it and update them as needed. (even in playlists)

M  +4    -1    shared/tag_helpers/TagHelper.cpp

http://commits.kde.org/amarok/23c180b6bc05d54d9c4c40e2f9dd559621a2a1ab

Sorry but I don't get it. 

Does this mean now just I get informed that 1500 files are ignored? 
How is this a fix? 

I already know that they are ignored if I look up an album and track 2, track 4, and track 7 are missing. How can I ever listen to this album as it was meant by the artist when there is no crosslink to the omitted files? 

What am I missing here? 

thx for clarifying, 
piedro

I agree. The bug still exists, on my point of view. The devs tried to solve a too-smart-for-its-own-sake feature-bug by throwing more smartness at it. But some fires cannot be fought with more fire. I think the whole feature should be nuked, but I honestly don't care about Amarok anymore. This feature-bug made me go for Clementine and I see no reason to go back, specially now that it is here to stay.

Anyways, at the very least I think this bug should have been closed with WONTFIX, since that is much closer to the opinions of the reporters. In the future, anyone else who notices the tracks missing and who looks into this bug report for answers will be really confused by the FIXED status - it doesn't make sense to those affected.

Hello Arthur! 

Don't you think this is a misunderstanding? 

No reasonable developer will consider the pure notification as a fix. This would be outright ridiculous and stupid. my guess is, we just don't understand exactly what has been done to solve the problem of missing tracks by omitted duplication. 

So I still hope one of the devs will explain how this is supposed to work now, please ... 

thx for reading and plz explain someone, 
thx piedro

(In reply to comment #65)
> Does this mean now just I get informed that 1500 files are ignored? 
> How is this a fix? 

Piedro, are you referring to an actual beheviour you experience with Amarok 2.7, or are you just extrapolating it from the comments?

> I already know that they are ignored if I look up an album and track 2,
> track 4, and track 7 are missing. How can I ever listen to this album as it
> was meant by the artist when there is no crosslink to the omitted files? 

With what Amarok version do you experience this?

Especially after commit 23c180b6bc05d54d9c4c40e2f9dd the "duplicate tracks" means bit-by-bit same copies, which cannot be for tracks that have different track number.

First of all thx Matej for your comments, I guess I get a better picture now! 

My question refered to the discussion, but I updated to amarok 2.7. I cannot test this reliably because I used the amarok_afttagger (-r -v -n *) utility to make sure that nothing gets counted as duplicate - so I can only check it out with new files (which I don't have so many at the moment to produce duplicates again). 

So the question for me is whether I still in the future have to run every new album through the amarok_afttagger process before adding it to the collection? 

The missing tracks I have had when using amarok 2.6 still and when not using the amarok_afttagger. 
I am glad to hear that the defintion (and thereby detection) of duplicates obviously has changed and that by bit by bit comparison there are only few scenarios conceivable in which this should be a problem. 

So thx for your remarks, I will post my experience with the new system ...

with best regards, 
piedro

(In reply to comment #69)
> My question refered to the discussion, but I updated to amarok 2.7. I cannot
> test this reliably because I used the amarok_afttagger (-r -v -n *) utility
> to make sure that nothing gets counted as duplicate - so I can only check it
> out with new files (which I don't have so many at the moment to produce
> duplicates again). 

You can remove the unique ids using the same utility. (it can eat your kittens though)

> So the question for me is whether I still in the future have to run every
> new album through the amarok_afttagger process before adding it to the
> collection?

Not at all! If it was the case, it was indeed a bug (and not this one).

So then it is a bug! 

I ripped the Randy Newman collection of my girlfriend, converted to mp3 and tried to imprt into amarok 2.7.0 on KDE 4.10. As you mentioned before I got a message about diplicates. the origin of the duplicates is different albums - but yes it is the same title. 

This cannot be the wanted behaviour in any way! 
The message telling me about the dupes doesn't help, because trying to reimport gives the same 
message over and over. This seems a waste of time ... - the only reliable solution so far is the suggested aft_tagger method. 

Why not use a aft_tagging process before importing automatically on every file? 
If you don't like that, make it optional: a "create Unique identifiers during import" tickbox in the options is all it does need ... 

piedro 

the importer message: 
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/2-23 Happy Ending.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Randy Newman's Faust/02 Can't Keep a Good Man Down.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/2-21 Can't Keep a Good Man Down.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Land of Dreams/03 Four Eyes.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/2-18 Four Eyes.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Born Again/10 William Brown.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/2-08 William Brown.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Little Criminals/09 I'll Be Home.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/2-04 I'll Be Home.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Little Criminals/06 In Germany Before the War.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/2-03 In Germany Before the War.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Little Criminals/01 Short People.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/2-01 Short People.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Little Criminals/10 Rider in the Rain.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-27 Rider in the Rain.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Little Criminals/08 Baltimore.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-26 Baltimore.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Sail Away/09 Memo to My Son.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-17 Memo to My Son.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Sail Away/08 Burn On.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-16 Burn On.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Sail Away/04 Last Night I Had a Dream.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-14 Last Night I Had a Dream.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Sail Away/02 Lonely at the Top.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-13 Lonely at the Top.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Sail Away/01 Sail Away.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-12 Sail Away.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/12 Songs/09 Old Kentucky Home.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-11 Old Kentucky Home.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/12 Songs/04 Suzanne.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-10 Suzanne.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Randy Newman Creates Something New Under the Sun/11 Davy the Fat Boy.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-06 Davy the Fat Boy.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Randy Newman Creates Something New Under the Sun/10 I Think It's Going to Rain Today.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-05 I Think It's Going to Rain Today.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Randy Newman Creates Something New Under the Sun/09 The Beehive State.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-04 The Beehive State.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Randy Newman Creates Something New Under the Sun/08 Cowboy.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-03 Cowboy.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Randy Newman Creates Something New Under the Sun/02 Bet No One Ever Hurt This Bad.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-02 Bet No One Ever Hurt This Bad.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Randy Newman Creates Something New Under the Sun/01 Love Story (You and Me).mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-01 Love Story (You and Me).mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-25 Kingfish.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Good Old Boys/08 Kingfish.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-24 Louisiana 1927.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Good Old Boys/06 Louisiana 1927.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-23 Guilty.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Good Old Boys/05 Guilty.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/1-21 Birmingham.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Good Old Boys/02 Birmingham.mp3
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Guilty_ 30 Years of Randy Newman/2-14 Take Me Back.mp3
/media/Turtle/Musik/Alles perfekt/Newman, Randy/Trouble in Paradise/09 Take Me Back.mp3

As expected: Aftzer I ran amarok_afttagger over the collection those duplicates disappeared. 

But there are still diplicates: 
- they are all *.m4a (aac I guess)  
- they are all afttagged 
- they are all perfectly tagged by musicbrainz picard 
- the definitely have different tracknumber and different albumnames and identifiers

Example: 
Duplicates found, the second file will be ignored:
/media/Turtle/Musik/Alles perfekt/Chemical Brothers, The/Exit Planet Dust/01 Leave Home.m4a
/media/Turtle/Musik/Alles perfekt/Various Artists/Cool Vibes, Volume 2/1-06 The Chemical Brothers - Leave Home.m4a 

Please offer at least an easy to hanle  workaround - at the momnet there isn't even a way to control which file is ignored! 

As in the example above - either in the album "Exit Planet Dust" track number 1 is missing in the album (and on the harddisk) or the "Cool Vibes" sampler misses track number 6. This behaviour means that after importing and deleting the imported source files there are files missing in the collection. If my girlfriend wants to listen to the album "Exit Planet Dust" on her portable or in the car, she is loading the album into her portable device only to find out later that track number 1 is missing. This  is even more irritating in particular by the fact that the most popular songs are more likely to be missing because they are the most likely to have duplicates ... 

discouraged, piedro

This is neither resolved nor fixed! 

p.

Couldn't there just be an option to disable duplicate detection? Seems like a win-win.

Exactly, Piedro - neither resolved nor fixed. As I have been trying to say, the silly dialog message and the afttagger trick are just symptomatic treatments - the problem lies underneath, and it is still there, being called a feature. The problem is that Amarok is trying to uniquify the tracks using some sort of smart and/or brute heuristic - and when it fails to do that, a message pops saying that it is somehow the user's fault and that the user has to jump through hoops to basically just see their own files on the library. It should not be trying to uniquify the tracks on the library - that is the problem. I understand if some smarter duplicate detection would be implemented in the "organize files" feature, but putting that kind of restriction, always enabled, on the library is very wrong. I have been working for many years with complex data aggregation for BI on DW and I know that forcing a natural key out of a data that has no natural key is the path to false positives on aggregates - the worst kind of error for aggregations - and hashing the data solves nothing (think of links, example below). You need a primary key that is completely artificial and to stop trying to uniquify the user's things - if it is not unique for the user, it shouldn't be for Amarok.
Piedro, I use an unorthodox media workflow using several directories in home. One for each media type, such as ~/Audio, one for new files, ~/Downloads, and one for LAN-shared read-write stuff, ~/pub and so on. This means that copying or moving files around would be very bad (wasteful and time consuming), so I usually just hard link files around - it is fast, no redundant copies lie around and I always find what I want in any place that makes sense for the data to be. Also, I can delete stuff from Downloads and pub with no fear of losing important files. But can you imagine the mess that Amarok does when I tell it to keep track of two or more of these directories and to keep the library up to date? The dialog shows up frequently and I can't keep afttagg'ing things all the time - files pop in and out of existence all the time here without my knowledge (which is fine in Clementine - a smart playlist shows me the new ones that people sent me over the LAN and the recently downloaded). And requiring user intervention every time Amarok gets confused is just silly.

Good suggestion, Brandon, I gave it two years ago, so +1.
I also suggested that the bug should stay open or, at least, marked as wontfix, since that is the opinion of (at least one of) the reporters/voters/affected users and others will come looking for this in the future, and fixed is definitely not right - not even in 2.7.0, which is what I retested a couple of weeks ago.

Arthur, I completely agree - it's silly to have to intervent as a user all the time. Even worse - I don't see what the right intervention might be ... 

- copying files into the amarok library manually to make sure I have them in the album folder? 
doesn't work cause I still miss them in amarok (and get duplication errors everytime the database refreshes ... 

- symlinking the removed files with the duplicates to keep the album folder integrity? 
same problem as above ... 

- as I stated before the AFTtagger workaround doesn't work anymore, the afttagger did give the files unique identifiers but still they are detected as duplicates? ... how, wtf? 

I "Rediscovered My Music" with clementine and found around 1000 tracks more than amarok found ... 
and I do get every album complete ... 
 
@devs: Have you ever tried to listen to a concept album (say "love over Gold", Dire Straits) with the most popular song missing? AWKWARD! 

plz reopen the bug and fix (or decide you don't want an active community cause we disagree), 
thx for reading, piedro

Piedro, Brandon, Arthur, please calm down. What you all refer to is bug 315329 which got fixed (by me, hehe) v2.7.0-96-g216c18b (git version 96 commits after 2.7.0).

Please try current git version (excellent & worry-free howto at http://blogs.fsfe.org/myriam/2009/09/compiling-amarok-from-git-locally-full-summary/ ), perform full rescan and retest. Any possible comments should go to bug 315329, not here.

As this title says, this bug is about *NOT SHOWING* them, not about misdetecting different tracks as duplicates and yes, it has been fixed since git commit 23c180b6 by Matěj Laitl. Committed on 03/01/2013 at 11:22.

If it was solved only after 2.7.0, then the "Version solved in" field on this bug should say something like 2.7.1, shouldn't it? The 2.7 there is a little misleading because I have been trying 2.7.0 up until now thinking that the commit 23c180b6 was present in the whole 2.7 family - I never bothered to check because of that. Just a thought.

(In reply to comment #79)
> If it was solved only after 2.7.0, then the "Version solved in" field on
> this bug should say something like 2.7.1, shouldn't it? The 2.7 there is a
> little misleading because I have been trying 2.7.0 up until now thinking
> that the commit 23c180b6 was present in the whole 2.7 family - I never
> bothered to check because of that. Just a thought.

Read again, please, you misunderstood.

(In reply to comment #79)
> If it was solved only after 2.7.0, then the "Version solved in" field on
> this bug should say something like 2.7.1, shouldn't it? The 2.7 there is a
> little misleading because I have been trying 2.7.0 up until now thinking
> that the commit 23c180b6 was present in the whole 2.7 family - I never
> bothered to check because of that. Just a thought.

Arthur, again. What you were suffering from is bug 315329, which has the correct Version Fixed In: 2.8. This bug is about *real* duplicates, not the misidentified ones.

thx matej! 

though i couldn't test yet - i compiled from git but i still get the same duplicates and the version number is still 2.7.0 - probably my mistake somehow ... 

I will have to wait for the next deb package i guess, 
thx for your efforts, 
piedro

(In reply to comment #82)
> thx matej! 
> 
> though i couldn't test yet - i compiled from git but i still get the same
> duplicates and the version number is still 2.7.0 - probably my mistake
> somehow ... 

Be sure to uninstall the distribution Amarok package first and that you follow the guide step by step (for example when setting the environment variables)

Hello Matej! 

Now I managed to use this GIT version and you are right the duplication of different album but same Title/artist tracks is gone. thx a lot for finally having addressed that. 

When will this be in a binary version? Though  this GIT stuff works somehow I lost all system integration by using this version. Could you provide a regulat deb file? 

thx anyways, 
I really appreciate your work, 
piedro

(In reply to comment #84)
> Now I managed to use this GIT version and you are right the duplication of
> different album but same Title/artist tracks is gone. thx a lot for finally
> having addressed that. 

Good to know.

> When will this be in a binary version? Though this GIT stuff works somehow
> I lost all system integration by using this version. Could you provide a
> regulat deb file? 

We don't provide packages, just the sources. Packaging is distributions's work - you should ask them instead. I'm also not aware of any "system integration" lost by using git version - many Amarok users use it w/out problems. I suggest you check that you've set up everything properly using the guide posted above.

Arrived in Arch Linux - works nicely, thx matej! 
p.