Bug 210096

Summary: Amarok doesn't scan my complete collection
Product: [Applications] amarok Reporter: Sayak Banerjee <sayakb>
Component: Collections/LocalAssignee: Amarok Developers <amarok-bugs-dist>
Status: RESOLVED FIXED    
Severity: normal CC: andreluizromano, joschi, mitchell, ralf-engels
Priority: NOR    
Version: 2.3.0   
Target Milestone: ---   
Platform: Compiled Sources   
OS: Linux   
Latest Commit: Version Fixed In: 2.3.2
Sentry Crash Report:

Description Sayak Banerjee 2009-10-10 18:37:45 UTC
Version:           2.1.1 (using Devel)
OS:                Linux
Installed from:    Compiled sources

Since I upgraded to KDE 4.3.2; Amarok does not scan my entire collection. It scans something around 3000-4000 tracks whereas other media players give me about  14000 tracks (which is the correct count).

REF: http://forum.kde.org/viewtopic.php?f=115&t=82805

I believe Amarok wasn't upgraded when the KDE version changed, although I definitely didn't face the issue before that.
Comment 1 Jeff Mitchell 2009-10-10 18:44:22 UTC
I've got the file -- thanks! You can take it down now. Will post here once I've had a chance to dig into this.
Comment 2 Sayak Banerjee 2009-10-10 18:55:20 UTC
Great! I've nuked the file down. :-)
Comment 3 Jochen Rüter 2009-10-26 22:22:58 UTC
Same issue here, I'm stuck at 766 files. However, debug output doesn't help me nailing down the file causing the trouble. The only suspicious part of the output is:

TagLib: MPEG::Header::parse() -- Invalid sample rate.                                                                                           
TagLib: MPEG::Header::parse() -- Invalid sample rate.                                                                                           
TagLib: MPEG::Header::parse() -- Invalid sample rate.                                                                                           
amarok:    [ERROR!] Collection scanner abort error:  3                                                                                          
amarok:    Trying to restart the QXmlStreamReader..                                                                                             
amarok:    Success. Committing result to database.                                                                                              
amarok:    temp_tracks:  ("766")                                                                                                                
amarok:    tracks before commit:  ("0")                                                                                                         
amarok:   BEGIN: void DatabaseUpdater::copyToPermanentTables()                                                                                  
amarok:   END__: void DatabaseUpdater::copyToPermanentTables() - Took 0.15s                                                                     
amarok:    tracks after commit:  ("766")                                                                                                        
amarok:   BEGIN: void DatabaseUpdater::removeTemporaryTables()                                                                                  
amarok:   END__: void DatabaseUpdater::removeTemporaryTables() - Took 0.0014s                                                                   
amarok:    Sending changed signal
Comment 4 Myriam Schweingruber 2009-10-26 23:40:23 UTC
Hm, Amarok 2.1.1 was quite some time ago, how about Amarok 2.2.0? Since 2.2.1 is around the corner, I bet this should be fixed in git by now. Jeff?
Comment 5 Jeff Mitchell 2009-10-27 00:17:29 UTC
Yeah -- unfortuantely, due to the absolutely massive number of collection scanning fixes in 2.2, I really can't help with 2.1 bugs anymore -- there's just no point, because there's nothing I can generally do about them.

The good news is that 2.2 fixes scanning problems for like 99% of people...

Setting to waitingforinfo to find out how 2.2 goes.
Comment 6 Jochen Rüter 2009-10-27 07:42:00 UTC
Sorry, I was not aware that this bug is only related to 2.1. I am actually running 2.2, should I open another bug?
Comment 7 Myriam Schweingruber 2009-11-03 20:03:15 UTC
(In reply to comment #6)
> Sorry, I was not aware that this bug is only related to 2.1. I am actually
> running 2.2, should I open another bug?

If you can reproduce this with current Amarok 2.2-git or the upcoming 2.2.1, feel free to reopen this bug
Comment 8 Jochen Rüter 2010-10-11 21:44:23 UTC
unfortunately, I'm still having collection scan issues now using Amarok 2.3.
Unlike the other bug reports I found, the scanner seems not to be stuck at a single file. Instead -as I read the logs- for some reason it thinks it has nothing to scan:

...
amarok: BEGIN: void ScanManager::startIncrementalScan(const QString&) 
amarok:   BEGIN: void ScanManager::checkTables(bool) 
amarok:   END__: void ScanManager::checkTables(bool) - Took 5.4e-05s 
amarok:   BEGIN: QStringList ScanManager::getDirsToScan() 
amarok:   END__: QStringList ScanManager::getDirsToScan() - Took 0.74s 
amarok:    GOING TO SCAN: 
amarok:    "(a *lot*: 25180 directories)" 
amarok:   BEGIN: void ScanManager::writeBatchIncrementalInfoFile() 
amarok:   END__: void ScanManager::writeBatchIncrementalInfoFile() - Took 0.037s 
amarok:   BEGIN: XmlParseJob::XmlParseJob(ScanManager*, SqlCollection*) 
amarok:     BEGIN: ProgressBar* ProgressBar::setAbortSlot(QObject*, const char*) 
amarok:        Setting abort slot for  "Musik wird durchsucht" 
amarok:        connecting to  1abort() 
amarok:     END__: ProgressBar* ProgressBar::setAbortSlot(QObject*, const char*) - Took 0.0001s 
amarok:   END__: XmlParseJob::XmlParseJob(ScanManager*, SqlCollection*) - Took 0.0011s 
amarok:   BEGIN: virtual void XmlParseJob::run() 
amarok:   END__: void ScanManager::startIncrementalScan(const QString&) - Took 0.78s 
amarok:    Setting up database 
amarok:   BEGIN: void DatabaseUpdater::createTemporaryTables() 
amarok:   END__: void DatabaseUpdater::createTemporaryTables() - Took 0.083s 
amarok:   BEGIN: void DatabaseUpdater::prepareTemporaryTables() 
amarok:   END__: void DatabaseUpdater::prepareTemporaryTables() - Took 0.24s 
amarok:   BEGIN: void ScanResultProcessor::populateCacheHashes() 
amarok:   END__: void ScanResultProcessor::populateCacheHashes() - Took 0.05s 
amarok:    Success. Committing result to database. 
amarok:   BEGIN: void ScanResultProcessor::copyHashesToTempTables() 
amarok:      obtained max_allowed_packet is  "1048576" 
amarok:      urls key size is  1696 
amarok:     BEGIN: void ScanManager::slotFinished() 
amarok:     END__: void ScanManager::slotFinished() - Took 0.00013s 
amarok:      tracks key size is  1530 
amarok:   END__: void ScanResultProcessor::copyHashesToTempTables() - Took 0.21s 
amarok:    temp_tracks:  ("1530") 
amarok:    tracks before commit:  ("1530") 
amarok:   BEGIN: void DatabaseUpdater::copyToPermanentTables() 
amarok:     BEGIN: virtual void Dynamic::BiasedPlaylist::invalidate() 
amarok:     END__: virtual void Dynamic::BiasedPlaylist::invalidate() - Took 9.6e-05s 
amarok:   END__: void DatabaseUpdater::copyToPermanentTables() - Took 0.45s 
amarok:    tracks after commit:  ("1530") 
amarok:   BEGIN: void DatabaseUpdater::removeTemporaryTables() 
amarok:   END__: void DatabaseUpdater::removeTemporaryTables() - Took 0.23s 
amarok:    Sending changed signal 
amarok: END__: virtual void XmlParseJob::run() - Took 1.9s 
...

Any help is greatly appreciated...
Comment 9 Ralf Engels 2010-10-14 13:46:59 UTC
Hi Jochen,
Could you please check if you have a directory with a newline in the directory name.
Unlikely as it might seem, that happened to me once and the collection scanner has a bug there.

Also you might try to start the collection scanner directly (it's called amarokcollectionscanner and you can start it with -r and then your collection directory)
This might also help you pinpointing the offending file.

I am currently working on a complete update of the collection scanner which probably would also correct this bug.
Comment 10 Jochen Rüter 2010-10-15 16:19:54 UTC
Hi Ralf,
doesn't look like I have any newlines in any file- or directory names.
I also tried calling amarokcollectionscanner manually. This seems to scan everything just fine! collection_scan.files contains 10746 entries, which seems to be complete!

-rw-rw-r--  1 joschi users   1057099 2010-10-15 16:06 collection_scan.files
-rw-rw-r--  1 joschi users        71 2010-10-15 16:09 collection_scan.log
[/media/data/backup/htpc/Musik/sortiert]$ wc -l collection_scan.files 
10746 collection_scan.files
[/media/data/backup/htpc/Musik/sortiert]$ 

So to me, the collection scanner itself doesn't seem to be the issue, right?
Comment 11 Jeff Mitchell 2010-10-15 21:29:39 UTC
If you can patch your Amarok install, try patching with http://gitweb.kde.org/amarok.git/patch/182f39c15575c3c1ce2b0425f9e3ec5997f3a9fd (this is against 2.3.2).
Comment 12 Myriam Schweingruber 2010-12-13 10:32:21 UTC
Is this still valid with 2.3.2 or 2.4 beta?
Comment 13 Myriam Schweingruber 2011-01-10 18:34:56 UTC
Sayak, Jochen: we need your input :)
Comment 14 Myriam Schweingruber 2011-03-20 12:36:24 UTC
Closing for lack of feedback, most likely fixed since quite some time.
Comment 15 andreluizromano 2011-05-08 04:06:18 UTC
This bug is not fixed in version 2.4 of Kubuntu 11.04 official repository...

How can I see what files are causing the ID3 problem?
Comment 16 Ralf Engels 2011-05-09 13:57:07 UTC
You have to start Amarok with the --debug option and then look at the debug output when scanning (there can be a lot).
You might see "duplicate id" or "sql error" messages.

Usually a run of the amarok AFT tagger (amarok_afttagger) solves most of these problems.
Comment 17 andreluizromano 2011-05-09 22:42:09 UTC
Thanks for your help...

This issue is still happening in Ubuntu 11.04 and Kubuntu 11.04 with Amarok 2.4.0...