I should say that the example track is Glósóli by Sigur Rós, since I've uploaded two other files to that directory for other bug reports. :)

Yeah that's a problem with the new SQLite 3.3.4 library, which we just upgraded to. See this SQLite bug report we made:

http://www.sqlite.org/cvstrac/tktview?tn=1681

*** Bug 122345 has been marked as a duplicate of this bug. ***

I have exactly the same results with SQLite as reported on www.sqlite.org. But with MySql I have the bug too! On stderr there is no ERROR message like with SQLite...

It's strange because i tried to compile amarok-SVN with --without-included-sqlite (using sqlite 3.2.8 from ubuntu dapper) and i have this bug too!  More than 1000 songs are not scanned.  I reinstalled amarok-1.4 beta 1 and the problem's gone!
I also tried with mysql and the bug persist!  Strange indeed!

Just to be sure, I went back to SQLite 3.2.7 and tested. No errors at all with this version.

So, it's definitely a SQLite 3.3.X issue.

*** Bug 122614 has been marked as a duplicate of this bug. ***

*** Bug 122615 has been marked as a duplicate of this bug. ***

SVN commit 513240 by markey:

Downgrade SQLite to version 3.2.7, until the problems with non-ASCII characters are fixed. 

NOTE: You may need to delete your collection.db, since the database format was changed between 3.2 and 3.3.

CCBUG: 122168


 M  +0 -1      ChangeLog  
 M  +0 -1      src/sqlite/Makefile.am  
 M  +31 -36    src/sqlite/alter.c  
 M  +11 -28    src/sqlite/analyze.c  
 M  +123 -264  src/sqlite/attach.c  
 M  +3 -10     src/sqlite/auth.c  
 M  +283 -1089 src/sqlite/btree.c  
 M  +2 -6      src/sqlite/btree.h  
 M  +220 -503  src/sqlite/build.c  
 M  +11 -20    src/sqlite/callback.c  
 M  +2 -2      src/sqlite/complete.c  
 M  +12 -14    src/sqlite/date.c  
 M  +24 -31    src/sqlite/delete.c  
 M  +98 -151   src/sqlite/expr.c  
 M  +46 -65    src/sqlite/func.c  
 M  +0 -5      src/sqlite/hash.c  
 M  +50 -60    src/sqlite/insert.c  
 M  +31 -32    src/sqlite/keywordhash.h  
 M  +6 -3      src/sqlite/legacy.c  
 M  +196 -369  src/sqlite/main.c  
 M  +101 -108  src/sqlite/opcodes.c  
 M  +111 -121  src/sqlite/opcodes.h  
 D             src/sqlite/os.c  
 M  +41 -274   src/sqlite/os.h  
 M  +1 -66     src/sqlite/os_common.h  
 M  +229 -697  src/sqlite/os_unix.c  
 M  +132 -708  src/sqlite/os_win.c  
 M  +363 -535  src/sqlite/pager.c  
 M  +1 -7      src/sqlite/pager.h  
 M  +2087 -1941 src/sqlite/parse.c  
 M  +145 -151  src/sqlite/parse.h  
 M  +58 -84    src/sqlite/pragma.c  
 M  +87 -180   src/sqlite/prepare.c  
 M  +15 -17    src/sqlite/printf.c  
 M  +199 -369  src/sqlite/select.c  
 M  +8 -211    src/sqlite/sqlite3.h  
 M  +107 -217  src/sqlite/sqliteInt.h  
 M  +1 -6      src/sqlite/table.c  
 M  +9 -19     src/sqlite/tokenize.c  
 M  +52 -61    src/sqlite/trigger.c  
 M  +14 -16    src/sqlite/update.c  
 M  +4 -30     src/sqlite/utf.c  
 M  +239 -664  src/sqlite/util.c  
 M  +18 -37    src/sqlite/vacuum.c  
 M  +299 -552  src/sqlite/vdbe.c  
 M  +1 -13     src/sqlite/vdbe.h  
 M  +7 -18     src/sqlite/vdbeInt.h  
 M  +34 -112   src/sqlite/vdbeapi.c  
 M  +117 -257  src/sqlite/vdbeaux.c  
 M  +29 -73    src/sqlite/vdbemem.c  
 M  +75 -145   src/sqlite/where.c  

I have tried with downgraded sqlite and have got the same result. To be sure I have renamed (old) ~/.kde/share/apps/amarok dir. And, to be more sure :-), I have found the sqlite3.h file has this string:

#define SQLITE_VERSION         "3.2.7"

I tried the new svn code, tags with non-ascii chars works fine, however amarok somehow ignored the directory with non-ascii chars, maybe that was the problme Andrew mentioned, should I open a new bug?

Dir's name? Not in my case. A dir name and 'ls' out are below. Only 4th and 6th tracks were hooked with scanner. All tracks with 'é' char in file name are skipped.

# cd /x/sound/ARCHIVE/Jazz/jacquesLoussier/bach1
anli bach1 # ls
01 - Prélude #1.flac                                                05 - Choral 'Jésus Que Ma Joie Demeure'.flac
02 - Partita En Si Bémol - Allemande, Courante.flac                 06 - Aria.flac
03 - Partita En Si Bémol - Sarabande, Menuet1, Menuet2, Gigue.flac  07 - Toccata Et Fugue En Ré Mineur.flac
04 - Choral #1.flac                                                 08 - Prélude #12.flac

I should say dirs and files instead of dirs :)

Yeah it's a regression in the directory reading code, which I've just ported 
over from 1.3. Gonna look at it sometime.

SVN commit 513565 by markey:

Fix problems with non-ASCII paths.

CCBUG: 122168



 M  +1 -1      collectionscanner.cpp  


--- trunk/extragear/multimedia/amarok/src/collectionscanner/collectionscanner.cpp #513564:513565
@@ -229,7 +229,7 @@
         }
 
         else if( S_ISREG( statBuf.st_mode ) )
-            entries.append( entry );
+            entries.append( QFile::decodeName( entry ) );
     }
 
     closedir( d );

Mark,

Thanks! The problem is resolved for me. I'm ready to try with sqlite 3.3.x.

Now, some news: Richard Hipp (SQLite creator) has replied to our bug report. Check this:

http://www.sqlite.org/cvstrac/tktview?tn=1681

Not too encouraging. Any SQLite expert here who could look at what we might be doing wrong?

Hi Mark! 

Thanks a lot for you investment! Amarok is incredible!
I first had the bug with mysql which you resolved with your patch. After reading your latest post I tried it with sqlite and everything is working perfectly for me. So why should you be doing something wrong? ;-)

Thanks and greetings, Lukas Wolf

SVN commit 518398 by illissius:

fix the sqlite-doesn't-like-weird-characters bug with later versions (3.3.4 e.g.)
the bug turned out to be that sqlite3_prepare() expects length as the number of bytes, not characters -- so using string.length() just happened to work when only the ASCII subset of UTF-8 was used, but otherwise broke. This is fixed by telling it to just read until the null at the end. (Maybe previous versions of SQLite do this always, which is why they don't trigger the bug?)
BUG:122168


 M  +2 -2      collectiondb.cpp  


--- trunk/extragear/multimedia/amarok/src/collectiondb.cpp #518397:518398
@@ -3148,7 +3148,7 @@
     sqlite3_stmt* stmt;
 
     //compile SQL program to virtual machine
-    error = sqlite3_prepare( m_db, statement.utf8(), statement.length(), &stmt, &tail );
+    error = sqlite3_prepare( m_db, statement.utf8(), -1, &stmt, &tail );
 
     if ( error != SQLITE_OK )
     {
@@ -3208,7 +3208,7 @@
     const char* tail;
     sqlite3_stmt* stmt;
     //compile SQL program to virtual machine
-    error = sqlite3_prepare( m_db, statement.utf8(), statement.length(), &stmt, &tail );
+    error = sqlite3_prepare( m_db, statement.utf8(), -1, &stmt, &tail );
 
     if ( error != SQLITE_OK )
     {