If installed on MacOSX (10.11.6 El Capitan) digikam seemed to freeze in the process of reading the database after starting it for the first time with default settings. This happens if digikam scans the default user's Pictures folder AND in case one already has used the Apple application Photos before that created a huge Photos Library.photoslibrary in the very same folder. Photos Library.photoslibrary is a packaged db in itself. The size of this file is in my case 26 GB as I managed almost 10,000 photos with it. One has to wait a looong time (about half an hour) to see digikam finish reading the content of Photos Library.photoslibrary in its own database and later on it seems to have some performance problems with so many photos or maybe with this file that it represents as a folder structure. It took me a long time to figure out that this is the reason for the apparent freeze – at the beginning I just thought that digikam 5.0.1 is simply not working on MacOSX El Capitan. Reproducible: Always Steps to Reproduce: 1. (for testing purpose one might) create a huge Photos Library.photoslibrary file in Pictures-folder (default) for many fotos (in my case almost 10.000) with the Photos app of Apple MacOSX El Capitan (it's done automatically if you start it) 2. Install pre-compiled digikam app 3. Start digikam with default settings and let it scan the Pictures folder (happens automatically) with huge Photos Library.photoslibrary in it Actual Results: Digikam hangs on 'Reading database' during startup Expected Results: At least the software should warn the user that it is going to take a very long time to read a file of a certain size or package files like Photos Library.photoslibrary and that it might cause performance problems later on and give the opportunity to skip this file from being written to the databas/to delete it/to move it elsewhere.
Looking on my macbook pro how Apple iPhoto or Photos applications store the data (image and metadata), it's look like a big mess, with a lots of redondant subdirectories relies on image, preview, thumbnails, sidecars for files and folders, database, etc... I think the best way to store images on my is to host all in dedicated directory heirarchies, outside the extra data for application. But of course, we need to be able to handle pro photo applications. On think that i know is the improvements to do in image scanner, to ignore some king of files or directories, hidden or not. There is a non finalized patch about this subject on this bugzilla entry : https://bugs.kde.org/show_bug.cgi?id=123097 Gilles Caulier
With current implementation we fix some lock with database on special conditions discovered under Windows. I think this problem can also appears with MacOS. So i just build a new pre 5.2.0 installer for MacOS : https://drive.google.com/file/d/0BzeiVr-byqt5RHU5LWRKaWh1YXM/view?usp=sharing Please try with this version... Gilles Caulier
I encountered this problem also with a large Photos lib. Removed the folder from the config to fix it. I installed the new 5.2.1-01 and configured it with a local SQLite and pointed it to $HOME\Pictures where my Photos lib is. It started importing, after a few minutes it shot to 27% and incremented all the way up to 43% while the local sqlite grew to 61Mb during that time. After +- 15 min it stopped at 43% done, activity monitor reports it as 99% usage & hanging / not responding and the sqlite db is no longer increasing in size, stuck at 61Mb for > 5 minutes, the thumbnail db is at 76Kb. $iostat doesn't show any significant disk access anymore so I guess it's dead. Some stats on the Photos lib: 170.000 media files in 70.000 subfolders (that’s Masters,Previews & Thumbnails dirs together)
Gilles, thank you very much for the new pre installer. I tried it. Unfortunately, the problem remains. For the time being a hint like you proposed in the releasenotes.html, a readme and the FAQ might suffice. "The best way to store images on Mac Book Pro is to host them in dedicated directory hierarchies, outside the extra data for other applications: Avoid having e.g. iPhoto' or Photos' library in the same directory in which your images are stored and scanned by digikam. " Andrea
I think the scanning actually went ok when seeing the console msg output. The problem seems to appear after the folder scan, or during consequent startup if you killed it. This is the console after it finished scanning Photos, it seemed to actually have finished. digikam.database: Complete scan took: 1339440 msecs. digikam.general: total scan value : 569565 digikam.database: Complete scan (file scanning deferred) took: 203521 msecs. digikam.general: Event is dispatched to OSX desktop notifier digikam.general: Camera XML data: "/Users/wowbagger/Library/Application Support/digikam/cameras.xml" digikam.facedb: FaceDB SelectFaceSetting val ret = 0 digikam.facedb: FaceDB SelectFaceSetting val ret = 0 digikam.facedb: Face database: have a structure version "2" digikam.facesengine: Face database ready for use digikam.facesengine: Face database ready for use digikam.geoiface: "setting backend marble" digikam.general: Stacked View Mode : 0 digikam.geoiface: "setting backend marble" digikam.general: "browse_album" digikam.general: "browse_tag" digikam.general: "browse_labels" digikam.general: "browse_date" digikam.general: "browse_timeline" digikam.general: "browse_search" digikam.general: "browse_fuzzysearch" digikam.general: "browse_gpssearch" digikam.general: "browse_people" digikam.widgets: Paths to color scheme : ("/opt/digikam/Applications/KF5/digikam.app/Contents/Resources//digikam/colorschemes") digikam.widgets: "" :: "" QFSFileEngine::open: No file name specified digikam.dimg: Invalid variant value for device! digikam.dimg: updating data digikam.dimg: updating data digikam.general: Using 4 CPU core to run threads digikam.general: new search text settings: "" : hasResult = false , validRows = 0 QFSFileEngine::open: No file name specified digikam.geoiface: ---- digikam.geoiface: ---- digikam.general: Added root album called: "Pictures" This is a lldb bt: * thread #1: tid = 0xd0980, 0x00000001065714ff QtCore`QHash<QString, QHashDummyValue>::insert(QString const&, QHashDummyValue const&) + 207, stop reason = signal SIGSTOP * frame #0: 0x00000001065714ff QtCore`QHash<QString, QHashDummyValue>::insert(QString const&, QHashDummyValue const&) + 207 frame #1: 0x000000010656f64b QtCore`QtPrivate::QStringList_removeDuplicates(QStringList*) + 251 frame #2: 0x0000000100df7a7d libdigikamcore.5.2.0.dylib`___lldb_unnamed_function10472$$libdigikamcore.5.2.0.dylib + 29 frame #3: 0x0000000100df7408 libdigikamcore.5.2.0.dylib`Digikam::ModelCompleter::slotDataChanged(QModelIndex const&, QModelIndex const&) + 1560 frame #4: 0x0000000100ea5e3f libdigikamcore.5.2.0.dylib`___lldb_unnamed_function12112$$libdigikamcore.5.2.0.dylib + 207 frame #5: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*, int, int, void**) + 742 frame #6: 0x000000010675c69d QtCore`QAbstractItemModel::dataChanged(QModelIndex const&, QModelIndex const&, QVector<int> const&) + 61 frame #7: 0x0000000100326621 libdigikamgui.5.2.0.dylib`Digikam::AbstractCountingAlbumModel::updateCount(Digikam::Album*) + 753 frame #8: 0x00000001003266e5 libdigikamgui.5.2.0.dylib`Digikam::AbstractCountingAlbumModel::includeChildrenCount(QModelIndex const&) + 117 frame #9: 0x00000001003555bd libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13361$$libdigikamgui.5.2.0.dylib + 61 frame #10: 0x000000010035548a libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13359$$libdigikamgui.5.2.0.dylib + 90 frame #11: 0x0000000100355655 libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13362$$libdigikamgui.5.2.0.dylib + 133 frame #12: 0x00000001003562c3 libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13381$$libdigikamgui.5.2.0.dylib + 51 frame #13: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*, int, int, void**) + 742 frame #14: 0x000000010675c88e QtCore`QAbstractItemModel::rowsInserted(QModelIndex const&, int, int, QAbstractItemModel::QPrivateSignal) + 78 frame #15: 0x000000010663b740 QtCore`QAbstractItemModel::endInsertRows() + 80 frame #16: 0x00000001066590b6 QtCore`QSortFilterProxyModelPrivate::insert_source_items(QVector<int>&, QVector<int>&, QVector<int> const&, QModelIndex const&, Qt::Orientation, bool) + 1430 frame #17: 0x0000000106659b06 QtCore`QSortFilterProxyModelPrivate::source_items_inserted(QModelIndex const&, int, int, Qt::Orientation) + 2070 frame #18: 0x0000000106661d03 QtCore`QSortFilterProxyModel::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) + 1443 frame #19: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*, int, int, void**) + 742 frame #20: 0x000000010675c88e QtCore`QAbstractItemModel::rowsInserted(QModelIndex const&, int, int, QAbstractItemModel::QPrivateSignal) + 78 frame #21: 0x000000010663b740 QtCore`QAbstractItemModel::endInsertRows() + 80 frame #22: 0x00000001003258ad libdigikamgui.5.2.0.dylib`Digikam::AbstractAlbumModel::slotAlbumAdded(Digikam::Album*) + 93 frame #23: 0x0000000100334cd7 libdigikamgui.5.2.0.dylib`___lldb_unnamed_function12637$$libdigikamgui.5.2.0.dylib + 167 frame #24: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*, int, int, void**) + 742 frame #25: 0x000000010039174d libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::signalAlbumAdded(Digikam::Album*) + 77 frame #26: 0x00000001003743b8 libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::insertPAlbum(Digikam::PAlbum*, Digikam::PAlbum*) + 328 frame #27: 0x0000000100375782 libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::scanPAlbums() + 4034 frame #28: 0x0000000100374739 libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::refresh() + 25 frame #29: 0x000000010037402a libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::startScan() + 1578 frame #30: 0x00000001000e8d71 libdigikamgui.5.2.0.dylib`Digikam::DigikamApp::DigikamApp() + 3953 frame #31: 0x0000000100011b7f digikam`main + 7343 frame #32: 0x00007fff9221b5ad libdyld.dylib`start + 1 frame #33: 0x00007fff9221b5ad libdyld.dylib`start + 1 And this is the sqlite db: sqlite> select count (*) from images; 120444 sqlite> select count(*) from albums; 106600 Maybe the Album parsing to create the tree make it hang/unresponsive due to the very high amount of albums?
Created attachment 100805 [details] attachment-13291-0.html 2016-08-27 13:50 GMT+02:00 Lars Van Casteren via KDE Bugzilla < bugzilla_noreply@kde.org>: > https://bugs.kde.org/show_bug.cgi?id=367853 > > --- Comment #5 from Lars Van Casteren <larsvancasteren@gmail.com> --- > I think the scanning actually went ok when seeing the console msg output. > The problem seems to appear after the folder scan, or during consequent > startup > if you killed it. > > This is the console after it finished scanning Photos, it seemed to > actually > have finished. > > digikam.database: Complete scan took: 1339440 msecs. > digikam.general: total scan value : 569565 > digikam.database: Complete scan (file scanning deferred) took: 203521 > msecs. > digikam.general: Event is dispatched to OSX desktop notifier > digikam.general: Camera XML data: "/Users/wowbagger/Library/Application > Support/digikam/cameras.xml" > digikam.facedb: FaceDB SelectFaceSetting val ret = 0 > digikam.facedb: FaceDB SelectFaceSetting val ret = 0 > digikam.facedb: Face database: have a structure version "2" > digikam.facesengine: Face database ready for use > digikam.facesengine: Face database ready for use > digikam.geoiface: "setting backend marble" > digikam.general: Stacked View Mode : 0 > digikam.geoiface: "setting backend marble" > digikam.general: "browse_album" > digikam.general: "browse_tag" > digikam.general: "browse_labels" > digikam.general: "browse_date" > digikam.general: "browse_timeline" > digikam.general: "browse_search" > digikam.general: "browse_fuzzysearch" > digikam.general: "browse_gpssearch" > digikam.general: "browse_people" > digikam.widgets: Paths to color scheme : > ("/opt/digikam/Applications/KF5/digikam.app/Contents/Resources//digikam/ > colorschemes") > digikam.widgets: "" :: "" > QFSFileEngine::open: No file name specified > digikam.dimg: Invalid variant value for device! > digikam.dimg: updating data > digikam.dimg: updating data > digikam.general: Using 4 CPU core to run threads > digikam.general: new search text settings: "" : hasResult = false , > validRows > = 0 > QFSFileEngine::open: No file name specified > digikam.geoiface: ---- > digikam.geoiface: ---- > digikam.general: Added root album called: "Pictures" > > This is a lldb bt: > > * thread #1: tid = 0xd0980, 0x00000001065714ff QtCore`QHash<QString, > QHashDummyValue>::insert(QString const&, QHashDummyValue const&) + 207, > stop > reason = signal SIGSTOP > * frame #0: 0x00000001065714ff QtCore`QHash<QString, > QHashDummyValue>::insert(QString const&, QHashDummyValue const&) + 207 > frame #1: 0x000000010656f64b > QtCore`QtPrivate::QStringList_removeDuplicates(QStringList*) + 251 > frame #2: 0x0000000100df7a7d > libdigikamcore.5.2.0.dylib`___lldb_unnamed_function10472$$ > libdigikamcore.5.2.0.dylib > + 29 > frame #3: 0x0000000100df7408 > libdigikamcore.5.2.0.dylib`Digikam::ModelCompleter:: > slotDataChanged(QModelIndex > const&, QModelIndex const&) + 1560 > frame #4: 0x0000000100ea5e3f > libdigikamcore.5.2.0.dylib`___lldb_unnamed_function12112$$ > libdigikamcore.5.2.0.dylib > + 207 > frame #5: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*, > int, > int, void**) + 742 > frame #6: 0x000000010675c69d > QtCore`QAbstractItemModel::dataChanged(QModelIndex const&, QModelIndex > const&, > QVector<int> const&) + 61 > frame #7: 0x0000000100326621 > libdigikamgui.5.2.0.dylib`Digikam::AbstractCountingAlbumModel:: > updateCount(Digikam::Album*) > + 753 > frame #8: 0x00000001003266e5 > libdigikamgui.5.2.0.dylib`Digikam::AbstractCountingAlbumModel:: > includeChildrenCount(QModelIndex > const&) + 117 > frame #9: 0x00000001003555bd > libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13361$$ > libdigikamgui.5.2.0.dylib > + 61 > frame #10: 0x000000010035548a > libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13359$$ > libdigikamgui.5.2.0.dylib > + 90 > frame #11: 0x0000000100355655 > libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13362$$ > libdigikamgui.5.2.0.dylib > + 133 > frame #12: 0x00000001003562c3 > libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13381$$ > libdigikamgui.5.2.0.dylib > + 51 > frame #13: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*, > int, > int, void**) + 742 > frame #14: 0x000000010675c88e > QtCore`QAbstractItemModel::rowsInserted(QModelIndex const&, int, int, > QAbstractItemModel::QPrivateSignal) + 78 > frame #15: 0x000000010663b740 QtCore`QAbstractItemModel::endInsertRows() > + > 80 > frame #16: 0x00000001066590b6 > QtCore`QSortFilterProxyModelPrivate::insert_source_items(QVector<int>&, > QVector<int>&, QVector<int> const&, QModelIndex const&, Qt::Orientation, > bool) > + 1430 > frame #17: 0x0000000106659b06 > QtCore`QSortFilterProxyModelPrivate::source_items_inserted(QModelIndex > const&, > int, int, Qt::Orientation) + 2070 > frame #18: 0x0000000106661d03 > QtCore`QSortFilterProxyModel::qt_static_metacall(QObject*, > QMetaObject::Call, > int, void**) + 1443 > frame #19: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*, > int, > int, void**) + 742 > frame #20: 0x000000010675c88e > QtCore`QAbstractItemModel::rowsInserted(QModelIndex const&, int, int, > QAbstractItemModel::QPrivateSignal) + 78 > frame #21: 0x000000010663b740 QtCore`QAbstractItemModel::endInsertRows() > + > 80 > frame #22: 0x00000001003258ad > libdigikamgui.5.2.0.dylib`Digikam::AbstractAlbumModel:: > slotAlbumAdded(Digikam::Album*) > + 93 > frame #23: 0x0000000100334cd7 > libdigikamgui.5.2.0.dylib`___lldb_unnamed_function12637$$ > libdigikamgui.5.2.0.dylib > + 167 > frame #24: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*, > int, > int, void**) + 742 > frame #25: 0x000000010039174d > libdigikamgui.5.2.0.dylib`Digikam::AlbumManager:: > signalAlbumAdded(Digikam::Album*) > + 77 > frame #26: 0x00000001003743b8 > libdigikamgui.5.2.0.dylib`Digikam::AlbumManager:: > insertPAlbum(Digikam::PAlbum*, > Digikam::PAlbum*) + 328 > frame #27: 0x0000000100375782 > libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::scanPAlbums() + 4034 > frame #28: 0x0000000100374739 > libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::refresh() + 25 > frame #29: 0x000000010037402a > libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::startScan() + 1578 > frame #30: 0x00000001000e8d71 > libdigikamgui.5.2.0.dylib`Digikam::DigikamApp::DigikamApp() + 3953 > frame #31: 0x0000000100011b7f digikam`main + 7343 > frame #32: 0x00007fff9221b5ad libdyld.dylib`start + 1 > frame #33: 0x00007fff9221b5ad libdyld.dylib`start + 1 > > And this is the sqlite db: > > sqlite> select count (*) from images; > 120444 > > sqlite> select count(*) from albums; > 106600 > > Maybe the Album parsing to create the tree make it hang/unresponsive due > to the > very high amount of albums? > If it's the case, this can be a sqlite internal settings problem. This will give the same result with Mysql internal to replace sqlite ? (not a remote Mysql server). Mysql must be better to handle big set of data. Does it pass until the end of scan in this situation ? Gilles Caulier
I got it up & running with some major trickery using MariaDB 10.1.14 (lots of manual config stuff & creating mysql. innoDB tables etc before DK would continue) but in the end I get the same result when adding $HOME/Pictures collection where the Photos library is located. The scan hangs at the same 43% in DK frontend using the local MariaDB. Exactly the same behaviour like SQLite. MariaDB [(none)]> select count(*) from digikam.albums; +----------+ | count(*) | +----------+ | 106600 | +----------+ 1 row in set (0.02 sec) 106600 seemed like a strange round number to me for the amount of subfolder but it checks out, $find . -type d | wc -l returns 106600 subfolders. Another test I did was to create a large number of directories in a collection to see what would happen. *note: it might take a minute or 3 to create the 110000 subdirs... $for ((i=1;i<=110000;i++)); do mkdir-p testdir/$i; done I ran a clean DK instance with SQLite and point to testdir as a collection so it would scan it and add the albums. Same problem applies. It starts scanning but it hangs after a while and doesn't seem to generate the (huge) tree although it’s just a single tree without subfolders or images. Rgds, Lars
digiKam 5.4.0 pre-release is available in this repository : https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM Can you test with this version please ? Gilles Caulier
What's about this file using 5.8.0 pre-release buncle : https://files.kde.org/digikam/ Thanks in advance Gilles Caulier
Andrea, Can you reproduce the dysfunction using 6.1.0 pre-release package for MacOSX available here : https://files.kde.org/digikam/ Thanks in advance Gilles Caulier
This system have similar problem in a newer version. The total image size is huge. Because of the big collection, the program interface is never shown except the scanning dialogue at startup but freeze most of the time. ADDITIONAL INFORMATION It is the setup: Mac OS 10.12.16 Network Speed: Local is Giga Lan, Wan 1 Giga, Wifi connected as 1300Mbps Using a folder in a network SMB connected as a volume (60000 - 70000 images in the image folder), total 445GB images Digikam 6.1.0 macOS disk image download from official website
digiKam 7.0.0 stable release is now published: https://www.digikam.org/news/2020-07-19-7.0.0_release_announcement/ We need a fresh feedback on this file using this version. Best Regards Gilles Caulier
https://bugs.kde.org/show_bug.cgi?id=426938 --- Comment #4 from caulier.gilles@gmail.com --- Hi, digiKam 7.2.0-beta2 pre-release PKG installer now support BigSur and is compiled with last stable Qt 5.15.2. https://files.kde.org/digikam/ Problem still reproducible with this version. Thanks and happy Christmas in advance Best Regards Gilles Caulier
digiKam 7.2.0 official release is published with more than 360 files closed from bugzilla: https://www.digikam.org/news/2021-03-22-7.2.0_release_announcement/ Can you reproduce the dysfunction with this version ? Thanks in advance for your feedback Gilles Caulier
Andrea, Stable digiKam 7.4.0 is published. Please check if problem is reproducible. https://www.digikam.org/download/ Thanks in advance
@Andrea digiKam 8.0.0 is out. Problem still reproducible ? Best regards Gilles Caulier
@Andrea, digikam 8.2.0 pre-release have been rebuilt using last Qt 5.15.11 + KDE 5.110 frameworks. Installer is available at usual place : https://files.kde.org/digikam/ Can reproduce the problem with this version? Thanks in advance Gilles Caulier
@Andrea, digiKam 8.3.0 stable version is released and available at usual place : https://www.digikam.org/download/ Can you reproduce the dysfunction on your computer ? Thanks in advance Gilles Caulier