| Summary: | GUI hangs when adding a network folder | ||
|---|---|---|---|
| Product: | [Applications] digikam | Reporter: | Stefan S <stefan.szekeres> |
| Component: | Database-Scan | Assignee: | Digikam Developers <digikam-bugs-null> |
| Status: | REOPENED --- | ||
| Severity: | crash | CC: | caulier.gilles, iwannaberich, joseph, metzpinguin, stefan.szekeres |
| Priority: | NOR | Keywords: | efficiency |
| Version: | 7.4.0 | ||
| Target Milestone: | --- | ||
| Platform: | macOS (DMG) | ||
| OS: | macOS | ||
| Latest Commit: | Version Fixed In: | ||
| Sentry Crash Report: | |||
| Attachments: | digiKam-7.0.0-beta2-MacOS-x86-64.pkg GUI hang bug | ||
|
Description
Stefan S
2020-04-20 10:39:02 UTC
Found this piece of software on softwarerecs on stackexchange and also on alternativeto.net to Google Picasa (discontinued) - the best management photo app there was. Searching for an alternative as that piece of software was discontinued. *** This bug has been marked as a duplicate of bug 414320 *** The scanning process is a separate thread. I can continue to work here on Linux, albeit a bit slower. Keep in mind that the scanning process is massively loading your file system. Depending on the database (SQLite, MySQL) on which drive (SSD / HDD) it is located also has an influence. Maik Created attachment 127725 [details]
digiKam-7.0.0-beta2-MacOS-x86-64.pkg GUI hang bug
Hi,
This still reproduces on digiKam-7.0.0-beta2-MacOS-x86-64.pkg
Same steps, added a few million more photos, my connection to the NAS is gigabit.
Regards,
Stefan
(In reply to Stefan S from comment #4) > Created attachment 127725 [details] > digiKam-7.0.0-beta2-MacOS-x86-64.pkg GUI hang bug > > > Hi, > > This still reproduces on digiKam-7.0.0-beta2-MacOS-x86-64.pkg > Same steps, added a few million more photos, my connection to the NAS is > gigabit. > > Regards, > Stefan My laptop has SSD where the digiKam software is installed FYI. The image is not very meaningful. You can no longer select or open anything in the GUI? If I integrate a new network collection here under Linux and the scan is running, I can work with digiKam without any problems. Gilles, can you confirm that MacOS doesn't work? Possibly the number of threads per program limited under MacOS? Maik I cannot select or do anything in the GUI. It is completely blocked. Uploading a video to youtube with screen recording. The only non-default configuration I'm aware of is that i selected to write all metadata to all images. Like faces / any metadata. But since this is the first time it is indexing, there is nothing new to store, I suppose. Here is the video link to the screen recording, will delete this in 24 hours for privacy. https://www.youtube.com/watch?v=m98cbtU1Hac Stefan The video is still processing by Youtube, will be 1080p in a few minutes, sorry. You must to check with 7.0.0-beta3, not beta2, available here : https://files.kde.org/digikam/ Gilles Caulier (In reply to caulier.gilles from comment #9) > You must to check with 7.0.0-beta3, not beta2, available here : > > https://files.kde.org/digikam/ > > Gilles Caulier Installing right now. Forced quit beta2, here is the call stack i got: https://gist.github.com/0wnrepo/c8953e0df7089e5f702afe058c9ad6c5 Installed version beta3, can't even open digikam, i hangs on startup. Had to kill it after 4 minutes. Reinstalling now with video capture, will come back with video link. Meanwhile, here is the stack for beta3: https://gist.github.com/0wnrepo/6873bd040404e0d4ea9bc1749e0a9218 Found your problem: You are doing work, on startup, on the main thread. (com.apple.main-thread (serial))
1970 Thread_18642693 DispatchQueue_1: com.apple.main-thread (serial)
+ 1970 start (in libdyld.dylib) + 1 [0x7fff5999b3d5]
+ 1970 main (in digikam) + 10140 [0x10114fedc]
+ 1970 Digikam::DigikamApp::DigikamApp() (in libdigikamgui.7.0.0.dylib) + 4136 [0x10118c628]
+ 1970 Digikam::AlbumManager::startScan() (in libdigikamgui.7.0.0.dylib) + 1567 [0x1013d2fcf]
+ 1970 Digikam::AlbumManager::refresh() (in libdigikamgui.7.0.0.dylib) + 25 [0x1013d3459]
+ 1969 Digikam::AlbumManager::scanPAlbums() (in libdigikamgui.7.0.0.dylib) + 4023 [0x1013dfea7]
+ ! 1969 Digikam::AlbumManager::insertPAlbum(Digikam::PAlbum*, Digikam::PAlbum*) (in libdigikamgui.7.0.0.dylib) + 328 [0x1013e0c98]
+ ! 1969 Digikam::AlbumManager::signalAlbumAdded(Digikam::Album*) (in libdigikamgui.7.0.0.dylib) + 87 [0x101389127]
+ ! 1956 ??? (in QtCore) load address 0x103bfa000 + 0x1544bc [0x103d4e4bc]
+ ! : 1956 ??? (in libdigikamgui.7.0.0.dylib) load address 0x101160000 + 0x228007 [0x101388007]
+ ! : 1943 ??? (in libdigikamgui.7.0.0.dylib) load address 0x101160000 + 0x244f73 [0x1013a4f73]
+ ! : | 1943 QFileSystemWatcher::addPath(QString const&) (in QtCore) + 50 [0x103ce283e]
+ ! : | 1943 QFileSystemWatcher::addPaths(QStringList const&) (in QtCore) + 287 [0x103ce2771]
full gist: https://gist.github.com/0wnrepo/fc794421ccd1bd5c1e767e1cefbd8621
video : https://www.youtube.com/watch?v=cz_Q7GXVoUY
The main thread is what the GUI uses, a good heuristic is to avoid doing any work on the main thread, on macOS (full disclosure, i'm a macOS developer) This message makes me wonder: QFileSystemWatcher::addPath(). Did you reactivate the monitoring of albums in the settings? This option is deactivated by default. Under MacOS, it takes hours to load albums from network drives. Go to the config file "digikamrc" and set Album Monitoring = false. Maik (In reply to Maik Qualmann from comment #14) > This message makes me wonder: QFileSystemWatcher::addPath(). Did you > reactivate the monitoring of albums in the settings? This option is > deactivated by default. Under MacOS, it takes hours to load albums from > network drives. Go to the config file "digikamrc" and set Album Monitoring = > false. > > Maik I did enable that, thought it won't index if i disconnect the drive and reconnect it after a while (this happened in Picasa) if i didn't check that box. Disabled this now, it doesn't hang anymore. Thanks for the suggestion! Not sure if it's indexing, though, no visual cue (there are so many files, 0% for 5 minutes now), Picasa was popping up a lower corner of the screen notification if it found any new images/videos. Later edit: Checked the sample of digikam, seems to be doing the indexing in the background :-) ??? (in libdigikamgui.7.0.0.dylib) load address 0x107cb1000 + 0x3c6863 [0x108077863] + 2424 Digikam::NewItemsFinder::slotStart() (in libdigikamgui.7.0.0.dylib) + 412 [0x10808935c] + 2424 Digikam::ScanController::completeCollectionScanInBackground(bool) (in libdigikamgui.7.0.0.dylib) + 42 [0x107e2368a] + 2424 Digikam::ScanController::completeCollectionScanCore(bool, bool) (in libdigikamgui.7.0.0.dylib) + 168 [0x107e23628] https://gist.github.com/0wnrepo/c14ebb4709311c5a581701f6eaed931f Also, if anybody wants to get some inspiration from Picasa (from Google) - discontinued project, here are the clean installation packages, they are a backup i downloaded from google a few years ago. I suggest you scan them to be extra safe on virustotal https://drive.google.com/open?id=1h3YzoPH4Ahg0qmH5u2O_Aqpcgg3h3u1G Regards! I'm very happy i found digiKam, seems to be only product that has the same versatility as Picasa. Great support, thanks Maik, Gilles. Hmm, and the GUI no longer hangs when new items are added in the background? I would then add an additional warning / explanation for MacOS to the album monitoring option. Maik (In reply to Maik Qualmann from comment #16) > Hmm, and the GUI no longer hangs when new items are added in the background? > I would then add an additional warning / explanation for MacOS to the album > monitoring option. > > Maik The GUI doesn't hang when new items are added in the background. Seems a good enough solution for now to add the warning / explanation. Might consider moving the monitoring to background in the future :-) Ok, thanks for the feedback. The albums are also monitored in the background (if enabled) with the QFileSystemWatcher. The problem is the QFilesystemWatcher itself, that adding a folder path (only if it is a network path?) under MacOS takes extremely long and the system is blocked. So a bug in the QFileSystemWatcher or MacOS. Maik (In reply to Maik Qualmann from comment #18) > Ok, thanks for the feedback. The albums are also monitored in the background > (if enabled) with the QFileSystemWatcher. The problem is the > QFilesystemWatcher itself, that adding a folder path (only if it is a > network path?) under MacOS takes extremely long and the system is blocked. > So a bug in the QFileSystemWatcher or MacOS. > > Maik The network path is treated as a local folder mounted in /Volumes/<path_name> via smb:// QFileSystemWatcher - why not create a new thread and run it there? maybe dispatch_async ? or some such (In reply to Maik Qualmann from comment #3) > The scanning process is a separate thread. I can continue to work here on > Linux, albeit a bit slower. Keep in mind that the scanning process is > massively loading your file system. Depending on the database (SQLite, > MySQL) on which drive (SSD / HDD) it is located also has an influence. > > Maik Picasa used an in memory database, that was written to disk, upon the shutdown (exit) of the application. Why not consider using this for performance? I see the files are pretty small, except the thumbnails-digikam.db which is over 2GiB on my machine, and going up. How many images / album folder are you trying to add to digiKam? Maik If you want performances and not to be limited in space size, use a SSD, it work perfectly, and not too expensive. Gilles Caulier (In reply to Maik Qualmann from comment #21) > How many images / album folder are you trying to add to digiKam? > > Maik As mentioned above, millions of images. also, digiKam just crashed: https://gist.github.com/0wnrepo/6e9ddb5f69f5e7eb09942f89c6306388 Did you use the bug version of OSX PKG installer ? Why there is no debug symbols in your backtrace ? Where is located the thread responsible of the crash ? The debug bracktrace is less clear than Linux version... Gilles Caulier It probably crashes in Exiv2 when loading the metadata. Probably through an image with wrong metadata. I don't want to disappoint you, we have never tested digiKam with such a number of imges. Through my photography hobby I have about 60K images over the years, Gilles probably has over 100K. By random generator, I created 200K images as a test. But millions... I think it won't work. A fast internal MySQL DB on SSD is then mandatory anyway. Since we keep all albums in the tree view model, from about 50K albums / folder there is a strong performance problem... Maik (In reply to caulier.gilles from comment #24) > Did you use the bug version of OSX PKG installer ? Why there is no debug > symbols in your backtrace ? Where is located the thread responsible of the > crash ? I was using the https://files.kde.org/digikam/ digiKam-7.0.0-beta3-20200419T173721-MacOS-x86-64.pkg . Currently installing the digiKam-7.0.0-beta3-20200419T175921-MacOS-x86-64-debug.pkg hoping to reproduce it. The crash points to thread 46 Crashed Thread: 46 Thread (pooled) Thread 46 Crashed:: Thread (pooled) 0 libdigikamcore.7.0.0.dylib 0x0000000108a65aaf 0x108a51000 + 84655 1 libdigikamcore.7.0.0.dylib 0x0000000108ccbc27 Digikam::MetaEngine::load(QString const&) + 39 2 libdigikamcore.7.0.0.dylib 0x0000000108d1ee82 Digikam::DMetadata::load(QString const&) + 434 3 libdigikamcore.7.0.0.dylib 0x0000000108a70f44 Digikam::DImgLoader::readMetadata(QString const&) + 244 4 DImg_JPEG_Plugin.so 0x000000012229f36f 0x122297000 + 33647 5 libdigikamcore.7.0.0.dylib 0x0000000108c75931 Digikam::DImg::load(QString const&, int, Digikam::DImgLoaderObserver*, Digikam::DRawDecoding const&) + 1745 6 libdigikamcore.7.0.0.dylib 0x0000000108c76622 Digikam::DImg::load(QString const&, Digikam::DImgLoaderObserver*, Digikam::DRawDecoding const&) + 50 7 libdigikamcore.7.0.0.dylib 0x0000000108d8bbb5 0x108a51000 + 3386293 8 libdigikamcore.7.0.0.dylib 0x0000000108da5124 Digikam::LoadSaveThread::run() + 372 9 libdigikamcore.7.0.0.dylib 0x0000000108ddd4b3 0x108a51000 + 3720371 10 org.qt-project.QtCore 0x000000010a772147 0x10a755000 + 119111 11 org.qt-project.QtCore 0x000000010a76fbd4 0x10a755000 + 109524 12 libsystem_pthread.dylib 0x00007fff59b8f2eb _pthread_body + 126 13 libsystem_pthread.dylib 0x00007fff59b92249 _pthread_start + 66 14 libsystem_pthread.dylib 0x00007fff59b8e40d thread_start + 13 > > The debug bracktrace is less clear than Linux version... > > Gilles Caulier Indeed, because i installed the non-debug version. You can symbolicate crash log with the atos tool. You need to manually copy the exact .dSYM file for libdigikamcore.7.0.0.dylib - from the release build, not from the debug build, they are not equivalent as the builds on macOS are not deterministic. Next fire up atos atos -arch x86_64 -o <path to your dSYM>/libdigikamcore.7.0.0.dylib.dSYM/Contents/Resources/DWARF/libdigikamcore.7.0.0.dylib -l 0x108a51000 where 0x108a51000 is the load path of the binary from the crash log https://gist.github.com/0wnrepo/6e9ddb5f69f5e7eb09942f89c6306388 And plug in 0x0000000108ccbc27 and 0x0000000108a65aaf to se exactly what's in there. > It probably crashes in Exiv2 when loading the metadata. Probably through an image with wrong metadata. I don't want to disappoint you, we have never tested digiKam with such a number of imges. Through my photography hobby I have about 60K images over the years, Gilles probably has over 100K. By random generator, I created 200K images as a test. But millions... I think it won't work. A fast internal MySQL DB on SSD is then mandatory anyway. Since we keep all albums in the tree view model, from about 50K albums / folder there is a strong performance problem... > Maik All good, I'm testing it *thoroughly*, although I only have macOS 10.14.5. The debug stage under OSX hurt me. It's definitively too much complicated... As Maik, said, the cardh is located in DMetadata::load() interface which call Exiv2 libary API to handle file metadata. It's know that Exiv2 is not stble enough everywhere... but i normally wrapped all Exiv2 call with C++ exception and protected all Exiv2 calls with Mutex to be thread safe. I tested myself with unit test to see if Exiv2 support separated thread operations, and it's not the case. I reported this problem to Exiv2 team, but making thread safe API is delayed to next release (even if this point is very important for DK, as all the multithreaded inside...) So i suspect a crash due to a cases where Exiv2 code is not protected by C++ exceptions handling (there still plenty of places in source code). To resolve you problem, you need to found which image crash DK while parsing, try to reproduce the crash with Exiv2 CLI tool, and report the problem to Exiv2 Github bugzilla. Gilles Caulier (In reply to caulier.gilles from comment #27) > The debug stage under OSX hurt me. It's definitively too much complicated... > > As Maik, said, the cardh is located in DMetadata::load() interface which > call Exiv2 libary API to handle file metadata. It's know that Exiv2 is not > stble enough everywhere... but i normally wrapped all Exiv2 call with C++ > exception and protected all Exiv2 calls with Mutex to be thread safe. I > tested myself with unit test to see if Exiv2 support separated thread > operations, and it's not the case. I reported this problem to Exiv2 team, > but making thread safe API is delayed to next release (even if this point is > very important for DK, as all the multithreaded inside...) > > So i suspect a crash due to a cases where Exiv2 code is not protected by C++ > exceptions handling (there still plenty of places in source code). > > To resolve you problem, you need to found which image crash DK while > parsing, try to reproduce the crash with Exiv2 CLI tool, and report the > problem to Exiv2 Github bugzilla. > > Gilles Caulier Can you provide me with a debug build log for all relevant files accessed by digiKam that could cause this crash? I can just run that from scratch, after i delete the current database, and the last line in the log should contain the file in question that crashes the app. Another hang on the Timeline tab (beta3-debug): https://gist.github.com/0wnrepo/6bd62ec27806c08de9bcb38cd3563336 Seems that there is too much work done on the main thread (the one the GUI is using) and this is why it blocks the app. DispatchQueue_1: com.apple.main-thread (serial) Stephan, we use Qt loggin rules to enable/disable debug traces. Look here in MacOS section: https://www.digikam.org/contribute/ Warning : this will generate a very verbose output on the console. Typically, the file open by Exiv2 library while scanning will appear. Crash must follow just later. Depending of many core you have on your mac, the real file crashing the library can be one from _n_ steps/cores in the trace, no more. I recommend to isolate album first containing many image. Import only this album and see if crash is reproducible. If yes, removing images step by step will permit to isolate the problematic file. Gilles Caulier (In reply to caulier.gilles from comment #30) > Stephan, > > we use Qt loggin rules to enable/disable debug traces. > > Look here in MacOS section: > > https://www.digikam.org/contribute/ > > Warning : this will generate a very verbose output on the console. > Typically, the file open by Exiv2 library while scanning will appear. Crash > must follow just later. Depending of many core you have on your mac, the > real file crashing the library can be one from _n_ steps/cores in the trace, > no more. > > I recommend to isolate album first containing many image. Import only this > album and see if crash is reproducible. If yes, removing images step by step > will permit to isolate the problematic file. > > Gilles Caulier Thanks, will do this later today. I'll come back if i find something of use. joined you on IRC to avoid spamming here https://webchat.freenode.net/#digikam All the best! Git commit 52c79f56432d1ba894760165d4aef5b9a59ae327 by Maik Qualmann. Committed on 21/04/2020 at 18:10. Pushed by mqualmann into branch 'master'. add note to the album monitoring option M +11 -3 core/utilities/setup/collections/setupcollections.cpp https://invent.kde.org/kde/digikam/commit/52c79f56432d1ba894760165d4aef5b9a59ae327 digiKam 7.0.0 stable release is now published: https://www.digikam.org/news/2020-07-19-7.0.0_release_announcement/ We need a fresh feedback on this file using this version. Best Regards Gilles Caulier I can confirm this still happens in the stable 7.0.0 release under MacOS Catalina. I copy what I wrote in a related bug report (Bug 389652): "I experienced this again in the 7.0.0 stable version on a macbook with MacOS Catalina (but also in Ubuntu some days ago). During the initial scan, while exploring for new pictures (less than 100) on a NAS, the interface was completely frozen, showing the spinning "beach ball" icon, and indicating that digiKam was "Not responding" in MacOS's task manager. The interface responded briefly after each new file that was added (~10MB each, since they were in PNG format), but became immediately irresponsible again. I think this should not happen. A regular user might thing that the program has crashed and try to force close it. The interface could provide more feedback on what digiKam is doing at that moment (showing which file is importing, how many files are left, etc. instead of a simple % bar, Bug #389868). Ideally, the interface should never freeze, so maybe there could be some way of separating the import process from the main interface so one doesn't block the other?" I cannot reproduce the problem on Linux or Windows. The scanning process is carried out in a separate thread and should not stop the GUI main thread. My question, can the problem only be reproduced with a network drive / path or also with a local collection? Maik I think it also happens with a local collection, but since the latency is much lower, the time it takes to import each individual picture is also lower and the effect is less pronounced. I'll run some tests this afternoon, comparing local / network files, and in Windows, Linux and MacOS. The cause could be a QEventLoop in the ScanController thread. There are hints on the web that a QEventLoop blocks the main thread under MacOS. Maik Ok, I did some tests, all of them using the stable 7.0.0 x64 version. I used a set of ~500 pictures, 4.5GB in size, and using a blank database. Basically, when storing the pictures locally, the import process is so fast (around half a minute) that there isn't time to notice any stutters. So no problemes there. When the pictures are in a NAS, the situation changes, but there are differences between OS. - In Windows 10, the interface never freezes. It may stutter every now and then due to the pictures being loaded, but that's it, works just fine. - In Linux (Ubuntu 20.04), it's a hit and miss. Most of the time the interface is responsive, but every now and then it will freeze for a few seconds at a time and showing the "this application is not responding" dialog. It happened probably 4 or 5 times when importing these pictures. I think I might have triggered it if I browsed pictures from the search panel while it was still scanning. - In MacOS (Catalina) the situation was worse; the interface was almost frozen for each file that was being imported, and responded briefly after it finished and before it started importing the next one. I have to clarify that the freezes only happen when new files are being imported. If the initial scan is not finding anything new, the interface will be ok even if the progress bar is still present. https://bugs.kde.org/show_bug.cgi?id=426938 --- Comment #4 from caulier.gilles@gmail.com --- Hi, digiKam 7.2.0-beta2 pre-release PKG installer now support BigSur and is compiled with last stable Qt 5.15.2. https://files.kde.org/digikam/ Problem still reproducible with this version. Thanks and happy Christmas in advance Best Regards Gilles Caulier digiKam 7.2.0 official release is published with more than 360 files closed from bugzilla: https://www.digikam.org/news/2021-03-22-7.2.0_release_announcement/ Can you reproduce the dysfunction with this version ? Thanks in advance for your feedback Gilles Caulier Hi, I can confirm that this problem still persists in Ubuntu 20.04 using the latest flatpak build (Build date: 22/8/21 10:27 (target: Debug) Rev.: 699538c1b3ff85650d617ab716493135fe44c999) Even now that the initial scan is much faster, when there are new images to import the interface becomes really sluggish, not responding to clicks or scroll. Only in the brief period after a picture has been imported to the database, the interface seems to respond again. Obviously the time it takes to import the data depends on the speed of the media, so importing via network this effect is much more noticeable. (this was discussed as part of bug 389652 too) Marc, Stable digiKam 7.4.0 is published. Please check if problem is reproducible. https://www.digikam.org/download/ Thanks in advance Mmm, I don't think nothing changed in that regard. I believe that, in slow networks, the database must be locked when it's being accessed. When scanning new pictures, the interface freezes during the import of each single picture, so it's only responsible for a few moments after each picture has been imported, I guess it's when the database is available again for new queries. So, in practical terms, digikam is only usable once it has finished importing new pictures (either because a new collection has been added, or new pictures have been found during the initial scan). I can confirm that this is still an issue in the 7.7.0 versions. I had an issue with my database and I had to reconstruct it. Right now my pictures are in a remote NAS and while Digikam is importing a few hundreds of GB of pictures (will take several hours) the UI is basically frozen and cannot be used. @Stefan digiKam 8.0.0 is out. Problem still reproducible ? Best regards Gilles Caulier @stefan, digikam 8.2.0 pre-release have been rebuilt using last Qt 5.15.11 + KDE 5.110 frameworks. Installer is available at usual place : https://files.kde.org/digikam/ Can reproduce the problem with this version? Thanks in advance Gilles Caulier @stefan, digiKam 8.3.0 stable version is released and available at usual place : https://www.digikam.org/download/ Can you reproduce the dysfunction on your computer ? Thanks in advance Gilles Caulier @Stefan Please try the new 8.5.0 pre-release PKG installer for MacOS Silicon (arm64) available here: https://files.kde.org/digikam/ Thanks in advance Gilles Caulier |