Bug 479711

Summary:	Find new items at startup very slow (Mac)
Product:	[Applications] digikam	Reporter:	Marshalleq <atoms>
Component:	Database-Albums	Assignee:	Digikam Developers <digikam-bugs-null>
Status:	REPORTED ---
Severity:	normal	CC:	caulier.gilles, metzpinguin
Priority:	NOR
Version:	8.3.0
Target Milestone:	---
Platform:	macOS (DMG)
OS:	macOS
Latest Commit:		Version Fixed In:
Sentry Crash Report:

Description Marshalleq 2024-01-12 21:31:47 UTC

SUMMARY
***
NOTE: If you are reporting a crash, please try to attach a backtrace with debug symbols.
See https://community.kde.org/Guidelines_and_HOWTOs/Debugging/How_to_create_useful_crash_reports
***
Further to this bug here: https://bugs.kde.org/show_bug.cgi?id=478674 scanning for new files on Mac is still abominably slow. Digikam is effectively completely unusable in this state. I'm talking multiple hours for a single 'fast' scan on either the NAS or the locally connected HDD. SSD seems to be OK, but I don't have many files on that. I have a large library of 90k images I've taken over the years. I've come from various commercial software (Lightroom, Exposure etc) none of them seem to exhibit this problem though Exposure without having a catalog is slowish to load the previews, but even that is not this slow.

I have the good fortune of being able to test SSD, external HDD and NAS alongside Windows, Mac and Linux and I haven't so far been able to find anything the performs well except local SSD. The Linux client I have skipped for now based on the other bug needing Digikam 8.3 and no 8.3 version is easily available for OpenSuse. I could try another bistro, but I may be better to downgrade everything to 8.1 and try again.

I am at a loss to understand what is causing this, I've turned scan for new files at startup off as it's completely pointless at present and I did have the fast scan option turned on.

STEPS TO REPRODUCE
1. Open Digikam
2. Wait for hours for scan to finish finding new files
3. Check

OBSERVED RESULT
Many hours to scan

EXPECTED RESULT
seconds if not minutes to scan

SOFTWARE/OS VERSIONS
macOS: Sonoma, Windows 10 (can try linux)

ADDITIONAL INFORMATION
My main library is on a ZFS pool on a NAS connected via a 1G cable connection. Mac never seems to be quite as fast so I've also tried connecting digikam via windows. I simply can't buy disks big enough and reliable enough to not use a NAS. Some of the library does include some large 300M tiff files, though even if I isolate those, the problem still persists.

I have copied a portion to an external drive to see if that would speed it up, but it doesn't.

I read somewhere that I can turn on some kind of debugging mode and provide logs, but I can't seem to find how yet, that might be the best place to start?

Thanks.

Comment 1 Maik Qualmann 2024-01-12 21:57:59 UTC

How a log is created under macOS is described here:

https://www.digikam.org/contribute/

Well, 90,000 images isn't that much. In a normal scan, we read the file date information from all of these images. In a quick scan, only the file date information of the albums (folders), so probably only 1-5 thousand date information. If even that takes a long time, something must be very wrong. The code for the scan function is actually already highly optimized, I don't see any potential there anymore. There is of course the possibility that there is poor interaction between Qt and macOS or something else. I'll add a debug this weekend that will output the current number of files processed per second.

What also surprises me and I have read several times is that it is hardly possible to work with digiKam during the scanning process. After all, the scanning process runs in its own thread.
I just added a large external hard drive to my Windows test computer today with initial scan. I was able to work with digiKam without any problems.

Maik

Comment 2 Marshalleq 2024-01-13 02:43:09 UTC

Thanks for the reply.  The unstable comment is not at all related to the performance of the non scanning parts of digikam, just that to manage my media, I need to know that the media is up to date, and to know that I have to wait hours for a scan to finish.

I'm sure the logs will provide something useful.

I'm just going to check my sidecar (XMP) settings as I do have sidecar files, perhaps even reset to default somehow, but even so I don't think that would make it slow to over an hour right?

Comment 3 Marshalleq 2024-01-13 03:04:35 UTC

I've enabled debug and console logging thank you.

Running digikam like this, and adding files from a local HDD, the find new items got to 100% quite quickly, but it's stuck on 100% now and presently there are a lot of these in the logs spinning by.  One for every file I assume.

```
digikam.metaengine: Loading metadata with "Exiv2" backend from "/Volumes/5TB/pictures/Family/Masters/2015/January/Trip to Whangarei/_MG_5465.CR2"
digikam.dimg: "/Volumes/5TB/pictures/Family/Masters/2015/January/Trip to Whangarei/_MG_5465.CR2" : "RAW" file identified
digikam.database: Adding new item "/Volumes/5TB/pictures/Family/Masters/2015/January/Trip to Whangarei/_MG_5465.CR2"
digikam.database: Recognized "/Volumes/5TB/pictures/Family/Masters/2015/January/Trip to Whangarei/_MG_5465.CR2" as identical to item 11650
digikam.database: Scanning took 39 ms
digikam.database: Finishing took 0 ms

```
These files were on the NAS and are now on a local drive, so I think what it's doing is matching them up in the cache?  Perhaps this will be a one time thing.

Will see when it finishes if it repeats itself.

Comment 4 caulier.gilles 2024-09-17 12:06:31 UTC

@atoms

Please try the new 8.5.0 pre-release PKG installer for MacOS Silicon (arm64)
available here:

https://files.kde.org/digikam/

Thanks in advance

Gilles Caulier