| Summary: | Very slow maintenance process with mariadb | ||
|---|---|---|---|
| Product: | [Applications] digikam | Reporter: | maderios <leoutation> |
| Component: | Database-Mysql | Assignee: | Digikam Developers <digikam-bugs-null> |
| Status: | REPORTED --- | ||
| Severity: | normal | CC: | metzpinguin |
| Priority: | NOR | ||
| Version: | 8.3.0 | ||
| Target Milestone: | --- | ||
| Platform: | Arch Linux | ||
| OS: | Linux | ||
| Latest Commit: | Version Fixed In: | ||
| Sentry Crash Report: | |||
| Attachments: | rescan debug log | ||
|
Description
maderios
2023-12-15 12:57:24 UTC
I haven't looked at the valgrind file yet, I'll do that late this evening. But just a thought, ExifTool has been on board since digiKam-8.0.0. So if you have a lot of files (videos or similar) or Exiv2 has an exception with your images, ExifTool will automatically read the metadata. This of course slows down the process. I definitely can't reproduce a drop in speed myself. Maik I have only 2 video files (2 .mkv) in my test directory/album, other files are .jpeg and .png (225 files) From other bug reports I know that you also have XCF files. Maik (In reply to Maik Qualmann from comment #3) > From other bug reports I know that you also have XCF files. > > Maik Yes, two XCF files in directory/album i used to test with valgrind today. They don't slow down but I deleted them. In an other account, I have no XCF files but big png files. I see scanning these big png takes about from 300 ms to 1100 ms. (In reply to Maik Qualmann from comment #1) > I haven't looked at the valgrind file yet, I'll do that late this evening. > But just a thought, ExifTool has been on board since digiKam-8.0.0. So if > you have a lot of files (videos or similar) or Exiv2 has an exception with > your images, ExifTool will automatically read the metadata. This of course > slows down the process. I definitely can't reproduce a drop in speed myself. > > Maik Did you try to reproduce it with big files, like png? They slow down process. It takes more one second to scan some of them. Yes, they may be scanned with ExifTool, because PNG is definitely a candidate where Exiv2 fails. A debug log from the terminal can quickly clarify things. Maik Created attachment 164223 [details]
rescan debug log
See maintenance rescan debug log (about 400 images)
I use a fresh account for these tests.
In fact, ExifTool has to intervene a few times to read metadata, with times between 60-500-1100ms per image. But what is noticeable is that images are processed twice. You also selected the “All Tags” option. Depending on the tag, this can actually lead to images being processed twice. I will change this situation. Maik (In reply to Maik Qualmann from comment #8) > In fact, ExifTool has to intervene a few times to read metadata, with times > between 60-500-1100ms per image. Not really "A few times": if we say it takes only 200 ms per image, 20 000 images will take 4000 s to scan, more than one hour. I think it's much more because I have many big files like png, xcf, .nef, with many sidecars/xmp tags (write to XMP sidecar only) > But what is noticeable is that images are processed twice. You also selected > the “All Tags” option. Depending on the tag, this can actually lead to > images being processed twice. I will change this situation. > > Maik I forgot to say i found other errors with an other digikam user account. I have many messages like this: digikam.metaengine: Exiv2 ( 2 ) : IPTC dataset Iptc.0x001c.0x0002 has invalid size 16640; skipped. digikam.metaengine: Exiv2 ( 2 ) : Failed to decode IPTC metadata. I don't use IPTC Git commit bc4a263fefbbf58cffd5ad55d50387e4acd9a4e6 by Maik Qualmann. Committed on 16/12/2023 at 17:32. Pushed by mqualmann into branch 'master'. prevent double processed images when tags are also selected M +14 -2 core/utilities/maintenance/autotagsassignment.cpp M +14 -2 core/utilities/maintenance/fingerprintsgenerator.cpp M +14 -2 core/utilities/maintenance/imagequalitysorter.cpp M +15 -2 core/utilities/maintenance/thumbsgenerator.cpp https://invent.kde.org/graphics/digikam/-/commit/bc4a263fefbbf58cffd5ad55d50387e4acd9a4e6 (In reply to maderios from comment #10) > I forgot to say i found other errors with an other digikam user account. I > have many messages like this: > digikam.metaengine: Exiv2 ( 2 ) : IPTC dataset Iptc.0x001c.0x0002 has > invalid size 16640; skipped. > digikam.metaengine: Exiv2 ( 2 ) : Failed to decode IPTC metadata. > I don't use IPTC Exiv2 throws an exception because it detected unresolvable problems/errors in the images. In these cases we then read the metadata using ExifTool, which is a little slower. But it's still better than having incorrect metadata or no metadata at all. Maik I have a comparison here with the AppImage digiKam-8.0.0 (older AppImages unfortunately no longer work here) and my current Qt6 version shows no difference when creating thumbnails and fingerprints. Maik |