Bug 465635 - Geolocate is too slow: it reads way too much data
Summary: Geolocate is too slow: it reads way too much data
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Geolocation-Workflow (other bugs)
Version First Reported In: 7.9.0
Platform: Kubuntu Linux
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-02-12 19:23 UTC by fotograaf
Modified: 2023-07-17 20:54 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed/Implemented In: 8.2.0
Sentry Crash Report:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description fotograaf 2023-02-12 19:23:56 UTC
SUMMARY
***
My pictures are on a networked drive over a gigabit network.
When I geolocate hundreds of images at once, it takes an extremely long time.
System usage monitor reveals that geolocate is reading many, many gigabytes of data - presumably the pictures. 
This takes a LOOOONG time.  And it's unneccesary; all the data is already available.

On applying the data, same story: It only needs to write the XMP files. Instead it reads many many gigabytes of data - again. 
***


STEPS TO REPRODUCE
1. Have 500+ pictures on a network drive 
2. Geolocate them
3. be surprised at the very long time it takes to read all the data that is already available
4. be surprised at the very long time it takes to apply the geolocation which should only be a small XMP-write.

OBSERVED RESULT
System monitor reports digikam is reading gigs of data, as if it is importing these pictures for the very first time.

EXPECTED RESULT
To be much faster since all the data is already available.

SOFTWARE/OS VERSIONS
Linux Kubuntu, all versions

ADDITIONAL INFORMATION
Comment 1 Maik Qualmann 2023-02-13 12:15:51 UTC
In order to write metadata, the file must first be read. Regardless of whether it is Exiv2 or ExifTool, the file is always completely rewritten. The metadata of the file is also required when creating sidecars. The digiKam database contains only a selection of metadata. There won't be any major optimization here, but you could still have a problem.
Which digiKam version are you using exactly? Is the Ubuntu digiKam Snap package used with the sandbox restriction?
Please enable internal debugging in the digiKam Setup under Miscellaneous-> System and restart digiKam in the terminal. Perform a write operation on some files and post the output in the terminal.

Maik
Comment 2 fotograaf 2023-02-13 14:29:28 UTC
I am using the Linux appimage, not snap. 

I see what you mean however, when adding tags this does not seem to work this way.  They can be added with far less diskreads&writes.
And at the end of the day, geolocation data is also just a tag. 
They seem to work different, tags and geolocation. 
It's like the tagging says 'ok, I know this file, add this tag and its XMP data, just write it in the XMP'.
And when Geolocation is started, it's like it's a completely new program which has not seen these files ever before - and therefore has to read them from scratch.

Again, in the main window when you tag a file, it's not completely re-read.
So there the necessary information to perform tagging is present.
Only the Geolocation module acts like it does not get this info. 


Extra info: It's much faster when files already have a geolocation. It doesn't read the entire file first.



Debug: This is for a single file.  Is it me or does it read the metadata (which the tagging module already has) 3 times??


^[[34mDigikam::DMetadata::load^[[0m: Loading metadata with "Exiv2" backend from "/media/erin_only/home/Photos/Mobile/PhoneCamera/Camera/IMG_20140717_093012.jpg"
^[[34mDigikam::DMetadata::load^[[0m: Loading metadata with "Exiv2" backend from "/media/erin_only/home/Photos/Mobile/PhoneCamera/Camera/IMG_20140717_093012.jpg"

^[[34mDigikam::DMetadata::load^[[0m: Loading metadata with "Exiv2" backend from "/media/erin_only/home/Photos/Mobile/PhoneCamera/Camera/IMG_20140717_093012.jpg"
^[[34mDigikam::MetaEngine::save^[[0m: MetaEngine::metadataWritingMode 1
^[[34mDigikam::MetaEngine::save^[[0m: Will write XMP sidecar for file "IMG_20140717_093012.jpg"
^[[34mDigikam::ItemLister::listSearch^[[0m: Search result: 33973
^[[34mDigikam::MetaEngine::Private::saveOperations^[[0m: wroteComment:  false
^[[34mDigikam::MetaEngine::Private::saveOperations^[[0m: wroteEXIF:  true
^[[34mDigikam::MetaEngine::Private::saveOperations^[[0m: wroteIPTC:  true
^[[34mDigikam::MetaEngine::Private::saveOperations^[[0m: wroteXMP:  true
^[[34mDigikam::MetaEngine::save^[[0m: Metadata for file "IMG_20140717_093012.jpg" written to XMP sidecar.
Comment 3 caulier.gilles 2023-05-03 04:02:32 UTC
@fotograaf

digiKam 8.0.0 is out. This entry still valid with this release ?

Best regards

Gilles Caulier
Comment 4 Maik Qualmann 2023-07-17 20:54:01 UTC
Git commit 2211b7b2b3a5db1b6c48f6e47f55f43023d6abef by Maik Qualmann.
Committed on 17/07/2023 at 20:52.
Pushed by mqualmann into branch 'master'.

under digiKam the geolocation tool now has a pure DB interface
Move ItemGPS class so we can link against the MetadataHub.
FIXED-IN: 8.2.0

M  +1    -1    NEWS
M  +7    -7    core/libs/database/CMakeLists.txt
R  +25   -12   core/libs/database/utils/ifaces/itemgps.cpp [from: core/libs/database/item/containers/itemgps.cpp - 080% similarity]
R  +0    -0    core/libs/database/utils/ifaces/itemgps.h [from: core/libs/database/item/containers/itemgps.h - 100% similarity]
M  +6    -17   core/utilities/geolocation/geoiface/items/gpsitemcontainer.cpp
M  +0    -4    core/utilities/geolocation/geoiface/items/gpsitemcontainer.h

https://invent.kde.org/graphics/digikam/-/commit/2211b7b2b3a5db1b6c48f6e47f55f43023d6abef