SUMMARY After upgrading from DK5.9 to DK6.0, moving any RAW file to a new path can cause loss of metadata. Tested with DK6.0b3 (20181215) STEPS TO REPRODUCE 1. Start with DK5.9 or earlier database, with RAW files in database (tested with CR2 and RAF). 2. Enable metadata setting 'read from sidecar' 3. Disable metadata setting 'write to sidecar' 4. Apply any metadata to RAW, such as caption or tags. 5. Run DK6.0, and using DK6.0 interface move the location of any RAW file or folder containing files to a new path within the collection. OBSERVED RESULT RAW files lose any metadata that is not saved to sidecar and lose any other features such as groups. In the database the RAW files at the new path are given new Image IDs even though their uniqueHash is unchanged - the old Image IDs remain unused. EXPECTED RESULT RAW files re-use same Image IDs and keep all metadata and features, even if it is not saved to sidecar. ADDITIONAL INFORMATION When 'read from sidecar' is used the Image records modification date is collected from the sidecar file, not the RAW itself. If this modification date does not match to the millisecond then the file does not match and a new Image ID is created. Coming from DK5.9 the modification date was stored with second accuracy only, so the first time any RAW is moved it will trigger this bug.
It has nothing to do with a DATETIME(3) field. I'm watching tonight if I can reproduce the problem. When reading sidecar files, the modification date is now also queried to determine if there was an external change. In your configuration, if you do not write in sidecars, the metadata option "Cleanup Database if reread metadata" should NOT be enabled, otherwise the metadata in the database will be replaced by the sidecar/image metadata. If the images are moved within the same collection? Or between different drives/partitions? Maik
I can not confirm or reproduce the loss of metadata. Therefore, I suspect that you have enabled the metadata option "Cleanup Database ...". The creation of a new image ID is done only when moving between different collections because the album root ID changes. This behavior is exactly the same in digiKam 5.9.0. I will see if I can optimize this behavior. Maik
I have not enabled "Cleanup Database". I am testing with a single collection, on a single partition. I believe there is an additional condition that triggers this bug and it is related to the filesystem timestamps of the photos. Using 'stat' on my system shows nanosecond accuracy. Some files have not been modified for some years and show all zeros for the trailing digits, eg. Modify: 2013-01-29 14:31:22.000000000 other files have been modified recently and have use all the trailing digits, eg. Modify: 2018-12-23 01:06:52.322555833 This bug appears to be related to the files timestamp has non-zeros in the trailing digits. Furthermore this bug only triggers when a FOLDER is moved, it does not appear to trigger when an individual file is moved. STEPS TO REPRODUCE 1. Create at least 2 test folders with RAW files (include xmp sidecars to test with sidecars enabled) At least one RAW/xmp file should have full resolution timestamp. 'touch' file if necessary to update timestamp and use 'stat' to confirm. 2. Start DK5.9 and add collection with these folders. At this point you can choose whether to test the RAW or xmp timestamps. Select Metadata option 'Read from sidecar files' to test the xmp timestamps. 3. Add some metadata to the file that you 'touched', such as caption or tag. 3. Close DK5.9 4. Open DK6.0 5. Using Digikam album list, move the folder - such as dragging and dropping onto a subfolder. OBSERVED RESULT The metadata added in step 3 will have disappeared. And inspecting the Images database table will show duplicate records with same filename and uniqueHash. ADDITIONAL INFORMATION I have tested this on Ubuntu 18.04.1 with external database MariaDB v10.2.19 Altering the Images table to use DATETIME(3) for 'modificationDate' and then running 'Reread metadata from image' appears to mitigate this bug.
It definitely does not matter, because SQLite has always saved with 3 additional digits. Such a date is scanned at most once and now the more accurate date is stored. But no metadata are lost. Are there any LR sidecar files in the directory? Digikam-6b3 will first look for sidecars in LR format and use these if they exist. I will go over their instructions step by step again. I see it right that their database is in the state of digiKam-5.9.0, since a back from 6.0.0 would not be possible because of the higher version number. Then the 3 additional digits will not exist after the start of digiKam-6.0.0 with the migration to V10. To get the extra 3 digits you need to create a new database and use the migration tool to convert the database. I'll check... Maik
Git commit 245740b1f390c9998a32a3e81f31c7be6cfb26a5 by Maik Qualmann. Committed on 22/12/2018 at 21:22. Pushed by mqualmann into branch 'master'. various fixes to find a moved item in the database and use it FIXED-IN: 6.0.0 M +2 -1 NEWS M +14 -15 core/libs/database/coredb/coredb.cpp M +6 -8 core/libs/database/coredb/coredb.h M +1 -1 core/libs/database/item/scanner/itemscanner.h M +11 -6 core/libs/database/item/scanner/itemscanner_database.cpp https://commits.kde.org/digikam/245740b1f390c9998a32a3e81f31c7be6cfb26a5
I could reproduce the problem. The fact that it did not occur in digiKam-5.9.0 is also a bug when writing metadata to the database. Now that we use the modification date from either the image or the sidecar, it can not be part of the recognition of an image when searching for images with album id == NULL. Maik
Git commit a24ab41458ec5a73a6aeb470cccefdd7ffbe449f by Maik Qualmann. Committed on 23/12/2018 at 08:08. Pushed by mqualmann into branch 'master'. reset grouped items cache M +4 -0 core/libs/database/item/containers/iteminfocache.cpp https://commits.kde.org/digikam/a24ab41458ec5a73a6aeb470cccefdd7ffbe449f
Git commit db18a967f684e7c35e27f9c4c06b72b7fc7d170d by Maik Qualmann. Committed on 23/12/2018 at 13:15. Pushed by mqualmann into branch 'master'. disable external album scans when processing internal files M +6 -0 core/libs/album/engine/albumwatch.cpp M +24 -11 core/libs/database/utils/ifaces/dio.cpp M +6 -2 core/libs/database/utils/ifaces/dio.h https://commits.kde.org/digikam/db18a967f684e7c35e27f9c4c06b72b7fc7d170d