Summary: | MP4 scans inconsistently chooses wrong "creation date" with ffmpeg - Use ExifTool instead | ||
---|---|---|---|
Product: | [Applications] digikam | Reporter: | Kurt Granroth <granroth> |
Component: | Metadata-ExifTool | Assignee: | Digikam Developers <digikam-bugs-null> |
Status: | RESOLVED FIXED | ||
Severity: | normal | CC: | caulier.gilles, metzpinguin |
Priority: | NOR | ||
Version: | 7.1.0 | ||
Target Milestone: | --- | ||
Platform: | macOS (DMG) | ||
OS: | macOS | ||
Latest Commit: | Version Fixed In: | 8.0.0 | |
Sentry Crash Report: |
Description
Kurt Granroth
2020-09-25 01:07:03 UTC
It figures that after months of seeing this with no known cause or workaround, I have potentially found both literally minutes to hours after posting this bug. Maybe a bit of rubber duck debugging. Anyway, the light bulb went off initially when doing an Import->Add Images rather than a copy-and-auto-scan and seeing that not only were all of the dates correct, but it also took a decent amount of time to import. Why so much time? Well, these files are quite large -- some are multi-gigabyte and it simply takes notable time to copy that much data on a mechanical HDD. Oh... could that be it? I did a test were I did a manual copy of some video files into another directory and noticed that MacOS reports the 'creation time' of the files as 'right now' for the duration of the copy and only sets the actual 'creation time' to match the original AFTER the copy is completed. So could this be a race condition? Is Digikam maybe monitoring the albums/directories on a set polling basis and could it simply be encountering the "wrong date" video files BEFORE the copying is completed? Seems plausible. With that in mind, there is a potential workaround (not counting the Import) and that is to run Tools -> Maintenance -> Sync Metadata and Database on the given albums after everything is successfully copied over. I make sure the sync direction is "From image metadata to database". That appears to work like a charm. Another option might be to disable auto-scan and just remember to Tools -> Scan for new items regularly. If it really is just a race condition then it is still a bug (shouldn't be adding the file until its state has solidified) but maybe that's not a bug worth fixing since it's such an esoteric case and has known workarounds? Hi, First video metadata are handle by ffmpeg, not Exiv2 and ExifTool. Second, ffmpeg will try to parse files to extract metadata. If large files are under process to be copied and are uncomplete, the parse will fail and current file date from file system will be used instead. This explain the dysfunction about the date. Gilles Caulier Normally we append a "digikamtempfile" to the file name during the copying process. The parser ignores these files, so there shouldn't be any problems during the copying process. I'll take a look at it. Maik Hi, digiKam 7.2.0-beta2 pre-release PKG installer now support BigSur and is compiled with last stable Qt 5.15.2. https://files.kde.org/digikam/ Problem still reproducible with this version. Thanks and happy Christmas in advance Best Regards Gilles Caulier Kurt, To solve this issue, if it relevant of Exiv2 or ffmpeg, we will have the possibility later to use ExifTool with Video File instead ffmpeg. With DK 7.3.0, ExifTool is now supported has metadata backend. For the moment only the viever in right sidebar use ExifTool, but we plan to use ExifTool to handle special cases or dysfunctions not working with Exiv2 or ffmpeg. If you can share the famous video file taken with your gopro (or a short sample) to test here, it will help to investigate. Best regards Gilles Caulier Giles, thank you so much for investigating this. You are definitely going down an excellent trail. Unfortunately, it's not feasible for me to upload a representative video sample. As I mentioned in the original report, this behavior almost exclusively happens with large files. The GOPR0026.MP4 given as an example, for instance, is 1.35GB!! Alas, I simply don't see this issue with smaller and more reasonable files. Git commit b3c56d4cdedb4ad83c7baca89c32d011f1738090 by Gilles Caulier. Committed on 26/04/2021 at 08:58. Pushed by cgilles into branch 'master'. Use EXV container to translate automatically all metadata from ExifTool to Exiv2 Related: bug 421464, bug 416516, bug 360714 M +23 -76 core/libs/metadataengine/exiftool/exiftoolparser.cpp M +2 -5 core/libs/metadataengine/exiftool/exiftoolparser.h M +13 -38 core/tests/metadataengine/exiftool/exiftoolexport_cli.cpp https://invent.kde.org/graphics/digikam/commit/b3c56d4cdedb4ad83c7baca89c32d011f1738090 Git commit 183a05476b1fa485e5143e0bb28d84e3a8daa471 by Gilles Caulier. Committed on 26/04/2021 at 09:08. Pushed by cgilles into branch 'master'. Remove obsolete ExifToolTranslator class. We use now the auto conversion to Exiv2 through EXV container processed by ExivTool Related: bug 421464, bug 416516, bug 360714 M +0 -6 core/libs/metadataengine/CMakeLists.txt M +10 -114 core/libs/metadataengine/exiftool/exiftoolparser.cpp D +0 -81 core/libs/metadataengine/exiftool/exiftooltranslator.cpp D +0 -65 core/libs/metadataengine/exiftool/exiftooltranslator.h D +0 -196 core/libs/metadataengine/exiftool/exiftooltranslator_exif.cpp D +0 -51 core/libs/metadataengine/exiftool/exiftooltranslator_ignoredgroups.cpp D +0 -164 core/libs/metadataengine/exiftool/exiftooltranslator_iptc.cpp D +0 -76 core/libs/metadataengine/exiftool/exiftooltranslator_p.cpp D +0 -73 core/libs/metadataengine/exiftool/exiftooltranslator_p.h D +0 -439 core/libs/metadataengine/exiftool/exiftooltranslator_xmp.cpp M +0 -1 core/libs/widgets/metadata/exiftool/exiftoollistview.cpp M +0 -1 core/tests/metadataengine/exiftool/CMakeLists.txt D +0 -318 core/tests/metadataengine/exiftool/exiftoolxmldb_cli.cpp https://invent.kde.org/graphics/digikam/commit/183a05476b1fa485e5143e0bb28d84e3a8daa471 Git commit 5b097916d8bfa42fef5a2966fa3b34d0793cf0e2 by Gilles Caulier. Committed on 26/04/2021 at 11:02. Pushed by cgilles into branch 'master'. separate ExitTool parser and process classes internal container implementations Related: bug 416516, bug 360714 M +2 -0 core/libs/metadataengine/CMakeLists.txt M +15 -154 core/libs/metadataengine/exiftool/exiftoolparser.cpp M +0 -11 core/libs/metadataengine/exiftool/exiftoolparser.h A +132 -0 core/libs/metadataengine/exiftool/exiftoolparser_p.cpp [License: GPL (v2+)] A +83 -0 core/libs/metadataengine/exiftool/exiftoolparser_p.h [License: GPL (v2+)] M +10 -209 core/libs/metadataengine/exiftool/exiftoolprocess.cpp M +10 -20 core/libs/metadataengine/exiftool/exiftoolprocess.h A +171 -0 core/libs/metadataengine/exiftool/exiftoolprocess_p.cpp [License: GPL (v2+)] A +107 -0 core/libs/metadataengine/exiftool/exiftoolprocess_p.h [License: GPL (v2+)] M +0 -1 core/tests/metadataengine/exiftool/exiftoolmulticore_cli.cpp M +0 -1 core/tests/metadataengine/exiftool/exiftooloutput_cli.cpp M +1 -1 core/tests/metadataengine/exiftool/exiftoolparserout_cli.cpp https://invent.kde.org/graphics/digikam/commit/5b097916d8bfa42fef5a2966fa3b34d0793cf0e2 Git commit 1dff0b016bc15918440a807863757376e297c53e by Gilles Caulier. Committed on 27/04/2021 at 09:27. Pushed by cgilles into branch 'master'. With load EXV chunk method, also backport all possible metadata to XMP. This permit to get all main information from video files not supported by Exiv2, as Quicktime for MP4 Related: bug 416516 M +1 -0 core/libs/metadataengine/exiftool/exiftoolparser.cpp https://invent.kde.org/graphics/digikam/commit/1dff0b016bc15918440a807863757376e297c53e Git commit febc26188fe4e6037e5c8a9b1840f65729972a47 by Gilles Caulier. Committed on 06/04/2022 at 20:38. Pushed by cgilles into branch 'master'. Add ExifTool backend to load metadata for image or video. Exiv2 is always used in priority. If it fail, ExifTool is used instead. ExifTool has the capability to create an EXV container for Exiv2. All metadata extracted with ExifTool are concatened to the EXV container with all possible translations with Exif, Iptc, and Xmp tags. Of course tags supported by ExifTool but not supported by Exiv2 are lost. Related: bug 385726, bug 416516, bug 360714, bug 264210, bug 419801, bug 419375, bug 103247 M +1 -0 core/libs/metadataengine/CMakeLists.txt M +7 -0 core/libs/metadataengine/dmetadata/dmetadata.h A +94 -0 core/libs/metadataengine/dmetadata/dmetadata_exiftool.cpp [License: GPL (v2+)] M +42 -17 core/libs/metadataengine/dmetadata/dmetadata_fileio.cpp M +7 -6 core/libs/metadataengine/engine/metaengine.h M +5 -0 core/libs/metadataengine/engine/metaengine_fileio.cpp M +4 -2 core/libs/metadataengine/exiftool/exiftoolparser_command.cpp https://invent.kde.org/graphics/digikam/commit/febc26188fe4e6037e5c8a9b1840f65729972a47 ExifTool backend is now used to scan video metadata and date is now properly extracted. Gilles Caulier |