Bug 385726 - Add Support For Motion Photos - Use ExifTool to extract data
Summary: Add Support For Motion Photos - Use ExifTool to extract data
Status: REOPENED
Alias: None
Product: digikam
Classification: Unclassified
Component: Metadata-ExifTool (show other bugs)
Version: 7.0.0
Platform: unspecified All
: NOR wishlist with 36 votes (vote)
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-10-14 03:31 UTC by thbarnes
Modified: 2022-04-06 20:54 UTC (History)
9 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
Screenshot digikam Live Photo as two separate entries (193.59 KB, image/png)
2019-06-11 07:26 UTC, Glassed Silver
Details
Apple Photos Live Photo View (2.41 MB, video/quicktime)
2019-06-11 07:27 UTC, Glassed Silver
Details

Note You need to log in before you can comment on or make changes to this bug.
Description thbarnes 2017-10-14 03:31:21 UTC
macOS users who switch to Linux should not have to lose their Live Photos.

They are simply photos with an H.264 video embedded.

There are FOSS libraries that can be ported.

https://developer.apple.com/documentation/livephotoskitjs
Comment 1 thbarnes 2017-10-14 03:34:12 UTC
A library from Tumblr
https://github.com/tumblr/laphs

"a Live Photo consists of a JPG combined with a MOV file that contains 45 frames playing back at around 15 frames per second." https://www.macrumors.com/2015/09/22/iphone6s-live-photos-file-details/
Comment 2 caulier.gilles 2017-10-14 08:29:46 UTC
Currently, how is detected this kind of file in digiKam ? As a video or as an image ?

We need also some samples to test. Please provide files. Thanks in advance...

Gilles Caulier
Comment 3 caulier.gilles 2017-10-14 08:34:52 UTC
It's not clear here :

https://stackoverflow.com/a/34335994

one or 2 separate files are provided ?

Gilles Caulier
Comment 4 caulier.gilles 2017-10-20 04:11:28 UTC
All link to this file about HEIF support for digiKAm are JSon library.

DK is based on C++ code, not JSon or another language.

Nokia provide an open source lib to handle HEIF images :

https://github.com/nokiatech/heif

Gilles Caulier
Comment 5 caulier.gilles 2019-04-08 15:23:04 UTC
Git commit 86a40546894c7b053ab3aad85a11a0c709e60220 by Gilles Caulier.
Committed on 08/04/2019 at 15:18.
Pushed by cgilles into branch 'master'.

Add ImageMagick codecs support as QImage fail-back image loader to be able
to render thumbnails and load image in editor with extra image format as HEIC, FITS, JPEG-XR, XCF.
For the moment, this will support image loading in this formats. Later we can plan to add writting
support if ImageMagick codecs can do it.
Related: bug 360806, bug 393408
FIXED-IN: 6.1.0

M  +1    -0    Mainpage.dox
M  +1    -0    NEWS
M  +8    -1    core/CMakeLists.txt
M  +4    -0    core/app/DigikamCoreTarget.cmake
M  +3    -0    core/app/utils/digikam_config.h.cmake.in
M  +10   -0    core/libs/dialogs/libsinfodlg.cpp
M  +2    -1    core/libs/dimg/loaders/README
M  +58   -2    core/libs/dimg/loaders/qimageloader.cpp
M  +5    -0    core/libs/dimg/loaders/qimageloader.h

https://commits.kde.org/digikam/86a40546894c7b053ab3aad85a11a0c709e60220
Comment 6 Glassed Silver 2019-06-11 06:26:17 UTC
Just tried adding a Live Photo to digikam 6.1 on macOS and all it produces is two entries for the image and the video part each. It doesn't show the two files as one entry as you'd expect coming from Apple Photos.

If this is how it's supposed to work, it's a wrong implementation and not doing anything more than just importing the two files as presented by ANY other software. I would not call this Live Photo support then. However I assume this might be an incompatibility with the macOS port yet?

Should I try the GNU/Linux version or is there any other thing to keep in mind?
Comment 7 caulier.gilles 2019-06-11 07:09:43 UTC
2 entries for one image and on video ?

I don't understand the dysfunction exactly. Please :

- take a shot of digiKam dysfunction.
- take a shot of iphoto view with files
- share files + XMP sidecars to try to reproduce the problem here.
- take a shot of DK/Setup/Metadata pannel.

Gilles Caulier
Comment 8 Glassed Silver 2019-06-11 07:26:02 UTC
Wooops, no not 2 for each, I changed my phrasing mid-sentence and then that happened. Sorry.

It's one entry for the image and video part each. So it shows the technically correct two files as what they are - technically - 2 files. However the whole point of Live Photos isn't that the capture process is bundled, but also viewing the image alongside the video seamlessly.

Default settings basically beyond some preferences during install wizard. SQLite db, locally stored. No sidecar files.

Attached screenshot of digikam and a short video screencap of Apple Photos viewing a Live Photo. As you can see it only displays one entry visually for the IMG_nnn.JPG & IMG_nnn.MOV pair and on mouse hover it will start playing the video bit without sound and if you go into view mode you can play with sound there. (latter not shown)
Comment 9 Glassed Silver 2019-06-11 07:26:42 UTC
Created attachment 120774 [details]
Screenshot digikam Live Photo as two separate entries
Comment 10 Glassed Silver 2019-06-11 07:27:49 UTC
Created attachment 120775 [details]
Apple Photos Live Photo View
Comment 11 caulier.gilles 2019-06-11 08:30:30 UTC
Ok, now i understand.

So typically, it's a items group import in database. I'm sure that it's not implemented in DK.

The JPEG + MOV file are grouped in Iphoto. In DK we can do the same but in the GUI there is nothing to switch from a JPEG to MOV thumbnails view when there are grouped.

This must be a topic well described in a new bugzilla entry as a wish.

Gilles Caulier
Comment 12 Glassed Silver 2019-06-11 13:09:50 UTC
I think if we're going to implement a unified view for the JPG (or HEIF) and movie pair we shouldn't use the regular grouping function, because then we couldn't group multiple Live Photos without creating a messy situation when you ungroup them again.

So in a way, we'd probably need a "level zero grouping" so to say. (sorry, can't find a good way to phrase it)

You say we should create a new ticket for that then? Not sure about that, but I could do that. Whilst we're at it we might as well implement other popular Live Photo-like implementations as well and use the same unified view for that. Samsung has Motion Photos, Google has Motion Photos as well on Pixel devices and some Android One phones as well I think (I know the Nokia 3.1 has that too for example) and then Huawei has an implementation very much like Apple's too.

They all work a little different, but technically it's always a pair of image and movie files being fused into one element visually. Samsung for example embeds the video into the image file itself (making the JPG bigger in size).

I know this might sound like a lot at once, but I swear to God, despite me owning a nice and lovely Canon DSLR often times I will at least take a picture a second time with my smartphone just to capture a Live Photo. It really adds that much more to pictures and lets you relieve moments a bit more vividly that didn't seem like a "video moment" but the little bit of motion can often times add a lot.
Comment 13 Ritesh Raj Sarraf 2019-07-10 09:45:29 UTC
Google has a similar feature in the form of motion images. Should I file a separate but report? Our can we track it on this report?
Comment 14 caulier.gilles 2019-07-10 14:11:30 UTC
Yes please, fill a new file, as this one is already closed.

Thanks in advance

Gilles Caulier
Comment 15 Glassed Silver 2020-03-01 12:40:22 UTC
Why is this item flagged as resolved?

Live Photos are still just presented as a separate JPG/HEIC and a .mov.


Here's some further reading if it helps grasp the concept of how it all works how it's designed to be a) detected b) treated/viewed.

1) Very nice conclusive overview post on Reddit, also links to sample images for Apple's version, but also Google's. https://www.reddit.com/r/DataHoarder/comments/cuk06h/help_preserving_iphone_live_photos_how_they_seem/exx2rj6/

2) This is another program written in C++ that to my knowledge correctly implements Live Photos: https://github.com/LycheeOrg/Lychee-Laravel/issues/378

3) Apple's HIG on how Live Photos should be treated: https://developer.apple.com/design/human-interface-guidelines/live-photos/overview/

If desired I can provide a sample for Samsung's implementation, too.
Comment 16 Nick Cross 2020-03-07 19:11:38 UTC
I would love to see support for embedded movies within jpg (be it Apple Live Photo, Google Photo or Samsung Motion Photos).

I see this has been reopened and couldn't see any other tickets for Google/Samsung support. I can supply sample Samsung images from my S10 if needed :-)
Comment 17 Glassed Silver 2020-03-08 00:11:40 UTC
https://drive.google.com/drive/folders/1PgxB3JBql7kRzOkEzhBwqKQUXtddcLAj?usp=sharing

The filesize limit doesn't allow me to include the files directly, which is a shame, not a fan of link rot and I gotta admit, this link WILL go down once this issue is resolved, because I don't like lingering public links on my Drive account.

Either way, I included a sample for Huawei's "Moving Picture" implementation. (consumer info page: https://consumer.huawei.com/uk/support/faq/how-to-take-moving-pictures/)

The pic was taken on my MediaPad M5 8.4" tablet. Not amazing quality on that one, but should do the trick of showing the file's logic.

I also included a sample for Samsung's "Motion Photo". (consumer info page: https://www.samsung.com/global/galaxy/what-is/motion-photo/ (it doesn't list S20 as compatible device, but it's been reported that it is indeed still a maintained feature, guess they forgot to update that page so far)

That pic was taken with my Galaxy Note9.

Forgive the dirty keyboard...
Comment 18 Maik Qualmann 2020-03-29 11:58:42 UTC
*** Bug 419375 has been marked as a duplicate of this bug. ***
Comment 19 Ritesh Raj Sarraf 2020-05-21 07:59:57 UTC
@Gilles Caulier

Here's a link to Google Motion Photo.

https://photos.app.goo.gl/g37hyTahF69qqQ2n6
Comment 20 Ritesh Raj Sarraf 2020-05-21 08:25:50 UTC
(In reply to Ritesh Raj Sarraf from comment #19)
> @Gilles Caulier
> 
> Here's a link to Google Motion Photo.
> 
> https://photos.app.goo.gl/g37hyTahF69qqQ2n6

In case Google post-processes and messes it up.

https://drive.google.com/file/d/1EFsJBFL1ZTNhIq5J_EqkOuLcDb8jpATo/view?usp=sharing
Comment 21 caulier.gilles 2020-05-21 08:35:21 UTC
Thanks Ritesh. I will take a look.

Gilles Caulier
Comment 22 Glassed Silver 2020-06-22 18:22:19 UTC
(In reply to caulier.gilles from comment #21)
> Thanks Ritesh. I will take a look.
> 
> Gilles Caulier

Forgive me if I may come across as impatient, I am just very thankful for finally being close to getting the long awaited Aperture replacement... (yep... I held out this long!)

My question is: is this feature still on track for 7.0 final?

Again, thank you so so much for even getting on this, bridging the different kinds of live photo-like implementations no less. I really think it's a very transformative format of our time and I cannot tell you how often I've been enjoying the little bit of extra life Live Photos and similar approaches have brought to me.
Comment 23 caulier.gilles 2020-06-22 20:00:20 UTC
yes, HEIF is supported in digiKam 7.0.0 right now, but not yet the video embeded.

The problem is with Exiv2 which refuse to support HEIF due to possible patents.

If HEIF will host video, we will need to be able to extract this data, and Exiv2 sound like the best way to operate.


The problem between HEIF and EXiv2 do not touch only HEIF, but all formats based on ISO base media format:

https://en.wikipedia.org/wiki/ISO/IEC_base_media_file_format

This include also the famous Canon CR3 raw files, and it's very problematic for the future...

https://github.com/Exiv2/exiv2/issues/236

So we need to have a plain support of ISO/IEC 14496-12 format first before to support extra video from this kind of container...

Gilles Caulier
Comment 24 Glassed Silver 2020-06-25 10:45:07 UTC
(In reply to caulier.gilles from comment #23)
> yes, HEIF is supported in digiKam 7.0.0 right now, but not yet the video
> embeded.
> 
> The problem is with Exiv2 which refuse to support HEIF due to possible
> patents.
> 
> If HEIF will host video, we will need to be able to extract this data, and
> Exiv2 sound like the best way to operate.
> 
> 
> The problem between HEIF and EXiv2 do not touch only HEIF, but all formats
> based on ISO base media format:
> 
> https://en.wikipedia.org/wiki/ISO/IEC_base_media_file_format
> 
> This include also the famous Canon CR3 raw files, and it's very problematic
> for the future...
> 
> https://github.com/Exiv2/exiv2/issues/236
> 
> So we need to have a plain support of ISO/IEC 14496-12 format first before
> to support extra video from this kind of container...
> 
> Gilles Caulier

Maybe I'm missing something here, but if we rely on a third-party dependency to resolve this and they don't release a solution in code before July (and even that isn't certain) how can you be sure that this feature is on track for 7.0 final scheduled to release on or around July 12?

Do you have an intermediary workaround in mind to use until this gets resolved?

Thanks for the wealth of background info as well! Very interesting to see this unfold and at the same time I cannot stop but cry over software patents being a thing... ESPECIALLY for things like file formats. Real innovation stifling at work...
Comment 25 caulier.gilles 2020-06-25 11:14:29 UTC
digiKam already rely on Exiv2 to all file format metadata support. There is no exception for HEIF and CR3. We need to still homogeneous and not to make a big puzzle. I'm fighting against this idea since a while, as digiKam is already a big application with many external dependencies.

There is no work around planed until July release...

If you want to see a better HEIF/CR3 support in the future, go to Exiv2 project from github and vote to implement metadata support for these format.

Gilles Caulier
Comment 26 Glassed Silver 2020-06-25 11:29:07 UTC
Thank you so much for your input.

I wasn't trying to advocate for a work-around, I was just trying to make sense of why you're optimistic, since this seems like it could go either way over at that project. At least from what I read regarding the potential legal implications.

I gave the OP post a thumbs up, I guess that's what you meant by voting for it.

Again thank you so much and I'll just adopt your hopefulness and anticipate the release, hopefully with CR3 and HEIC metadata support.

I guess one way or another a way has to be found either way eventually, so if I don't completely misinterpret this, if the feature doesn't make it into 7.0, hope wouldn't be completely lost?

Cheers!
Comment 27 caulier.gilles 2020-06-25 12:02:25 UTC
Don't be afraid. HEIF and CR3 support are popular format, and Bugzilla is here to remember the important entry (you can vote for this support here too).

HEIF is Android and Apple used format -> very popular
CR3 is Canon RAW file -> a major actor in photography.

So...

Gilles Caulier
Comment 28 Glassed Silver 2020-06-25 13:47:18 UTC
(In reply to caulier.gilles from comment #27)
> Don't be afraid. HEIF and CR3 support are popular format, and Bugzilla is
> here to remember the important entry (you can vote for this support here
> too).
> 
> HEIF is Android and Apple used format -> very popular
> CR3 is Canon RAW file -> a major actor in photography.
> 
> So...
> 
> Gilles Caulier

Well, I'm a CR2 user thanks to my 60D, I know it's pretty much inevitable a solution will be found. HEIF isn't just Apple and Android though, I think in the future demand for it will rise quite a bit more, because it does provide substantial space savings which can be applied elsewhere too.

Oh also I wanted to let you know that I've donated to Digikam a few days ago because seeing the Live Photo format getting picked up alerted me to help out a bit myself. When this issue gets resolved I'll be looking at weeks if not months of carefully moving data to Digikam and trialing productive use with a large library. Finally a home for my photography and videoclips again.

Much love from Germany! :)
Comment 29 caulier.gilles 2021-03-16 03:43:36 UTC
Git commit c23ccd2c0e40c624631e748237aacbd26527eda8 by Gilles Caulier.
Committed on 16/03/2021 at 03:36.
Pushed by cgilles into branch 'master'.

Exiv2 0.27.4 will introduce Base Media File format support for CR3, HEIF, HEIC, and AVIF image containers.
Enable Exiv2 compilation option for BMFF supports.
Add new static method to get BMFF support from Exiv2.
Report BMFF support in help/components dialog.
Fix MetaEngine::initializeExiv2() to register std::atexit C++11 method and cleanup automatically XMP SDK allocation at application exit.
Remove obsolete cleanupExiv2() method from MEtaEngine and relevant calls everywhere.

M  +0    -2    core/app/main/main.cpp
M  +3    -2    core/libs/dialogs/libsinfodlg.cpp
M  +20   -8    core/libs/metadataengine/engine/metaengine.cpp
M  +5    -8    core/libs/metadataengine/engine/metaengine.h
M  +6    -0    core/libs/metadataengine/engine/metaengine_p.cpp
M  +1    -0    core/libs/metadataengine/engine/metaengine_p.h
M  +2    -2    core/showfoto/main/main.cpp
M  +0    -4    core/tests/dimg/dimgabstracthistorytest.cpp
M  +0    -2    core/tests/dimg/dimgfilteractiontest.cpp
M  +0    -2    core/tests/dimg/testcolorbalancefilter.cpp
M  +0    -2    core/tests/dimg/testdimgloader.cpp
M  +0    -2    core/tests/dimg/testequalizefilter.cpp
M  +0    -2    core/tests/dplugins/loadandrun_generic.cpp
M  +0    -2    core/tests/fileio/loadsavethreadtest.cpp
M  +0    -2    core/tests/filters/testautocrop.cpp
M  +0    -2    core/tests/filters/testlensfuniface.cpp
M  +0    -2    core/tests/filters/testnrestimate.cpp
M  +0    -2    core/tests/geolocation/editor/test_gpsimageitem.cpp
M  +0    -3    core/tests/geolocation/geoiface/demo/mainwindow.cpp
M  +1    -2    core/tests/metadataengine/abstractunittest.h
M  +0    -1    core/tests/metadataengine/commentreadwritetest.cpp
M  +0    -1    core/tests/metadataengine/ratingreadwritetest.cpp
M  +0    -1    core/tests/metadataengine/tagsreadwritetest.cpp
M  +0    -2    core/tests/video/framesencoder.cpp
M  +1    -9    project/bundles/3rdparty/ext_exiv2/CMakeLists.txt
M  +1    -0    project/bundles/flatpak/org.kde.digikam.json
M  +1    -1    project/scripts/bootstrap-exiv2.linux.sh
M  +1    -0    project/scripts/bootstrap-exiv2.mxe.sh

https://invent.kde.org/graphics/digikam/commit/c23ccd2c0e40c624631e748237aacbd26527eda8
Comment 30 Glassed Silver 2021-03-30 12:55:04 UTC
Just checked the release notes for 7.2 and noticed that Apple Live Photo Support has been scheduled for 7.4, which is fantastic!

I have one question though, are competing, similar formats like Samsung's Motion Photos, Huawei's Moving Picture and possibly other contenders that may be relevant in later feedback still on track for release as well?

Personally for my own needs I am using Apple and Samsung's formats, so those would be my main concern, but Huawei is also pretty popularly used. Not sure what other manufacturers put out there, but having Apple, Huawei, Google and Samsung included would cover a large share of the market.

Photo enthusiasts with mobile device choices around their creative needs may even go for some other brands, but I'm pretty sure those need not be the focus for now and could get added once the main share of the market is implemented and has gone through some wider trialing in the field I guess?

Also, if desired I could add a new Samsung Motion Photo sample taken with a newer device (Note20 Ultra) just in case there are alterations to Samsung's implementation. (doubt it, but I want to be as assisting in this implementation as possible and providing samples is something I can do pretty nicely as someone who doesn't develop themselves)
Comment 31 caulier.gilles 2021-03-30 13:16:29 UTC
About Apple Live Photo, Samsung's Motion Photos, and Huawei's Moving Picture, i'm not sure where the differences are located.

I'm sure that Apple Live Photo is HEIF based. 

in all cases libheif must support these format to access to image and video data in container. About metadata, libexiv2 will support Base Media Format (HEIF and AVIF containers) with next 0.27.4 release planed for June 2021.
Comment 32 caulier.gilles 2021-03-30 13:23:48 UTC
Samsung's Motion Photo format consists of a picture (jpeg) and movie (mp4) merged into a single file. Between them there is a special marker. The script finds the marker and splits it accordingly.

It's perhaps HEIF container, as HEIF con embed JPEG image. This need investiguations
Comment 33 caulier.gilles 2021-03-30 13:37:59 UTC
Huawei's Moving Picture file format => i cannot found technical details about this container on the Web. So we need reverse engineering...
Comment 34 Ritesh Raj Sarraf 2021-03-30 17:06:39 UTC
(In reply to Ritesh Raj Sarraf from comment #20)
> (In reply to Ritesh Raj Sarraf from comment #19)
> > @Gilles Caulier
> > 
> > Here's a link to Google Motion Photo.
> > 
> > https://photos.app.goo.gl/g37hyTahF69qqQ2n6
> 
> In case Google post-processes and messes it up.
> 
> https://drive.google.com/file/d/1EFsJBFL1ZTNhIq5J_EqkOuLcDb8jpATo/
> view?usp=sharing

Sorry. It seems that the picture that I linked, for whatever reason, is not what I intended to share as a motion photo from an Android phone.

Please see this one instead:
https://photos.app.goo.gl/rcmuoCg6ZifZo4jD8

As of today, I have verified that this indeed is an Android motion photo and is accessible anonymously.
Comment 35 Glassed Silver 2021-04-14 09:06:14 UTC
(In reply to caulier.gilles from comment #31)
> About Apple Live Photo, Samsung's Motion Photos, and Huawei's Moving
> Picture, i'm not sure where the differences are located.
> 
> I'm sure that Apple Live Photo is HEIF based. 
> 
> in all cases libheif must support these format to access to image and video
> data in container. About metadata, libexiv2 will support Base Media Format
> (HEIF and AVIF containers) with next 0.27.4 release planed for June 2021.

Modern implementations yes, but before Apple adopted HEIF Live Photos existed as well and there's plenty of JPEG-based Live Photo backlog that would be in I bet many users' libraries. :)
Comment 36 hmueller 2022-01-04 08:16:21 UTC
You may find more details about GoogleMotion Photos at https://github.com/photoprism/photoprism/issues/1739 and the links there. Quite easy to implement. Would be happy to see extraction (jpg and mp4) as well as the preview of video in digikam. Tx a lot.
Comment 37 caulier.gilles 2022-04-06 20:54:17 UTC
Git commit febc26188fe4e6037e5c8a9b1840f65729972a47 by Gilles Caulier.
Committed on 06/04/2022 at 20:38.
Pushed by cgilles into branch 'master'.

Add ExifTool backend to load metadata for image or video.
Exiv2 is always used in priority. If it fail, ExifTool is used instead.
ExifTool has the capability to create an EXV container for Exiv2. All metadata extracted with ExifTool
are concatened to the EXV container with all possible translations with Exif, Iptc, and Xmp tags.
Of course tags supported by ExifTool but not supported by Exiv2 are lost.
Related: bug 416516, bug 360714, bug 264210, bug 426938, bug 419801, bug 419375, bug 103247

M  +1    -0    core/libs/metadataengine/CMakeLists.txt
M  +7    -0    core/libs/metadataengine/dmetadata/dmetadata.h
A  +94   -0    core/libs/metadataengine/dmetadata/dmetadata_exiftool.cpp     [License: GPL (v2+)]
M  +42   -17   core/libs/metadataengine/dmetadata/dmetadata_fileio.cpp
M  +7    -6    core/libs/metadataengine/engine/metaengine.h
M  +5    -0    core/libs/metadataengine/engine/metaengine_fileio.cpp
M  +4    -2    core/libs/metadataengine/exiftool/exiftoolparser_command.cpp

https://invent.kde.org/graphics/digikam/commit/febc26188fe4e6037e5c8a9b1840f65729972a47