Bug 271030 - Windows Live Photo Gallery people tagging Compatibility [patch]
Summary: Windows Live Photo Gallery people tagging Compatibility [patch]
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Unclassified
Component: Faces-Engine (show other bugs)
Version: 2.0.0
Platform: Unlisted Binaries Linux
: NOR wishlist (vote)
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-04-15 15:11 UTC by harry
Modified: 2012-06-27 10:20 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In: 2.1.0


Attachments
Example image from Windows Live Photo Gallery w/ 1 tagged face (22.91 KB, image/jpeg)
2011-04-16 13:12 UTC, harry
Details
Sample Image with two faces tagged including email addresses (39.83 KB, image/jpeg)
2011-04-16 13:26 UTC, harry
Details
DigiKam having correctly parsed the face tags from image (106.65 KB, image/png)
2011-07-10 07:33 UTC, Leif Huhn
Details
Initial read support for WLPG face tags (6.87 KB, patch)
2011-07-10 17:02 UTC, Leif Huhn
Details
Minor fix to previous patch (7.03 KB, patch)
2011-07-10 19:51 UTC, Leif Huhn
Details
Revision 3 (21.10 KB, patch)
2011-07-12 05:32 UTC, Leif Huhn
Details
Revision 2 to 3 interdiff (17.12 KB, patch)
2011-07-12 05:33 UTC, Leif Huhn
Details

Note You need to log in before you can comment on or make changes to this bug.
Description harry 2011-04-15 15:11:39 UTC
Version:           2.0.0
OS:                Linux

It would be really good if digiKam was compatible with the way WLPG stores people tags, it would make it a lot easier to migrate to digikam.
WLPG uses XMP to store people tags and digiKam also uses XMP so it can't be to complicated?
The specs for how WLPG stores people tags is here: http://msdn.microsoft.com/en-us/library/ee719905%28VS.85%29.aspx

Reproducible: Didn't try
Comment 1 caulier.gilles 2011-04-16 09:40:29 UTC
This M$ paper talk about XMP Win7 photo properties. I cannot see this schema in Exiftool :

http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/XMP.html

Gilles Caulier
Comment 2 harry 2011-04-16 10:40:30 UTC
The schema is here:
http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Microsoft.html#MP

example exiv2 output (exiv2 -PX) for a photo with one person tag:
 Xmp.MP.RegionInfo                            XmpText     0  type="Struct"
 Xmp.MP.RegionInfo/MPRI:Regions               XmpText     0  type="Bag"
 Xmp.MP.RegionInfo/MPRI:Regions[1]            XmpText     0  type="Struct"
 Xmp.MP.RegionInfo/MPRI:Regions[1]/MPReg:Rectangle XmpText    38  0.817434, 0.427997, 0.055099, 0.073465
 Xmp.MP.RegionInfo/MPRI:Regions[1]/MPReg:PersonDisplayName XmpText     3  Bob
 Xmp.MP.RegionInfo/MPRI:Regions[1]/MPReg:PersonEmailDigest XmpText    40  AD6224CAD65B77EB326CB7E846DD21256E45169A
 Xmp.MP.RegionInfo/MPRI:Regions[1]/MPReg:PersonSourceID XmpText    13  FB:1612692345

In the Microsoft document it looks like tags starting with MP: are actually in Xmp.MP.
Comment 3 caulier.gilles 2011-04-16 11:48:11 UTC
Great. Can you attach to this entry an image with these information set by M$ software ?

Gilles Caulier
Comment 4 harry 2011-04-16 13:12:23 UTC
Created attachment 59036 [details]
Example image from Windows Live Photo Gallery w/ 1 tagged face

The emotion in this image is tagged Example, no extra data (email address, etc) is added.
Exiv2 -PX output for this file:
Xmp.MP.RegionInfo                            XmpText     0  type="Struct"
Xmp.MP.RegionInfo/MPRI:Regions               XmpText     0  type="Bag"
Xmp.MP.RegionInfo/MPRI:Regions[1]            XmpText     0  type="Struct"
Xmp.MP.RegionInfo/MPRI:Regions[1]/MPReg:Rectangle XmpText    38  0.328185, 0.445
860, 0.397683, 0.328025
Xmp.MP.RegionInfo/MPRI:Regions[1]/MPReg:PersonDisplayName XmpText     7  Example
Comment 5 harry 2011-04-16 13:26:18 UTC
Created attachment 59038 [details]
Sample Image with two faces tagged including email addresses

In this photo I've tagged two faces.
The red one is tagged Bob with email address bob@example.com
The green one is tagged Example Face (input as: first name last name) and had the email address face@example.com

exiv2 -PX
Xmp.MP.RegionInfo                            XmpText     0  type="Struct"
Xmp.MP.RegionInfo/MPRI:Regions               XmpText     0  type="Bag"
Xmp.MP.RegionInfo/MPRI:Regions[1]            XmpText     0  type="Struct"
Xmp.MP.RegionInfo/MPRI:Regions[1]/MPReg:Rectangle XmpText    38  0.438073, 0.301
923, 0.185780, 0.311538
Xmp.MP.RegionInfo/MPRI:Regions[1]/MPReg:PersonDisplayName XmpText    12  Example
 Face
Xmp.MP.RegionInfo/MPRI:Regions[1]/MPReg:PersonEmailDigest XmpText    40  E43A034
3081CABFE3D1ED5C9F414B68E9F1D7EF8
Xmp.MP.RegionInfo/MPRI:Regions[2]            XmpText     0  type="Struct"
Xmp.MP.RegionInfo/MPRI:Regions[2]/MPReg:Rectangle XmpText    38  0.126147, 0.232
692, 0.178899, 0.300000
Xmp.MP.RegionInfo/MPRI:Regions[2]/MPReg:PersonDisplayName XmpText     3  Bob
Xmp.MP.RegionInfo/MPRI:Regions[2]/MPReg:PersonEmailDigest XmpText    40  F4F6C93
8D35ACA3A4B5EBFD68A2A4B207EB327EE
Comment 6 Leif Huhn 2011-07-09 23:05:17 UTC
*** Bug 277429 has been marked as a duplicate of this bug. ***
Comment 7 Leif Huhn 2011-07-10 07:33:52 UTC
Created attachment 61747 [details]
DigiKam having correctly parsed the face tags from image

I have created a patch that will load the data from windows live photo gallery tags.  It needs a little cleaning up (probably will do this tomorrow).  For now, here is a screenshot of DigiKam showing the tagged faces from comment #5.
Comment 8 caulier.gilles 2011-07-10 09:24:41 UTC
These data are load _and_ registered to digiKam database ?

The typical case is to restore a DB using image metadata, if necessary.

Also, the goal is to be compatible with M$ photo management software (read/write)

Do you know an another software which manage face tags in metadata, as picasa for ex ? Where information are recorded ?

Gilles Caulier
Comment 9 Leif Huhn 2011-07-10 17:00:31 UTC
Hi Gilles,

Right now my patch forces a load of the faces data each time (for testing).  I found it surprising that the menu option "Image, Reread Metadata from Image" did not reload most of the metadata (but only a small amount).  At least I think that's what's happening (I didn't test that much).

How do we handle the situation where a user already has the image in DigiKam (and so we tend not to reload the metadata) but we want to pick up these WLPG tags now?  I'd be inclined to make the "Reread Metadata from Image" always load the face data (and more).

I haven't written write support yet.  I would like to hear your comments on my patch before I continue to write code, in case you can give me some pointers on how DigiKam arranges functions, etc.

Personally, I do not use Windows Live Photo Gallery.  I merely wanted DigiKam to read and write faces to tags.  My web research indicated that there are only few programs that write faces to XMP, and most of them understand WLPG.  I do not have time now to summarize my research right now but I will try later today or tomorrow.

I think Picasa only keeps its face data inside its own database (and not in tags).  I have seen web postings that say that.  It may have changed (I have not tested).

I will upload the patch now.
Comment 10 Leif Huhn 2011-07-10 17:02:40 UTC
Created attachment 61763 [details]
Initial read support for WLPG face tags
Comment 11 Leif Huhn 2011-07-10 19:51:57 UTC
Created attachment 61766 [details]
Minor fix to previous patch
Comment 12 Leif Huhn 2011-07-10 20:09:37 UTC
You asked about what other programs are using for face tags on images.

iPhoto doesn't export faces to tags at all (says the web...I haven't tested).  However, the program "Phoshare" (formerly iphotoexport) is a program that allows you to export iPhoto photos.  It is apache 2.0 licensed.  It uses the same WLPG tags that we are adding to DigiKam.

https://sites.google.com/site/phosharedoc/
"Faces: exports the face tags and face regions using the Microsoft Photo Region Schema, and merge the face tags with your other keywords, and written in the keywords field in the image meta data (JPEG only)"
Comment 13 caulier.gilles 2011-07-11 11:13:26 UTC
You patch sound in the good way.

About your question:

+        // FIXME is there a function to build this string somewhere?
+        QString imageTagProperty("<rect x=\"%1\" y=\"%2\" width=\"%3\" height=\"%4\"/>");

I'm not sure. I don't know if Marcel has factored this code somewhere. grep do not give results here.

In your loop :

while(1) {
+        // FIXME QString::ascii() deprecated...use latin1 instead?
+        QString person = getXmpTagString(personPath.arg(i).ascii(), false);
+        QString rect   = getXmpTagString(rectPath.arg(i).ascii(), false);
+        if (person.isEmpty())
+            break;
+        i++;
+        faces[person] = rect;
+    }

XMP support UTF-8. We must use it to get person name. For rect, it sound fine. Need to test of course. Do you have any test picture faced with M$ tags ?

Your patch implement face scanning at startup. It miss some mechanism to read and write face tags on the fly when face are edited in digiKam. In order :

- implement DMetadata::setImageFacesPath()
- in digiKam, there are 2 classes named MetadataHub and MetadataManager used to play info to read/store in image depending of user actions. Look in digikam/digikam/metadata dir. You will see that face tags are not yet managed.

In Exiv2, some code need to be written too to handle XMP M$ face tags as well. I can take a look.

Gilles Caulier
Comment 14 caulier.gilles 2011-07-11 12:00:02 UTC
Leif,

About Exiv2 code to handle XMP M$ Face Tags, code to patch is here :

http://dev.exiv2.org/projects/exiv2/repository/entry/trunk/src/properties.cpp

Currently, only ExpressionMedia schema is know :

http://dev.exiv2.org/projects/exiv2/repository/entry/trunk/src/properties.cpp#L119
http://dev.exiv2.org/projects/exiv2/repository/entry/trunk/src/properties.cpp#L931

... which is this one in Exiftool :

http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Microsoft.html#XMP

The M$ MP schema is not yet implemented :

http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Microsoft.html#MP

...but "RegionInfoMP Tags" and "Regions Tags" structure details are not yet available in Exiftool web site...

Gilles Caulier
Comment 15 Marcel Wiesweg 2011-07-11 16:57:34 UTC
Thanks a lot for digging through all this.
Some comments on your patch:

1) I would suggest to return a map name -> QRectF from DMetadata, so that all string parsing of the specific storage format is done in the DMetadata class and we get a rectangle in relative coordinates

2)There is elaborate code in FaceIFace to create a tag for a given name and assign it, mostly FaceIface::FaceIfacePriv::tagForName and FaceIface::FaceIfacePriv::add.
It seems now we need this in libdigikamdatabase. So we'd need another refactor of this class (removing the Priv class, which holds no more data, just create the FaceIface on the stack, give the class a better name) and splitting it, because some methods call MetadataManager which is not in the libraries.

> I'd be inclined to make the "Reread Metadata from Image" always load
> the face data (and more).

Yes. This is scan mode Rescan (in ImageScanner) which mostly uses the same code paths as NewScan, but at some places take care not to overwrite existing information if there is no information from the metadata.
Comment 16 Leif Huhn 2011-07-11 17:49:26 UTC
Thank you Gilles, Marcel,

I will take your suggestions and update my patch tonight.  When I am finished I will post another revision.
Comment 17 caulier.gilles 2011-07-11 17:55:55 UTC
Great. thanks Leif for your help

Gilles Caulier
Comment 18 Leif Huhn 2011-07-12 05:30:49 UTC
Gilles,

Re: comment #13.  The name in my code is retrieved via unicode.  The .ascii() call was converting the QString representing the XMP path to const char *.  I don't know if there is a better way to do it (but it does produce a warn).

Thanks for looking up the info for Exiv2.

I spent most of today reading the code trying to understand how it all fits together.  I am confused about what situations MetadataHub::load is called.  When I click "Reread metadata" it seems to call ImageScanner::rescan.  In what situation is MetadataHub::load called?

I have written a new patch.  There is not much new.

Marcel,

I used QRectF for passing the image info and I have merged FaceIface and FaceIfacePriv (moving the methods from FaceIfacePriv into private methods in FaceIFace).  I think that is what you wanted.  What methods should be broken off?  Only the ones that use MetadataManager?
Comment 19 Leif Huhn 2011-07-12 05:32:08 UTC
Created attachment 61791 [details]
Revision 3
Comment 20 Leif Huhn 2011-07-12 05:33:15 UTC
Created attachment 61792 [details]
Revision 2 to 3 interdiff
Comment 21 Leif Huhn 2011-07-12 15:07:17 UTC
I have looked at bug 268688 (DigiKam won't delete tags in metadata).  Until that bug is resolved, I suppose I could implement the same logic for faces as we have for tags now.  The downside is that it would be impossible to delete a face from the tags.  For tags there is a workaround (bug 268688 comment #8).  I don't believe the workaround would work for faces because there is no panel for editing face tag data.

Also, faces are more complicated.  What do we do in this situation?

1. User adds a file to DigiKam that already has a face Person/Alice.  DigiKam imports the face data.
2. Close DigiKam.  Open WLPG.  Move Alice's rectangle.
3. Close WLPG.  Open DigiKam.  DigiKam still has the old rectangle.  Save Metadata.
   Do we overwrite the metadata?
4. Change the rectangle inside DigiKam.  Save metadata.
   Do we overwrite the metadata?

Metadatahub doesn't have enough information to know to not overwrite the metadata in #3 but to make a change in #4.

I have proposed a fix in bug 268688 for the issue with tags.  We could do something similar for faces.

I propose that for right now, we import faces from WLPG, but only read/write a DigiKam specific face tag.  I think telling the user that DigiKam can't write WLPG faces (but can import them) is easier to understand than DigiKam being able to add WLPG faces (but not delete or rename them).
Comment 22 Marcel Wiesweg 2011-07-12 18:13:38 UTC
> I spent most of today reading the code trying to understand how it all fits
> together.  I am confused about what situations MetadataHub::load is called. 
> When I click "Reread metadata" it seems to call ImageScanner::rescan.  In what
> situation is MetadataHub::load called?

MetadataHub was written to support loading multiple images in the right sidebar, which is mostly complicated for tags. You load one or multiple images, have a merged result, and write it back. In more places it is used for simple load one - write one.

> I used QRectF for passing the image info and I have merged FaceIface and
> FaceIfacePriv (moving the methods from FaceIfacePriv into private methods in
> FaceIFace).  I think that is what you wanted.  What methods should be broken
> off?  Only the ones that use MetadataManager?

The problem is that using MetadataManager cannot be moved to the libdigikamdatabase library, where we need to put FaceIface when used from ImageScanner. 
MetadataManager is asynchronous, has UI progress feedback, and writes to database and metadata, which we do not want at all when reading from metadata.

I believe would be sufficient to move the calls to a virtual method, the default implementation calling directly ImageInfo.setTag etc, and a subclass at higher level using MetadataManager (asynchronous, with UI progress etc.)

Further on, we should see to replace most of the code you have in ImageScanner::scanFaces() with calls to FaceIface. Something like
int tagid = iface.getOrCreateTagForPerson(i.key())
iface.add(m_scanInfo.id, tagId, TagRegion(rect), false);
Comment 23 Leif Huhn 2011-07-13 20:22:49 UTC
I'm still working on writing a patch for exiv2.  This page gives a schema definition:

http://metability.editme.com/mmf-scavenger-xmp-schemaex-MSPeopleTagging

I'm figuring out how to translate that into the data structure for exiv2.
Comment 24 Leif Huhn 2011-07-13 20:26:04 UTC
Actually, nevermind.  That post just contains information from

http://msdn.microsoft.com/en-us/library/ee719905%28VS.85%29.aspx

in the original post.  I'll just go there.
Comment 25 caulier.gilles 2011-07-13 20:34:16 UTC
Leif,

Andreas Huggel is in CC of this file. He is the Exiv2 lead developer. He can guide you in to implement Microsoft MP schema support in XMP.

Gilles Caulier
Comment 26 Leif Huhn 2011-07-13 22:10:38 UTC
Oh, I didn't make the connection.  I just added a bug under exiv2:

http://dev.exiv2.org/issues/778
Comment 27 Leif Huhn 2011-07-16 18:09:12 UTC
In order to write the faces to the metadata, I need to specifically know the id of the image, in order to look up the ImageTagProperties (or use FaceIface).  Writing the data happens in MetadataHub::write.  That class was designed to be used in the case of multiple images (so it doesn't know the image id for what it's being used for, because it might be used for multiple images).

Without changing the way things are called, I can't add logic to MetadataHub::write to write the image tags.

How should we resolve this?
Comment 28 Marcel Wiesweg 2011-07-23 13:57:32 UTC
Sorry for late answer.

We do not need and cannot do conflict resolution/multiple file merging for faces. So the way to go is the same as for the comment: Provide a QList<DatabaseFace>, a status field, and load with loadSingleValue().

For loading from ImageInfo, FaceIFace::databaseFaces(info.id()) load the faces for you. Omit the implementation in MetadataHub::load(DMetadata) because that method is unused nowadays as I just checked.

Note: As soon as 2.0 is released, we should apply your patch as it is by then and work on it together in git.
Comment 29 Marcel Wiesweg 2011-08-08 17:39:35 UTC
Git commit 1f389e3556758a50fd0171a28856b48f8029710d by Marcel Wiesweg.
Committed on 08/08/2011 at 18:49.
Pushed by mwiesweg into branch 'master'.

Split the FaceIface class in three parts

1) FaceTags in libs/database/faces: Contains utility code to find or create face tags
   with TagsCache help
2) FaceTagsEditor: Code to add, remove or confirm face entries in the database.
   Strictly only database dependency, no libkface
DatabaseFace is moved to libs/database/faces as well
3) FaceIface: Code to detect and recognize with libkface, and merge the results.
   Inherits FaceTagsEditor and reimplements the methods for normal tags
   using MetadataManager

CCBUG: 271030

M  +4    -5    utilities/facedetection/facegroup.cpp
M  +0    -2    utilities/facedetection/facepipeline.h
R  +2    -2    libs/database/faces/databaseface.h [from: utilities/facedetection/databaseface.h - 097% similarity]
M  +32   -30   utilities/facedetection/facepipeline.cpp
A  +109  -0    libs/database/faces/facetags.h     [License: GPL (v2+)]
R  +0    -0    libs/database/faces/databaseface.cpp [from: utilities/facedetection/databaseface.cpp - 100% similarity]
A  +327  -0    libs/database/faces/facetags.cpp     [License: GPL (v2+)]
M  +1    -1    digikam/items/assignnameoverlay.cpp
M  +27   -1032 utilities/facedetection/faceiface.cpp
M  +42   -202  utilities/facedetection/faceiface.h
M  +4    -1    CMakeLists.txt
A  +411  -0    libs/database/faces/facetagseditor.cpp     [License: GPL (v2+)]
M  +0    -1    utilities/facedetection/facepipeline_p.h
A  +160  -0    libs/database/faces/facetagseditor.h     [License: GPL (v2+)]

http://commits.kde.org/digikam/1f389e3556758a50fd0171a28856b48f8029710d
Comment 30 Marcel Wiesweg 2011-08-08 17:39:35 UTC
Git commit ea30232a07d2e1e2a788035da53cbf9806d3c01b by Marcel Wiesweg.
Committed on 08/08/2011 at 19:38.
Pushed by mwiesweg into branch 'master'.

Add scanning of XMP face tags to the ImageScanner

Part 3 of Leif Huhn's patch, rewritten to reuse code
as much as possible.
Test image provided in bug report is correctly parsed.

BUG: 271030

M  +2    -1    NEWS
M  +1    -0    libs/database/imagescanner.h
M  +42   -0    libs/database/imagescanner.cpp
M  +8    -0    libs/database/tagregion.cpp
M  +3    -0    libs/database/tagregion.h

http://commits.kde.org/digikam/ea30232a07d2e1e2a788035da53cbf9806d3c01b
Comment 31 Marcel Wiesweg 2011-08-08 17:40:28 UTC
commit 1ccda1b93daa5e1ea47cfe1f82db1fa2e6bf4388
Author: Marcel Wiesweg <marcel.wiesweg@gmx.de>
Date:   Mon Aug 8 19:36:19 2011 +0200

    Add parsing of WLPG face tags in XMP to DMetadata
    
    Part 2 of Leif Huhn's patch. Parses the metadata and returns a map
    name -> relative rectangle.
    
    CCBUG: 217030