Bug 345220

Summary: HUB : Import ACDSee Categories to hierarchical Tags [patch]
Product: [Applications] digikam Reporter: Alan Pater <alan.pater>
Component: Metadata-HubAssignee: Digikam Developers <digikam-bugs-null>
Status: RESOLVED FIXED    
Severity: wishlist CC: alan.pater, caulier.gilles, McFiddich, metzpinguin, simonandric5
Priority: NOR    
Version: 4.8.0   
Target Milestone: ---   
Platform: unspecified   
OS: Linux   
Latest Commit: Version Fixed In: 4.10.0
Attachments: Test image from ACDSee
Sample image from Heikki Rekki
Sample image from Shotwell report
Code example
acdseetags.patch
acdseetags2.patch
result.txt
Additional ACDSee properties
acdseetags3.patch
complex example image
Sample with Rating and Caption
acdseetags4.patch
acdseetags5.patch
acdseetags6.patch
digikam.jpg
libkexiv2.patch
acdseetags7.patch

Description Alan Pater 2015-03-16 16:54:19 UTC
The ACDSee family of photo management applications has it's own schema to store hierarchical keywords or tags. digikam does not recognize these as tags and does not import them. Users trying to migrate from ACDSee to digikam lose access to these tags.

digikam has implemented it's own schema as has Microsoft and Adobe Lightroom. The MWG also has proposed a standard schema. However, the Adobe schema can be considered as the defacto standard, as many other imaging applications have adopted it.

I've documented how various applications implement hierarchical keywords/tags. In each application I created the same 6 level hierarchy and tagged a sample image with level 2 and level 5. I then used exiv2 to read the resulting metadata. These are just the main hierarchical fields written, flat tags are also in xmp.dc.subject, for example.

    6 Level
(X)     6.2 Level
            6.3 Level
                6.4 Level
(X)                 6.5 Level
                        6.6 Level


* Digikam 4.8
Xmp.digiKam.TagsList        XmpSeq  2  6 Level/6.2 Level/6.3 Level/6.4 Level/6.5 Level, 6 Level/6.2 Level

* Adobe Lightroom 4.4
Xmp.lr.hierarchicalSubject  XmpBag  2  6 Level|6.2 Level, 6 Level|6.2 Level|6.3 Level|6.4 Level|6.5 Level

* ACDSee 8 Pro
Xmp.acdsee.categories   XmpText   282   <Categories><Category Assigned="0">6 Level<Category Assigned="1">6.2 Level<Category Assigned="0">6.3 Level<Category Assigned="0">6.4 Level<Category Assigned="1">6.5 Level</Category></Category></Category></Category></Category></Categories>


Reproducible: Always
Comment 1 Alan Pater 2015-03-16 16:59:21 UTC
Created attachment 91582 [details]
Test image from ACDSee
Comment 2 Alan Pater 2015-03-18 16:04:23 UTC
Created attachment 91621 [details]
Sample image from Heikki Rekki

Issue was originally brought up on digikam users mailing list. 
   
   https://mail.kde.org/pipermail/digikam-users/2015-March/020752.html
This sample image was provided.

   https://mail.kde.org/pipermail/digikam-users/2015-March/020769.html
Comment 3 Alan Pater 2015-03-18 16:06:44 UTC
Created attachment 91622 [details]
Sample image from Shotwell report

This issue has also been brought up as a Shotwell bug report and this image was provided as a sample.

   https://bugzilla.gnome.org/show_bug.cgi?id=718586
Comment 4 Alan Pater 2015-04-06 14:03:21 UTC
Created attachment 91907 [details]
Code example

This example code in perl was supplied by Robin Mills over at exiv2. It has two issues:

1) I have no idea how this code might be converted for use in digikam.
2) It needs to only write the string if it encounters <Category Assigned="1">
Comment 5 Alan Pater 2015-04-13 00:56:35 UTC
I've been trying to figure out what tool to use for this string
conversion. I don't have any QT experience and very little c++. What
I've found are things like QString replace, QRegExp, QXmlStreamReader.

Which of these would be the right tool for the job? Or is there
something better and easier?
Comment 7 Maik Qualmann 2015-04-14 10:43:26 UTC
Created attachment 92014 [details]
acdseetags.patch

Here is a first version of a patch for import tags from ACDSee. It only uses QString and QStringList. Please test...

Maik
Comment 8 Maik Qualmann 2015-04-14 16:31:30 UTC
Created attachment 92022 [details]
acdseetags2.patch

Minor code optimizations.

Maik
Comment 9 caulier.gilles 2015-04-14 17:40:08 UTC
Maik,

I look your code (not yet tested). I sound right.

Some suggestions : 

- export from digiKam to ACDSEE (the revert way) must be implemented to setImageTagsPath() for interroperability purpose, as i already do for LR for ex.

- Not mandatory, but better, is to implement a small QTest code in core/tests to check if these methods work fine.

 For "get" method :
   * make a metadata container with ACDSee tag paths
   * get tag paths to string list and look results.

For "set" method :
   * pass tag paths to string list
   * look if metadata container is set properly.

Note : this can be extended to LR, M$ Windows Live Photo Gallery, Imach, and Windows keywords.

Gilles
Comment 10 caulier.gilles 2015-04-14 17:41:05 UTC
ALan, 

If you can give a feedback about Maik patch to see if it's the right way, we can plan to integrate this feature in next 4.10.0

Gilles
Comment 11 Maik Qualmann 2015-04-14 17:55:20 UTC
Created attachment 92023 [details]
result.txt

On request of Alan the results for the attachments in this bug report.

Maik
Comment 12 Alan Pater 2015-04-14 18:45:20 UTC
Looks good Maik!

Gilles, if you want to export to acdsee, I'll add the acdsee namespace and properties directly to exiv2.
Comment 13 caulier.gilles 2015-04-14 20:21:02 UTC
Alan,

Yes, this is the usual way to expand compatibility. If Exiv2 has more namespace defined, this will help...

but...
 
The most important is too release Exiv2 quickly now. Library have not been updated since a while. In OpenSource, release plan is very important. Do no lets to much time between releases...

Just my experience to share

Gilles
Comment 14 Alan Pater 2015-04-14 20:24:38 UTC
Gilles, agreed. Wait for the next exiv2 release before submitting more changes.
Comment 15 Alan Pater 2015-04-14 20:32:04 UTC
Unless I'm formatting things wrong, it looks to me like the result from attachment 91621 [details] is not quite right. Here it is what I think it should look like, expanded to make it easier to read.

>>> acdsee <<< categories : 
<Categories>

    <Category Assigned="0">Ihmiset
        <Category Assigned="1">Oma perhe
            <Category Assigned="1">Marjatta
            </Category>
            <Category Assigned="1">Jouni
            </Category>
        </Category>
    </Category>

    <Category Assigned="0">Paikat
        <Category Assigned="1">koti
            <Category Assigned="1">Kouvola
            </Category>
        </Category>
    </Category>

</Categories>

>>> digiKam-Schema <<< Tags List : 

    Ihmiset/Oma perhe, 
    Ihmiset/Oma perhe/Marjatta, 
    Ihmiset/Oma perhe/Jouni, 

    Paikat/koti,
    Paikat/koti/Kouvola, 

>>> lr <<< hierarchicalSubject : 

  Ihmiset|Oma perhe,
  Ihmiset|Oma perhe|Marjatta, 
  Ihmiset|Oma perhe|Jouni,
   
  Paikat|koti,
  Paikat|koti|Kouvola
Comment 16 Alan Pater 2015-04-15 17:35:57 UTC
Created attachment 92063 [details]
Additional ACDSee properties

In addition to the Categories property, acdsee also adds the following:

Xmp.acdsee.caption                           XmpText     0  
Xmp.acdsee.datetime                          XmpText    23  2012-03-30T19:05:24.000
Xmp.acdsee.author                            XmpText     0  
Xmp.acdsee.rating                            XmpText     1  0
Xmp.acdsee.notes                             XmpText    91  Quebec - Cours de perfectionnement de la Chambre des notaires du Quebec - Chateau Frontenac
Xmp.acdsee.tagged                            XmpText     5  False
Xmp.acdsee.keywords                          XmpBag      2  Coca, River

The attached tries to address the "rating", "notes", and "keywords" properties, and includes Maik's work on "Categories".
Comment 17 Maik Qualmann 2015-04-15 19:10:11 UTC
Created attachment 92065 [details]
acdseetags3.patch

This new patch fixes the bug from Comment 15. Alan, can you share the complex example from https://mail.kde.org/pipermail/digikam-users/2015-March/020770.html?

Maik
Comment 18 Alan Pater 2015-04-15 19:31:22 UTC
Created attachment 92066 [details]
complex example image
Comment 19 Maik Qualmann 2015-04-15 20:09:49 UTC
Result from Comment 18:

>>> digiKam-Schema <<<

ACDSee/Pro/8,
1 Level,
2 Level,
2 Level/2.2 Level,
3 Level,
3 Level/3.2 Level,
3 Level/3.2 Level/3.3 Level,
4 Level,
4 Level/4.2 Level,
4 Level/4.2 Level/4.3 Level,
4 Level/4.2 Level/4.3 Level/4.4 Level,
5 Level,
5 Level/5.2 Level,
5 Level/5.2 Level/5.3 Level,
5 Level/5.2 Level/5.3 Level/5.4 Level,
5 Level/5.2 Level/5.3 Level/5.4 Level/5.5  Level,
6 Level,
6 Level/6.2 Level,
6 Level/6.2 Level/6.3 Level,
6 Level/6.2 Level/6.3 Level/6.4 Level,
6 Level/6.2 Level/6.3 Level/6.4 Level/6.5 Level,
6 Level/6.2 Level/6.3 Level/6.4 Level/6.5 Level/6.6  Level,
Comment 20 Alan Pater 2015-04-15 20:26:10 UTC
Great Maik, looks correct to my eyes.
Comment 21 Alan Pater 2015-04-15 20:46:51 UTC
I'm not 100% sure what to do with the other three acdsee properties.

I think "Xmp.acdsee.caption" should map to "Xmp.dc.title" but could not find in the code where that mapping should be.

"Xmp.acdsee.author" should map to "Xmp.dc.creator", but again I am not sure where that goes.

In the samples "Xmp.acdsee.datetime" has the same DateTime as "Exif.Photo.DateTimeOriginal". I suspect that can be ignored, as all the samples already have that data in EXIF.
Comment 22 Maik Qualmann 2015-04-16 16:54:42 UTC
Hi Alan,
can you create a test image with "Xmp.acdsee.caption" and "Xmp.acdsee.rating". The value range of the rating is of interest.

Maik
Comment 23 Alan Pater 2015-04-16 18:24:49 UTC
Created attachment 92085 [details]
Sample with Rating and Caption

In the ACDSee GUI, Rating is a number between 1-5. In the attached sample, I choose 4.

~$ exiv2 -pa Pictures/tests/acdsee/acdsee18.rating.jpg 
Xmp.acdsee.caption                           XmpText    14  ACDSee Caption
Xmp.acdsee.datetime                          XmpText     0  
Xmp.acdsee.author                            XmpText    13  ACDSee Author
Xmp.acdsee.rating                            XmpText     1  4
Xmp.acdsee.notes                             XmpText    12  ACDSee Notes
Xmp.acdsee.tagged                            XmpText     5  False
Xmp.acdsee.categories                        XmpText     0
Comment 24 Maik Qualmann 2015-04-16 21:08:49 UTC
Created attachment 92088 [details]
acdseetags4.patch

The following fields are now supported when importing images:

Xmp.acdsee.categories
Xmp.acdsee.caption
Xmp.acdsee.author
Xmp.acdsee.rating

The "Xmp.acdsee.caption" field is interpreted as "Xmp.dc.description".

Maik
Comment 25 Maik Qualmann 2015-04-17 04:59:30 UTC
Or is "Xmp.acdsee.notes" -> "Xmp.dc.description" and "Xmp.acdsee.caption" -> "Xmp.dc.title"?

Maik
Comment 26 caulier.gilles 2015-04-17 06:24:04 UTC
> Or is "Xmp.acdsee.notes" -> "Xmp.dc.description" and "Xmp.acdsee.caption" -> "Xmp.dc.title"?

What's about comments i18n management here, as comment can be in English (default), in French, in German, etc...

ACDsee mamespace support this feature, as standard XMP namespace ? Exiftool which have the most advanced list of tags reference as these entries :

http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/XMP.html#acdsee

Note : why application create new namespace for usual tags if no new properties are not add ? But it's another subject...

Gilles
Comment 27 Alan Pater 2015-04-17 17:19:50 UTC
(In reply to Maik Qualmann from comment #25)
> Or is "Xmp.acdsee.notes" -> "Xmp.dc.description" and "Xmp.acdsee.caption" ->
> "Xmp.dc.title"?
> 
I think it should be that way, not "Xmp.acdsee.caption" -> "Xmp.dc.description". In the ACDSee documentation, "Caption" is also referred to to as "Title".

"Notes" appear more like "Description". We have a sample image with the following:

Xmp.acdsee.notes    XmpText    91  Quebec - Cours de perfectionnement de la Chambre des notaires du Quebec - Chateau Frontenac
Comment 28 Maik Qualmann 2015-04-17 19:08:09 UTC
Created attachment 92108 [details]
acdseetags5.patch

Final version of the patch to import ACDSee XMP fields.

Xmp.acdsee.categories
Xmp.acdsee.caption -> Xmp.dc.title
Xmp.acdsee.author
Xmp.acdsee.rating
Xmp.acdsee.notes -> Xmp.dc.description

I think that only a few users want to write the ACDSee Metadata back into the image.
That should be optional -> https://community.kde.org/GSoC/2015/Ideas#Project:_Advanced_Metadata_HUB

Maik
Comment 29 caulier.gilles 2015-04-17 20:53:31 UTC
>I think that only a few users want to write the ACDSee Metadata back into the image.
>That should be optional -> >https://community.kde.org/GSoC/2015/Ideas#Project:_Advanced_Metadata_HUB

Maik,

Optional, yes, but implemented in DMetadata. This GSoC 2015 project is to route metadata reading and writing rules.

It will be great to have ACDSee Export code as i do for LR and Microsoft namespace, for interroperability reasons.

As Alan is able to check if export from digiKam to ACDSee will work, all conditions to validate this kind of code are here...

Gilles
Comment 30 Alan Pater 2015-04-21 19:51:22 UTC
I've added the acdsee namespace and properties to exiv2.

http://dev.exiv2.org/projects/exiv2/repository/revisions/3715
Comment 31 caulier.gilles 2015-04-21 20:01:31 UTC
Alan,

This will help to make code to export from digiKam to ACDSee ?

Maik,

Did you manage this king of feature ?

Gilles
Comment 32 Alan Pater 2015-04-21 20:31:05 UTC
(In reply to Gilles Caulier from comment #31)

> This will help to make code to export from digiKam to ACDSee ?
> 
Correct. This makes the acdsee properties writable, assuming that the user has at least exiv2 rev 3715.
Comment 33 Maik Qualmann 2015-04-21 20:52:32 UTC
I can write from digiKam the ACDSee properties without the patch. The export to ACDSee is in work.

Maik
Comment 34 Maik Qualmann 2015-04-26 10:36:41 UTC
Created attachment 92226 [details]
acdseetags6.patch

This patch is now with write support from digiKam to ACDSee.

Maik
Comment 35 Maik Qualmann 2015-04-26 10:39:35 UTC
Created attachment 92227 [details]
digikam.jpg

Test image with categories for ACDSee created with digiKam.

Maik
Comment 36 Alan Pater 2015-04-28 12:19:41 UTC
Cool, looks like everything is working!

I asked to Heikki Rekki to test the image created on digikam, and he reports success.

On Tue, Apr 28, 2015 at 4:19 AM, Heikki Rekki wrote:
> Wow! Thank You all! You have done fine job!
> The categories of the test image are all fine in my system (Win7 64b,
> ACDSeePro7 64b).

https://mail.kde.org/pipermail/digikam-users/2015-April/020904.html
Comment 37 caulier.gilles 2015-04-29 15:31:14 UTC
Maik,

The feedback sound good for this entry, at least ACDSee Interoperability work fine in both way.

Gilles
Comment 38 Maik Qualmann 2015-04-29 20:07:18 UTC
I think we need a new version of exiv2 with the patch from Alan out comment 30. I have a strange problem here. If exiv2 has not seen (open) a image with ACDSee metadata, no metadata can be written. 

digikam(5938)/KEXIV2: Cannot remove Xmp tag using Exiv2   (Error # 35 :  No namespace info available for XMP prefix `acdsee'

digikam(5938)/KEXIV2: Cannot set Xmp tag string into image using Exiv2   (Error # 35 :  No namespace info available for XMP prefix `acdsee'

After the first reading of an image with ACDSee metadata, the metadata can also be written. Reproducible after each start of digiKam. Has exiv2 a temporary cache for new metadata?

Maik
Comment 39 Alan Pater 2015-04-29 20:51:03 UTC
Same goes for the "lr" namespace I believe.  exiv2 does not have a temporary space, I thought digikam does. Or does digikam pre-register the "lr" namespace?

Otherwise exiv2 0.25 should be out in the wild in 20 days.
    http://dev.exiv2.org/projects/exiv2/roadmap

 But, of course, you can't guarantee that a given user has that version. Can you prompt the user to upgrade?

      IF exiv2 < 0.25 THEN politely inform the user that this function needs a newer exiv2 library ..
Comment 40 caulier.gilles 2015-04-29 21:16:19 UTC
I think you don't need to have an update of Exiv2. Just declare namespace in Libkexiv2 instance, as i do with LR xmp namespace here :

https://projects.kde.org/projects/kde/kdegraphics/libs/libkexiv2/repository/revisions/master/entry/libkexiv2/kexiv2.cpp#L110

Note that namespace need to be unregistered at end of libkexiv2 instance, else there is a memory leak in Adobe XMP SDK (thanks valgrind)

Gilles
Comment 41 Maik Qualmann 2015-04-30 18:41:55 UTC
Created attachment 92344 [details]
libkexiv2.patch

Thanks Gilles, the namespace of ACDSee was added to libkexiv2.

Maik
Comment 42 caulier.gilles 2015-04-30 18:45:19 UTC
Well, if all patches work together, it's fine to commit.

Gilles
Comment 43 Maik Qualmann 2015-04-30 18:54:54 UTC
Created attachment 92345 [details]
acdseetags7.patch

Small update for the patch. Gilles, if you have no objections, I would the patch now write into branch master.

Maik
Comment 44 Maik Qualmann 2015-04-30 19:08:36 UTC
Git commit 01c9cb4e62efd2211680a87db6e1a422a2b8186b by Maik Qualmann.
Committed on 30/04/2015 at 19:04.
Pushed by mqualmann into branch 'master'.

namespace from ACDSee added

M  +2    -0    libkexiv2/kexiv2.cpp

http://commits.kde.org/libkexiv2/01c9cb4e62efd2211680a87db6e1a422a2b8186b
Comment 45 Maik Qualmann 2015-04-30 19:29:43 UTC
Git commit 0ffb27e37a2af5eb40657730d28e5f891bc42c37 by Maik Qualmann.
Committed on 30/04/2015 at 19:20.
Pushed by mqualmann into branch 'master'.

apply patch #92345 to import ACDSee hierarchical categories
FIXED-IN: 4.10.0

M  +2    -1    NEWS
M  +177  -5    libs/dmetadata/dmetadata.cpp

http://commits.kde.org/digikam/0ffb27e37a2af5eb40657730d28e5f891bc42c37
Comment 46 caulier.gilles 2015-05-01 07:42:40 UTC
Git commit bba8e5f2ccaa0047a81c376e0d2efd0f57b3e8de by Gilles Caulier.
Committed on 01/05/2015 at 07:41.
Pushed by cgilles into branch 'frameworks'.

backport commit #01c9cb4e62efd2211680a87db6e1a422a2b8186b from git/master to frameworks branch

M  +2    -0    src/kexiv2.cpp

http://commits.kde.org/libkexiv2/bba8e5f2ccaa0047a81c376e0d2efd0f57b3e8de
Comment 47 Maik Qualmann 2015-05-01 09:49:45 UTC
Git commit b472006f71c9774ab5194770dc2a079241e64a19 by Maik Qualmann.
Committed on 01/05/2015 at 09:46.
Pushed by mqualmann into branch 'master'.

correction if special characters in tag names exist

M  +7    -1    libs/dmetadata/dmetadata.cpp

http://commits.kde.org/digikam/b472006f71c9774ab5194770dc2a079241e64a19
Comment 48 caulier.gilles 2015-05-01 12:10:24 UTC
Git commit 1d4848f72664815f8e39eb19b5bcd1a4b8c6feab by Gilles Caulier.
Committed on 01/05/2015 at 12:09.
Pushed by cgilles into branch 'frameworks'.

backport commit #0ffb27e37a2af5eb40657730d28e5f891bc42c37 from git/master to frameworks branch

M  +178  -5    libs/dmetadata/dmetadata.cpp

http://commits.kde.org/digikam/1d4848f72664815f8e39eb19b5bcd1a4b8c6feab
Comment 49 caulier.gilles 2015-05-01 12:13:01 UTC
Git commit ad04aa610c3425c98e59c508cf27d82f5391aca8 by Gilles Caulier.
Committed on 01/05/2015 at 12:12.
Pushed by cgilles into branch 'frameworks'.

backport commit #b472006f71c9774ab5194770dc2a079241e64a19 from git/master to frameworks branch

M  +7    -1    libs/dmetadata/dmetadata.cpp

http://commits.kde.org/digikam/ad04aa610c3425c98e59c508cf27d82f5391aca8
Comment 50 caulier.gilles 2015-05-01 12:16:17 UTC
Maik,

At end of commit from comment #45, you have performed these changes :

             return fromIptcOrXmp("Iptc.Application2.Headline", "Xmp.photoshop.Headline");
         case MetadataInfo::Title:
         {
-            QVariant var = fromXmpLangAlt("Xmp.dc.title");
-
-            if (!var.isNull())
-            {
-                return var;
-            }
-
-            return fromIptcEmulateLangAlt("Iptc.Application2.ObjectName");
+            QString str = getImageTitles()[QString("x-default")].caption;
+
+            if (str.isEmpty())
+            {
+                return QVariant(QVariant::Map);
+            }
+
+            QMap<QString, QVariant> map;
+            map["x-default"] = str;
+            return map;
         }
         case MetadataInfo::DescriptionWriter:
             return fromIptcOrXmp("Iptc.Application2.Writer", "Xmp.photoshop.CaptionWriter");

You drop fromIptcEmulateLangAlt() call. This is intentional ? New code will replace it properly ?
Comment 51 Maik Qualmann 2015-05-01 16:00:20 UTC
(In reply to Gilles Caulier from comment #50)
> You drop fromIptcEmulateLangAlt() call. This is intentional ? New code will
> replace it properly ?

Yes, in order to avoid duplicate code, getImageTitles() is now used. This includes "Iptc.Application2.ObjectName". It is right that we should first check for XMP and IPTC as last? But I will check it again.

Maik
Comment 52 caulier.gilles 2015-05-01 16:01:23 UTC
I would just to be sure that no regression is included...

Gilles
Comment 53 caulier.gilles 2015-08-14 08:48:03 UTC
*** Bug 351260 has been marked as a duplicate of this bug. ***
Comment 54 caulier.gilles 2015-08-17 15:12:45 UTC
*** Bug 351400 has been marked as a duplicate of this bug. ***