Bug 103201

*** This bug has been confirmed by popular vote. ***

The bug #113997 looks like the same (the #113997 is a technical solution, the #103201 is a functional wish).

I envision an export function that could work something like:
Choose a series of photos, by tag, by album, by whatever.
Choose Export digiKam Archive.
Choose a destination directory.
Choose options like preserve album hierarchy.
digiKam then generates 
* a meta-data file (SQL dump or similar) for each photo in the set
* a tarball containing the images and meta-data file.

There's no real point in compressing the tarball, as most content will be compressed already (JPEG for instance).  The user now has a tarball that any compatible form of digiKam should be able to import, generating albums, tags, ratings as needed.

Combined with the concept of md5/sha1 hashes of each file, duplicate detection becomes possible (only import files that aren't in the digiKam db already for instance).  Must file that md5 idea some day.

Ok. This is the role of the future digiKam backup/restore tool planed later than 0.9.0. I have no time to work actually on this subject, but i recommend to all users/developpers to post all comments about in this thread. Thanks in advance...

Gilles

> I envision an export function that could work something like:
> Choose a series of photos, by tag, by album, by whatever.
> Choose Export digiKam Archive.
> Choose a destination directory.
> Choose options like preserve album hierarchy.
> digiKam then generates
> * a meta-data file (SQL dump or similar) for each photo in the set
> * a tarball containing the images and meta-data file.

I'm not sure about creating a tarball. Personally, I'd prefer to have a folder 
containing the images (organized by albums) and an XML file with the 
metadata. I could then store this folder on a CD-R and access the pictures 
with any program, while still being able to import the pictures on another 
computer. You'd lose some flexibility by "hiding" the images in a tarball.
If you really needed to have a single file containing everything you could 
still create a tarball from that folder.

> Combined with the concept of md5/sha1 hashes of each file, duplicate
> detection becomes possible (only import files that aren't in the digiKam db
> already for instance).  Must file that md5 idea some day.

See wish #110066.
There's no mention yet of using MD5 sums to identify duplicates.

why not just add to the export dialog a checkbox "make tarball"?
Give the user a choice :)

> I'm not sure about creating a tarball. Personally, I'd prefer to have a
> folder containing the images (organized by albums) and an XML file with the
> metadata.

I'd like to slightly amend my proposal. IMO opinion it would be best to store 
such an XML file in *every* subfolder (i.e. album). These XML files would 
only contain the metadata of the pictures contained in the same directory.
Subalbums of such a backup could simply be extracted with konqueror. 
Drag'n'drop would be everything that's needed. So, the backup would consist 
of the current directory hierarchy but with multiple "XML databases" instead 
of a single binary database file.

This would make exporting almost trivial.
Importing would be another matter, but importing is always more complicated, 
since duplicates (pictures, tags, albums) must be handled intelligently.

Why don't you store this extra data (tags, comments, etc...) on the IPTC metadata of the photo file !
If it could be slowlier, why don't you use an flag in the option menu to do that ?

> This is the role of the future digiKam backup/restore tool
> planed later than 0.9.0.
Cool, so that's planned in the not-so-long term ? (and bug #117073 will surely be fixed along the way)

Regarding metadata, I think it would be good to store as much of it as possible in the files themselves (like picasa does) in metadata headers, at least when exporting. But from what I read elsewhere, this is also planned...

No need XML file here. Other photo management sofware using this way like picasa of photoshop album.

The universal way to store complex informations in pictures is XMP.

To support XMP in digikam, we need a library:

1- Extended Exiv2 to support XMP (my preferred way)
2- Using Adobe library, recently ported to linux, but not GPL !!!

I prefer Exiv2 in all case because it's a C++ free sofware library. Concept are excelent and work very well. Add XMP support is not very complicated, but require some free time... 

I have planed to contrib in Exiv2 project later than 0.9.0-beta1 to add more RAW file formats parser, like MRW (I have a Minolta camera).

I'm using this thread to say that all possible contributors to Exiv2 project are welcome. All improvements to Exiv2 lib will improve digiKam as well.


About #117073 file, this is relevant of libkipi improvements. The widget to display all album in kipi-plugins come from libkipi. Please Nicolas, move this B.K.O to kipi-plugins area. Thanks in advance.

Gilles

Ah forget to said that IPTC is a limited way to store data. Its an old method to embeded information in media. Strings size are limited and it do not suppot UTF8 encoding.

XMP is the future, but IPTC is very used by pro-photographers.

Gilles

Note: Nicolas, you working on Eurocopter. Are you living near to Marseille ? Me, yes...

> ------- No need XML file here. Other photo management sofware using this
> way like picasa of photoshop album.
>
> The universal way to store complex informations in pictures is XMP.

I didn't know about XMP, but it sure sounds interesting. But I'm wondering if 
XMP has to be stored *in* the image file? I would be astonished if *all* 
supported file formats also allow saving XMP data. For example I have some 
(crappy) videos taken with my camera. Digikam shows these files in the album 
view and even allows me to tag them.
Since this wish concerns *lossless* transport of albums and everything, XMP 
would *not* realise my wish. Due to the variety of supported file formats, 
only an external metadata file can guarantee an exact replication on the 
destination computer.

Yes, XMP is a metadata embedded in File. Support JPEG, TIFF, PNG, PSD, and others major file format used in digital photography. More info here :

http://www.adobe.com/products/xmp/overview.html

The Adobe SDK :

http://partners.adobe.com/public/developer/xmp/sdk/index.html

Look the licence... sounds like an Open Source lib but with any restrictives points.

Gilles

XMP looks interesting indeed, BUT it is proprietary.
Is there no free alternative? If not, maybe an initiative should be started.
I have a bad feeling with trusting on a proprietary format, I'd rather not use it for my data if I have the choice...

My 2 Euro cents:

I think the current way to store pictures in a strict album tree while using a database is no longer necessary and I think its a restriction that can be overcome.

Part of the the database should be the path to a picture and a hash ID. The database should be part of ~/.kde/share/apps/digikam/ and not part of an export or backup. Todays harddisks are big enough to store really huge database files or if sqlite failed to do so digikam can also support a "real" sql service

To exchange (this means duplicate) photos between digikam users its fine also to exchange digikams own metadata like Rating, Tags or allocation of CMS infos.
This exchange should be based on embedded metadata like XMP or EXIF. I dont think it is a good idea to squeeze it into limited IPTC structure better create a own tag. Of course this need exellent r/w metadata support. There is hope that exiv2 improves in a nearer future.

One problem is in my concept the integration of photos on removable media. A  photo on a CD,DVD or CF-card on a Linux system can have the same name but different content. That why I think a unique hash value as content ID is necessary to identify those structures.

To speed-up the handling of such a ID structure this maybe leeds to a caching system like digikam already uses for thumbnails. 

Bye

   Thorsten 

> ------- Additional Comments From caulier.gilles free fr  
> 2006-04-05 17:27 ------- Yes, XMP is a metadata embedded in 
> File. Support JPEG, TIFF, PNG, PSD, and others major file 
> format used in digital photography. 

What happens if you try to view, say, a JPEG file containing XMP
metadata with an application that is not XMP-enabled ?

SVN commit 526983 by cgilles:

digikam from trunk : IPTC and digiKam tags :

- Tags are saved in IPTC Keywords, but this tag can be redondant. This is want mean that each digiKam tag will be saved in a IPTC keywords tag, each one limited to 64 characters (before all digiKam tags been saved in only one IPTC Keywords tag). It's more powerfull (:=)))...

- Rating are now saved in IPTC Urgency tag, not FixtureId tag. This way is used by other photo management programs, like mapivi for example.

CCMAIL: digikam-devel@kde.org
CCBUGS: 103201, 113997

CCMAIL: digikam-devel@kde.org

 M  +107 -55   dmetadata.cpp  


--- trunk/extragear/graphics/digikam/libs/dmetadata/dmetadata.cpp #526982:526983
@@ -669,6 +669,22 @@
     return false;
 }
 
+/*
+Iptc.Application2.Urgency <==> digiKam Rating links:
+
+digiKam     IPTC
+Rating      Urgency
+
+0 star  <=>  8          // Least important
+1 star  <=>  7
+1 star  <==  6
+2 star  <=>  5
+3 star  <=>  4
+4 star  <==  3
+4 star  <=>  2
+5 star  <=>  1          // Most important
+*/
+
 int DMetadata::getImageRating() const
 {
     try
@@ -680,25 +696,29 @@
         {
             Exiv2::IptcData iptcData;
             iptcData.load((const Exiv2::byte*)m_iptcMetadata.data(), m_iptcMetadata.size());
-            Exiv2::IptcKey key("Iptc.Application2.FixtureId");
+            Exiv2::IptcKey key("Iptc.Application2.Urgency");
             Exiv2::IptcData::iterator it = iptcData.findKey(key);
             
             if (it != iptcData.end())
             {
-                QString IptcComment(it->toString().c_str());
+                QString IptcUrgency(it->toString().c_str());
     
-                if (IptcComment == QString("digiKamRating=0"))
+                if (IptcUrgency == QString("1"))
+                    return 5;
+                else if (IptcUrgency == QString("2"))
+                    return 4;
+                else if (IptcUrgency == QString("3"))
+                    return 4;
+                else if (IptcUrgency == QString("4"))
+                    return 3;
+                else if (IptcUrgency == QString("5"))
+                    return 2;
+                else if (IptcUrgency == QString("6"))
+                    return 1;
+                else if (IptcUrgency == QString("7"))
+                    return 1;
+                else if (IptcUrgency == QString("8"))
                     return 0;
-                else if (IptcComment == QString("digiKamRating=1"))
-                    return 1;
-                else if (IptcComment == QString("digiKamRating=2"))
-                    return 2;
-                else if (IptcComment == QString("digiKamRating=3"))
-                    return 3;
-                else if (IptcComment == QString("digiKamRating=4"))
-                    return 4;
-                else if (IptcComment == QString("digiKamRating=5"))
-                    return 5;
             }
         }
     }
@@ -729,12 +749,31 @@
             iptcData.load((const Exiv2::byte*)m_iptcMetadata.data(), m_iptcMetadata.size());
 
         setImageProgramId(iptcData);
+        QString urgencyTag;
         
-        QString temp;
-        QString ratingString("digiKamRating=");
-        ratingString.append(temp.setNum(rating));
+        switch(rating)
+        {
+            case 0:
+                urgencyTag = QString("8");
+                break;
+            case 1:
+                urgencyTag = QString("7");
+                break;
+            case 2:
+                urgencyTag = QString("5");
+                break;
+            case 3:
+                urgencyTag = QString("4");
+                break;
+            case 4:
+                urgencyTag = QString("3");
+                break;
+            case 5:
+                urgencyTag = QString("1");
+                break;
+        }
         
-        iptcData["Iptc.Application2.FixtureId"] = ratingString.latin1();
+        iptcData["Iptc.Application2.Urgency"] = urgencyTag.ascii();
         setIptc(iptcData.copy());
 
         return true;
@@ -749,6 +788,41 @@
     return false;
 }
 
+// Warning: this method haven't be tested yet!
+QStringList DMetadata::getImageKeywords() const
+{
+    try
+    {    
+        if (m_iptcMetadata.isEmpty())
+        {
+            Exiv2::IptcData iptcData;
+            iptcData.load((const Exiv2::byte*)m_iptcMetadata.data(), m_iptcMetadata.size());
+            QStringList keywords;
+            
+            for (Exiv2::IptcData::iterator it = iptcData.begin(); it != iptcData.end(); ++it)
+            {
+                QString key = QString::fromLocal8Bit(it->key().c_str());
+                
+                if (key == QString("Iptc.Application2.Keywords"))
+                {
+                    QString val(it->toString().c_str());
+                    keywords.append(val);
+                }
+            }
+            
+            return keywords;
+        }
+    }
+    catch( Exiv2::Error &e )
+    {
+        kdDebug() << "Cannot get Keywords from image using Exiv2 (" 
+                  << QString::fromLocal8Bit(e.what().c_str())
+                  << ")" << endl;
+    }        
+    
+    return QString();
+}
+
 bool DMetadata::setImageKeywords(const QStringList& keywords)
 {
     try
@@ -756,18 +830,28 @@
         if (keywords.isEmpty())
             return false;
 
+        QStringList keys = keywords;
+        
         Exiv2::IptcData iptcData;
         if (!m_iptcMetadata.isEmpty())
             iptcData.load((const Exiv2::byte*)m_iptcMetadata.data(), m_iptcMetadata.size());
         
         setImageProgramId(iptcData);
-
-        // Keywords IPTC tag is limited to 64 char.
-        QString keywordsString = keywords.join(" ");
-        keywordsString.truncate(64);
-        kdDebug() << m_filePath << " ==> Keywords: " << keywordsString << endl;
+        kdDebug() << m_filePath << " ==> Keywords: " << keywords << endl;
         
-        iptcData["Iptc.Application2.Keywords"] = keywordsString.latin1();
+        // Keywords IPTC tag is limited to 64 char but can be redondancy.
+        
+        for (QStringList::iterator it = keys.begin(); it != keys.end(); ++it)
+        {
+            QString key = *it;
+            key.truncate(64);
+            Exiv2::IptcKey iptcTag("Iptc.Application2.Keywords");
+            
+            Exiv2::Value::AutoPtr val = Exiv2::Value::create(Exiv2::asciiString);
+            val->read(key.latin1());
+            iptcData.add(iptcTag, val.get());        
+        }
+        
         setIptc(iptcData.copy());
         return true;
     }
@@ -853,36 +937,4 @@
     return false;
 }
 
-// -- METHODS BELOW ARE UNTESTED ---------------------------------
-
-QStringList DMetadata::getImageKeywords() const
-{
-    try
-    {    
-        if (m_iptcMetadata.isEmpty())
-        {
-            Exiv2::IptcData iptcData;
-            iptcData.load((const Exiv2::byte*)m_iptcMetadata.data(), m_iptcMetadata.size());
-            Exiv2::IptcKey key("Iptc.Application2.Keywords");
-            Exiv2::IptcData::iterator it = iptcData.findKey(key);
-            
-            if (it != iptcData.end())
-            {
-                QStringList keywords;
-                QString keywordsString(it->toString().c_str());
-                keywords.split(" ", keywordsString);
-                return keywords;
-            }
-        }
-    }
-    catch( Exiv2::Error &e )
-    {
-        kdDebug() << "Cannot get Keywords from image using Exiv2 (" 
-                  << QString::fromLocal8Bit(e.what().c_str())
-                  << ")" << endl;
-    }        
-    
-    return QString();
-}
-
 }  // NameSpace Digikam

My 2 `centimes d'euro` :
For what I've seen on photoshop generated jpeg with xmp, I don't
really like xmp :
   - Huge chunk of non compressed text (I enclose the copy of a nearly
empty one from a file I've found on the web, I've seen files with much
more data with several time the same info)
  - Of course, we all have plenty of memory, disc space and cpu power
we want to throw away don't we ?

That's what I like in IPTC: it only uses the space for the data we
want to include and for me, there no big missing feature in it.
I think I remember reading something about the possible use of UTF-8
in IPTC but I can't find it back. There is something on page 52 of the
IIMV4.1 specification about UTF-8. It seems that other applications
don't care about character set. I think that for that character set
problem, a flag in the settings to use utf-8 or pure ascii will be
nice. Perhaps also a flag in an export function to convert to pure
ascii. With that two things, people like me with languages full of
accents can be happy, and still be able to share the pictures with
people with badly programmed windows applications ;-)
( I can speak of badly programmed things as I just updated not so long
ago my hack/patch on the kfile_jpeg plugin to support character sets)

Unfortunatly, I don't have enought time to take part in digikam main
developpment (I can hardly follow all these interesting discussions in
this mailing list). I may try to work a little on some kipi plugins
but I can't engage myself as it will be very slow)

Please see also my comment in the following bug about a possible way
to handle some IPTC (and maybe XMP) information :
http://bugs.kde.org/show_bug.cgi?id=91812#c15

     Loic
P.S.: that was a big message.

Yes, i have seen your report into IPTC editor B.K.O entry. IPTC editor is planed later 0.9.0 release. We will talking about indeep in the future. Actually, i'm look like "mapivi" working with IPTC. Take a look and give me a feedback

Gilles Caulier

What about this one? Since XMP can be used now, isn't this satisfying the topic here?

Andi

Yes and no, because not all files can be writted with metadata (some RAW format for ex.)

The ultimate way is to have a tools to export all digiKam metadata into an XMP sidecar file. This can be done in a future backup/restore tool. Still TODO

Note : Exiv2 0.18 can read/write now XMP sidecar files. So it will be easy to implement.

Gilles

Since XMP sidecar is fully supported by Exiv2 and digiKam, and as it's easy to process file metadata sync with DB into XMP sidecar through Maintenance tool, this file can be considerate as fixed now.  All digiKam DB data are saved into XMP sidecar through all standard XMP namespace. For non standard value, we use a dedicated digiKam XMP namespace.

Gilles Caulier