Bug 268688

Marcel,

I'm sure that we have already work on this problem in the past. At least me, if i remember...

There is 2 situations :

1/ To have Tags recorded to item metadata exactly as digiKam database (This is my case). It's typically the case of user only use digiKam to manage items.

2/ To preserve old tags already assigned to image with another photo-management program. Typically, tags are created to DB when image are imported. Sometime, people want that digiKam do not touch to old tags.

For me, the case 2/ is very dangerous. DB and item metadata are not synchronized properly...

I resume fine the situation ?

Gilles Caulier

Gilles, you summarize very well. However, I think it is very dangerous to assume that digikam is the user's only image tool.
And I think there will always be metadata in images that digikam doesn't necessarily understand but that metadata needs to be preserved anyway.

If I understand Mark correctly, he is actively changing/renaming/deleting a tag and then he wants *that specific change* to be reflected in the metadata of the image. He doesn't want other metadata to be affected.

Best regards,
Mikkel

ok. 

I will explain better indeep. Tags are stored in many places :

1/ IPTC keywords (name only, not path)
2/ XMP keywords (name only, not path) = IPTC keywords without ASCII char and strings size limitations 
3/ XMP digiKam namespace : full tag path.

For 1/ and 2/, they contents full history, and not reflect the exact database contents. Typically old keywords can be still assigned to image.

For 3/, it's THE dedicated place to store digiKam tag paths from database. This must be the exact copy of DB for backup. It's this point which is not fine currently. It's the current problem.

NOTE : what's keywords and tag paths :

for a digiKam tag as "City/Paris/Monuments/Seine/Events" with "City", "Monuments", and "Events" assigned to and item

=> Keywords are : "City", "Monuments", and "Events"
=> Tag Paths are : "City", "City/Paris/Monuments", and "City/Paris/Monuments/Seine/Events"

If this item is imported to a new computer with digiKam, by priority, tags will be created in database using :

1/ XMP digiKam namespace
2/ XMP keywords
3/ IPTC keywords

Gilles Caulier

Gilles, thank you very much for the thorough explanation. I understand it much better now. It also gives me some ideas for other feature requests, but they don't belong in this bug.

Best regards, Mikkel

Looks like I've managed to remove obsolete tags from Xmp.dc.subject by using Image > Metadata > Edit XMP by manually editing every single photo in my collection, but there doesn't seem to be a way of editing Xmp.digiKam.TagsList without using exiv2 from the command line.

Which as I keep a copy of all images on at least 3 other computers and if Gilles is right by saying the import order is:
1/ XMP digiKam namespace
2/ XMP keywords
3/ IPTC keywords
Then another copy of DigiKam on another computer will always show the wrong tags.

I remember a very old bug report about this situation. The problem is that we'd need to record the change when the tag is renamed, possibly ask for confirmation that all images should be edited, and then edit all affected images. This is not implemented.

Marcel, Yes, there is an old entry for this subject.

I suspect that it have been implemented and removed because another entry ask the oposite...

Why not to integrate an option to Metadata settings panel ?

For ex, i don't want to backup old tags in metadata. I want a pure copy of current tags from DB.

Gilles Caulier

Hi,
I've tried today to rename a tag and stumbled upon this bug (using digikam 2.0-beta4). After playing a little bit I've found a workaround which should allow me to rename tags for multiple images at once without the tedious manual editing.

This may be interested to OP (Mark Fraser) and other users affected by the bug, as well as to developers as an example how renaming may be easily implemented on top of existing features.

In order to rename tag "foo" into "bar" do:
0. if tag "bar" doesn't exist, create one
   (e.g. choose in the main menu: "Tag->New...")
1. in the main menu choose "Browse->Tags".
   "My Tags" hierarchy is opened on the left
2. choose the existing "foo" tag.
   all files assigned this tag are displayed in the main view
3. select all files in the main view (Ctrl+A)
4. open "Caption/tags" right pane
5. choose (check) "bar" tag in "My Tags" tree in the right pane
6. uncheck "foo" tag in the same tree
7. press "Apply" in the right pane
8. in the main menu choose "Image->Write Metadata to Selected Images".
   I'm not sure whether this step is always required, since in my
   configuration metadata is written to files automatically.

I've tested this workflow using digikam 2.0-beta4 as the only photo management software, using XMP sidecar.

BTW, this bug is relevant to tag *removal* and *move* as well:
- if you delete a tag which is assigned to some files, the tag is removed from DB, but not from these files.
- if you move the tag which is assigned to some files across tag hierarchy (e.g. foo/test_tag -> bar/test_tag), DB is updated but not the file metadata. Actually, this is equivalent to regular tag renaming.

The workflow I've mentioned in comment #8 is far from being intuitive to the end user, so I propose to add the following behavior to digikam:

When user performs some change (delete/move/rename/something else?) in tags for the tag assigned to some files display a dialog like this:
-------------------------------------------------
You have requested to rename/move tag "/People/Alice" into "/People/Bob".
"/People/Alice" is currently assigned to 42 files.
Do you want to update these files metadata or just the Digikam database?

| Update database only |
| Update database and files |
| Cancel |

[ ] Remember my decision
-------------------------------------------------
(in case of tag deletion the wording in the first line is a little bit different)

A new (drop-down 3-value list) option can be added under "Settings->Metadata->Behavior->Common Metadata Actions", which will control the above dialog:

When changing properties of a tag assigned to files:
- update database only
- update database and files
- ask me
With default to "ask me".

When user uncheck the option "Save image tags as "Keywords" tags in metadata embedded in files" the new option (the proposed one) can be grayed out and the dialog won't be displayed at all, since the user don't want to touch tags in file's metadata at all.

I would very much like to see this implemented as proposed by Ilia in Comment #9. The ability to write changes in tag hierarchy to all the images affected, would simplify things nicely.

Shouldn't DigiKam just remove the tag from the file?  If I didn't want DigiKam removing tags, I would tell it not to write metadata to the files.

From reading the code, I think this is what DigiKam will do:

1. Add image already tagged Person/Alice
2. Close DigiKam.  Open another program and tag the image with Person/Bob
3. Close other program.  Open DigiKam.  Tag Person/Charles.  Write the metadata.
DigiKam won't delete Person/Bob.  That's good!
4. Remove Person/Alice.  Write metadata.
DigiKam won't delete Person/Alice.  That's bad.

However, if I ever delete a tag in the DigiKam UI _and_ I have DigiKam set to write metadata, I expect the tag to be deleted.  If I didn't want to write metadata I would tell DigiKam not to write metadata at all.

I don't think DigiKam should prompt the user if they should write the metadata to the file when they remove a tag (there are already many options for what kind of metadata we should write to the file in the settings menu).

When DigiKam decides whether to write data in MetadataHub, we have in loadTags:

    // Those tags which had been set as MetadataAvailable before,
    // but are not contained in this set, have to be set to MetadataDisjoint

That means the change won't be written to the file.

What I think we need is a marker in the DigiKam DB that the tag was explicitly removed.  So DigiKam would have a record that the tag does not exist because the user removed it inside DigiKam.  Then we can avoid removing Person/Bob in the example but know to remove Person/Alice.

Leif,

Well, this bug come back periodically in bugzilla, into 2 different forms:

- Some peoples want to add new tags in image and not to remove older one (similar to a backup or a versioning of tags)

- Other peoples, like me, want to have a strictly synchronization between tags from DB and tags registered to image. This must be the default value.

To solve definitively this problem, an option must be added to setup/metadata. 

Gilles Caulier

Mark,

This file still valid using digiKam 2.4 ?

Gilles Caulier

*** This bug has been confirmed by popular vote. ***

*** Bug 288842 has been marked as a duplicate of this bug. ***

*** Bug 300407 has been marked as a duplicate of this bug. ***

This is a seriouse bug and the main reason that I have to use Windows (and Windows Live Photo Gallery) every now and again, is there any updates on when this might be fixed?
At the moment I can only use digikam to view my photos not edit the metadata due to this issue.

Bump

Regardless of the suggested switch for strong synchronisazion for the various metadata keys in the settings dialog, I would expect DigiKam to strongly synchronize the DB content to the key Xmp.digiKam.TagsList in the digikam namespace of xmp, if metadata export is enabled. 
This includes changes to all affected files when renaming/reoranizing tags.
Otherways there is no way to reliably reconstruct this information in case of an incident to the DB and the people who use this feature would loose much time (they spent into tagging) and faith in this software.

Hi All, Is there any progres on this one? 

/Mårten Jansson

I would like to see some progres as well. Now I have to call `find -iname "*.jpg" -execdir /usr/bin/exiv2 -d i {} +` to delete all tags on all images (Digikam is the only application storing IPTC tags on my system) and then store the tags using Digikam back to files. Strange is that Digikam is not able to keep its own tags in sync.

I have described another "in-digikam-workaround" at bug #288842 comment #2. 
But of course, the best solution would be if this issue is fixed.

I think that two or more ideas are being conflated here.
This is not a request for digikam to auto-remove tags that are not present in the db. ie it is not a request for strong synchronisation of tags. That would be the wrong way to fix this issue IMO.
It is a request for changes made to the tags hierarchy to be reflected in all images containing those tags. So only those tags changed should be updated but it should be done in all relevant images. It should not affect other tags already applied to the image regardless of whether they exist in the digikam db or not.
I struggle to believe that anyone would want the result of moving a tag in a hierarchy to actually just be to copy it to a new location or if they delete a tag, for it to not actually be deleted. Which is actually what the current behaviour is.
If there is a desire to have a history of changes made to tags that could be done as part of non-destructive editing and versioning. It does not mean that this bug should not be fixed as requested.

Yes, and because of this very old issue I consider digiKam's tagging capabilities for new images functional but for already tagged photos completely unusable. digiKam is primarily a DAM tool, and if tagging files causes a cabbage-soup effect of duplicate old and new tags then the program's primary function it broken.
Please prioritize this!

It is possible to rearrange tags on the tag tree and synchronize tags and the database IF you let digiKam write to the images. The trick is to reread metadata from the images to get all the mismatched tags into the digiKam database, then select ALL tags (not just the unwanted tags) on the Filters pane before rearranging and pruning the tag tree. Use a combination of albums and tags on the left pane to actually select the files to work with.

Unfortunately the above process doesn't seem to work if you are only writing to XMP sidecar files. digiKam doesn't seem to reread the XMP file and/or doesn't seem to want to delete any tags already written in the XMP file. 

I can see how some people might want digiKam to leave existing tags alone in the XMP file, if other applications are also using the same XMP file. But for myself, digiKam is the only application that writes to the XMP file and the current behavior makes using only XMP files impossible if I want the XMP files and the digiKam database to be in synch, which I do.

The problem is also that using _only_ Digikam to organise your tags is not well usable. It is able to add tags well, but that is all.

The bug also exists for deleting tags, they are not deleted from metadata.
As someone who has to work with a photo collection on multiple computers and sometimes access photos using other programs, it really is a serious bug. (and one of the very few reasons that I still need to use Windows)

Elle: The XMP sidecar problem seems to be unrelated. Please try to reduce it to a simple test case and report a separate bug.

Marcel, per your request, I submitted a bug report:
https://bugs.kde.org/show_bug.cgi?id=309058

On 10/26/12, Marcel Wiesweg <marcel.wiesweg@gmx.de> wrote:
> https://bugs.kde.org/show_bug.cgi?id=268688
>
> --- Comment #28 from Marcel Wiesweg <marcel.wiesweg@gmx.de> ---
> Elle: The XMP sidecar problem seems to be unrelated. Please try to reduce it
> to
> a simple test case and report a separate bug.
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
>

That is incorrect. I only store metadata in the image files themselves, not in sidecar files. I cannot effectively delete tags so that they will be removed from the images' embedded metadata. Removing the digikam.db file doesn't help either as it will get repopulated with crap when the images are reread.

(In reply to comment #25)
> It is possible to rearrange tags on the tag tree and synchronize tags and
> the database IF you let digiKam write to the images. The trick is to reread
> metadata from the images to get all the mismatched tags into the digiKam
> database, then select ALL tags (not just the unwanted tags) on the Filters
> pane before rearranging and pruning the tag tree. Use a combination of
> albums and tags on the left pane to actually select the files to work with.
> 
> Unfortunately the above process doesn't seem to work if you are only writing
> to XMP sidecar files. digiKam doesn't seem to reread the XMP file and/or
> doesn't seem to want to delete any tags already written in the XMP file. 
> 
> I can see how some people might want digiKam to leave existing tags alone in
> the XMP file, if other applications are also using the same XMP file. But
> for myself, digiKam is the only application that writes to the XMP file and
> the current behavior makes using only XMP files impossible if I want the XMP
> files and the digiKam database to be in synch, which I do.

On 10/26/12, DrSlony <bugs@londonlight.org> wrote:
> https://bugs.kde.org/show_bug.cgi?id=268688
>
> --- Comment #30 from DrSlony <bugs@londonlight.org> ---
> That is incorrect. I only store metadata in the image files themselves, not
> in
> sidecar files. I cannot effectively delete tags so that they will be
> removed
> from the images' embedded metadata. Removing the digikam.db file doesn't
> help
> either as it will get repopulated with crap when the images are reread.
>
> (In reply to comment #25)
>> It is possible to rearrange tags on the tag tree and synchronize tags and
>> the database IF you let digiKam write to the images. The trick is to
>> reread

See http://digikam.1695700.n4.nabble.com/How-to-remove-quot-left-behind-quot-tag-tree-td4660139.html

About the fifth post down I posted step by step how to remove all the
outdated tags, if you are writing to the image rather than to an XMP
sidecar. The steps are not intuitively obvious, it's really easy to
miss a step, and sometimes a few outdated tags are still left behind,
necessitating a redo to get the remaining tags. But it can be done.

I moved/broke up a very large tag tree, with tags applied to around
3500 images, leaving behind duplicate, outdated tags in the images.
That was just last week using digiKam 2.9. So I experimented and
searched the internet and came up with a process that worked for me.
As a precaution, probably best to test the process on some throwaway
tags/images before deploying over important images.

Also see this post: http://duq.ca/duqamuq/?p=270. It's for an older
version of digiKam, but the process outlined looks similar.

See "Synchronizing digiKam 2.9.0 database and image metadata"
http://ninedegreesbelow.com/photography/digikam-how-to-synchronize-images-database.html
for how to resynchronize the database and the image metadata. This
only works if digiKam writes to the image files. It doesn't work if
digiKam writes to XMP sidecar files.

Thank you for the guide Elle, I'll try it, but such things should not be necessary, this should just work.

I agree with you, it should just work, and quite frankly, I don't
think my "workaround" is worth the hassle for anyone who does a lot of
tag-tree rearranging.

Sometimes even when all the tags are selected in the tags filter
before rearranging the tag tree, things still get out of
synchronization. This happened to me this AM and the only way I could
get digiKam to recognize the old tags still in the image metadata was
to delete the old database, create a new database, and then unassign
all the outdated tags, tag by tag, a tedious, slow process. I've
modified my guide to emphasize that the procedure isn't a real
solution.

On 10/28/12, DrSlony <bugs@londonlight.org> wrote:
> https://bugs.kde.org/show_bug.cgi?id=268688
>
> --- Comment #33 from DrSlony <bugs@londonlight.org> ---
> Thank you for the guide Elle, I'll try it, but such things should not be
> necessary, this should just work.
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
>

When i just migrated from f-spot to digikam some weeks ago, i thought digikam is totally broken - i was doing so much rearranging (basically build the tree as f-spot didn't put the structure into any tags, so i rearranged every tag. In the end I exported 
To me this behaviour is absolutely rediculus, arguing it is a feature seems strange to me. New tags are written (nonexistance of tag not preserved), removed tags are deleted (nothing preserved at all), but renamed/moved tags are too important to be not-preserved... 

Anyway, I'm attaching my workaround to this. Its a perl script using exiftool to apply the tags it reads from the database. Its not beautiful (non-recursive, fixed maximum depth tagtree resolving, dirty hacks about when to write the metadata and the list of files (if -a is not specified) has to be given newline-delimited, which only happens if I paste to gedit, not to the CLI. Yet it allows you to either straighten the tags of your full collection or to a digikam-selected subset. Maybe its helpful to someone else, too.

Created attachment 77121 [details]
Script to manage tags externally to digikam. German details: https://glumpundzeug.wordpress.com/2013/02/11/arbeiten-mit-digikam-was-es-kann-was-es-nicht-kann-und-was-man-dagegen-machen-kann/

Is the issue solved in the 3.0.0 version?
I was waiting for it before doing a complete tags reorganization.

Marie-Noëlle

2013/2/11 Sven <sven-kdebug@geekbox.info>

> https://bugs.kde.org/show_bug.cgi?id=268688
>


> --
> You are receiving this mail because:
> You voted for the bug.
>

Created attachment 77122 [details]
attached wrong file before, sorry.

digiKam is first and foremost a digital asset management program. This issue should be given the highest priority.

Any updates?

A year later :/

I will fix this issue as part of my Tag Manager proposal, please wait a little bit longer :)

Hi all here,

I discovered a dummy bug in maintenance tool about metadata synchronization which in fact do exactly the opposite. Typically, tool read back image metadata to DB against the inverse. 

So, Tags are always restored from file data to DB... It's weird.

I fixed this problem for next 3.4.0 release planed in September. I cannot reproduce this Tags tree problem, but it would be nice if somebody in this file can check behavior using current implementation from git/master.

Thanks in advance

Gilles Caulier

Created attachment 81960 [details]
Patch : debug statements and quick fix

I found where is bug.

It's in loop from MetadataHub::write() where tags list is recreated before to apply on metadata.

This patch add debug statement to see how is handle tags status from DB and to create tags list to record.

The quick fix here is to disable the line #712 from metadatahub.cpp to prevent loading old path tags list from image metadata.

In fact the loop processed just after run on tags status from DB. It do not handle tags which are not in DB (but always present in image metadata). So, loop process fine DB tags but ignore to remove obsolete one.

If we disable line #712, all sound work fine. 

This cannot be the ultimate solution. The loop must be simplified as there is nothing to handle from a previous tags list loading from image metadata.

I see also that there is always an empty tag from DB checked by the loop. By chance this one is drop when tags list is recorded to image metadata. I don't know where it come from. Perhaps it's a bug in TagsCache class...

Please review my patch and test. If this solution is fine, i will make a better patch for next 3.4.0 release planed on next Monday.

Gilles Caulier

I applied Gille's patch and so far everything seems to work like it should, meaning I haven't (yet?) been able to cause a discrepency between the image/sidecar metadata and the database.

Thanks Elle,

The more advanced test to do is to cha,ge tas on right sidebar to a multiple selection, with combined values, as common tags shared between images, tags removed, tags added, and tags renamed.

There is also a test to check with and without XMP sidecar support enabled.

If all work fine as expected, well i will improve my patch as i explained before.

Thanks in advance for your feedback

Gilles Caulier

:| We ended up doing the same work. Before I could read your post I also found the source of bug.

But your patch s not complete, there is a special case that I fixed:

setXmpKeybords from dmetadata called AddToTagStringBag instead of setXmpTagStringBag

which also causes duplicates.

Ok.. I will commit my changes.

I think it should be done now.

Please test and report if everything work as it should :)

Git commit e15b54f988db686994dea2c1711d80dbf5c8c45d by Veaceslav Munteanu.
Committed on 27/08/2013 at 17:14.
Pushed by munteanu into branch 'master'.

BUG: 268688 Disable read of old tag paths and use setXmpTagStringBag in dmetadata

M  +1    -1    digikam/fileaction/metadatahub.cpp
M  +1    -1    libs/dmetadata/dmetadata.cpp

http://commits.kde.org/digikam/e15b54f988db686994dea2c1711d80dbf5c8c45d

diff --git a/digikam/fileaction/metadatahub.cpp b/digikam/fileaction/metadatahub.cpp
index 115104a..6cb67c8 100644
--- a/digikam/fileaction/metadatahub.cpp
+++ b/digikam/fileaction/metadatahub.cpp
@@ -709,7 +709,7 @@ bool MetadataHub::write(DMetadata& metadata, WriteMode writeMode, const Metadata

         // create list of keywords to be added and to be removed
         QStringList tagsPathList, oldKeywords, newKeywords;
-        metadata.getImageTagsPath(tagsPathList);
+        //metadata.getImageTagsPath(tagsPathList);

         for (QMap<int, TagStatus>::iterator it = d->tags.begin(); it != d->tags.end(); ++it)
         {
diff --git a/libs/dmetadata/dmetadata.cpp b/libs/dmetadata/dmetadata.cpp
index c465572..e8abb6a 100644
--- a/libs/dmetadata/dmetadata.cpp
+++ b/libs/dmetadata/dmetadata.cpp
@@ -3037,7 +3037,7 @@ QStringList DMetadata::getXmpKeywords() const

 bool DMetadata::setXmpKeywords(const QStringList& newKeywords, bool setProgramName) const
 {
-    return addToXmpTagStringBag("Xmp.dc.subject", newKeywords, setProgramName);
+    return setXmpTagStringBag("Xmp.dc.subject", newKeywords, setProgramName);
 }

 bool DMetadata::removeXmpKeywords(const QStringList& keywordsToRemove, bool setProgramName)

This is from before the new patch!

Original Settings: Reading and Writing Metadata:
Read from sidecar files
Write to XMP sidecar only
(not checked: 'If possible write Metadata to RAW files')
I removed several tag heirarchies altogether from the list of tags, rearranged several of the tags that were left, renamed one of the tags, selected all the images, wrote the information to disk, closed and reopened digiKam. Spot-checking with exiftool, the sidecars are exactly synchronized with the database. The images, as expected, are not. At some point along the way I added a tag with Exiftool to one image. digiKam has not noticed this new tag, which is what should happen if the database really controls what goes into the xmp sidecar files.

Change to New Settings: Reading and Writing Metadata:
Read from sidecar files
Write to image and XMP Sidecar 
('If possible write Metadata to RAW files' is not checked)
Now changing digiKam settings so that digiKam writes to the image as well as the sidecar, and writing to all images: The tag that I added to the image with Exiftool was removed from the image file.  So the database is controlling what goes into the image file and not the other way around. All jpeg metadata is now synchronized to the database. The raw files of course are not, but the raw file sidecars are synchronized.