The digiKam database can't be synchronized with XMP sidecars. digiKam writes new tags to the sidecar file, but doesn't delete a tag that has been deleted from the database. So when rearranging tags on the tag tree, the xmp file contains the old tag tree and the new tag tree. The same thing happens when renaming rather than rearranging tags. Reproducible: Always Steps to Reproduce: 1. Set digiKam metadata settings to only read and write to xmp sidecar files. 2. Create a tag tree and assign it to an image. The sidecar file and database are in synch. 3. Move the tag tree. Now the sidecar file and database are no longer in synch. The database only has one copy of the tag tree, in the correct new location. The xmp file has two copies, in the old location and in the new location. 4. Rename the tag and apply/write. Now the tag is in the xmp file under both the old and the new name, as well as in two different tag trees. Actual Results: In the database, the tag tree is moved. But in the xmp file, instead of one tag tree in the new location, now there are two tag trees, one in the new location, one in the old location. In the database, a tag is renamed. But in the xmp file, instead of the tag having the new name, now there are two tags, one with the old name and one with the new name. Sometimes, unpredictably, an outdated tag IS removed from the xmp file. See http://ninedegreesbelow.com/temp/digikam-sidecar.html for an example. But not all of the outdated tags are removed, and I can't reproduce what happened when one of the outdated tags was removed. Expected Results: There should be a way to keep the xmp sidecar file in synch with the digiKam database. There should be a way to write to the sidecar file that deletes out-of-date tags, renames tags when they are renamed in the database, etc. At present, digiKam only adds new tags to xmp sidecar files, but never deletes old tags. Reading from and writing to the sidecar files doesn't behave like reading from and writing to the image itself, but it should, or else the sidecar files can't be used as a way to avoid writing to the image file. Sometimes, unpredictably, an outdated tag IS removed from the xmp file. See http://ninedegreesbelow.com/temp/digikam-sidecar.html for an example. I'm running OpenSuse 12.2, kde 4.9.2 release 511. I left the severity at "normal" but basically this bug is a show-stopper for me. I'm trying out the possibility of creating "image sidecars" for all my jpegs, so digiKam can write to "an" image but not to the original image. But that means doubling the number of images in the database and the whole process is turning out to be time-consuming and awkward.
The underlying problem of missing complete sync is described in 268068. My question here is: what is the difference between writing to a sidecar and writing to a file? There should be none.
The difference between writing to an XMP sidecar file and writing to an image file is: *It really is possible to resynchronize the image metadata with the database if you set digiKam up to write to the image file. "How" is not intuitive, but it can be done. *I haven't found any way to synchronize the image metadata with the databse if I set digiKam up to write to an XMP sidecar file instead of to the image file. See http://ninedegreesbelow.com/photography/digikam-how-to-synchronize-images-database.html for step-by-step instructons on how to synchronize the image metadata with the database after rearranging the tag tree. See http://ninedegreesbelow.com/temp/digikam-sidecar.html for my failed attempts to get the XMP sidecar files to synchronize with the database. On 10/26/12, Marcel Wiesweg <marcel.wiesweg@gmx.de> wrote: > https://bugs.kde.org/show_bug.cgi?id=309058 > > --- Comment #1 from Marcel Wiesweg <marcel.wiesweg@gmx.de> --- > The underlying problem of missing complete sync is described in 268068. My > question here is: what is the difference between writing to a sidecar and > writing to a file? There should be none. > > -- > You are receiving this mail because: > You reported the bug. >
I am aware that editing the tags tree has no effect on any pictures which carry this tag in their metadata. To remove a tag from the metadata of the currently selected pictures, uncheck a checked tag in the right side bar metadata panel and click Apply. Let's keep out of the current discussion the case of editing the tag tree while a picture with this tag is currently selected for the captions/tags sidebar. Is there any difference in this step between writing to a the image file or writing to the sidecar?
If all you do is select an image, uncheck a checked tag in the right side bar "Captions/Tags" panel, and then click Apply, writing to the sidecar and writing to the image behave exactly the same. The unchecked tag is removed from the sidecar and removed from the image metadata. I set up two test databases, one for writing to the sidecar and one for writing to the image file. Both behave exactly the same. Sorry to be so obtuse, but: What does "uncheck a checked tag in the right side bar metadata panel" mean? As far as I can tell, the panel labelled "metadata" is read-only. What does "Let's keep out of the current discussion the case of editing the tag tree while a picture with this tag is currently selected for the captions/tags sidebar" mean? The only way I know of to assign or unassign a tag to an image is to have the captions/tags right sidebar panel open and then to uncheck the appropriate tag in the tags list. What does "editing the tags tree has no effect on any pictures which carry this tag in their metadata" mean? Do you mean that rearranging tags on the tag tree doesn't affect tags in the image metadata? Because if you do it the way I posted to my website (first select all the tags in the left side tags filter - if somewhere I said select all the tags in the captions/tags panel, then I miswrote - sorry), yes, it does affect tags already in the image metadata. But NOT in the xmp sidecar file. That's the (at least a) difference between writing to the xmp sidecar file and writing to the image file. On 10/27/12, Marcel Wiesweg <marcel.wiesweg@gmx.de> wrote: > https://bugs.kde.org/show_bug.cgi?id=309058 > > --- Comment #3 from Marcel Wiesweg <marcel.wiesweg@gmx.de> --- > I am aware that editing the tags tree has no effect on any pictures which > carry > this tag in their metadata. > To remove a tag from the metadata of the currently selected pictures, > uncheck a > checked tag in the right side bar metadata panel and click Apply. > Let's keep out of the current discussion the case of editing the tag tree > while > a picture with this tag is currently selected for the captions/tags > sidebar. > > Is there any difference in this step between writing to a the image file or > writing to the sidecar? > > -- > You are receiving this mail because: > You reported the bug. >
There is another difference that I see between writing to the XMP sidecar file, compared to writing to the image metadata. It has to do with the orientation flag. In the digiKam metadata settings, in both cases, with Settings/Metadata/Rotation set to "Rotate by only setting a flag" and with "Write flag to metadata if possible" checked, and with "Show images/thumbnails rotated according to orientation flag" also checked: When writing to the image, everything works as expected. If you rotate the image, the image metadata changes to reflect the rotation. Reread the image metadata or reread the "More/file metadata" (right panel), and in both cases, the image stays rotated as it was last set. When reading and writing to the sidecar file only, if you rotate the image, the xmp file orientation tag changes to reflect the rotation. But as soon as you reread the image metadata or reread the "More/file metadata" (right panel), in both cases the image goes back to the state it was in before it was rotated. The xmp file rotation tag also changes back to the state it was in before it was rotated. This is contrary to what I would expect, as I would expect writing to the XMP file to be equivalent to writing to the image file, so "write to the image" should really mean "write to the xmp file" and "read from the image" should really mean "read the xmp file". Is "More/read metadata from file to database" different from "Image/Reread metadata from image"? Is "More/write metadata to each file" different from "Image/Write metadata to image"? When digiKam is set to read and write only to the xmp sidecar file, are "More/read", "More/write", "Image/read", and "Image/write" all supposed to read and write to the XMP file? When digiKam is set to read and write only to the image file, are "More/read", "More/write", "Image/read", and "Image/write" all supposed to read and write to the image file? There are two options under "More" in the right panel. When hovering over the top option, an oval is drawn around "read metadata from file to database", which gives feedback that something might happen. There is no oval when hovering over the second option, "write metadata to each file", and I don't know if that option actually does anything or not.
The "More->" actions essentially do the same but are old code, not multithreaded, I should remove them or rather replace them with the "Image->" actions. Everything honors the settings regarding XMP sidecar reading or writing. Your observation regarding the orientation is true: Here the Exif value is given priority over the corresponding XMP value. This is a bug for your scenario. In the old days, we had IPTC and Exif; then came XMP, and the question was how to detect conflicts if only IPTC or Exif was edited by an older program and the change not updated into XMP. With XMP sidecar files, relevant Exif and IPTC fields are copied into corresponding XMP fields in the sidecar file. This is a quite different situation. I tend to define that if a sidecar exists, sidecar information always takes precedence over file information, not only for XMP, but also for Exif and IPTC. This is done by simple merging. The situation is unclear when a tag does not exist in the sidecar, but in the file. Assuming we have "write only to sidecars", removing properties will only happen in the sidecar. If all no keywords or rating are found the sidecar but in the file, this can be because a) These fields are stored in the file and were not copied to the sidecar. They are valid. b) These fields were removed, changes applied to the sidecar, but not to the file. They are invalid. Ideas for a clean solution?
On 10/28/12, Marcel Wiesweg <marcel.wiesweg@gmx.de> wrote: > https://bugs.kde.org/show_bug.cgi?id=309058 > > --- Comment #6 from Marcel Wiesweg <marcel.wiesweg@gmx.de> --- > The "More->" actions essentially do the same but are old code, not > multithreaded, I should remove them or rather replace them with the > "Image->" > actions. Ah. Replacing them with "Image->" would be nice, as the placement is very convenient. > Everything honors the settings regarding XMP sidecar reading or writing. Except that "write only" isn't paired with "read only". Once the sidecar has been created, digiKam keeps also reading the image metadata. > Your observation regarding the orientation is true: Here the Exif value is > given priority over the corresponding XMP value. This is a bug for your > scenario. > In the old days, we had IPTC and Exif; then came XMP, and the question was > how > to detect conflicts if only IPTC or Exif was edited by an older program and > the > change not updated into XMP. > With XMP sidecar files, relevant Exif and IPTC fields are copied into > corresponding XMP fields in the sidecar file. This is a quite different > situation. I tend to define that if a sidecar exists, sidecar information > always takes precedence over file information, not only for XMP, but also > for > Exif and IPTC. This is done by simple merging. This implies that if I tell digiKam to read from and write to the XMP sidecar file, digiKam writes ONLY to the XMP but reads from BOTH the image file and the XMP file. Which means any time a user enters XMP data in the sidecar file, if there is a conflict between an XMP tag and other tags, the XMP tag gets overidden by non-XMP data in the image file. Would it be possible to let XMP always take priority over IPTC and Exif? Obviously digiKam has to read the image file in order to create the XMP sidecar in the first place, assuming no other application has already created an XMP file. My assumption was that once the XMP file is created, digiKam would read ONLY the XMP file. I think my mistaken assumption makes more sense than what actually happens. For example: With regards to orientation, there are actually three (hopefully only three) places in the metadata where the image orientation can be store - two exif locations: IFD0 and IFD1, and one xmp location: XMP-tiff. Only XMP-tiff is copied to the XMP file. A test: exiftool -a -s -G1 -orientation 070512_102713.*======== 070512_102713.jpg [IFD0] Orientation : Rotate 90 CW [IFD1] Orientation : Rotate 270 CW [XMP-tiff] Orientation : Horizontal (normal) ======== 070512_102713.jpg.xmp [XMP-tiff] Orientation : Rotate 90 CW Upon reading the Image metadata (which again, I assumed would only read the XMP sidecar file, but I was completely wrong), digiKam chooses IFD1 over IFD0, and chooses IFD0 over XMP-tiff. Then it rewrites the XMP-tiff tag in the XMP sidecar file to match the IFD1 (or IDF0, if the IDF1 tag is missing, or presumably vice versa) tag in the image. Another test: exiftool -a -s -G1 -orientation 070512_102713.*======== 070512_102713.jpg [XMP-tiff] Orientation : Horizontal (normal) ======== 070512_102713.jpg.xmp [XMP-tiff] Orientation : Rotate 90 CW This time digiKam respects the XMP sidecar tag over the tag in the image, which is how I would expect it to act. On the other hand, if I set digiKam up to read and write to the *image* metadata, instead of to the XMP sidecar file, then before changing the orientation in the database, digiKam uses the IFD0 tag over the XMP-tiff tag: exiftool -a -s -G1 -orientation 070512_102713.jpg [IFD0] Orientation : Rotate 90 CW [XMP-tiff] Orientation : Horizontal (normal) After changing the orientation flag in digiKam, BOTH tags are rewritten in the image metadata: exiftool -a -s -G1 -orientation 070512_102713.jpg [IFD0] Orientation : Rotate 270 CW [XMP-tiff] Orientation : Rotate 270 CW But when writing only to the XMP file, digiKam can't rewrite the non-XMP tag. Regardless of how one dices up the metadata pie between IPTC, XMP, and EXIF, it seems to me that if the user says "read (only, except that's not presently an option) and write only to the XMP sidecar, that the orientation tag (and any other tag) in the XMP sidecar is the tag that ought to be respected. Let's say I tag 1000 images, and 200 of them needed to have the orientation tag changed in order to display properly. If digiKam resets the XMP-tiff tag in the sidecar every time the image metadata is read, that's a lot of work to have to redo over and over again. > The situation is unclear when a tag does not exist in the sidecar, but in > the > file. >Assuming we have "write only to sidecars", removing properties will > only > happen in the sidecar. If all no keywords or rating are found the sidecar > but > in the file, this can be because > a) These fields are stored in the file and were not copied to the sidecar. > They > are valid. On the one hand, if digiKam created the sidecar, presumably digiKam copies all the relevant information that can be copied to XMP. In the case of IPTC or Exif storing contradictory information, why not let XMP take priority? XMP is very well established at this point. > b) These fields were removed, changes applied to the sidecar, but not to > the > file. They are invalid. > Ideas for a clean solution? The difference in "what happens", between writing to the image and writing to the XMP sidecar, at least in part and maybe entirely seems to be that by definition writing to the sidecar only means the image metadata is untouched. But digiKam keeps reading non-XMP information, and then rewrites it as the corresponding XMP tag, which gets rewritten again the next time the image metadata is read. The cleanest solution would be to always give precedence to what is in the XMP file and assume that the user will eventually reconcile the differences. Otherwise the database and the XMP file can't be kept synchronized. Presumably any user who says "don't write to the image file" realizes that the XMP and the database will soon be out of synch with one another, and either has plans to deal with that situation later, or really doesn't care what is in the image metadata.
>Presumably any user who says "don't write to the image >file" realizes that the XMP and the database will soon be out of synch >with one another, and either has plans to deal with that situation >later, or really doesn't care what is in the image metadata. Sorry, I meant to say "realizes that the image file metadata and the database will soon be out of synch," not "the XMP [file] and the database", as the goal is to keep the XMP file and the database in synch with one another.
There's a problem in digikam that Exif takes precedence over XMP regarding the orientation. For all other fields, XMP takes precedence. In Exif more than in IPTC, there are a lot of field with typically read-only information, including makernotes, which cannot be copied to the sidecar file. Only a defined subset of Exif and IPTC fields is copied to the sidecar. That means we cannot just completely ignore the main file. I agree to say that if there is a sidecar, the file's XMP should be ignored. If there are keywords, rating etc. defined in XMP, these entries always take precedence over entries in Exif and IPTC. The problem comes when e.g. all keywords are removed from the image and thus the sidecar, but IPTC in the file remains unedited. When not finding an entry in XMP, there will be a fallback to IPTC. During the step of reading file and sidecar metadata, we thus need to cancel from the read Exif and IPTC those fields which would be copied to the sidecar. This is possible. I'm not quite sure if there are implications and cases where this construct brings other problems.
> --- Comment #9 from Marcel Wiesweg <marcel.wiesweg@gmx.de> --- > There's a problem in digikam that Exif takes precedence over XMP regarding > the > orientation. For all other fields, XMP takes precedence. Is there a "use case" reason why exif takes precedence over XMP for orientation? Tangentially related - digiKam writes the tiff:orientation tag to the xmp sidecar for jpeg image files. But it won't write the tiff:orientation tag to the xmp sidecar for Canon cr2 raw files. Is this a feature or a known bug? It throws a wrench in my on-going effort to work around digiKam as it now handles XMP sidecars. > If there are keywords, rating etc. defined in XMP, these entries always > take > precedence over entries in Exif and IPTC. The problem comes when e.g. all > keywords are removed from the image and thus the sidecar, but IPTC in the > file > remains unedited. When not finding an entry in XMP, there will be a fallback > to > IPTC. During the step of reading file and sidecar metadata, we thus need to > cancel from the read Exif and IPTC those fields which would be copied to > the > sidecar. > This is possible. I'm not quite sure if there are implications and cases > where > this construct brings other problems. When writing to the image file (instead of the sidecar), doesn't digiKam write dc:subject to dc:subject and also to iptc:keywords (and possibly other places)? And also for rating-related tags, etc? Just as it writes the orientation information to exif as well as to xmp-tiff:orientaion? If so, then telling digiKam to ignore iptc/exif **in the specific cases where writing to the image would change the iptc/exif equivalents as well as the xmp tags** would seem to just make writing to a sidecar equivalent to writing to the image. I would like to see digiKam ignore the iptc/exif as above when the xmp is different in the sidecar, but I wouldn't want to see a change in the current digiKam behavior mess up someone else's workflow. Perhaps a question should be posted on the user forum to ask for input from other people who use sidecars?
Orientation: I suspect this is done to keep compatibility with old tool adjusting the Exif flag (a very common operation) and ignoring the XMP value. I'm trying to set up a process which does the integration of the sidecar information as you describe it, based on the specification available for mapping XMP, Exif and IPTC (there is plenty of such specs and guidance docs). In the end, there is a clearly defined and limited set of affected metadata fields. Unfortunately, the sidecar file itself is very much underdocumented regarding implementation guidelines, the MWG guidance doc is completely ignoring the case we have here. So we are on our own.
If there is any testing/input/etc that you would like from me, let me know. I'm reasonably familiar with the exif/iptc/xmp fields and which fields can/should map to one another.
Git commit bdd6666e438ee239def7c50ddba8e93fa6975002 by Marcel Wiesweg. Committed on 01/03/2013 at 22:43. Pushed by mwiesweg into branch 'master'. Improve merging of sidecar with image metadata. If there is a sidecar, image metadata may remain, but can be outdated if image shall not or cannot be edited. Even more, a field can be removed from a sidecar as part of an explicit editing operation, in which case it may remain in the image metadata, but needs to be ignored. For those fields were a mapping between EXIF, IPTC to XMP is cleanly defined, implement merging which takes into account the points listed above. M +2 -6 libkexiv2/kexiv2.cpp M +88 -15 libkexiv2/kexiv2_p.cpp M +118 -5 libkexiv2/kexiv2_p.h M +18 -18 libkexiv2/kexiv2image.cpp http://commits.kde.org/libkexiv2/bdd6666e438ee239def7c50ddba8e93fa6975002
This needs testing with some more real-life use cases.
This patch is not sufficient. I have been using the patch for a month or so, as I am using my newly written kipi plugin for photivo. The plugin uses sidecar data when running photivo on a file. When I rename a tag, the old tag stays in the sidecar file, along with the new tag. When I correct a misspelling, I get the correct spelling as well as the incorrect spelling. My workaround is: rename the tag. open all images with the new tag within digikam reread metadata from image verify that old tag still exists on the images go to the Caption/Tags sidebar, remove the old tag from all images, Apply delete the old tag Then repeat above steps, just in case. I think that the above steps are sufficient, but sometimes I do not catch all of the bad tags on the first round. I h
Here are my steps to reproduce a bug. Under Configure=>Metadata, choose: Read from sidecar files Write to sidecar files Choose an raw image. Set the tag "Sidecar" for this image. Under Caption/Tags, For the tag "Sidecar" Go to My Tags=>Properties Change the title of this tag to "Sidecar-new" The tag appears to change. Examine the sidecar file, in my case ~/Nikon/DCIM/106ND700/DSC_3148.NEF.xmp Note the Sidecar is listed, rather than Sidecar-new <digiKam:TagsList> <rdf:Seq> <rdf:li>Sidecar</rdf:li> </rdf:Seq> </digiKam:TagsList> <MicrosoftPhoto:LastKeywordXMP> <rdf:Seq> <rdf:li>Sidecar</rdf:li> </rdf:Seq> </MicrosoftPhoto:LastKeywordXMP> <lr:hierarchicalSubject> <rdf:Bag> <rdf:li>Sidecar</rdf:li> </rdf:Bag> <dc:subject> <rdf:Bag> <rdf:li>Sidecar</rdf:li> </rdf:Bag> </dc:subject> Now, go to Image=>Reread Metadata From Image Both tags now appear in digikam, "Sidecar" and "Sidecar-new" The sidecar file still only has "Sidecar". Click on Image=>Write Metadata to Image Now the sidecar file contains both tags. However, if you reverse the order of the steps, Click on Image=>Write Metadata to Image Image=>Reread Metadata From Image then everything is OK. After clicking on Image=>Write Metadata to Image the sidecar file looks fine. This suggests a solution to the problem, which I will examine later.
Git commit 8fe3b75c6cb7dd02fe37a7f2acaa26f88523ac46 by Kevin Dalley. Committed on 03/12/2013 at 08:21. Pushed by kdalley into branch 'bug309058'. When tags are modified, synchronize tag changes by writing to files, using WriteFromDatabaseToFile M +20 -0 digikam/tags/tagmodificationhelper.cpp M +27 -0 utilities/maintenance/metadatasynchronizer.cpp M +8 -0 utilities/maintenance/metadatasynchronizer.h http://commits.kde.org/digikam/8fe3b75c6cb7dd02fe37a7f2acaa26f88523ac46
This patch fixes 2 problems related to this bug. However, as I was writing this report, I found a 3rd part of this bug which is not yet fixed. I'll look at this bug in the next few days. For now, I will leave this bug open. Here are the tests which I ran: I choose 2 files, a jpeg file, and a NEF file with an xmp sidecar. I assigned the tag "Sidecar" to the 2 images. Test 1, rename a tag, passes: Under Caption/Tags, For the tag "Sidecar" Go to My Tags=>Properties Change the title of this tag to "Sidecar-new" examine both the jpeg and the xmp file exiftool -a file |grep Sidecar Verify that the tag "Sidecar-new" exists in both files, and that the tag "Sidecar" does not exist. Now select both images, go to Image=>Reread Metadata From Image Only the tag "Sidecar-new" appears. Test 2, delete a tag, passes: Now delete the tag "Sidecar-2". examine both the jpeg and the xmp file exiftool -a file |grep Sidecar Verify that the tag "Sidecar-new"does not exist in either files. Now select both images, go to Image=>Reread Metadata From Image The tag "Sidecar-new" does not appear. Test 3, move a tag, fails: After recreating the tag "Sidecar", move it to "Top/Sidecar" Examine the jpeg and the xmp sidecar exiftool -a file |grep Sidecar Note that "Top/Sidecar" does not appear, though "Sidecar" does appear. go to Image=>Reread Metadata From Image This does not change "Top/Sidecar". "Sidecar" does not reappear. That seems odd. I am fairly positive that this is a bug, but I am willing to be corrected.
I pushed another patch into 'bug309058', which solves the problem of moving a tag within the tree. When this patch is placed into master, then the bug can be closed. When tags are moved to a different position in the tree, synchronize tag changes by writing to files, using WriteFromDatabaseToFile I repeated Test 3 from above. Test 3, move a tag, passes: After recreating the tag "Sidecar", move it to "Testing/Sidecar" Examine the jpeg and the xmp sidecar exiftool -a file |grep Sidecar Note that "Testing/Sidecar" does appear now. Note that Sidecar-new shows up under Subject. This seems to be the result of early testing. The following does not fix the problem. I consider this either a different bug, or possibly a feature, where Subject does not get updated to match the tags. Click on Image=>Write Metadata to Image kevin@nereocystis:~/src/digikam-git/digikam$ exiftool -a /home/kevin/Nikon/DCIM/106ND700/DSC_3148.NEF.xmp|grep Sidecar Tags List : Testing/Sidecar, Testing Last Keyword XMP : Testing/Sidecar, Testing Hierarchical Subject : Testing|Sidecar, Testing Subject : Sidecar, Testing kevin@nereocystis:~/src/digikam-git/digikam$ exiftool -a /home/kevin/Nikon/DCIM/Android/newstuff/2013-11-03/IMG_20131103_102820.jpg |grep Sidecar Tags List : Testing/Sidecar, Testing Last Keyword XMP : Testing/Sidecar, Testing Hierarchical Subject : Testing|Sidecar, Testing Subject : Sidecar, Testing Keywords : Sidecar-new, Sidecar, Testing kevin@nereocystis:~/src/digikam-git/digikam$ exiftool -a /home/kevin/Nikon/DCIM/Android/newstuff/2013-11-03/IMG_20131103_102820.jpg.xmp |grep Sidecar Tags List : Testing/Sidecar, Testing Last Keyword XMP : Testing/Sidecar, Testing Hierarchical Subject : Testing|Sidecar, Testing Subject : Sidecar-new, Sidecar, Testing
Hi Kevin, What's happen if a lots of images relevant of tags hierarchy changes needs to be updated ? This can take a while to process. To update files as well can decrease digiKAm performance a lots until files update is done. Do you tried in production ? I know that Aperture or Lightroom put all changes in a list to play when users can do it (for ex when he don't use application in workflow). This is the goal of Maintenance tool in digiKam. End users run it at the most adapted moment... Best Gilles Caulier
That's a good question. I have tried changing the name of a tag which was used by about 100 files. It does take a little while at that level. Of course, a user could modify the tags for every file. That could be slow. Here are the options I see: 1. Use the code changes I suggested. Advantages: Much lower risk that database and files are out of sync. Disadvantages: Could significantly slow digikam. Repeated operations are even worse. 2. Use Maintenance Tool. Perhaps user could be warned when operation is performed. Advantages: Users control when files are updated. Disadvantages: User has to remember to run Maintenance. User has to know what to run, and which direction. A very big disadvantage, I think. If database is read from out of date files, old tags return, creating a lot of work for the user. 3. Use digikam Preferences to determine behavior from 1 and 2. 4. Allow users to specify behavior when operations are performed. 5. Modify MetadataSynchronizer so that synchronization is slower and less painful for the user. This is easy to say, but more challenging to implement. 6. A variation on 5. Keep a list of images which are out of sync, stored in the database. Occasionally process a few. This method avoids having the same image listed twice. Occasionally process some files. I somewhat like 6, but I am not sure. Any of these methods allow for the possibility of unsynchronized data. Completely removing this risk requires a lot of work. On 12/09/2013 02:49 PM, Gilles Caulier wrote: > https://bugs.kde.org/show_bug.cgi?id=309058 > > --- Comment #20 from Gilles Caulier <caulier.gilles@gmail.com> --- > Hi Kevin, > > What's happen if a lots of images relevant of tags hierarchy changes > needs to be updated ? This can take a while to process. To update > files as well can decrease digiKAm performance a lots until files > update is done. Do you tried in production ? > > I know that Aperture or Lightroom put all changes in a list to play > when users can do it (for ex when he don't use application in > workflow). This is the goal of Maintenance tool in digiKam. End users > run it at the most adapted moment... > > Best > > Gilles Caulier >
I performed some additional tests, changing the tag name for a tag which had over 1000 images. Then I changed it back again. During the change back, I hit a deadlock, at least that is my best guess. I'm investigating it now, attaching gdb to the running process. Digikam hung up. I did try performing other operations while the writing was finishing. I suspect that writing that many images tickled a bug which had existed before. In addition to the deadlock, the intial change was slow, perhaps 5 minutes. Performing other operations was slow, but they did complete. On 12/09/2013 02:49 PM, Gilles Caulier wrote: > https://bugs.kde.org/show_bug.cgi?id=309058 > > --- Comment #20 from Gilles Caulier <caulier.gilles@gmail.com> --- > Hi Kevin, > > What's happen if a lots of images relevant of tags hierarchy changes > needs to be updated ? This can take a while to process. To update > files as well can decrease digiKAm performance a lots until files > update is done. Do you tried in production ? > > I know that Aperture or Lightroom put all changes in a list to play > when users can do it (for ex when he don't use application in > workflow). This is the goal of Maintenance tool in digiKam. End users > run it at the most adapted moment... > > Best > > Gilles Caulier >
"thr appl all bt" gives you backtraces of all threads. If there's a deadlock, it can often be identified with this snapshot.
Elle, This file still valid using last digiKam 4.2.0 ? Gilles Caulier
Gilles, Thanks for checking. I'll see whether I can test this. I am still actively using digikam, but I'm not carefully tracking my bugs. It will probably take a few days to a week before I have an answer.
This file still valid using last digiKam 4.10.0 ? Gilles Caulier
digiKam 4.11.0 is out. Still reproducible ? Thanks in advance Gilles Caulier
Did my changes ever get merged into the main code base?
yes, this must be. Gilles Caulier
I set up a test folder with 3 images, and a test digiKam database. I told digiKam to only write to the XMP files and only read from the XMP files. I don't see any XMP files being created. So I am unable to test to see whether the original bug has been fixed. Maybe I'm doing something wrong. How does one create the XMP side car files?
Change/update something metadata to create XMP sidecar. Ex : open file in editor, rotate image and save it. Gilles Caulier
I already made a whole lot of changes, rearranging the tag tree and renaming some tags. No XMP files were created. I tried reading and writing the metadata. The old metadata got pulled in from the images to the database, which is not desirable behavior because now the database the "prechange" information and also the "postchange" information. But no XMP sidecars were created. Per your suggestion, I rotated an image, which created a sidecar called "JpegRotator-M20394.digikamtempfile.jpg.XML". That XML file doesn't contain any informationl except the line "<?xml version="1.0" encoding="UTF-8"?>".
Go to Help/Components Info dialog and look in XMP is supported by Exiv2... Gilles Caulier
digiKam version 4.10.0 Exiv2 supports XMP metadata: Yes
To rotate image use image editor as well. Run digiKam form a console while this operation. Did you seen some debug traces ? Gilles Caulier
After rotating an image with showFoto, nothing is written to the terminal. While asking digiKam to write to the image (which should mean write to the XMPfile) and then read from the image (which should mean read from the XMP file), the following (doesn't seem relevant) is printed to screen: digikam(20825)/digikam (core) Digikam::DNotificationWrapper: parent is null Thread::requestAbort: not running. Thread::requestAbort: not running. digikam(20825)/KEXIV2: Cannot find Xmp key 'Xmp.acdsee.caption' into image using Exiv2 (Error # 35 : No namespace info available for XMP prefix `acdsee' digikam(20825)/KEXIV2: Cannot find Xmp key 'Xmp.acdsee.author' into image using Exiv2 (Error # 35 : No namespace info available for XMP prefix `acdsee' digikam(20825)/KEXIV2: Cannot find Xmp key 'Xmp.acdsee.notes' into image using Exiv2 (Error # 35 : No namespace info available for XMP prefix `acdsee' Thread::requestAbort: not running. Thread::requestAbort: not running.
No. Do not use Showfoto, but editor from digiKam as well. Turn on right Metdata settings before. The "acdsee" namespace error from Exiv2 must not be a problem. It's appear because you use 0.24 release, not the last 0.25 Gilles Caulier
Ah, my apologies. I didn't actually have digikam set to write any information at all. Now one jpeg has an XMPfile. I don't know what it says. There are too many things going on at the moment to continue trying to see if this bug has been fixed. I'll try to check again tomorrow or maybe later today. Sorry!
I will take a look at the current digikam code, as well as my patches. Unfortunately, this may take a few days.
digiKam 4.12.0 is out : https://www.digikam.org/node/741 We need a fresh feedback using this release please... Thanks in advance.
I updated digikam to 4.12.0, made a test database, and started adding and rearranging tags. Adding, deleting, and rearranging tags and tag trees, and then using Tools/Maintenance/Sync Metadata, choosing "Database/From database to image metadata" seems to work just fine, but I'm planning to do more testing just to see if things get out of step. I haven't finished checking Image/Write and Image/Read metadata and I haven't checked changing the orientation. Are there any specific checks I should try? What should happen when syncing an image or the entire database from the image to the database? What about if I edit a tag directly in the xml file and then sync from image metadata to database? I'm assuming that when digiKam is set up to read and write metadata only from the sidecar files, that reading and writing from the "image" really means reading and writing from the xml file, if there is one, and otherwise read from the image and create the xml file?
Veaceslav, As you work currently on metadathub re-write, you will certainly give best explanations about Elle questions from comment #41. Gilles
(In reply to Elle Stone from comment #41) > I updated digikam to 4.12.0, made a test database, and started adding and > rearranging tags. > > Adding, deleting, and rearranging tags and tag trees, and then using > Tools/Maintenance/Sync Metadata, choosing "Database/From database to image > metadata" seems to work just fine, but I'm planning to do more testing just > to see if things get out of step. > > I haven't finished checking Image/Write and Image/Read metadata and I > haven't checked changing the orientation. Are there any specific checks I > should try? Everything that works with normal image metadata should work with sidecars. I'm not a fan of them, tried them only once. > What should happen when syncing an image or the entire database from the > image to the database? What about if I edit a tag directly in the xml file > and then sync from image metadata to database? When You sync FROM image TO Database, all metadata from image is written to database. > I'm assuming that when digiKam is set up to read and write metadata only > from the sidecar files, that reading and writing from the "image" really > means reading and writing from the xml file, if there is one, and otherwise > read from the image and create the xml file? Look in the settings menu: When you activate read/write to sidecars, you have a combo box to choose between: 1. Use sidecar for read-only images 2. Use only sidecar 3. Use both image metadata and sidecar Please notice, writing to sidecars is not managed by digiKam. Libkexiv2/exiv2 is the one who does this. digiKam only feed the options, as far as I saw when working with the whole metadata subsystem.
Here are my digiKam settings: Under Reading and Writing Metadata: "Read from sidecar files" is checked. "Write to sidecar files" is checked. "Write to XMP sidecar only" is also checked. "If possible write Metadata to RAW files (experimental)" is NOT checked. Under Rotation: "Rotate by only setting a flag" is selected. "Write flag to metadata if possible" is also checked. With the above metadata settings, testing shows that: * Syncing from the database to the image writes new/modified metadata to the XMP file, and completely replaces with old tag trees with new tag trees. * Rotating an image sets a flag in the database, and syncing from the database to the image writes the flag to the XMP file. * The XMP tag trees and rotation flags survive deleting the database and making a new one. Regarding syncing: * Syncing from database to image: I closed digiKam and modified an XMP file with a text editor, changing one item in a "tag branch". Then I opened digiKam and synced from the database to the "image" (actually XMP file). The externally modified tag branches were rewritten by the digiKam database tag branches. This is exactly what I want to happen. I don't intentionally use other software to add or modify XMP tags, but sometimes other software tries to write tags anyway. * Synching from image to database: I closed digiKam and modified an XMP file using a text editor, again changing one item in a tag "branch". Then I opened digiKam and synced from the "image" to the database. digiKam picked up the new "tag branch", but the original "tag branch" was still there. This is exactly what I want to happen: If I ever accidentally sync from the image to the database, I don't want syncing to remove or modify tags that I've added using digiKam. I didn't test syncing from image to database very carefully, and I'm not sure how this behavior would cohere with workflows that depend on digiKam reading in data written by other software. All the above seems to work as I'd expect it to, given the metadata settings. The original problems seem to be fixed. I've only tested with a very small database of images. Hopefully results will be the same on the real database. But I think this bug can be closed. If working with the real database reveals remaining or other issues, maybe open a new bug report? (In reply to Veaceslav Munteanu from comment #43) > Everything that works with normal image metadata should work with sidecars. > I'm not a fan of them, tried them only once. Many people never write metadata to their original image files, because there's always an element of risk.
>If working with the real database reveals remaining or other issues, maybe open a new bug report? Or re-open this one... Thanks for your feedback Gilles Caulier