Bug 278935

What happens if there is file1.jpg, file1.tiff and file1.png?

I see how fully supporting this naming scheme for sidecar files could be problematic, but from what I see on the web, Adobe itself also uses $BASENAME.xmp. Thus at least partial support for those files in Digikam is needed (e.g. only reading).

I currently use JPhotoTagger (http://jphototagger.org/) on win-pc and APhotoManger (https://github.com/k3b/APhotoManager/) on android to manage my 12000 photos. 

JPhotoTagger has hard coded "write changes only to sidecar file" and currently both use the naming convention $BASENAME.xmp so i cannot import the meta data to digiKam without renaming 12000 xmp sidecar files.

I also agree that $BASENAME.$EXT.xmp makes more sense but adding this option would increase interoperability and allows me to use both JPhotoTagger and digiKam at the same time

I have just added this issue to APhotoManager https://github.com/k3b/APhotoManager/issues/84

BTW. Current Geeqie GIT version supports both sidecar naming schemes.

I see why this could be desirable for interoperability (I see no other benefit?) However I don't see how to do this conceptually. Digikam doesn't store images in a managed library, but reads it from disk. So how do you enforce unique base filenames (i.e. that image.jpg and image.png never exist alongside)?

Does anyone know how the mentioned projects (JPhotoTager, AphotoManager, Adobe) handle this?

I can comment about Geeqie only. Even before this issue was resolved Geeqie grouped files with the same basenames together (e.g. img.cr2 and img.jpg). By default it included img.xmp in the same group and used it to read/write tags for other image files in this group.

Now, the recent patch makes Geeqie add img.cr2.xmp to the group if img.cr2 is a member. It also offers an option to write metadata to img.cr2.xmp files rather than img.xmp files. The issue was discussed here: https://github.com/BestImageViewer/geeqie/issues/147

I agree that it can have unintended consequences if two unrelated files share basename. That's why I believe that using img.xmp naming scheme should be optional. Moreover, since the longer naming scheme makes more sense, I don't think there is a need to make Digikam able to write those files. However if they do exist, they could be read (or even migrated, but that might create a mess).

I see, the "unique base filename" makes it much more complicated ;-(

how frequent are these naming conflicts so that code must take care of it?

On the android/APHotoManager side i will ignore the problem in the first run. 

Move/rename/delete of img.jpg will also 
handle img.xmp and img.jpg.xmp if they exist 
(no matter what xmp-naming-convention the seggings dialog has)

if there is imp.jpg and img.jpeg the first one beeing moved/deleted/renamed wins

the media scanner will assing the content of img.xmp to both: img.jpg and img.jpeg

xmp update policy: 

* if there is imp.jpg.xmp update this file
* else if there is img.xmp update this file
* else if there is no xmp file yet create it according to xmp-naming-convention settings

I see two possible ways to avoid conflicts...

1) Other programs add this line to the .xmp files photoshop:SidecarForExtension="cr2"
Can't this just be expanded to a list to reflect all extentions the sidecar is used for?
I'll admit I have not read through the xmp specifications so i apologize in case this is complete nonsense.

2) Since most of the time sidecar files are only needed for RAW files (jpg and tiff both support embedded xmp data), why not add two options of which only one can be turned on:

- write xmp sidecar data only for raw files and embed metadata in other image formats
- The way it is handled right now with .ext.xmp

I store raw and jpeg files in the same directory (same file name different extension). I geotag, tag and rate the photos storing the information in the sidecar. Raw and jpeg files share the same information. It would be very useful to share the same sidecar for both files.

Created attachment 109721 [details]
patch to change xmp filename to be compatible with LR

This simple patch is a proof of concept to be compatible with LR XMP sidecar file name. This is for testing LR import metadata.

Typically a sidecar will be : path + basename + .xmp

Applying this patch will break the previous behavior. All older sidecars generated with previous behavior will be ignored.

The patch must be improved to add option in GUI as :

- an option to use older naming scheme or new naming scheme while import.
- an option to rename old sidecar files using old naming scheme exists to new scheme automatically.

With an automatic sidecar file renaming from old scheme to new scheme, we have a problem. Typical case in a common album : 

- foo.jpg
- foo.tif
- foo.nef
- foo.jpg.xmp
- foo.tif.xmp
- foo.nef.xmp

=> how to generate foo.xmp ? which file we must take to generate the single xmp file ? Merging is not possible.

Gilles Caulier

I would condsider this to be an ok solution:

Read the metadata fields from all three

1. Add all metadata that is the same
2. If 2 xmp files have different data for a field then take the data from the last modified one

A confirm dialogue and backup of the original xmps if there was a conflict

There is now a patch from Jakob Malm, a digiKam user, in Phabricator at this url:

https://phabricator.kde.org/D9648

This must be review for next 5.9.0.

Gilles Caulier

As windows user for ages I wanted to give dk another try and found that it would just read meta data from my tiffs but none from my raw files. What a mess!

Browsing the web I found a few discussions on this problem and finally arrived here.

I always felt dropping the extension and adding ".ext" to a file name was a bad decision by adobe. I sometimes stumbled in this trap too with my ".cr2" and ".nef" files using the same base name. But it is widely accepted since many years and I have more than 350,000 side car files following this rule.

I also love the idea to allow side cars for all file types, even ".jpg", ".tiff" or whatever. Using side cars is much smarter than writing meta data into the image files. Makes backup much smaller and faster :-)

So here's my suggestion to allow all thinkable naming schemes and leave the decision to the users. There already exists a dialogue to add other extensions like ".pp3". But this only can give "{basename}{ext}.pp3".

Why not drill this procedure and allow to type full templates like
"{basename}{ext}.ext" (used by dk)
"{basename}.ext" (the standard from adobe and widely used outside the linux world)
"{basename}{ext}.pp3" (the sample in the dialogue in dk)
or even 
"{subfolder}/{basename}{ext}.ext" (for people who love to browse their file base and down't want to see the side cars)
and also allow to type any other combination the users might want.

Templates like
"tiff:{basename}{ext}.ext"
"jpg:{basename}{ext}.ext"
"nef:{basename}.ext"
"cr2:{basename}.ext"

would even allow to have different naming schemes for different file types.

Then make the top most template the default for writing if no other side car can be found and try the list top down to find a side car for reading.

Thanx for reading, knick

Created attachment 114950 [details]
sidecarForLR.patch

It is actually relatively simple to support LR users, if a sidecar exists in LR format, it will be used. If not, the digiKam format.

Maik

You're saying patching the source code and compiling the whole app with all it's dependencies relative easy?

No miracle why linux doesn't make it's way to desktop users.

knick,

It's not much difficult, but if you never experienced that, We will provide an AppImage linux bundle for you.

Maik,

As your code is really simple, there is no risk to introduce side effects. I recommend to patch git/master as well. While this week end i will rebuild all bundles.


Gilles

Ok, Gilles You were faster. I also wanted to write that it only served to get feedback and he did not have to compile anything by himself.

https://files.kde.org/digikam/

Maik

Git commit 8f2d1d2df23df8540c47971d8f487122ce256ccd by Maik Qualmann.
Committed on 14/09/2018 at 17:08.
Pushed by mqualmann into branch 'master'.

use sidecar in LR format if available

M  +10   -0    core/libs/dmetadata/metaengine.cpp

https://commits.kde.org/digikam/8f2d1d2df23df8540c47971d8f487122ce256ccd

Hello,
Is it planned to publish the patch by Maik Qualmann? If I'm not wrong, digiKam 6.0 doesn't support $BASENAME.xmp files under windows (at least those created by Capture One, XnViewMP or ON1). digiKam seems very nice indeed but I cannot use it without this support.
Thanks,
Sebastien.

DigiKam definitely supports $BASENAME.xmp files in digiKam-6.0.0 if they are present they will be used. Otherwise please upload XMP files from Capture One, XnViewMP or ON1.

Maik

Re-openned

My mistake, it does indeed works, sorry for the wrong message.
Thank you for your reactivity!
Sebastien.

Could someone clarify Comment 22? What does it meant by 'used', does it simply mean read?

Perhaps I am missing it but I cannot find a way to write to a sidecar named $BASENAME.xmp, which is the use case that most people are concerned about when discussing interoperability (ie read _and_ write).

For better or worse many RAW processors use $BASENAME.xmp, including Capture One (which doesn't have DAM features!), and so writing to these files from digiKam is an essential aspect of interoperability.

If writing to $BASENAME.xmp is provided as an option it will greatly increase the appeal of digiKam to new users. Think of the glory, and the donations! I would happily make a donation if digiKam suited this use case (sadly, without that option this powerful application is useless to me).

Digikam reads and writes $BASENAME.xmp if the file already exists. If the file does not exist, digiKam creates $BASENAME.EXT.xmp and uses it.

Maik

Thanks for your quick response Maik.

So digiKam will default to reading and writing to $BASENAME.xmp if it finds one already existing. That's very helpful to know.

This still presents a problem for a typical use case, where the user imports and manages new images through digiKam but processes the RAW files in (say) Capture One. In this case digiKam will create $BASENAME.EXT.xmp files, which Capture One will not be aware of. A common workflow would be rating newly imported images in digiKam, and sorting by these in Capture One for editing (and even changing the ratings or some other metadata in the RAW processor. eg to indicate editing complete and ready for printing).

It remains the case that user control over xmp file naming would be very helpful and open digiKam to other users.

I've just realised this issue was raised against Linux. Is there any difference in behaviour regards xmp file naming, between Linux and Windows?

Git commit a5f0380220cbebb7247d535302670e6374179412 by Maik Qualmann.
Committed on 09/10/2019 at 21:34.
Pushed by mqualmann into branch 'master'.

add new Option to use compatible file name for sidecar files

M  +1    -0    core/libs/metadataengine/dmetadata/dmetadata.cpp
M  +10   -0    core/libs/metadataengine/engine/metaengine.cpp
M  +10   -2    core/libs/metadataengine/engine/metaengine.h
M  +2    -2    core/libs/metadataengine/engine/metaengine_fileio.cpp
M  +4    -1    core/libs/metadataengine/engine/metaengine_p.cpp
M  +1    -0    core/libs/metadataengine/engine/metaengine_p.h
M  +5    -0    core/libs/metadataengine/engine/metaenginesettingscontainer.cpp
M  +1    -0    core/libs/metadataengine/engine/metaenginesettingscontainer.h
M  +11   -0    core/utilities/setup/metadata/setupmetadata.cpp

https://invent.kde.org/kde/digikam/commit/a5f0380220cbebb7247d535302670e6374179412

! I look forward to using digiKam :)