Bug 469423

Summary: digiKam creates duplicate files when local directory is a onedrive folder
Product: [Applications] digikam Reporter: Jürgen <juergen>
Component: Setup-CollectionsAssignee: Digikam Developers <digikam-bugs-null>
Status: RESOLVED FIXED    
Severity: normal CC: caulier.gilles, josjonkeren, juergen, metzpinguin
Priority: NOR    
Version First Reported In: 8.5.0   
Target Milestone: ---   
Platform: Microsoft Windows   
OS: Microsoft Windows   
Latest Commit: Version Fixed/Implemented In: 8.3.0
Sentry Crash Report:

Description Jürgen 2023-05-06 17:28:17 UTC
images files stored in a local folder. The folder is a onedrive folder with all files "always keep local". 
When using the facial recognition and quickly updating many images file are duplicated. One version with the Original Name and another version with "-Computer name" appended to the file name 


STEPS TO REPRODUCE
1. Use image stored locally in a onedrive folder with files marked as "always keep local"
2. Edit many files via the facial recognition 


OBSERVED RESULT
1. Files are duplicated - with two different names. One with the original name. One with "-Computer-Name" appended to the file name 

The duplicate is actually created by onedrive but it's triggered by digikam. Digikam is triggering a download from ondrive of a file which already is stored locally. Onedrive detects that and creates the second version with the "-Computer name" added to the file name

EXPECTED RESULT
well, no duplication 

SOFTWARE/OS VERSIONS
Windows:  11
Comment 1 Maik Qualmann 2023-05-06 17:33:38 UTC
digiKam does not support OneDrive virtual folders. Note this information in the documentation.

https://docs.digikam.org/en/setup_application/collections_settings.html#the-network-shares-specificity

Maik
Comment 2 Jürgen 2023-05-07 06:35:12 UTC
adding more details on the configuration
- digikam is installed on a "physical windows 11" PC
- digikam database is set to SQLlite sitting on a local folder which is NOT syncronized via onedrive. 
- there is no second installation of digikam accessing the same image files
- the image files are stored on the same PC on a local drive (no sharepoint, NSA, NFS, ...)
- Onedrive is installed on the PC as well. The only purpose is to have a remote copy of the files (including but not limited to  to images)

I'm don't have sufficient detailed knowledge on the technology but I would think the digikam does not see the file/folders are sycronized using onedrive to a network location.
Comment 3 Maik Qualmann 2023-05-07 06:51:57 UTC
A virtual OneDrive folder must not be part of a digiKam collection. We have no way of recognizing a virtual folder with the Qt-API we use. digiKam does not create new files with the computer name in the filename, this behavior must come from the virtual folder itself.

Maik
Comment 4 Jürgen 2023-05-10 16:17:16 UTC
(In reply to Maik Qualmann from comment #3)
> A virtual OneDrive folder must not be part of a digiKam collection. We have
> no way of recognizing a virtual folder with the Qt-API we use. digiKam does
> not create new files with the computer name in the filename, this behavior
> must come from the virtual folder itself.
> 
> Maik

agreed. It's not digikam that creates it. It is onedrive https://support.microsoft.com/en-us/topic/duplicate-files-in-onedrive-fd47ce5e-8dd0-465e-9e3a-461e1a3cf613 but it happens ONLY when I use digikam and modify the images. The solution in the link from MS does not work. THere is no credentials issue. It seems digikam is accessing the same file with different "users"?? or with concurrent access?? 

Any idea of a solution and/or workaround.

I was looking for some image admin software and digikam is just great but this duplication issue is killing it :-(
Comment 5 caulier.gilles 2023-05-20 06:29:06 UTC
>It seems digikam is accessing the same file with different "users"?? 

No, definitively.

But look if no zombie ExifTool or Mysqld process still in memory from older digiKam instance.

Gilles Caulier
Comment 6 Jürgen 2023-05-20 08:03:59 UTC
(In reply to caulier.gilles from comment #5)
> >It seems digikam is accessing the same file with different "users"?? 
> 
> No, definitively.
> 
> But look if no zombie ExifTool or Mysqld process still in memory from older
> digiKam instance.
> 
> Gilles Caulier



That seems to be it!... I see multiple "exiftool.exe" running. Guess that creates the concurrent access to the fiiles writing meta data into the image file ... which then triggers onedrive to create duplicates :-(   

any idea how to prevent that?
Comment 7 caulier.gilles 2023-05-20 12:16:07 UTC
digiKam start exiftool.exe (included in windows bundle) is used to replace Exiv2 shared library in special use cases to hanlde metadata.

We have a file in bugzilla about the non closed exiftool.exe when digiKam crash. It's specific to Windows system.

Gilles
Comment 8 caulier.gilles 2023-10-15 03:41:41 UTC
@Jürgen,

This problem still reproducible with the new digiKam 8.2.0 pre-release Windows
installer available at usual place:

https://files.kde.org/digikam/

This new bundle is based on last Qt framework 5.15.11 and KDE framework 5.110.

Thanks in advance

Gilles Caulier
Comment 9 Jürgen 2024-02-15 16:36:29 UTC
(In reply to caulier.gilles from comment #8)
@Gilles,

I use the software quite infrequently... so my answer took a while.  The good news: The error did not occur with the new version!
Comment 10 Jürgen 2024-12-30 15:14:06 UTC
Bug shows up in 8.5.0 again :-(  Original bug description is still valid. 

Assigning a person to a "unknow" face created a file duplication on the local drive. The filename equals the names of the existing file + "-Computer name". Face tag is properly updated in the new file, the existing file is no modified.

The issue does not occur each time a file is edited. It occurs in approx 1 of 5 edits.
Comment 11 Jonkeren 2025-01-17 13:53:45 UTC
Jürgen exactly this issue I (and others) have. 
I think it is not a Digikam issue, but something between Exiftool <> Windows file system <> Onedrive.
Onedrive re-downloads older files. I suspect because of an incorrect file changed time stamp (not correctly updated by Exiftool?)
I wrote some suggestions to "mitigate" this issue here --> https://bugs.kde.org/show_bug.cgi?id=498798

Hopefully this can be implemented in Digikam. Some Exiftool command line setting. 
Exiftool itself offers various ways to "write to" the files --
1. no option specified - ExifTool always creates a new output file.
2. -overwrite_original (in which case the output file is renamed to replace the original), 
3. -overwrite_original_in_place (in which case the contents of the output file are used to overwrite the original, and the output file is deleted).

I GUESS one of hese things is the culprit.
Comment 12 Jürgen 2025-01-17 15:51:32 UTC
Jonkeren, thanks for the feedback. One more thing I would want to highlight is that the error occurs randomly. Sometimes I edit many files with no problem at all. Sometimes I touch 2 files and the bug occurs 2 times. I was looking at the files from all possible angles trying to figure out any "rule" when the issue occurs and when it does not - but no success.... Also I tried all variations of enabling / disabling onedrive, connecting and disconneting from the networt and reboots to see if there is an approach that reliably stops the error - but likewise failed :-(
Comment 13 caulier.gilles 2025-01-17 18:51:18 UTC
As the online documentation said "digiKam DOES NOT support virtual placeholder folders"

https://docs.digikam.org/en/setup_application/collections_settings.html#the-network-shares-specificity