Bug 455091 - Renaming files on import loses image order in some cases
Summary: Renaming files on import loses image order in some cases
Status: RESOLVED WORKSFORME
Alias: None
Product: digikam
Classification: Applications
Component: Import-Gphoto2 (show other bugs)
Version: 7.5.0
Platform: Ubuntu Linux
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-06-09 16:27 UTC by Such A Shame
Modified: 2022-06-24 19:50 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In: 7.7.0


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Such A Shame 2022-06-09 16:27:23 UTC
SUMMARY
Renaming images on import will lose original image order for images that are taken within same second.

STEPS TO REPRODUCE
1. Import multiple photos taken at same second and use [date] to rename them during import.

OBSERVED RESULT
New filenames progress in random order for files taken in same second.

EXPECTED RESULT
Filenames should progress in original image order in all cases.

SOFTWARE/OS VERSIONS
Linux/KDE Plasma: Ubuntu 22.04 LTS
KDE Plasma Version: ?
KDE Frameworks Version: 5.92.0
Qt Version: 5.15.3
Comment 1 Such A Shame 2022-06-09 16:30:33 UTC
Correction: this may actually occur when later similarly using rename option for bunch of files, or in both cases.
Comment 2 Maik Qualmann 2022-06-09 16:57:38 UTC
Add the "{unique}" modifier to the [date] option. There can be no other solution if date and time are identical. The "{unique}" modifier only respects the file extension correctly in digiKam-7.6.0.

Maik
Comment 3 Such A Shame 2022-06-09 19:39:52 UTC
No, it doesn't help {unique} behaves similarly.
Comment 4 Such A Shame 2022-06-09 19:49:53 UTC
It actually seems to work with [date]{unique} IF I also sort items in growing order in import view, when selecting newst first they were renamed backwards when testing... Latter shouldn't be any criteria.

There's still one wish, digikam album view should view files in correct order. Now they seem to be:

123456_01.jpg
123456_02.jpg
123456_03.jpg
123456_04.jpg
123456.jpg

when correct is:

123456.jpg
123456_01.jpg
123456_02.jpg
123456_03.jpg
123456_04.jpg

but this is actually not rename bug.

So {unique} actually seems to help in random order which plain [date] gives, but the view order affects the way it shouldn't. Also album sorting needs minor fix.
Comment 5 Maik Qualmann 2022-06-09 20:01:06 UTC
The sort order is correct, it corresponds to the "natural" string sorting. If you look in the Dolphin file manager, it sorts the files exactly the same way. We make an exception in versioning if the counter still contains the "v", as 123456_v01.
We won't change it, you'll have to come up with another renaming string to make it more unique.

Maik
Comment 6 Such A Shame 2022-06-09 20:17:31 UTC
Adding v manually seems to change the order. Could the {unique} identifier then result to 

_v1
_v2
_v3

instead of current

_1
_2
_3

? 

This would make things work as expected. There's no easy way to add another unique identifier (such as adding original filename after date) because different cameras seems to give different results to hours (possibly because of inconsistensies in time zones??) and I normally do double renaming (which complicate that): first from camera without using metadata, then separate renames for photos and videos which work differently. I actually have to rename (fine tune hours for) videos manually since I don't get correct hours by any method.
Comment 7 Maik Qualmann 2022-06-09 21:16:17 UTC
Definitely no, it would break the natural numerical file order for other users. It would be inconsistent with file managers and we would have the next bug report. Accept it.

Maik
Comment 8 Maik Qualmann 2022-06-10 04:45:28 UTC
We can't technically implement it either, we would have to check each item with a regular expression, which would greatly slow down the sorting. By the way, the sorting in this case also depends on the platform, on Windows the order is as requested by you. So you would have to contact Qt. I'm still checking whether we can possibly introduce another option besides the character string comparison Options "Normal", "Natural" to disable/enable the consideration of punctual notation.

Maik
Comment 9 Such A Shame 2022-06-10 05:39:52 UTC
Ok. 

Also what possibly was earlier left unnoticed, the way files are arranged in import view (ascending/descending) shouldn't be involved in any way in which order unique filenames are formed. Renaming should follow only original filenames (in ascending order) - or possibly metadata if used - which possibly means that actual import should always be done in that order regardless of what the view says (I haven't looked at the actual code, this is just how it appears).

But this is possibly not biggest problem for me in practice, just a complication possibly deeper in systems which I think shouldn't exist.
Comment 10 Maik Qualmann 2022-06-10 06:03:59 UTC
I've played around with the options that QCollator offers, but we can't change the behavior. It depends on whether QCollator uses the libicu or not. The libicu is not used on Windows.

Maik
Comment 11 Such A Shame 2022-06-11 12:40:00 UTC
Still one more question. Are you sure this {uniques} actually works on import, because since switching to it, I'm constantly having images donwloaded second time afterwards? They seem not to be marked as already downloaded by digikam, which I constantly have to do manually. There must be reason why I originally dropped it out from import phase, and chose to do more reliable second rename afterwards, and it could be this behavior (can't remember anymore).
Comment 12 Such A Shame 2022-06-11 14:14:42 UTC
Latest test didn't offer to import duplicates and did mark all correctly imported. 

I did two changes which could have caused this: 

i) replaced renaming from [date] to [date]{import}
ii) started using metadata to directly get correct hours for photos (videos handle +2 as +4 and reduces too many hours). 

However problem doesn't seem to occur every time.
Comment 13 Maik Qualmann 2022-06-11 15:18:18 UTC
The {unique} modifier only considers the set of files just imported, not whether any already exist in the album. During the download, the option in the digiKam camera settings would take effect here, whether duplicate files should be overwritten, skipped or renamed with a _vxx extension.

I already wrote that renaming just by date is a very bad idea.
Why don't you rename the files after importing them? Then all database renaming functions are available. The digiKam view can sort by date, then rename all files with date and a running counter (###).

Detecting whether files have already been imported is a complicated story. We can't use a fingerprint at the moment because the file isn't available to us locally when importing from the GPhoto2 driver. The detection is currently a mix of device ID + file name + file size and date.

Yes, the import tool needs some major improvements that are on my todo list. We already have a bug report on the problem of recognizing files that have already been downloaded, I'll close it here.

Maik
Comment 14 Maik Qualmann 2022-06-15 17:13:25 UTC
Git commit eeb5ba940b0b7e523200e791b9f98623ab75b65b by Maik Qualmann.
Committed on 15/06/2022 at 17:12.
Pushed by mqualmann into branch 'master'.

fix read creation date from imported items from Import Tool
- Metadata for renaming will only be read after saving in the album
- Correct creation of date based albums.
- Add metadata and camera renaming options in the Import Tool.
- Removed live renaming preview in import view,
  preview for selected item only.
Related: bug 452088, bug 444928, bug 386072, bug 381214
FIXED-IN: 8.0.0

M  +5    -1    NEWS
M  +9    -1    core/utilities/advancedrename/advancedrenamemanager.cpp
M  +1    -0    core/utilities/advancedrename/advancedrenamemanager.h
M  +2    -0    core/utilities/advancedrename/common/parsesettings.h
M  +1    -3    core/utilities/advancedrename/parser/importrenameparser.cpp
M  +4    -1    core/utilities/advancedrename/parser/options/filepropertiesoption.cpp
M  +129  -201  core/utilities/import/backend/cameracontroller.cpp
M  +1    -18   core/utilities/import/backend/cameracontroller.h
M  +1    -1    core/utilities/import/backend/camiteminfo.h
M  +1    -1    core/utilities/import/items/importdelegate.cpp
M  +339  -267  core/utilities/import/main/importui.cpp
M  +9    -10   core/utilities/import/main/importui.h
M  +7    -1    core/utilities/import/main/importui_p.h
M  +8    -28   core/utilities/import/widgets/renamecustomizer.cpp
M  +2    -1    core/utilities/import/widgets/renamecustomizer.h

https://invent.kde.org/graphics/digikam/commit/eeb5ba940b0b7e523200e791b9f98623ab75b65b
Comment 15 Such A Shame 2022-06-24 19:50:44 UTC
Just to let you know detecting already imported files appears to be cleared while *changing the setting* whether metadata is read or not, and all files are detected as new afterwards. 

Will check the updates when it's handy to upgrade, but this works well enough now for this version.