Bug 375703

Yes, resolving the group is currently a workaround for the same or as copied recognized images. Otherwise, the groups would double in the source folder and would be linked. Because the leader of the group may not be copied first, it is difficult to recreate the group during the copying process.

Maik

Is there no clean solution for this? AFAIK the groups only refer to the images anyway (not the paths). so if images are moved, only the image path must be updated. Group information in the database could stay completely unchanged. 

Right?

Hi Jens, I just uploaded a patch in https://bugs.kde.org/show_bug.cgi?id=374591 that, apart from garbage collection also leads to generating less junk.
Meaning, if files are moved/renamed, the old images table entries are reused. This way, the image id of the image is preserved. This could solve your problem.
This patch does not work for images that were already moved before applying the patch. You can test it if you compile from source and apply the patch.
But be advised: The patch is in testing phase. If you do not run the garbage collector in maintenance, nothing bad should happen to your databases. But just to be sure, backup your databases before you test.

Oh, that is perfect. Garbage collection was on my list too - especially since some garbage has accumulated while developing iphoto2xmp: https://github.com/jensb/iphoto2xmp :-)

Thank you! Will try as soon as there is an appimage.

New AppImage will be ready tomorrow morning, not before...

Gilles Caulier

New 5.5.0 AppImage is done with garbage database collector patches.
Uploading to GDrive is under progress. It will be online in few minutes at usual
place :

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

New database Garbage Collector options are there :

https://www.flickr.com/photos/digikam/32549923912/in/dateposted-public/
https://www.flickr.com/photos/digikam/32549923632/in/dateposted-public/

Gilles Caulier

*** Bug 376014 has been marked as a duplicate of this bug. ***

I have for a test the workaround code removed.

https://cgit.kde.org/digikam.git/tree/libs/database/item/imagescanner.cpp#n393

But the problem with the wrongly linked grouped image IDs is also not fixed with the garbagecollection branch.

Maik

Is this (wrongly linked image IDs) an issue that should concern me? what exactly happens?
Thank you :)

(In reply to Maik Qualmann from comment #8)
> I have for a test the workaround code removed.
> 
> https://cgit.kde.org/digikam.git/tree/libs/database/item/imagescanner.
> cpp#n393
> 
> But the problem with the wrongly linked grouped image IDs is also not fixed
> with the garbagecollection branch.
> 
> Maik

Maik,
I just compiled the current state of the garbage collection branch and did the following:
1) Define 3 images as grouped
2) move the group to a different directory
Result:
The images are now located in the target directory and are still grouped.

My garbage avoidance for moved items works like follows: If items are moved, I set the album to the target album. Thus, the ImageScanner should not detect the moved image files as new and thus also should not detect the images as identical. Moreover, the images entries stay intact and the image relation grouped should not be touched.

I can not reproduce the problem in the current state of the garbage collection branch.

Can you both test against this branch?

I have to correct myself. Moving the group back suddenly destroyed the group relation... That's odd.

Jens,

One point : Which database type do you use ? sqlite or mariadb ?

Mario,

Problem reproducible here too (sqlite)

Gilles Caulier

Mario,

for testing you must remove the workaround code:

https://cgit.kde.org/digikam.git/tree/libs/database/item/imagescanner.cpp#n393

Yes, it now works with moved images within digiKam. But not with a copied group or when the same grouped images are added with import. Also externally copied grouped images with the file manager makes problems.

Maik

Maik

I am using MySQL on Ubuntu 16.04

Okay,

I thought about the problems for some time.

I try to resume the expected behaviour:
1) Moving a group of items should preserve the grouping relation on the moved items (Bug description by Jens) - This works in garbage collection branch without workaround.
2) Moving an item that is grouped should preserve its grouping relation - This also works in garbage collection branch without the workaround.
3) Copying a group within digiKam should also copy the grouping relation for the newly generated items - This is not done.
4) Copying an item of a group within digiKam - This is not clear. Should the new item belong to the group?
5) Importing images into digiKam that are identical to grouped items - Should the imported images be grouped automatically?
6) What do you mean with
> Also externally copied grouped images with the file manager makes problems.
   I assume that you mean that some user copies grouped images that are imported in digiKam with a file manager.
   But this then also makes problems when grouped images are moved with a file manager.

I give here my opinion to 3-6 and I am open for discussion. I will try to make my arguments as precise and founded as possible.

I would state the following for 3 and 4: Copy means copy and we should give the user what can be expected by the terms we use. Since we copy all image properties, the grouping relation should be copied in principle, too. But the problem here is: What does the user expect? Should the copies have a grouping relation with the source group leader or should the copied group be a new group with the copy of the old group leader being the new group leader? I cannot answer this question. Only the users can. So the default would be for me to not set group relations for copied items and (potentially) give the user an option to decide about the relations to established.

I would state the following for 5: If a user imports some images into digiKam, we cannot know whether he wants to establish a group or add the images to a group. Also, the grouping relation is existent in digiKam but not in file managers - at least not that I know. Thus, the imported images should not be grouped automatically. We could ask the user or give an option as import. But I would not take the decision from the user.

I would state the following for 6: I like good user experience. But this case is a clear misuse for me. One cannot expect some complex software to work properly when tampering with the resources managed by the software. If you just copy an image tag relation in our database, you will also have unexpected behaviour if our constraints do prohibit that. And this also holds for other software projects. If you copy some bundles from a OSGI container, he will probably yell at you to stop tampering with his internal resources. Probably with a crash. But we cannot prohibit changes in the file system level. We cannot even detect them if digiKam is not started. This is - at least for me - something that should be stated in the documentation.

To conclude: 
cases 3 to 5 are no bugs for me. I think they are usability issues and we could try to give the user the control about what is done with the group
1) establish a new group with copy of group leader being new group leader
2) link to existent group leader
3) do nothing (default)
case 6 is misuse and should be documented.

Opinions?

All the best
Mario

Mario,

I agree to 1) and 2): Moving a group or a single items should preserve the group relation.

With 3) I don't quite understand the remark at the very end of your comment:
> 3) do nothing (default)
However, I agree to
> 3) Copying a group within digiKam should also copy the grouping relation for the newly generated items
but of course it should be a completly new group independent of the original.

4) Here I would vote for not establishing relations to the original group. We could in addition implement an option to keep the copy in the group but I don't like the idea of too many options/settings.

5) I would (as a user) never expect anything to be grouped automatically at import. I why should we group identical images anyway? That's what Find Duplicates is for. And here applies the same as under 4): I don't like the idea ...

6) I agree absolutely.

Other than that I am of the opinion that improving metadata handling is the most important issue in digiKam.

I agree to the previous points. But I have one thought I'd like to share.

We are getting to the point where "groups" of images become basically identical to sub-albums (subfolders within an album) except for the fact that they are displayed within the main folder and sorted into the parent's images, not separately.

So do we *want* the difference between grouped images and subfolders/albums to exist at all? Is there a difference? Does there need to be a difference?

For me, 
- grouped images represent images that are very similar (e.g. taken with my camera's "machine gun mode" to catch the right moment) but I almost never want to see all of the images. They are part of an event (album) and a chronological series of images.
- (sub)albums represent "subevents" of their own, part of a larger event (e.g. one day's tour within a vacation) with different and unique images that all want to be viewed (and some can be grouped).

But both could be used for both purposes with very little change in the UI. Both subalbums and groups can have a "master image" (thumbnail) and both subalbums and groups can be opened and closed.

Maybe the whole idea of grouped images needs to be rethought? Maybe we can solve many of the grouping issues by just redefining "groups === subalbums"? Very little would need to be done, IMHO:

- Include an option for each album (and a global default) to display subalbums in the main thumbnail view (with a folder icon or the defined album thumbnail) as an image within the other images
- allow ratings, tags, etc. to be applied to albums (which would effectively apply them to all contained images)

Is this maybe worth a second thought? It would make the whole concept a lot simpler and - I think - solve a lot of the consistency issues that the concept of grouping images has right now.

But then I can throw all my wonderful doc about grouping into the bin :-((
No, but seriously: It's for sure worth a second thought! You are right: the differences are minim. But right now I don't have the time for the second thought. And the devs have to say anyway whether it is worth it from the technical point of view.
I did use the function up to now only to get familiar with it for wrting the doc. Will do a bit more soon (if I don't get washed away by the ongoing discussion anyway).
Just one thing comes to mind: if we cancel grouping we need to provide some kind of conversion for those users who used it already. We cannot just kill their existing groups ...

True - removing features must always be well thought out and there must be a migration path (= database migration). I thought about this some more. The concepts are indeed very similar, only little change would need to be done to enable "pseudo grouping" using subfolders:

+ Albums would get a new property setting whether 
  - the album should be treated as a folder as it is now, with
    - a entry in the albums tree view
    - a separate section in the main view)
  or
  - as an image group inside the main thumbnails view, not in the albums sidebar
    - using the album's folder thumbnail image as thumbnail
    - and the thumbnail image's metadata to sort this thumbnail

+ Such "group" albums would also allow metadata changes to the main thumbnail image and optionally apply them to all images inside the group.

+ There must be a migration function which converts grouped images to "group folders"
  for all future Digikam versions (since we don't know what versions people will use)
  which automatically activates on startup so future Digikam versions don't modify old databases.

Advantages:
- We can get rid of a huge ton of complexity regarding grouped images
  (moving, copying, external manipulation, reimport, etc.)
- Image groups are represented in the file system and not just in the database
  (better for external image manipulation)
- Import grouped images into Digikam is much easier since (almost) all you need to do is create the appropriate subfolders. Maybe it would even be possible to create ".xmp" files inside folders with appropriate folder properties for Digikam, this would make data transfer even easier.

Disadvantages:
- There will be two types of "albums" for the future.
  This might require modification of existing code regarding albums.
- 

I think it would be worth it. Image groups are a really great feature but right now the feature is somewhat unstable and so I am still a bit hesitant to use it intensively.

*** Bug 377782 has been marked as a duplicate of this bug. ***

IMHO I don't think grouping images is so similar to subalbums. One typical use of groups is group the RAW and the JPG file of the same image. This way you only see one picture (the JPG probably) of your images instead of having the RAW image, the camera JPG and your own developed JPG. 

Also, about mentioned advantages, it's said: "Image groups are represented in the file system and not just in the database (better for external image manipulation)". This may be true, but is it really an advantage? Using groups for each picture and its derivatives (RAW, JPGs, etc) means each 'real' album, in the filesystem, would be a bunch of directories, with all versions of the same picture on each. Is this better than having all in the same directory? IMHO is not.

Using directories for each group probably will end in a cluttered album view with thousand of tiny albums. Also, it makes necessary to select the main picture of each subalbum to be shown in the parent album: this needs a convenient GUI to do it. And, are all subalbums going to be shown in the parent album alongside the parent's picture? This will be a change in current Digikam's behavior, potentially affecting many users.

Also, grouping is a feature present in other image software like Darktable or Adobe Lightroom.


I just wanted to add my point of view, of course I don't know the problems that grouping brings to the Digikam base code, I'm a talking only as a user :)

Ricardo, I wasn't intending to (much) change the *visual* appearance of grouped images. Just the technical way of storing them.

Right now, there are a huge heap of difficulties involved in managing grouped images and it is hard (if not impossible) to get at groups from without Digikam. Using folders (=albums) for groups, with a special tag, will keep files grouped if files are moved outside Digikam and not require a huge amount of database management when copying, moving or renaming albums with groups.

Hi Team,
I was hoping this would be fixed in 5.6 but the problem still occurs and it's a pain :-( losing alot of time re-doing the grouping of images ...

FWIW, On my system (Fedora25), the issue occurs only when moving albums from one collection to another one (like local SSD to NFS based NAS)
Moving albums inside the same collection preserves images groups ...

And it's still broken using the 5.7 appimage :-(

And by the way, problem also shows when moving form local collection to removable collection (USB) not just Network so it's quite easy to reproduce ...

Haven't looked at the code but I don't understand why it's behaving like this !?!

Looking at the Database structure, the ImageRelations table associates image Ids ..

So why is moving the images breaking the relation ? the imageIds shouldn't be changing, should they ?

Anyway, this is a very painful problem ... Am I the only person using image groups ?

I think Simon fix the problem while Randa reunion. The fix will be published on 5.8.0.

I build new bundles while the event, but I'm not sure if i included these fix at the right time.

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

In all cases, i will rebuild the bundle this week end including huge changes in new Media Server implementation. Please be patient.

Gilles Caulier

Don't worry Gilles, I'm a patient man :)
I Just wanted to make sure this bug wasn't lost ...

I'm already grateful for the tremendous work you guys are doing !
I wish I had more time to spend helping you ...


Thanks for the update ! I'll check out the next build.

*** Bug 385147 has been marked as a duplicate of this bug. ***

Yes Jens, In 5.6.0 I had the issue only when moving to remote collection.
With the 5.7.0 appimage I downloaded today to make a test, I was getting the issue with local collections as well ...

Can you test with the current AppImage 5.8.0 avaialble on the google drive repository, please.

Gilles Caulier

Still not working (downloaded two days ago).

The fix isn't committed yet. I wanted to do it "right" (e.g. not introduce new code duplication), but that was proven to be hard and I don't have much time for hacking on digikam right now. I will try to get a quick and dirty version committed by 5.8.0.

Issue is still present for me as well with the 5.8.0 appimage build.

Behaviour seems even worse as it seems images are physically moved but only one image from the group is visible in the new destination until you force a refresh.
The image count on the folder is also inaccurate.

looking at the content of the db, it looks like it has created some bogus records :/

https://www.dropbox.com/s/n06p4ex3dxhw3of/digikamdbmessedup.jpg?raw=1

Git commit 38c6058eddbef286de4b5652ee92ddad11d7c52e by Simon Frei.
Committed on 24/10/2017 at 09:07.
Pushed by sfrei into branch 'groupMoveDirty'.

do both file and db operations on grouped items

This is just a quick fix to be consistent again. It resolves groupes no matter
what, while grouping should be properly handled by views. This manifested in
drag&drop, where groups are not yet handled.
FIXED-IN: 5.8.0

M  +12   -7    libs/database/utils/dio.cpp

https://commits.kde.org/digikam/38c6058eddbef286de4b5652ee92ddad11d7c52e

The push below was a mistake, the patch will appear hear shortly after testing.

Created attachment 108534 [details]
patch to do both move all grouped files both physically and in db

With the attached patch I can't reproduce this problem anymore.
I originally intended to solve the problem properly (i.e. add grouping to drag&drop), but I didn't finish and don't have time at the moment to work on it. But I believe this is a cause for the problem in this issue -> please test.

Also if anyone is inclined to use phabricator, I gave it a try:
https://phabricator.kde.org/D8440

Created attachment 108537 [details]
removeWorkaround.patch

This workaround patch from me would need to be removed. But the copying of grouped images still does not work. It comes to incorrectly linked groups.

Maik

Indeed, my patch does only concern moving, not copying. For me it works with your workaround patch. So I assume it should be left in place until copying is fixed as well?

Simon,

please commit your patch. Moving grouped images over collections that reside on different partitions is now working.

Maik

Git commit f4f5779477787d8465c2d306e45ee62ebf381cca by Simon Frei.
Committed on 25/10/2017 at 20:06.
Pushed by sfrei into branch 'master'.

do both file and db operations on grouped items

Summary:
This is just a quick fix to be consistent again. It resolves groupes no matter
what, while grouping should be properly handled by views. This manifested in
drag&drop, where groups are not yet handled.
FIXED-IN: 5.8.0

Reviewers: #digikam

Differential Revision: https://phabricator.kde.org/D8440

M  +13   -7    libs/database/utils/dio.cpp

https://commits.kde.org/digikam/f4f5779477787d8465c2d306e45ee62ebf381cca

Guys,

I downloaded the digikam-5.8.0-20171212T132305-x86-64.appimage

I'm still experiencing the same problem of the image groups being lost when moving images to another collection.

Is the proposed patch incorporated in that build ??

As i can see, i will said yes...

Maik, Simon, Mario, can you confirm that all expected proposed patches are included in git/master ?

Gilles Caulier

Yes the patches are in master and for me moving (not copying!) works fine with preserved grouping.
Vincent, can you please give exact steps of what you are doing to produce the problem and what types of collections and which database (sqlite, mysql) you use.

Basically, I have 2 collections, one is my local SSD drive (/home/timetre/Pictures) and the second is my NAS mounted as NFS (/NAS/Pictures)

Hierarchy looks like the following in Digikam:

NAS
 |
  - 2015
  - 2016
  - 2017

SSD
 |
  - 04 To Publish
  - 05 To Archive
         |
          - 2017-11-25-Le_Petit_Cap_Roux

That folder named 2017-11-25-Le_Petit_Cap_Roux contains a bunch of grouped images.
Typically, I have 3 images per group (CR2, TIF, JPG) and the xmp sidecar files so let's say:

Tassy_20171125_3989.cr2
Tassy_20171125_3989.cr2.xmp
Tassy_20171125_3989.tif
Tassy_20171125_3989_WEB.jpg

So when I reach this stage in my workflow, I would drag and drop the folder 2017-11-25-Le_Petit_Cap_Roux onto the NAS/2017 folder and chose "Move" in the contrext menu.

After a little while (and no progress bar by the way), the folder has been moved in my NAS but the image groups are lost ... I can see the individual images and I have to do the grouping all over again.

Hope that's clear enough ... I can do screen captures if it helps ...
Vince.

You only have to move the images. If you move the folder, a copy operation takes place because the drive and partition are different. Moving folders works only within the collection or if both collections are on the same partition.

Maik

I see !!!
That explains it ...
Thanks for the clarification. That will have to do for now ...

Since this is now marked Fixed in 5.8, should another bug be created to keep track of the fact that the copy operation (and therefore folder move) is still not working ?

Yes, please do that. I have started to work again on grouping some time ago, but it took a lot of time and I don't have time at the moment to keep working on it. When/if it gets finished, it will fix group copying (along other stuff).