Bug 347937

Summary: SETUP : Collections added as "Removable Media" change as "Local Collections"
Product: [Applications] digikam Reporter: RJVB <rjvbertin>
Component: Database-MediaAssignee: Digikam Developers <digikam-bugs-null>
Status: RESOLVED FIXED    
Severity: major CC: caulier.gilles, marcel.wiesweg, reuben_p
Priority: NOR    
Version: 5.5.0   
Target Milestone: ---   
Platform: macOS (DMG)   
OS: macOS   
Latest Commit: Version Fixed In: 7.1.0
Sentry Crash Report:

Description RJVB 2015-05-19 09:50:39 UTC
Adding collections to the "Removable Media" category on OS X works as a "backdoor" to adding them to the "Local Collections" category.

Reproducible: Always

Steps to Reproduce:
1. Open the Preferences dialog, and go to the Collections panel
2. Use the "Add Collection" button next to the "Removable Media" category to add a folder to that category
3. Close the Preferences dialog
4. Reopen the Preferences dialog and go back to the Collections panel

Actual Results:  
The collection (album) just added shows up in the "Local Collections" category.

Expected Results:  
The collection (album) just added shows up in the "Removable Collections" category.

I have not attempted to verify, but I have a hunch the same would happen with the "Network Shares" category. This is based on my assumption that this bug has something to do with the way OS X mounts everything under /Volumes as opposed to how Linux organises things.

I'm labelling this a Major issue as basically it disables all support for removable media.
Comment 1 caulier.gilles 2015-05-19 09:54:38 UTC
I think it's more relevant of Solid do not work well under OSX. In fact Solid API is used here to register collection in DB...

Gilles Caulier
Comment 2 RJVB 2015-05-19 10:58:12 UTC
Hmm, if you can point me to where/how this is done in digikam's sources, I can have a look at Solid.

But I don't really see the point in providing the categories through the settings dialog if afterwards you're only going to use an external library to determine whether a collection is local, on a removable or on a network share. Why not store the information how a collection was added and use it as an additional property (or to override whatever Solid tells you on OS X and potentially other platforms where Solid dysfunctions)?
Comment 3 caulier.gilles 2015-05-19 11:07:17 UTC
Marcel,

Can you clarify/guide RJVB according with previous comment.

Thanks in advance

Gilles
Comment 4 RJVB 2015-05-19 11:22:24 UTC
(In reply to RJVB from comment #2)
> Hmm, if you can point me to where/how this is done in digikam's sources, I
> can have a look at Solid.

I can try to see what goes wrong in Solid when I know how it is being used by digiKam.

> share. Why not store the information how a collection was added and use it
> as an additional property (or to override whatever Solid tells you on OS X
> and potentially other platforms where Solid dysfunctions)?

Collections are added as `Local`, `Removable` or `Network`, and treated accordingly. Apparently this property isn't stored; instead you rely on Solid to keep track of which collections are on local, removable or network storage. If Solid isn't reliable in telling you if a collection directory is on local, removable or network storage, you should store that property with the other, non-hardware related properties (like the collection name).

This would probably also make it easier to work with temporary collections, "scratch" folders that are on local storage but not always present. The same could apply to collections on "fancy" filesystems: I have no idea how Solid would recognise a ZFS pool or dataset for instance.

There's an underlying question of principles here: should you use technology to second-guess the user, or should you just take it for given when s/he tells you "this collection is on removable media, treat it as such"?
Comment 5 Marcel Wiesweg 2015-05-20 18:38:17 UTC
Most of the relevant code is in libs/database/collectionmanager.cpp
You can find the UI part in utilities/setup/setupcollectionview.cpp

Network shares are indeed treated differently, they are stored by mount path. Solid does not care for network shares, nor did any other system library in KDE or Qt ar the time. I guess it's still the same.

The fact that there's an extra button to add removable media is due to UI design decisions, we need an extra button for network collections and there's no good way to have a common button for normal storage volumes.
Apart from that, to digikam there is no real difference if you remove your USB stick or screw out your harddisk, in either case the storage volume is not available. There are two flags in Solid::StorageDrive, isHotpluggable() and isRemovable(), which determine the "removable" property.
Comment 6 RJVB 2015-05-20 19:01:12 UTC
Thanks, I'll have a look.

Right now, I'm getting errors about all collections on non-mounted removables when I start digikam. Are you saying that would still be the case if it knew those missing collections were on removables instead of considering them to be on fixed storage?
Comment 7 Marcel Wiesweg 2015-05-21 18:03:42 UTC
What kind of errors do you get? Normally, the collection should silently be marked as not available. It's not an error if a removable drive is removed.
Comment 8 RJVB 2015-05-21 19:57:56 UTC
I agree it's not an error, but as I described above, on OS X:

1) collections on removable storage are classified among the ones on a fixed harddisk
2) as a result, an error is printed that they're not available when that's the case, because digikam doesn't know it's not an error because of the removable nature of their support.

That's why I suggested that the easiest and most straightforward fix that would work in all situations where Solid doesn't work reliably, would be to store the isRemovable state provided by the user when s/he adds the collection, rather than relying on Solid to reproduce it. Alternatively, remove the choice completely and let isRemovable be determined by Solid even when the collection is added. I don't know what it means that "there's no good way to have a common button for normal storage volumes". Either you rely on Solid to guess the nature of a volume (and you only need a separate button for networked collections), or you rely on what the user indicates (and you need a button for removable collections if you want to support that notion).

Again, I'm a proponent of listening to the user. If you support the notion of removable collections (however small the difference with non-removable collections) and let the user indicate which is what, you don't second-guess this choice. You could ask confirmation if  Solid tells you otherwise on platforms where this is reliable, and you could use subsequent changes as an indication that a disk was for instance moved to an external enclosure (and ask confirmation if this change should be recorded). But in my book, if the user tells you to consider a collection as removable  (or non removable), you conform as long as this instruction does not lead to conflicts or impossible situations. But then I try to pay attention to the little details that annoy me when using someone else's software :)
Comment 9 Marcel Wiesweg 2015-05-22 18:26:21 UTC
In any way there is now no compatible fix to store the "isRemovable" bit.
I meant to say that the fact that there are three buttons is more or less deliberate, in some other UI design there would be no separate buttons. It's like two doors leading into the same room.

IMO, Solid should be fixed on OS X. We have a set of high-level platform libraries and should in our applications not add complex workarounds for bugs in the underlying library.
Comment 10 RJVB 2015-05-22 21:24:34 UTC
What do you mean with "no compatible fix"? That everything about the defined collections is stored in a fixed-format database which doesn't provide the flexibility to add a parameter? I've wondered why the list of collections isn't stored in a standard rc file too.

Solid is incomplete on OS X, and I got dissuaded from spending much time on it IIRC. Not that that was hard to do: I'm not familiar with IOKit, and getting more of it into Solid looks to be writing a considerable amount of doubly navel-staring (pardon, introspective) code ...

Are you sure that Solid on Linux recognises all kinds of removable storage correctly? How does it handle ZFS pools/datasets, for instance? For example, I have datasets on my ZFS root pool that are not always mounted. They live on my internal harddisk, though. Come to think of it, there isn't even need to consider uncommon filesystems: one could keeps photo collections on partitions of the internal disk that aren't always mounted; the MS Windows partition would probably a good example.
Comment 11 caulier.gilles 2015-06-22 19:26:31 UTC
New digiKam 4.11.0 is available with official PKG installer for OSX.

https://www.digikam.org/node/740

Can you reproduce the problem with this release ?

Gilles Caulier
Comment 12 RJVB 2015-06-23 20:12:52 UTC
Indeed, as can be expected from the exchange above.
Comment 13 caulier.gilles 2015-08-13 08:01:45 UTC
digiKam 4.12.0 is out.

https://www.digikam.org/node/741

Problem still reproducible ?

Gilles Caulier
Comment 14 RJVB 2015-08-13 15:41:35 UTC
On my ToDo list for when temperatures drop !
Comment 15 caulier.gilles 2016-06-28 09:08:08 UTC
digiKam 5.0.0-beta7 Installer for MacOSX is now available for testing...

https://drive.google.com/open?id=0B7yq-xFihT0_ZjlIazRPb1lRTEE

Gilles Caulier
Comment 16 caulier.gilles 2016-07-01 12:51:07 UTC
RJVB,

Do you seen my previous comment ?

Gilles Caulier
Comment 17 RJVB 2016-07-01 13:47:15 UTC
Yes, but I suppose the detection of removable vs. local devices is still done via Solid? That library hasn't changed to my knowledge so I didn't run out to check if this issue is still present ....
Comment 18 caulier.gilles 2017-04-16 20:32:02 UTC
new 5.6.0 pre-release as bundle is available here :

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

Please check if this problem still reproducible with these versions.

Thanks in advance

Gilles Caulier
Comment 19 RJVB 2017-04-17 18:26:10 UTC
How could it NOT be reproducible as long as the Solid framework doesn't provide the required information, and digiKam itself doesn't remember under which category a collection was added?

I tested with an external harddisk: added as a removable collection, closed the Preferences dialog, re-opened that dialog; the new collection was listed under the regular, fixed collections.
Comment 20 caulier.gilles 2017-04-17 20:41:11 UTC
RJVB,

After all this time, KF5::Solid API do not support yet MacOS better... Why ?

Gilles Caulier
Comment 21 RJVB 2017-04-18 00:13:37 UTC
(In reply to caulier.gilles from comment #20)

> After all this time, KF5::Solid API do not support yet MacOS better... Why ?

Because no one bothered improving it, presumably in (large?) part because it's seen as something needed on Plasma desktops, not in KDE applications running in other environments?

The other thing is that it isn't the easiest kind of code to write, and also not something you want to develop on a virtual machine.

BTW, I tested with the 5.6.0 installer preview.
Comment 22 caulier.gilles 2017-06-22 21:43:16 UTC
digiKam 5.6.0 is now released and available as bundle for Linux, MacOS and Windows.

https://www.digikam.org/news/2017-06-21-5.6.0-release-announcement/

Can you check if problem still exists with this version ?

Thanks in advance

Gilles Caulier
Comment 23 caulier.gilles 2017-09-13 08:17:23 UTC
Just a remember link to Rene entry in Phabricator about Solid patches for OSX :

https://bugs.kde.org/show_bug.cgi?id=365337

Rene, Kdenlive will also package for MacOS and will use Solid for camera detection. So they have also a good candidate to test your patch.
Comment 24 RJVB 2017-09-13 08:23:18 UTC
That's interesting, but I haven't done anything about camera detection/support in Solid, and I doubt a bit that I ever will.
Comment 25 caulier.gilles 2017-12-27 09:57:40 UTC
To solve this issue, we pending this patch to be included in Solid API officially for a future KF5 release :

https://phabricator.kde.org/D7401

Until this stage passed, nothing can be done in digiKam. After that, the MacOS digiKam PKG installer must be rebuild with relevant KF5 release including the Solid patch.

Gilles Caulier
Comment 26 caulier.gilles 2018-02-11 18:01:16 UTC
Git commit ff9080d97b74867368dba034449f3cb3bc3d0cab by Gilles Caulier.
Committed on 11/02/2018 at 17:55.
Pushed by cgilles into branch '6.0.0'.

Add KF5::Solid IOKit support patch from René under MacOS through Macports.

This will permit to support Apple devices under MacOS as under Linux with the PKG digiKam bundle 6.0.0.

This patch need to be tested:

1/ compilation with mast Solid framework code ;
2/ At run time with digiKam, with new removable device connected as a camera or an hardrive.

See https://phabricator.kde.org/D7401 for details.
Related: bug 371535
CCMAIL: rjvbertin@gmail.com

M  +2    -1    project/bundles/3rdparty/ext_kf5/CMakeLists.txt
A  +2581 -0    project/bundles/3rdparty/ext_kf5/solid-iokitsupport-macports.patch

https://commits.kde.org/digikam-software-compilation/ff9080d97b74867368dba034449f3cb3bc3d0cab
Comment 27 RJVB 2018-02-11 18:11:28 UTC
I haven't tested this much, but as far as I've seen digiKam now supports removable media as it should on Mac.

My Solid patch doesn't include any improvement specific to cameras, so I don't think anything would change for cameras that don't behave as USB mass storage.
Comment 28 caulier.gilles 2018-08-17 21:30:24 UTC
Can you reproduce the dysfunction using digiKam 6.0.0 pre-release bundle
available here :

https://files.kde.org/digikam/
Comment 29 caulier.gilles 2018-12-31 11:50:26 UTC
Can you reproduce the dysfunction using the last digiKam 6.0.0-beta3 just
released ?

https://www.digikam.org/news/2018-12-30-6.0.0-beta3_release_announcement/
Comment 30 caulier.gilles 2020-07-14 09:41:37 UTC
Hi,

Can you check if this problem still exist with last weekly bundle build of digiKam 7.0.0 available here:

https://files.kde.org/digikam/

Thanks in advance

Gilles Caulier
Comment 31 Reuben 2020-07-29 11:09:14 UTC
(In reply to caulier.gilles from comment #30)
> Hi,
> 
> Can you check if this problem still exist with last weekly bundle build of
> digiKam 7.0.0 available here:
> 
> https://files.kde.org/digikam/
> 
> Thanks in advance
> 
> Gilles Caulier

I am running 
digiKam
Version 7.0.0-beta2
from Debian experimental
and the issue seems to have been fixed.
Comment 32 caulier.gilles 2020-07-29 12:25:24 UTC
Thanks for the feedback

Gilles Caulier