Bug 215486

sorry, mousing error....
continuing here

-------

the collection


(Folder /home/simon/Photos on the volume with the id:<uuid>)

is currently not found on your system
please choose the most appropriate action
A) database is located on temp storage
B) take no action
-----------------

This is wrong on a lot of levels:
- I don't _know_ about my collections, as I have only one
- I've been using this collection for a long time and had several updates (with similar problems, but never resulting in a total loss of all database information)
- the actual database is still there, and contains data 
- neither action proposed is or could be appropriate
- the reference to the setup dialog is misleading, since the actual path to the configuration in digikam is elsewhere.
- the actual database is still there _and_ is the cause of the problem, because it contains bad information about paths and volume-ids that earlier versions of digikam have put there themselves.
- I get two of these dialogs, for both entries in the database concerning my rootpaths or whatever.


I managed to fix it partially, or completely (I'm not sure yet) thanks to some hints on the ubuntu forum:
http://ubuntuforums.org/showthread.php?t=1192655
(The fix was to remove the rootpaths records in the database using sqlitebrowser, and letting digikam create a new entry. Then checking what this looks like and putting that in the old databasefile on each of my rootpath entries got me a single dialog like above and a working "collection" with my tags available again (I hope)

and the bug looks somewhat like this one:
https://bugs.kde.org/show_bug.cgi?id=193522

but my situation is different.

In general, I think a lot more care should be taken to deal with this database around upgrades and in general. I have growing reluctance to trust my photos and especially the metadata to digikam because of such an upgrade experience. I have 10s of thousands of images, (I know, way too much, but it takes time to weed them out). If I put in time to tag them, I want to know my data is safe!

How hard can it be to write a database recovery tool (inside digikam) to handle this kind of situation. If a used has backups, the data in it should be recoverable with a small bit of assistance from the user. It should recover from any "collection" with any version of digikam's database formats. The main thing is that images retain the links to their tags and ratings and comments, those things cannot be generated by the machine.

Cheers

Simon

The path I had was of the form:
6034d5dd-65c5-40a6-a231-d20f1e4e02a7
(though probably volumeid:?uuid=6034d5dd-65c5-40a6-a231-d20f1e4e02a7)

And it should be:
volumeid:?path=%2Fhome%2Fsimon%2FPhotos
And a / in the next field, instead of /home/simon/Photos

> - I don't _know_ about my collections, as I have only one
> - I've been using this collection for a long time and had several updates (with
> similar problems, but never resulting in a total loss of all database
> information)
> - the actual database is still there, and contains data 

Nothing is said about loss of database information.
Please report upgrade problems

> - neither action proposed is or could be appropriate
> - the reference to the setup dialog is misleading, since the actual path to the
> configuration in digikam is elsewhere.

??

> - the actual database is still there _and_ is the cause of the problem, because
> it contains bad information about paths and volume-ids that earlier versions of
> digikam have put there themselves.

The database did not change. Probably your partitions and or filesystems changed.
Please post the output of "solid-hardware list details"


> The path I had was of the form:
> 6034d5dd-65c5-40a6-a231-d20f1e4e02a7
> (though probably volumeid:?uuid=6034d5dd-65c5-40a6-a231-d20f1e4e02a7)

That is how it should be.

> 
> And it should be:
> volumeid:?path=%2Fhome%2Fsimon%2FPhotos
> And a / in the next field, instead of /home/simon/Photos

No it should not be like this. It works like this, it can be like this if you have reason.

> Nothing is said about loss of database information.
> Please report upgrade problems

Technically, the database lost no information, because the file digikam4.db was still there, containing the information (AFAICT).

The upgrade problem is that I was confronted with a newer version of digikam and I "lost" my entire "collection" due to some error regarding the uuid of the disk on which my collection is stored.

The volume is an lvm2 one, though it matters not a bit, because I could have copied all my data to a new and bigger disk, then my uuid would also be different.

When I removed all the entries for AlbumRoots from the database, digikam told me it was unable to determine the uuid and if it was ok to use the pathname (which is perfectly fine with me).


> 
> > - neither action proposed is or could be appropriate
> > - the reference to the setup dialogue is misleading, since the actual path to the
> > configuration in digikam is elsewhere.
> 
> ??

The error refers to the "setup dialogue", the actual path is menu -> Settings -> Configure Digikam -> Collections, if I understand correctly.


> > - the actual database is still there _and_ is the cause of the problem, because
> > it contains bad information about paths and volume-ids that earlier versions of
> > digikam have put there themselves.
> 
> The database did not change. Probably your partitions and or filesystems
> changed.
> Please post the output of "solid-hardware list details"

I usually work remotely (ssh X tunnelling) I get this output:
 virtual QStringList Solid::Backends::Hal::HalManager::allDevices()  error:  "org.freedesktop.DBus.Error.ServiceUnknown" 

> 
> > The path I had was of the form:
> > 6034d5dd-65c5-40a6-a231-d20f1e4e02a7
> > (though probably volumeid:?uuid=6034d5dd-65c5-40a6-a231-d20f1e4e02a7)
> 
> That is how it should be.
> 
> > 
> > And it should be:
> > volumeid:?path=%2Fhome%2Fsimon%2FPhotos
> > And a / in the next field, instead of /home/simon/Photos
> 
> No it should not be like this. It works like this, it can be like this if you
> have reason.

Well, whether or not this is right or wrong according to the current implementation or intention, I would like to argue that this way of referring to "collections" is error prone, due to uncaught assumptions (as far as I understand the intention, of course).

To me it seems that in order to handle collections on temporary storage, the concept of uuid was added as a required authentication of a collection. However, there are lots of situations that can occur with perfectly non-temporary storage of collections that make the uuid change or unavailable (as in my case, probably).

If my assumption is correct, I would suggest to change the behaviour of digikam to take the uuid as a hint and rely mostly on the pathname. If a mismatch is found in the uuid, or the stored collection on disk (the actual image files), then it might be that you're dealing with a (different) temporary storage device, rather than a permanently collected storage device. The latter being the common use-case, I presume.

In modern Linux systems, the pathname is usually /media/<volume-label> anyway, so it might very well be that the problem is commonly solved that way, without needing something as arbitrary as a uuid. (It's not like you can see from the uuid what kind of disk you're dealing with)

And furthermore, even if something seems to be wrong, it would definitely be helpful if digikam was able to "import" the old database for the meta-data of the files. 

If the database contains filename and exif-date linked with the unique database key used to store the tags and comments, I'd expect to get a 100% recovery in most cases.

I hope I've cleared up the situation!

/Simon

Probably the most important reason not to care about which disk the collection is on is that unix is designed to not care about where the physical files are stored, as long as the pathname is the same, the same file is intended.

Obviously, this breaks when partitions are added/removed willy nilly, like with usb storage. But it makes no sense to break the basic design assumption of unix (and Linux). To determine whether a different collection is meant by the user(!), some sort of identifier should be kept with the collection, in the form of a file, or the identification should be based on the image files themselves.

/Simon

> The volume is an lvm2 one, though it matters not a bit, because I could have
> copied all my data to a new and bigger disk, then my uuid would also be
> different.
> 
> When I removed all the entries for AlbumRoots from the database, digikam told
> me it was unable to determine the uuid and if it was ok to use the pathname
> (which is perfectly fine with me).
> 
> I usually work remotely (ssh X tunnelling) I get this output:
>  virtual QStringList Solid::Backends::Hal::HalManager::allDevices()  error: 
> "org.freedesktop.DBus.Error.ServiceUnknown" 

It seems that HAL is broken on your installation. That explains why digikam could not find the UUID of newly added collections and why it did not suggest the new place of your collection in the first place.

(and no, we do not add extra handling when HAL is broken - for one we cannot find out because Solid is cross-platform, and secondly it's a system service, so it's the problem of distributions)


> To me it seems that in order to handle collections on temporary storage, the
> concept of uuid was added as a required authentication of a collection.
> However, there are lots of situations that can occur with perfectly
> non-temporary storage of collections that make the uuid change or unavailable
> (as in my case, probably).

The intention is to support storage on hard-disks, optical disks, all kinds of pluggable USB storage devices, and - this is a different conceptual level - network drives. For all of the former, the UUID is by far the best means of identification, followed by volume label (can easily be the same for different media) and mount path (maybe based on volume label on Linux, arbitrary on Windows).

The UUID has a weakness when you repartition your harddisk and change the filesystem type, but do not change the mountpoint. In this case digikam will see that the old collection from hard-wired storage is missing, and there is a new candidate with the same relative paths (under the mountpoint), and offer you that as a candidate. Provided your HAL system services are not broken.


> If the database contains filename and exif-date linked with the unique database
> key used to store the tags and comments, I'd expect to get a 100% recovery in
> most cases.

Me too. Otherwise it's a bug ;-)

Regarding HAL, I'm running ubuntu karmic koala and in the release notes it says HAL is deprecated:

> hal deprecation
>
> Ubuntu 9.10 Beta's underlying technology for power management, laptop 
> hotkeys, and handling of storage devices and cameras maps has moved from 
> "hal" (which is in the process of being deprecated) to "DeviceKit-power", 
> "DeviceKit-disks" and "udev". When testing Ubuntu 9.10 Beta, please be alert 
> for regressions in those areas and report any bugs you find. 

So I suppose I should report this as a regression to ubuntu?

But the larger picture is still the one from my last comment, it's just not the unix way to depend on meta-information about the storage device for this kind of problems. If now HAL is deprecated and devicekit is used, in 5 years time perhaps some other solution will be used, but the underlying philosophy is the same old unix-way, so that's what you need to get things to work over longer periods (and photo collections tend to be LONG periods).

So the problem is that some albumroots contain a different collection at different times and digikam needs to differentiate between them?

The collection contains the actual photo-files and nothing else?

And the database is stored (theoretically) in a separate location (found by digikamrc configuration?)

(In my case the digikam4.db is in the same toplevel directory as my collection)

Doesn't it make sense to add a collection identifying file to each collection's root directory? That way you'll always know which is which. And even if you want to solve the problem using some arbitrary other route (like uuid), there's always a backup way to retain a working system.

Cheers

Simon

https://wiki.ubuntu.com/Halsectomy has an overview of regressions for ubuntu's decision to deprecate HAL

What we use is KDE's Solid API, which is a cross-platform API in kdelibs proper, on Linux usually traditionally on HAL. I don't know of the status of post-HAL support for Solid, but that is out of our scope. If a distribution decides to remove a piece of system software it should make sure KDE is not broken. (I guess at least Kubuntu will sooner or later be forced to fix that)

And yes, you can put your database wherever you want and can add as many collections to it (= photos on a storage somewhere) as you want.

It would be possible to add support for identifying collections by a file of defined filename and known contents (could contain a random seed, a name or a uuid)

Seed identification may be done in the future for network collections, I'm not sure.
The main problem here is clearly a DOWNSTREAM and/or UPSTREAM issue, we only rely on platform-independent kdelibs API

Hi Marcel,

I disagree with the closing of this bug. Please read again my comments (the prose at the end of)  https://bugs.kde.org/show_bug.cgi?id=215486#c4

This "feature" assumes too much about the OS, nothing guarantees that the uuid will remain the same during the timeperiod of any collection's life. The uuid is simply not able to uniquely identify the collection.

Relying on the kdelibs (solid I presume?) is also not robust, especially regarding low level stuff like uuid's. 

It's like you're identifying cars by the composition of the metal used for the wheel-hubs, where you should be using the number-plates (they're stuck on as well!)

Cheers

Simon

There are quite a few pros and cons to consider, and certainly one size doesn't fit it all. For good support of network shares we can't UUID either of course. Maybe there'll be an option to load a collection by path only, if the user really wants. But if you check the Solid API, there is mostly label and uuid to be used.
http://api.kde.org/4.x-api/kdelibs-apidocs/solid/html/classSolid_1_1StorageVolume.html

To me it seems like you're desperately set on using an API intended to get details about hardware when you really want to put a label on a directory-tree. 

How do you handle:
- more than one collection on the same disk?
- moving a collection from one disk to another?
- moving a collection to another linux install?
- sub-collections?
- fixing a collection from a broken disk?

Say you put a file: .digikamcollection containing some information about the collection like: owner, date of last access/additions, key for the digikamdb, etc.

This would solve a lot of problems and it is doesn't get in the way.

Additionally, I think it would be a good idea to more pro-actively try to match a physical collection to a collection in the database when a quick match (based on uuid or the file I propose) isn't possible.

I hope you can see that the bug (as I intended it) isn't resolved at all!

/Simon

Created attachment 39340 [details]
Hardware list provided by Solid

Please don't close this issue yet. I'm in a similar situation, and I tink this is a real problem.

I've experienced recently a disk failure. The disk contained my photo collections, but fortunately I had backup copy of all photos + the digikam database.

Now I've decided to build a RAID system on my desktop computer, to be even more protected against another eventual disk failure. So I bought 2 identical disks, set up a RAID array and created some LVM2 logical partitions over it. Finally, I restored my backup copy (in fact, a backup of whole /home containing /home/photos) and triedopened Digikam to check my collection, but digikam complained with similar message as the one of the fisrst reporter (Simon).

The 2 options offered were to do nothing and to consider the collection being in a removable storage. But this dialog should also offer the option to fix the drive's UUID stored in the database, as in my opinion the situation I face is not uncommon.

I'm also on kubuntu karmic, but I don't thing my HAL is broken, although I'm not an expert in this field and I can be wrong.

I attach the output of "solid-hardware", and explain a bit my setup, for the sake of clarity:

I've 2 identical, 1TB drives: /dev/sda and /dev/sdb. The two are formatted identically:

GPT partition table
  1 small "BIOS" partition (sda1, sdb1) to make room for GRUB2
  1 big partition (sda2, sdb2) to serve as a RAID volume

The created RAID (/dev/md1) is used as a LVM2 physical volume, and this volume is assigned to a "gv1" volume group. Finally, this volume group is divided into 3 logical partitions: root ("arrel" in my language), home and swap, mounted and used the obvious way.

The backup copy, previously stored in a LVM2 logical partition with UUID=5f5cb5e1-091b-4f05-a564-9bd29ab9c664 in the failed drive, now is restored in the new /home partition, with UUID=16013925-4625-4763-9246-22f1b80a65f2, and Digikam is complaining about that.

I think I'll be able to fix it editing the database (with some suitable tool) and change the old UUID with the new one, but obviously the average user won't, and so thisshould be considered a (data loss) bug.

I only see the 5f5fb... partition, no 16013... partition in the output. Look for "StorageAccess". So it seems that Solid does not understand the disk layout.
I dont know how to fix this.
In the dialog, maybe we can offer an option to switch to identification by filepath, to make things work then at least.

I have  the same issue. I moved my pictures from a ext4 on LUKS partition to an ext4 on LUKS on RAID1 partition.

After that move, digikam would complain about not finding the ols collections although they are still under the same absolute path and under the same relative path under the new partition.

Additionally when I create a new collection I have a warning message: "It is not possible on your system to identify the storage medium of this path".

ditto. i migrated to a new computer by copying over my  /home folder. this obviously resulted in a different uuid for the disk. my photos are currently inaccessible, and i'm trying to look up complicated steps to edit the sqlite db to find them again.

pretty terrible.

in fact, the solution suggested here:
http://ubuntuforums.org/showthread.php?t=1192655

does not work for me on ubuntu 10.10. i've tried every uuid my system has, and digikam still can't find my photos, which are nontheless still there in the same path they've always been -- /home/igor47/photos

is there an interim solution while this bug is fixed that will get me my data back?

The issues with LVM and RAID should be fixed by the fix for #273369. Not sure if that fixes the original problem though.

Simon, 

This file still valid using digiKam 2.4 ?

Gilles Caulier

Hi Gilles

it's very hard to reproduce this bug on a voluntary basis ;-)

I've had some issues upgrading to 2.1 I think, but I don't remember if I
had problems or what the outcome was. (Well, the outcome is that digikam
still has pictures, but I haven't been able to spend quality time with
digikam much, lately)

As for database problems, there's now another one, every time I start
digikam (2.1 and 2.4 now) I get an error that the path to mysqld isn't
set, but I'm not using mysql, so that shouldn't matter to me.

From the comments I see there were quite some a few other people with
the same type of problems, so perhaps someone else can confirm this in a
recent release.

Cheers

Simon

Do you experience this kind of problem with more recent digiKam release, as 4.2.0 ?

Gilles Caulier

I experience this problem with digikam 4.2 and right now I lost my 80.000 pics and tags it seems :-( after changing the filesystem from NTFS to ext4. The database is in my /home directory but the image colleactions reside on a second disk. 
Now I can see the configured collections in the config-dialogue of digikam but I can't edit them. Where is the path located ?  in the database only ?  And how to change that ? What if I want to move to another disk ...I would run in the same problem. 

My system ist Manjaro 0.8.10 and right now I cant get access to my pics , although i see my tags...but no pics...;

Ok I could solve the problem with the help of the sqlite browser modifying the uuid entries in the database. Now everything is working again. HolyJesus... pheww....

But I can't understand why that has to be done this unpracticable. That is NOT userfriendly :-)) Just editing the path insider the config dialogue would be so much better and doable by a "normal" user. Why rely on the UUID ?  Yeah I read it has to do with the KDE API blabla. But for a user that is totally impractical. The UUID is something I dont want to handle... ; I have path's and data. And I want to change that easily..;  If I will change my drives I will have the same problem again.. and again.. ;  Just a hint to think about in the settings. Why not implementing a switch to let the user decide HOW to handle to path to the collections ?   I thought LINUX is the system who does what tghe user wants ?  ;-) 

However:   thanks fo the great program... even if we have to fiddle with it...its superb ! :-)

Hm initially, UUID was needed to support removable media.
With hardwired disks, you can either change the UUID or the path.
If you change the mount path, the partition will be found via UUID and you dont notice it.
If you change the UUID and keep the path, there should be a short dialog asking you if this is still the same media etc. and all should be handled well if you press Ok.
If you change the pah and the UUID, it cannot be expected to be handled automatically.
If you have network shares, there will be only the mount path to work with, no UUID here.

I cannot tell you what went wrong on your setup.

Hmm.. couldnt remember correctly: it could be that I also changed the diskname slightly. But nothing changed on the path's for the pictures. So I think it would be very good to add the possibility to change the location of the collection in the setup menu for the case that it could not be found. And there are several reasons why that could happen:
- the user accidentally didnt click "yes" in the daialogue after a change of ONE parameter ( UUID or mount path) 
- the user changed several things like possibly me

ups... acidentally "saved changes" while i was writing.. :-/.  
So I mean adding an edit-field in the setup of the collection-location would be a real enhancement. Because right now you see the "old" collections and you cant even see the location nor can you change it !   Thats not so good. Normaly there should be the option to change the collection and/ or its location. My opinion ;-) 

Thanks for your comments Marcel and your hard work ! 
Mike

I just ran into an issue where, moving to a new laptop, I had a different disk layout. Instead of having /home mounted separately, it is just a directory in my root partition.

Because the database stored the paths relative to the mount point, I had a collection recorded as being at /alex/Photos, so Digikam didn't give me the option of using the new location. If went into the database, and changed it to /home/alex/Photos, and then Digikam allowed me to say that it had just moved to a local disk.

I think that local collections should either be stored differently (absolute paths or relative to the database file), or the option to say where a collection moved to should be given when it can't be found (so as not to lose metadata).

Simon,

Is the file still valid using digiKam 5.1.0?
Please test and provide feedback.

Git commit ceac0f1f71118e9b24316320ec1ab98237125f2b by Maik Qualmann.
Committed on 07/12/2018 at 20:48.
Pushed by mqualmann into branch 'master'.

add button to update an existing collection to the new drive or path
This satisfies a long-standing desire of many users to
easily adapt an existing collection to a new drive.
FIXED-IN: 6.0.0

M  +2    -1    NEWS
M  +1    -0    core/libs/database/collection/collectionmanager.h
M  +89   -3    core/libs/database/collection/collectionmanager_location.cpp
M  +7    -0    core/libs/database/coredb/coredb.cpp
M  +7    -0    core/libs/database/coredb/coredb.h
M  +290  -36   core/utilities/setup/collections/setupcollectionview.cpp
M  +17   -5    core/utilities/setup/collections/setupcollectionview.h

https://commits.kde.org/digikam/ceac0f1f71118e9b24316320ec1ab98237125f2b