For the config, this can be done by using KConfigGroup's read/writePathEntry. I dont see a problem here.

Inside the database there may be some complications due to the way we refer to a harddisk by uuid + path. I see the value of using $HOME though.

Hi,
sorry for the lengthy post, but I tried to be constructive and not only state
a problem, but propose a viable solution. What do you, developers, think?

This issue may also break the following use case (I actually need this!):
Alice and Bob use shared MySQL server for digikam database. The files
themselves are shared over NFS, but Alice mounts them on her computer as
/home/alice/Pictures and Bob on his computer as /mnt/photoes.

Other affected use cases may be thought of: DB & files on mobile HDD, backup
restoration, mount point move/rename, etc.

IMHO, DB should contain all file paths relative to each collection root,
e.g. use UUID as a first element of the name, e.g.
/mnt/photoes/album/file.jpg
becomes
a1a88989-5fb5-4621-b539-522f5aa02ff1/album/file.jpg

The question is where to store the paths for collection roots. For above
usecase, it seems logical to store them in local config.

There is a broader issue of guessing that different absolute file names refer
to the same object:

Alice configures her digikam and adds photos (DB records are created). Now Bob
wants to access the same files and DB entries. Although he knows path to the
collection, how can he know UUID? A possible solution: digikam writes uuid.txt
in a collection root with collection UUID, but I don't like it. What if
collection is read-only? What if Bob now uses his own DB?

Maybe, a better solution: allow to query collection list from DB and present
it to the user along with a descriptive name and probably a path hints (last
two are given by the user who already configured an access to the
collection). So collection settings tab can look like the following mockup.
ASCII tables and their headers are used just to signify the layout. Legend and
behavior are described in the next comment (presumably, comment #3 ).

---------------------------------------------------------------------------
Below are the locations of root albums used to store image files.
Write access is necessary to be able to edit images in these albums.
(Removable media and remote file systems are supported)

You have configured the following collections:
| Type  | Name         | Path                  | Action      |
|-------+--------------+-----------------------+-------------|
| [LOC] | Bob Photos   | /home/bob/Pictures    | [Del]       |

----

Other users of your database have configured the following collections.
To access these collections you will need to provide a path to image files
from your computer.

| Type  | Name         | Possible Path(s)      | Action      |
|-------+--------------+-----------------------+-------------|
| [LOC] | Alice Photos | /home/alice/Pictures  | [Add] [Del] |
|-------+--------------+-----------------------+-------------|
| [MED] | NIKON D90    | /media/NIKON D90      | [Add] [Del] |
|-------+--------------+-----------------------+-------------|
| [NET] | Archive      | /mnt/server/photos    | [Add] [Del] |
|       |              | /media/archive/photos |             |
|-------+--------------+-----------------------+-------------|

----

[Add new collection...]
---------------------------------------------------------------------------

cont. of comment #2

Legend for the ASCII mockup:
- [LOC] Icon for local collection + a tooltip:
       "Local Collection, directly stored on your computer hard disk"
- [MED] Icon for removable media collection + a tooltip:
       "Collection on Removable Media, such as CD, DVD and USB drives"
- [NET] Icon for network shares collection + a tooltip:
        "Collection on Network Shares, such as NFS and Samba/CIFS"
- [Add] add button (e.g. a standard green plus icon or the word "Add"),
        a tooltip: "Add this collection to your Digikam"
- [Del] delete button (e.g. a standard red X icon, or the word "Delete"),
        a tooltip: "Remove this collection from the database.
        This will not remove the files, but this collection will disappear
        from your Digikam and Digikam of *all other users* which use this
        database."
- [Add new collection...] a button with the obvious purpose

When there are no collections in the second table the whole table along with
the preceding description can be replaced by a simple (probably, grayed out)
message: "No other collections are configured in your database".

The interaction with the dialog.

[Add new collection...]
starts a dialog/wizard of the same name which allows user to specify:
1. Path
2. Type
3. Short name
After choosing a path, short name and type can be guessed by digikam, but
allowed to be changed.

[Add]
starts the same dialog/wizard as [Add new collection...], but short name is
not editable and type is preselected to [NET] or [MED] according to DB data.
A hint of paths at other user's machines is displayed. Possible addition: "try
this path" button near each item to choose the same path locally.
Anyway, after user chooses a local directory some sanity check may be done in
order to ensure consistency between DB and the chosen path.

[Del]
opens a confirmation dialog.
If collection to be deleted is locally bound and there are other DB users,
allow 3 options:
  1. Delete from this Digikam only
  2. Delete from this Digikam and Digikam of all other users (delete from DB)
  3. Cancel
If collection is not used locally, display options 2 & 3 only.
If DB is not shared, display options 1 & 3 only.

The above dialog mockup illustrates the following situation.
- Alice, Bob and Carol share the same DB.
- Alice has added her local ~/Pictures
- Alice or Carol has added removable card mounted at /media/NIKON D90
- Alice has added network share mounted at /mnt/server/photos
- Card has added the same network share mounted at his computer at
  /media/archive/photos
- Bob has added his local ~/Pictures
- Bob opened the said dialog

We must take care not to mix local hardware and network storage. Network storage is not well supported, and we dont have good support from the libraries to make this easy. It's more or less a hack.
When you say UUID you mean a hardware partition. You can share it by physically moving a USB hard disk, but it does not apply to network storage. So it's always more or less local. We need some different identifier for network storage, which we dont have atm, see above. 
I'm not sure you are aware that all normal collections are stored by UUID + relative path in the db? That's the problem with the original report here, it's not easily applicable depending on partition layout.

Be UUID I actually meant any unique string used to replace path to collection
in a stored file path. For example instead of
"/home/user/mycollection/album1/file.jpg"
one could store
"XXXX/album1/file.jpg"

Later (while fiddling with bug #272301 ) I discovered you made it even better
by storing relative path only ("/album1/file.jpg") along with a collection key
pointing to the relevant collection in AlbumRoots. Great!

However, this hasn't solved the problem for digikam thumbnails database. It
still stores the absolute paths. It's probably better to store relative path
there as well and make the client (digikam) prefix it by a collection
root. Since this information is stored in another database, it's probably
better to refer to it by some long identifier (e.g. random UUID) instead of
just a number (AlbumRoots primary key id). Being unique, AlbumRoots.identifier
may suit this. So a new AlbumRootsRef table may be added to thumbnails DB with
two columns only: int id, longtext reference. The later will hold
AlbumRoots.identifier or alike and the former may be referenced in FilePaths
table like it's done now in Albums table in the main DB.

But thumbnails DB is not the most important issue here. Currently AlbumRoots
allows only one path to collection root (specificPath). But two different
users may have the same collection under different paths. In DB layer this can
be handled this way for example:
- remove AlbumRoots.specificPath column
- add new table AlbumRootPaths with 3 columns
  - int albumRoot
    refers to AlbumRoots primary key
  - longtext user
    refers to specific digikam instance. This value is a single constant per
    client and is stored in local digikam config. May use UUID here too.
  - longtext specificPath
    path to collection root as accessed by specific digikam client

Now to access an album one should JOIN between Albums, AlbumRoots and
AlbumRootPaths WHERE AlbumRootPaths.user is equal to the string provided by
the current client.

BTW, this way we need not rely on "good" UUID supplied by KIO/OS/whatever as
long as a path does not change (as seen by the same client).

What is essential is that when digikam feels lost it should ask for a corrected root - not automatically remove all tags.
[Can it be "Scan for new items at startup (makes startup slower)" that causes this automatic tag deletion?]

Digikam should work on album restored from a backup to a different disk, currently it does not.

I mount my disks on label => the tree is the truth.
USB disks are nowadays also automatically mounted by label as /media/LABEL


Local/user mounted (NFS) disks are a difficult problem...
Those would require different album roots for different users...
(As would using private symbolic links in path to database)

I'm following this discussion with an uneasy feeling... Please try to avoid the mistake that the Amarok developers made, see bug 265567. There, the database with ratings and statistics for each song identifies each device by UUID. With the switch from HAL (KDE 4.4) to UDisks (KDE 4.6), each hard disk partition got a new UUID, which means that everyone loses all playlist statistics, ratings, ...

Yes Andreas, that's what I've written in comment #5!
Currently, digikam relies on KDE (Solid, if I'm not mistaken) to
supply volume id which must identify the volume. I propose to use such
volumeid as a hint at most and ask a user to supply correct local
path, probably relying on this hint. UUID I've mentioned is just a
random string *generated* by digikam itself.

You give an important example of volumeid change as a result of
OS/library update. This is one more reason to avoid relying on it.

@Ilia: The full path is only a fallback in the thumbs database. Primarily used is identifaction via file hash, with no path involved at all (and with 2.0, the file hash has been improved so as to be much more unique)

@Andreas: I heard of no problem with the KDE 4.6 release except for an uppercase/lowercase problem, so I assume there is no problem. We'd have tons of bugreports otherwise.

@Roger: If digikam has a collection that was known as "hard-wired" and does not find it, it will look for a different partition where the same path can be found and offer a dialog for confirmation. The default behavior is to hide collections which are not found (like on removable media). Problems arise if a collection is stored by mount path, because Solid/HAL/udev did not provide the necessary information. I've seen that a few times when HAL was unable to manage a complex logical volume layout.

Regarding the original request:
The config file is user data, so using KConfigGroup::read/writePathEntry with automatic $HOME expansion seems sensible.
Regarding the database, there's the case of a multi-user where photos are in a common dir. Now one adds photos in his home dir, if they were encoded as $HOME, digikam would think them gone with the next user. Yet such a kind of setup would be inherently broken. I could live with $HOME expansion as well.

But both changes would not be backwards compatible, so we'd do them only with the next non-backwards-compatible db change. (no such change is planned atm)

Git commit 9ed1a530b01ac4f94b98cef641eae56cc7e2067c by Marcel Wiesweg.
Committed on 21/07/2012 at 15:27.
Pushed by mwiesweg into branch 'master'.

Accept $HOME as part of the database file path in the config file

In contrast to substituting the home dir with $HOME by using writePathEntry,
this change is fully backwards compatible.

M  +10   -2    libs/database/core/databaseparameters.cpp

http://commits.kde.org/digikam/9ed1a530b01ac4f94b98cef641eae56cc7e2067c

*** Bug 317616 has been marked as a duplicate of this bug. ***

Will,

digiKam 4.11.0 is out.

This file still valid ?

Gilles Caulier

Confirming this bug exists in digiKam 4.6 and the issue has remained since 2009 when it was reported in https://bugs.kde.org/show_bug.cgi?id=215486

When a collection is copied to a new system/disk/partition which naturally has a different UUID, the database is unable to locate the images or thumbnails.

To rsolve the problem, the UUID and path must be adjusted by manually editing the digikam4.db AlbumRoots Table.

The bug is that, if digiKam will persist with this method of specifying the path with UUID in digiKam4.db, there should be a way to edit the path, including the UUID, from within digiKam settings. If such an option is added, it should be done so that it is easy for the novice user to browse to the album/collection and have digiKam handle the UUID lookup and digikam4.db edit behind the scenes.

Copying a collection to a new disk is a standard user operation for moving, upgrading, restoring from backups. These operations should not break digiKam to the point where it cannot be fixed from within the application.

Is the file still valid using digiKam 5.1.0?
Users are requested to test and provide necessary updates.

Yes this is still the same. There is no way to move a collection. The location of a collection is stored in the database, so it has to be manually adjusted in the database. Currently digikam sees that all files vanished, so deletes them in database and the only thing you can do is creating a new collection at the new location, obviously losing all information stored in the database. A solution within the current framework would be a function within digikam to move a collection.
In my opinion much better would be to take a different approach at storing collection locations: Keep them separate from the database either in the existing or a new configuration file. That means all the paths in the database should be relative to root and the path to root is stored in this configuration file. This allows for easily changing the location of a collection and even for two digikam instances to share a database (not simultaneously, of course, but if that is a worry, that should be solved by locking). A use case would be if data and database were shared over a network and both could be accessed on different computers.

digiKam 7.0.0 stable release is now published:

https://www.digikam.org/news/2020-07-19-7.0.0_release_announcement/

We need a fresh feedback on this file using this version.

Best Regards

Gilles Caulier

MAik,

No improvements done with last versions cannot help to solve this issue ?

Gilles

(In reply to Simon from comment #16)
> Yes this is still the same. There is no way to move a collection. The
> location of a collection is stored in the database, so it has to be manually
> adjusted in the database. Currently digikam sees that all files vanished, so
> deletes them in database and the only thing you can do is creating a new
> collection at the new location, obviously losing all information stored in
> the database. A solution within the current framework would be a function
> within digikam to move a collection.
> In my opinion much better would be to take a different approach at storing
> collection locations: Keep them separate from the database either in the
> existing or a new configuration file. That means all the paths in the
> database should be relative to root and the path to root is stored in this
> configuration file. This allows for easily changing the location of a
> collection and even for two digikam instances to share a database (not
> simultaneously, of course, but if that is a worry, that should be solved by
> locking). A use case would be if data and database were shared over a
> network and both could be accessed on different computers.

Has this been addressed yet? In particular, I'm interested in: "all the paths in the database should be relative to root and the path to root is stored in this configuration file" - for the purpose of sharing one database on a WIn+Linux dual boot machine, rather than needing to maintain double databases & double thumbnails for what is really the same photo collection. For other software, this is how it has always worked: I specify a base path, and as all other paths are relative, the same database can be used by both.

Edit: another common solution is to specify paths relative to the database location, i.e. ../../photos, which would also allow for the same.

In digiKam-8.0.0, additional base paths can be "attached" to a network collection in the collection settings. The respective base path that exists on the operating system is then used.

Maik

Ah, very cool!

So even if the images are technically stored in a shared partition of the internal local ssd, I should add it as "Collections on Network Shares"? (Meaning that if I have an existing Local Collection, I should remove that, & reestablish the same data as Collections on Network Shares?)

You can switch a local collection to a network collection in the collection settings without rescanning everything. A network collection can also be a local partition. The only difference is that with a network collection, the collection is identified by path and not by a drive UUID.

Maik

Cool - I didn't see that in v7 stable, perhaps it's a new option in v8.

I just found your other comment here https://bugs.kde.org/show_bug.cgi?id=456749#c4 - just curious, is the 8.0 feature simply a UI for appending a 2nd mountpoint query arg in the sqlite db? So I could effectively accomplish the exact same, while still using the v7 stable version? Or does v8 actually have other related fixes, so it's better to use v8 pre-release, even if it might be less stable?

Yes, the solution from Bug 456749 would also work with digiKam-7.x.x.

Maik

Apologies - I'm giving 8.0.0 a try, and I can't seem to find the option to move the existing collection from Local to Network. When adding a new Network collection from scratch, I do see the new "+" button, "Append a path to the collection." But I'm not sure how to "switch" my current Local connection to Network. If I remove Local & re-add the same path as Network, it does start a full re-scan (I backed up my previous db before trying).  I tried i.e. dragging that entry from "Local" to "Network" - just a guess - but that doesn't work. Could you specify where is the UI to switch a Local collection to Network?

Thanks again for all the replies

Sorry, nevermind, I found it. I thought the "refresh" icon was for refresh (it's the same icon as when you right-click an album & select "Refresh" - turns out that on this dialog, the icon means "switch location."

Got it & done, thanks!

Following Maik comment #2@ and later, i considerate this file as solved now. I close it.

Apologies if I shouldn't have reopened this (or reported it as a feature requested), however it seems that this still is a major usability issue.

Use case:
- Someone (like me) creates a Digikam database, spends lots of time editing it with careful backups.
- Migrates it to another disk (either because moving disks, or moving computers, or a failed disk, or views it on another computer).

The expected behavior is that everything still works.
Actual behavior is that database becomes corrupted, and I had to use a backup because I did not know how to tell digikam to "match up" files (comment #23). The instructions (comment #27) to hit the refresh-icon button next to the database to change its type were "mostly correct" but did not work, since it lost all my tags and seemingly corrupted the database. However I eventually got it to work (see far below).

People coming to this bug probably need a way to 'uncorrupt' the database and recover their tags, because they've likely run digikam thinking that nothing should go wrong, but it has. See comment #14 "Copying a collection to a new disk is a standard user operation for moving, upgrading, restoring from backups. These operations should not break digiKam to the point where it cannot be fixed from within the application." I believe some similar data loss caused me to give up on digikam a very very long time ago, without writing a bug for it.
People were seemingly confused http://digikam.185.s1.nabble.com/use-of-uuid-in-database-td3492022.html and it still seems to be causing problems for people today (because they eventually migrate their collection to new computers, because they don't keep their old computer forever).


Maybe someone could chime in how to recover from losing all tags by having digikam rescan the DB. However this is how I "recovered" the database from a backup once I realized something was going wrong:

# I had previously backed up the database
cp -r collection/digikam ~/temp/digikam-dir-backups/backup-17

# realize something going wrong
exit digikam
mv collection/digikam ~/temp/digikam-dir-backups/digikam-post-new-disk-corrupted
cp -r ~/temp/digikam-dir-backups/backup-17 collection/digikam
# THIS IS IMPORTANT: DO NOT mv YOUR BACKUP INTO PLACE THEN TRY TO EDIT IT; when recovering from a backup, copy FROM your backup (obviously)

# (then I changed the database type in Settings > Collections > "refresh" icon [should be changed to a text button with better symbology], but when this failed I noted the change in the db tables with sqlitebrowser, destroyed the db again, and recovered again from (e.g. backup-17), all for the express purpose to see what was going on; you don't have to do this as I noted the changes for you below...)

# edit the recovered-from-backup SQL database with your favorite tool
sqlitebrowser collection/digikam/digikam4.db


previous AlbumRoots table entries (before migration)

INSERT INTO "main"."AlbumRoots" 
("id", "label", "status", "type", "identifier", "specificPath", "caseSensitivity") 
VALUES 
('1', 'library', '0', '1', 'volumeid:?uuid=aeb623be-3509-47a4-b93f-2a4fcb5acad5&fileuuid=ab6XXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX', '/home/yourusername/path/to/collection', '2');

...should become (changing mountpath and fileuuid as appropriate)...:

INSERT INTO "main"."AlbumRoots" 
("id", "label", "status", "type", "identifier", "specificPath", "caseSensitivity") 
VALUES 
('1', 'library', '0', '3', 'networkshareid:?mountpath=/home/yourusername/path/to/collection&fileuuid=ab6XXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX', '/', '2');


In e.g. sqlitebrowser, you can edit the table cells individually. In summary, find the relevant row of the collection in the AlbumRoots table, then:
- change its .type from 1 to 3, and 
- change its .identifier from 'volumeid:?uuid=...&fileuuid=...' to 'networkshareid:?mountpath=/home/yourusername/path/to/collection&fileuuid=INSERT OLD FILEUUID' (not sure what file uuid is or is for)
- change its .specificPath to '/'

See if this works, let digikam load up and 'refresh' everything (not sure what steps should be taken to rescan database in e.g. maintenance), let it run overnight doing any upkeep until the CPU usage is low, then look if things seem to be working. Make a new backup snapshot.


To summarize, I think comment #6 said:
"What is essential is that when digikam feels lost it should ask for a corrected root - not automatically remove all tags.
[Can it be "Scan for new items at startup (makes startup slower)" that causes this automatic tag deletion?]
Digikam should work on album restored from a backup to a different disk, currently it does not."


The problem is that people expect the databases of type Local (is this the default? 'local' 100% seems like what you want to use, but is deceptive because it is 99% not what anyone wants to use and is a misnomer), and 99% of users have a ticking issue that will only bite them a few years down the road. The solution is to:

1) Not 'lose' or corrupt the entire collection (without any way to prompt the user to recover it) if the collection AlbumRoot merely seems moved (instead prompt the user about the problem with a description of the problem);
2) Create a third type of database, the default, called 'Default' or 'Local (v2)' or 'Default Local Path', which is just an alias to a network share with a local path, and prompt all users to migrate to it
3) Rename 'Local' to 'ByDiskUUID (obsolete)' and obsolesce it, unless there is a specific use case for disk UUID.
4) Force-migrate eveyone very carefully to the new Default (non-UUID) setup.
5) (Perhaps rename the Database Migration dialog to "Database sqlite<->MySQL Migration", or update the Database Migration dialog to handle the use case of "the AlbumRoot is not found; where is it?" and/or to recover the database (it's unclear if it's possible to migrate a database from /some/path to the same /some/path and if that would be a useful thing to do given the use of UUIDs). Currently it is unclear if that tool (which I had no idea exists) is for use before or after moving the database or what its purpose is, so maybe it should automatically show up when needed and tell the use what to do and how to use it. However upon closer inspection, it currently seems to only be for changing between sqlite and MySQL.)

When I read SQLiteBrowser in this context, I feel dizzy.
The update function in the collection settings does exactly what it's supposed to do. I often use it to test different scenarios or reproduce bugs.
We've already fully enabled the new file UUID for network drives. Not yet for local hard drives and removable media; there's still a fallback to the partition UUID.
There are various reasons for this, e.g., support for read-only media/partitions. A collection that isn't recognized won't be scanned. New hard drives that haven't been formatted are problematic, as they can contain the same partition UUID.
Image tags can also be lost with a new setup, e.g., if sidecars are used but reading from sidecars was forgotten.

Maik

(In reply to Maik Qualmann from comment #30)
> When I read SQLiteBrowser in this context, I feel dizzy.
> The update function in the collection settings does exactly what it's
> supposed to do. I often use it to test different scenarios or reproduce bugs.
> We've already fully enabled the new file UUID for network drives. Not yet
> for local hard drives and removable media; there's still a fallback to the
> partition UUID.
> There are various reasons for this, e.g., support for read-only
> media/partitions. A collection that isn't recognized won't be scanned. New
> hard drives that haven't been formatted are problematic, as they can contain
> the same partition UUID.
> Image tags can also be lost with a new setup, e.g., if sidecars are used but
> reading from sidecars was forgotten.
> 
> Maik

Maybe the update function worked, but the database was already corrupted because "digikam was started and somehow did a scan after an rsync"? All I know is that all tags disappeared and the Refresh icon into a Network DB didn't solve it, and most users would stop there and not know what to do.

I guess there are perhaps some use cases for UUID; it just seemed strange for users to be allowed to make that choice without numerous warnings, and incorrect nomenclature, and now it seems most users need to be rescued from this. I'm not quite sure how the internals of digikam work ("file UUID"; is that just a UUID of a file path?) so I can't comment. Though, perhaps if digikam stores its db by default in the collection, it could detect a copied/moved collection by noting its database was moved and no longer matches, and prompting.

All I was trying to say was that I notice a Critical problem that will punish 99% of regular users unless it is addressed, (nor mentioned in the docs or config UI or setup wizard?,) and for posterity posted the only way I know that can solve it with a backup (I have found no way to solve this without a backup unless there's a way to rescan the collection). I'll leave further discussion to you all. Cheers.

First, I'd clarify what you mean by "lost tags"? Did you lose tags assigned to images? Or did you generally lose tags in the tag tree?

The latter isn't possible at all, even if an empty or incorrect collection is scanned. In that case, something must have gone wrong when copying the database.

Maik

(Note to people about my previous suggestion: I am still experiencing database corruption with this method, so don't use it.)

(In reply to Maik Qualmann from comment #32)
> First, I'd clarify what you mean by "lost tags"? Did you lose tags assigned
> to images? Or did you generally lose tags in the tag tree?
> 
> The latter isn't possible at all, even if an empty or incorrect collection
> is scanned. In that case, something must have gone wrong when copying the
> database.
> 
> Maik

I don't have tags set to write back to files.
It seems my initial assessment was incorrect. I tried to replicate this with both the UUID backup (what would happen if you just rsync'd your collection over) and the backup where UUID was edited to be a local 'Network' share (what would happen if you rsync'd then edited before starting digikam). The results were both as follows:

- All pending image recognition tags such as pending face tags are lost.
- Some amount of tags remain in the tag tree (either some or all; I don't have the original numbers since digikam 8.6 seems to segfault when X-forwarded, but if I had to guess I'd say all of the tags are okay on the contents of digikam4.db).
- Digikam will start to Scan for New Files on its own, or you can stop/restart the process with Maintenance -> [X] All Albums -> [X] Scan for New Files.
- As progress goes from 0 to 100% on the scan, the tags in the tag tree decrease from x -> 0, for whatever initial value of x they had. e.g. if tags MyTag (260) had 260 images, and the scan takes 6-10 hours for example let's say 8hr, then after about 4hr there will be MyTag (130) images left in that tag, and eventually 0. At the end of the scan, after ensuring the scan is complete (progress bar on bottom), almost all tags will be gone.

Is this the same bug?
Is there some step-by-step instruction for changing hard disks with digikam?