Bug 267131

For the config, this can be done by using KConfigGroup's read/writePathEntry. I dont see a problem here.

Inside the database there may be some complications due to the way we refer to a harddisk by uuid + path. I see the value of using $HOME though.

Hi,
sorry for the lengthy post, but I tried to be constructive and not only state
a problem, but propose a viable solution. What do you, developers, think?

This issue may also break the following use case (I actually need this!):
Alice and Bob use shared MySQL server for digikam database. The files
themselves are shared over NFS, but Alice mounts them on her computer as
/home/alice/Pictures and Bob on his computer as /mnt/photoes.

Other affected use cases may be thought of: DB & files on mobile HDD, backup
restoration, mount point move/rename, etc.

IMHO, DB should contain all file paths relative to each collection root,
e.g. use UUID as a first element of the name, e.g.
/mnt/photoes/album/file.jpg
becomes
a1a88989-5fb5-4621-b539-522f5aa02ff1/album/file.jpg

The question is where to store the paths for collection roots. For above
usecase, it seems logical to store them in local config.

There is a broader issue of guessing that different absolute file names refer
to the same object:

Alice configures her digikam and adds photos (DB records are created). Now Bob
wants to access the same files and DB entries. Although he knows path to the
collection, how can he know UUID? A possible solution: digikam writes uuid.txt
in a collection root with collection UUID, but I don't like it. What if
collection is read-only? What if Bob now uses his own DB?

Maybe, a better solution: allow to query collection list from DB and present
it to the user along with a descriptive name and probably a path hints (last
two are given by the user who already configured an access to the
collection). So collection settings tab can look like the following mockup.
ASCII tables and their headers are used just to signify the layout. Legend and
behavior are described in the next comment (presumably, comment #3 ).

---------------------------------------------------------------------------
Below are the locations of root albums used to store image files.
Write access is necessary to be able to edit images in these albums.
(Removable media and remote file systems are supported)

You have configured the following collections:
| Type  | Name         | Path                  | Action      |
|-------+--------------+-----------------------+-------------|
| [LOC] | Bob Photos   | /home/bob/Pictures    | [Del]       |

----

Other users of your database have configured the following collections.
To access these collections you will need to provide a path to image files
from your computer.

| Type  | Name         | Possible Path(s)      | Action      |
|-------+--------------+-----------------------+-------------|
| [LOC] | Alice Photos | /home/alice/Pictures  | [Add] [Del] |
|-------+--------------+-----------------------+-------------|
| [MED] | NIKON D90    | /media/NIKON D90      | [Add] [Del] |
|-------+--------------+-----------------------+-------------|
| [NET] | Archive      | /mnt/server/photos    | [Add] [Del] |
|       |              | /media/archive/photos |             |
|-------+--------------+-----------------------+-------------|

----

[Add new collection...]
---------------------------------------------------------------------------

cont. of comment #2

Legend for the ASCII mockup:
- [LOC] Icon for local collection + a tooltip:
       "Local Collection, directly stored on your computer hard disk"
- [MED] Icon for removable media collection + a tooltip:
       "Collection on Removable Media, such as CD, DVD and USB drives"
- [NET] Icon for network shares collection + a tooltip:
        "Collection on Network Shares, such as NFS and Samba/CIFS"
- [Add] add button (e.g. a standard green plus icon or the word "Add"),
        a tooltip: "Add this collection to your Digikam"
- [Del] delete button (e.g. a standard red X icon, or the word "Delete"),
        a tooltip: "Remove this collection from the database.
        This will not remove the files, but this collection will disappear
        from your Digikam and Digikam of *all other users* which use this
        database."
- [Add new collection...] a button with the obvious purpose

When there are no collections in the second table the whole table along with
the preceding description can be replaced by a simple (probably, grayed out)
message: "No other collections are configured in your database".

The interaction with the dialog.

[Add new collection...]
starts a dialog/wizard of the same name which allows user to specify:
1. Path
2. Type
3. Short name
After choosing a path, short name and type can be guessed by digikam, but
allowed to be changed.

[Add]
starts the same dialog/wizard as [Add new collection...], but short name is
not editable and type is preselected to [NET] or [MED] according to DB data.
A hint of paths at other user's machines is displayed. Possible addition: "try
this path" button near each item to choose the same path locally.
Anyway, after user chooses a local directory some sanity check may be done in
order to ensure consistency between DB and the chosen path.

[Del]
opens a confirmation dialog.
If collection to be deleted is locally bound and there are other DB users,
allow 3 options:
  1. Delete from this Digikam only
  2. Delete from this Digikam and Digikam of all other users (delete from DB)
  3. Cancel
If collection is not used locally, display options 2 & 3 only.
If DB is not shared, display options 1 & 3 only.

The above dialog mockup illustrates the following situation.
- Alice, Bob and Carol share the same DB.
- Alice has added her local ~/Pictures
- Alice or Carol has added removable card mounted at /media/NIKON D90
- Alice has added network share mounted at /mnt/server/photos
- Card has added the same network share mounted at his computer at
  /media/archive/photos
- Bob has added his local ~/Pictures
- Bob opened the said dialog

We must take care not to mix local hardware and network storage. Network storage is not well supported, and we dont have good support from the libraries to make this easy. It's more or less a hack.
When you say UUID you mean a hardware partition. You can share it by physically moving a USB hard disk, but it does not apply to network storage. So it's always more or less local. We need some different identifier for network storage, which we dont have atm, see above. 
I'm not sure you are aware that all normal collections are stored by UUID + relative path in the db? That's the problem with the original report here, it's not easily applicable depending on partition layout.

Be UUID I actually meant any unique string used to replace path to collection
in a stored file path. For example instead of
"/home/user/mycollection/album1/file.jpg"
one could store
"XXXX/album1/file.jpg"

Later (while fiddling with bug #272301 ) I discovered you made it even better
by storing relative path only ("/album1/file.jpg") along with a collection key
pointing to the relevant collection in AlbumRoots. Great!

However, this hasn't solved the problem for digikam thumbnails database. It
still stores the absolute paths. It's probably better to store relative path
there as well and make the client (digikam) prefix it by a collection
root. Since this information is stored in another database, it's probably
better to refer to it by some long identifier (e.g. random UUID) instead of
just a number (AlbumRoots primary key id). Being unique, AlbumRoots.identifier
may suit this. So a new AlbumRootsRef table may be added to thumbnails DB with
two columns only: int id, longtext reference. The later will hold
AlbumRoots.identifier or alike and the former may be referenced in FilePaths
table like it's done now in Albums table in the main DB.

But thumbnails DB is not the most important issue here. Currently AlbumRoots
allows only one path to collection root (specificPath). But two different
users may have the same collection under different paths. In DB layer this can
be handled this way for example:
- remove AlbumRoots.specificPath column
- add new table AlbumRootPaths with 3 columns
  - int albumRoot
    refers to AlbumRoots primary key
  - longtext user
    refers to specific digikam instance. This value is a single constant per
    client and is stored in local digikam config. May use UUID here too.
  - longtext specificPath
    path to collection root as accessed by specific digikam client

Now to access an album one should JOIN between Albums, AlbumRoots and
AlbumRootPaths WHERE AlbumRootPaths.user is equal to the string provided by
the current client.

BTW, this way we need not rely on "good" UUID supplied by KIO/OS/whatever as
long as a path does not change (as seen by the same client).

What is essential is that when digikam feels lost it should ask for a corrected root - not automatically remove all tags.
[Can it be "Scan for new items at startup (makes startup slower)" that causes this automatic tag deletion?]

Digikam should work on album restored from a backup to a different disk, currently it does not.

I mount my disks on label => the tree is the truth.
USB disks are nowadays also automatically mounted by label as /media/LABEL


Local/user mounted (NFS) disks are a difficult problem...
Those would require different album roots for different users...
(As would using private symbolic links in path to database)

I'm following this discussion with an uneasy feeling... Please try to avoid the mistake that the Amarok developers made, see bug 265567. There, the database with ratings and statistics for each song identifies each device by UUID. With the switch from HAL (KDE 4.4) to UDisks (KDE 4.6), each hard disk partition got a new UUID, which means that everyone loses all playlist statistics, ratings, ...

Yes Andreas, that's what I've written in comment #5!
Currently, digikam relies on KDE (Solid, if I'm not mistaken) to
supply volume id which must identify the volume. I propose to use such
volumeid as a hint at most and ask a user to supply correct local
path, probably relying on this hint. UUID I've mentioned is just a
random string *generated* by digikam itself.

You give an important example of volumeid change as a result of
OS/library update. This is one more reason to avoid relying on it.

@Ilia: The full path is only a fallback in the thumbs database. Primarily used is identifaction via file hash, with no path involved at all (and with 2.0, the file hash has been improved so as to be much more unique)

@Andreas: I heard of no problem with the KDE 4.6 release except for an uppercase/lowercase problem, so I assume there is no problem. We'd have tons of bugreports otherwise.

@Roger: If digikam has a collection that was known as "hard-wired" and does not find it, it will look for a different partition where the same path can be found and offer a dialog for confirmation. The default behavior is to hide collections which are not found (like on removable media). Problems arise if a collection is stored by mount path, because Solid/HAL/udev did not provide the necessary information. I've seen that a few times when HAL was unable to manage a complex logical volume layout.

Regarding the original request:
The config file is user data, so using KConfigGroup::read/writePathEntry with automatic $HOME expansion seems sensible.
Regarding the database, there's the case of a multi-user where photos are in a common dir. Now one adds photos in his home dir, if they were encoded as $HOME, digikam would think them gone with the next user. Yet such a kind of setup would be inherently broken. I could live with $HOME expansion as well.

But both changes would not be backwards compatible, so we'd do them only with the next non-backwards-compatible db change. (no such change is planned atm)

Git commit 9ed1a530b01ac4f94b98cef641eae56cc7e2067c by Marcel Wiesweg.
Committed on 21/07/2012 at 15:27.
Pushed by mwiesweg into branch 'master'.

Accept $HOME as part of the database file path in the config file

In contrast to substituting the home dir with $HOME by using writePathEntry,
this change is fully backwards compatible.

M  +10   -2    libs/database/core/databaseparameters.cpp

http://commits.kde.org/digikam/9ed1a530b01ac4f94b98cef641eae56cc7e2067c

*** Bug 317616 has been marked as a duplicate of this bug. ***

Will,

digiKam 4.11.0 is out.

This file still valid ?

Gilles Caulier

Confirming this bug exists in digiKam 4.6 and the issue has remained since 2009 when it was reported in https://bugs.kde.org/show_bug.cgi?id=215486

When a collection is copied to a new system/disk/partition which naturally has a different UUID, the database is unable to locate the images or thumbnails.

To rsolve the problem, the UUID and path must be adjusted by manually editing the digikam4.db AlbumRoots Table.

The bug is that, if digiKam will persist with this method of specifying the path with UUID in digiKam4.db, there should be a way to edit the path, including the UUID, from within digiKam settings. If such an option is added, it should be done so that it is easy for the novice user to browse to the album/collection and have digiKam handle the UUID lookup and digikam4.db edit behind the scenes.

Copying a collection to a new disk is a standard user operation for moving, upgrading, restoring from backups. These operations should not break digiKam to the point where it cannot be fixed from within the application.

Is the file still valid using digiKam 5.1.0?
Users are requested to test and provide necessary updates.

Yes this is still the same. There is no way to move a collection. The location of a collection is stored in the database, so it has to be manually adjusted in the database. Currently digikam sees that all files vanished, so deletes them in database and the only thing you can do is creating a new collection at the new location, obviously losing all information stored in the database. A solution within the current framework would be a function within digikam to move a collection.
In my opinion much better would be to take a different approach at storing collection locations: Keep them separate from the database either in the existing or a new configuration file. That means all the paths in the database should be relative to root and the path to root is stored in this configuration file. This allows for easily changing the location of a collection and even for two digikam instances to share a database (not simultaneously, of course, but if that is a worry, that should be solved by locking). A use case would be if data and database were shared over a network and both could be accessed on different computers.

digiKam 7.0.0 stable release is now published:

https://www.digikam.org/news/2020-07-19-7.0.0_release_announcement/

We need a fresh feedback on this file using this version.

Best Regards

Gilles Caulier

MAik,

No improvements done with last versions cannot help to solve this issue ?

Gilles

(In reply to Simon from comment #16)
> Yes this is still the same. There is no way to move a collection. The
> location of a collection is stored in the database, so it has to be manually
> adjusted in the database. Currently digikam sees that all files vanished, so
> deletes them in database and the only thing you can do is creating a new
> collection at the new location, obviously losing all information stored in
> the database. A solution within the current framework would be a function
> within digikam to move a collection.
> In my opinion much better would be to take a different approach at storing
> collection locations: Keep them separate from the database either in the
> existing or a new configuration file. That means all the paths in the
> database should be relative to root and the path to root is stored in this
> configuration file. This allows for easily changing the location of a
> collection and even for two digikam instances to share a database (not
> simultaneously, of course, but if that is a worry, that should be solved by
> locking). A use case would be if data and database were shared over a
> network and both could be accessed on different computers.

Has this been addressed yet? In particular, I'm interested in: "all the paths in the database should be relative to root and the path to root is stored in this configuration file" - for the purpose of sharing one database on a WIn+Linux dual boot machine, rather than needing to maintain double databases & double thumbnails for what is really the same photo collection. For other software, this is how it has always worked: I specify a base path, and as all other paths are relative, the same database can be used by both.

Edit: another common solution is to specify paths relative to the database location, i.e. ../../photos, which would also allow for the same.

In digiKam-8.0.0, additional base paths can be "attached" to a network collection in the collection settings. The respective base path that exists on the operating system is then used.

Maik

Ah, very cool!

So even if the images are technically stored in a shared partition of the internal local ssd, I should add it as "Collections on Network Shares"? (Meaning that if I have an existing Local Collection, I should remove that, & reestablish the same data as Collections on Network Shares?)

You can switch a local collection to a network collection in the collection settings without rescanning everything. A network collection can also be a local partition. The only difference is that with a network collection, the collection is identified by path and not by a drive UUID.

Maik

Cool - I didn't see that in v7 stable, perhaps it's a new option in v8.

I just found your other comment here https://bugs.kde.org/show_bug.cgi?id=456749#c4 - just curious, is the 8.0 feature simply a UI for appending a 2nd mountpoint query arg in the sqlite db? So I could effectively accomplish the exact same, while still using the v7 stable version? Or does v8 actually have other related fixes, so it's better to use v8 pre-release, even if it might be less stable?

Yes, the solution from Bug 456749 would also work with digiKam-7.x.x.

Maik

Apologies - I'm giving 8.0.0 a try, and I can't seem to find the option to move the existing collection from Local to Network. When adding a new Network collection from scratch, I do see the new "+" button, "Append a path to the collection." But I'm not sure how to "switch" my current Local connection to Network. If I remove Local & re-add the same path as Network, it does start a full re-scan (I backed up my previous db before trying).  I tried i.e. dragging that entry from "Local" to "Network" - just a guess - but that doesn't work. Could you specify where is the UI to switch a Local collection to Network?

Thanks again for all the replies

Sorry, nevermind, I found it. I thought the "refresh" icon was for refresh (it's the same icon as when you right-click an album & select "Refresh" - turns out that on this dialog, the icon means "switch location."

Got it & done, thanks!

Following Maik comment #2@ and later, i considerate this file as solved now. I close it.