Bug 267131 - SETUP : digiKam config and data cannot be moved, absolute paths embedded in database and digikamrc
Summary: SETUP : digiKam config and data cannot be moved, absolute paths embedded in d...
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Setup-Database (show other bugs)
Version: 5.1.0
Platform: Unlisted Binaries Linux
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-02-25 09:03 UTC by Will Stephenson
Modified: 2023-04-20 14:47 UTC (History)
10 users (show)

See Also:
Latest Commit:
Version Fixed In: 8.1.0


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Will Stephenson 2011-02-25 09:03:21 UTC
Version:           1.8.0 (using Devel) 
OS:                Linux

I moved my KDE configs and my photos from /home/foo to /home/bar.  Most KDE programs allow this without hacking

Digikam records the path to the database absolutely in digikamrc and the paths to the albums in the AlbumRoots table of its database.

I had to hand-edit digikamrc and the sqlite database to allow digikam to start up without a database not found error, and then to find the moved albums.

Reproducible: Didn't try




Recommend encoding paths involving current $HOME as $HOME instead of the aboslute path, then expand these, allowing config and data to be migrated between homes.
Comment 1 Marcel Wiesweg 2011-04-18 22:05:48 UTC
For the config, this can be done by using KConfigGroup's read/writePathEntry. I dont see a problem here.

Inside the database there may be some complications due to the way we refer to a harddisk by uuid + path. I see the value of using $HOME though.
Comment 2 Ilia K. 2011-05-02 04:57:28 UTC
Hi,
sorry for the lengthy post, but I tried to be constructive and not only state
a problem, but propose a viable solution. What do you, developers, think?

This issue may also break the following use case (I actually need this!):
Alice and Bob use shared MySQL server for digikam database. The files
themselves are shared over NFS, but Alice mounts them on her computer as
/home/alice/Pictures and Bob on his computer as /mnt/photoes.

Other affected use cases may be thought of: DB & files on mobile HDD, backup
restoration, mount point move/rename, etc.

IMHO, DB should contain all file paths relative to each collection root,
e.g. use UUID as a first element of the name, e.g.
/mnt/photoes/album/file.jpg
becomes
a1a88989-5fb5-4621-b539-522f5aa02ff1/album/file.jpg

The question is where to store the paths for collection roots. For above
usecase, it seems logical to store them in local config.

There is a broader issue of guessing that different absolute file names refer
to the same object:

Alice configures her digikam and adds photos (DB records are created). Now Bob
wants to access the same files and DB entries. Although he knows path to the
collection, how can he know UUID? A possible solution: digikam writes uuid.txt
in a collection root with collection UUID, but I don't like it. What if
collection is read-only? What if Bob now uses his own DB?

Maybe, a better solution: allow to query collection list from DB and present
it to the user along with a descriptive name and probably a path hints (last
two are given by the user who already configured an access to the
collection). So collection settings tab can look like the following mockup.
ASCII tables and their headers are used just to signify the layout. Legend and
behavior are described in the next comment (presumably, comment #3 ).

---------------------------------------------------------------------------
Below are the locations of root albums used to store image files.
Write access is necessary to be able to edit images in these albums.
(Removable media and remote file systems are supported)

You have configured the following collections:
| Type  | Name         | Path                  | Action      |
|-------+--------------+-----------------------+-------------|
| [LOC] | Bob Photos   | /home/bob/Pictures    | [Del]       |

----

Other users of your database have configured the following collections.
To access these collections you will need to provide a path to image files
from your computer.

| Type  | Name         | Possible Path(s)      | Action      |
|-------+--------------+-----------------------+-------------|
| [LOC] | Alice Photos | /home/alice/Pictures  | [Add] [Del] |
|-------+--------------+-----------------------+-------------|
| [MED] | NIKON D90    | /media/NIKON D90      | [Add] [Del] |
|-------+--------------+-----------------------+-------------|
| [NET] | Archive      | /mnt/server/photos    | [Add] [Del] |
|       |              | /media/archive/photos |             |
|-------+--------------+-----------------------+-------------|

----

[Add new collection...]
---------------------------------------------------------------------------
Comment 3 Ilia K. 2011-05-02 04:58:21 UTC
cont. of comment #2

Legend for the ASCII mockup:
- [LOC] Icon for local collection + a tooltip:
       "Local Collection, directly stored on your computer hard disk"
- [MED] Icon for removable media collection + a tooltip:
       "Collection on Removable Media, such as CD, DVD and USB drives"
- [NET] Icon for network shares collection + a tooltip:
        "Collection on Network Shares, such as NFS and Samba/CIFS"
- [Add] add button (e.g. a standard green plus icon or the word "Add"),
        a tooltip: "Add this collection to your Digikam"
- [Del] delete button (e.g. a standard red X icon, or the word "Delete"),
        a tooltip: "Remove this collection from the database.
        This will not remove the files, but this collection will disappear
        from your Digikam and Digikam of *all other users* which use this
        database."
- [Add new collection...] a button with the obvious purpose

When there are no collections in the second table the whole table along with
the preceding description can be replaced by a simple (probably, grayed out)
message: "No other collections are configured in your database".

The interaction with the dialog.

[Add new collection...]
starts a dialog/wizard of the same name which allows user to specify:
1. Path
2. Type
3. Short name
After choosing a path, short name and type can be guessed by digikam, but
allowed to be changed.

[Add]
starts the same dialog/wizard as [Add new collection...], but short name is
not editable and type is preselected to [NET] or [MED] according to DB data.
A hint of paths at other user's machines is displayed. Possible addition: "try
this path" button near each item to choose the same path locally.
Anyway, after user chooses a local directory some sanity check may be done in
order to ensure consistency between DB and the chosen path.

[Del]
opens a confirmation dialog.
If collection to be deleted is locally bound and there are other DB users,
allow 3 options:
  1. Delete from this Digikam only
  2. Delete from this Digikam and Digikam of all other users (delete from DB)
  3. Cancel
If collection is not used locally, display options 2 & 3 only.
If DB is not shared, display options 1 & 3 only.

The above dialog mockup illustrates the following situation.
- Alice, Bob and Carol share the same DB.
- Alice has added her local ~/Pictures
- Alice or Carol has added removable card mounted at /media/NIKON D90
- Alice has added network share mounted at /mnt/server/photos
- Card has added the same network share mounted at his computer at
  /media/archive/photos
- Bob has added his local ~/Pictures
- Bob opened the said dialog
Comment 4 Marcel Wiesweg 2011-05-03 22:12:25 UTC
We must take care not to mix local hardware and network storage. Network storage is not well supported, and we dont have good support from the libraries to make this easy. It's more or less a hack.
When you say UUID you mean a hardware partition. You can share it by physically moving a USB hard disk, but it does not apply to network storage. So it's always more or less local. We need some different identifier for network storage, which we dont have atm, see above. 
I'm not sure you are aware that all normal collections are stored by UUID + relative path in the db? That's the problem with the original report here, it's not easily applicable depending on partition layout.
Comment 5 Ilia K. 2011-05-04 03:14:10 UTC
Be UUID I actually meant any unique string used to replace path to collection
in a stored file path. For example instead of
"/home/user/mycollection/album1/file.jpg"
one could store
"XXXX/album1/file.jpg"

Later (while fiddling with bug #272301 ) I discovered you made it even better
by storing relative path only ("/album1/file.jpg") along with a collection key
pointing to the relevant collection in AlbumRoots. Great!

However, this hasn't solved the problem for digikam thumbnails database. It
still stores the absolute paths. It's probably better to store relative path
there as well and make the client (digikam) prefix it by a collection
root. Since this information is stored in another database, it's probably
better to refer to it by some long identifier (e.g. random UUID) instead of
just a number (AlbumRoots primary key id). Being unique, AlbumRoots.identifier
may suit this. So a new AlbumRootsRef table may be added to thumbnails DB with
two columns only: int id, longtext reference. The later will hold
AlbumRoots.identifier or alike and the former may be referenced in FilePaths
table like it's done now in Albums table in the main DB.

But thumbnails DB is not the most important issue here. Currently AlbumRoots
allows only one path to collection root (specificPath). But two different
users may have the same collection under different paths. In DB layer this can
be handled this way for example:
- remove AlbumRoots.specificPath column
- add new table AlbumRootPaths with 3 columns
  - int albumRoot
    refers to AlbumRoots primary key
  - longtext user
    refers to specific digikam instance. This value is a single constant per
    client and is stored in local digikam config. May use UUID here too.
  - longtext specificPath
    path to collection root as accessed by specific digikam client

Now to access an album one should JOIN between Albums, AlbumRoots and
AlbumRootPaths WHERE AlbumRootPaths.user is equal to the string provided by
the current client.

BTW, this way we need not rely on "good" UUID supplied by KIO/OS/whatever as
long as a path does not change (as seen by the same client).
Comment 6 Roger Larsson 2011-05-04 09:46:59 UTC
What is essential is that when digikam feels lost it should ask for a corrected root - not automatically remove all tags.
[Can it be "Scan for new items at startup (makes startup slower)" that causes this automatic tag deletion?]

Digikam should work on album restored from a backup to a different disk, currently it does not.

I mount my disks on label => the tree is the truth.
USB disks are nowadays also automatically mounted by label as /media/LABEL


Local/user mounted (NFS) disks are a difficult problem...
Those would require different album roots for different users...
(As would using private symbolic links in path to database)
Comment 7 Andreas K. Huettel 2011-05-04 12:32:55 UTC
I'm following this discussion with an uneasy feeling... Please try to avoid the mistake that the Amarok developers made, see bug 265567. There, the database with ratings and statistics for each song identifies each device by UUID. With the switch from HAL (KDE 4.4) to UDisks (KDE 4.6), each hard disk partition got a new UUID, which means that everyone loses all playlist statistics, ratings, ...
Comment 8 Ilia K. 2011-05-04 15:27:51 UTC
Yes Andreas, that's what I've written in comment #5!
Currently, digikam relies on KDE (Solid, if I'm not mistaken) to
supply volume id which must identify the volume. I propose to use such
volumeid as a hint at most and ask a user to supply correct local
path, probably relying on this hint. UUID I've mentioned is just a
random string *generated* by digikam itself.

You give an important example of volumeid change as a result of
OS/library update. This is one more reason to avoid relying on it.
Comment 9 Marcel Wiesweg 2011-05-07 21:51:36 UTC
@Ilia: The full path is only a fallback in the thumbs database. Primarily used is identifaction via file hash, with no path involved at all (and with 2.0, the file hash has been improved so as to be much more unique)

@Andreas: I heard of no problem with the KDE 4.6 release except for an uppercase/lowercase problem, so I assume there is no problem. We'd have tons of bugreports otherwise.

@Roger: If digikam has a collection that was known as "hard-wired" and does not find it, it will look for a different partition where the same path can be found and offer a dialog for confirmation. The default behavior is to hide collections which are not found (like on removable media). Problems arise if a collection is stored by mount path, because Solid/HAL/udev did not provide the necessary information. I've seen that a few times when HAL was unable to manage a complex logical volume layout.
Comment 10 Marcel Wiesweg 2012-07-21 13:23:37 UTC
Regarding the original request:
The config file is user data, so using KConfigGroup::read/writePathEntry with automatic $HOME expansion seems sensible.
Regarding the database, there's the case of a multi-user where photos are in a common dir. Now one adds photos in his home dir, if they were encoded as $HOME, digikam would think them gone with the next user. Yet such a kind of setup would be inherently broken. I could live with $HOME expansion as well.

But both changes would not be backwards compatible, so we'd do them only with the next non-backwards-compatible db change. (no such change is planned atm)
Comment 11 Marcel Wiesweg 2012-07-21 13:31:21 UTC
Git commit 9ed1a530b01ac4f94b98cef641eae56cc7e2067c by Marcel Wiesweg.
Committed on 21/07/2012 at 15:27.
Pushed by mwiesweg into branch 'master'.

Accept $HOME as part of the database file path in the config file

In contrast to substituting the home dir with $HOME by using writePathEntry,
this change is fully backwards compatible.

M  +10   -2    libs/database/core/databaseparameters.cpp

http://commits.kde.org/digikam/9ed1a530b01ac4f94b98cef641eae56cc7e2067c
Comment 12 Teemu Rytilahti 2013-04-01 14:20:07 UTC
*** Bug 317616 has been marked as a duplicate of this bug. ***
Comment 13 caulier.gilles 2015-06-26 13:59:00 UTC
Will,

digiKam 4.11.0 is out.

This file still valid ?

Gilles Caulier
Comment 14 Net Codger 2015-07-17 16:55:02 UTC
Confirming this bug exists in digiKam 4.6 and the issue has remained since 2009 when it was reported in https://bugs.kde.org/show_bug.cgi?id=215486

When a collection is copied to a new system/disk/partition which naturally has a different UUID, the database is unable to locate the images or thumbnails.

To rsolve the problem, the UUID and path must be adjusted by manually editing the digikam4.db AlbumRoots Table.

The bug is that, if digiKam will persist with this method of specifying the path with UUID in digiKam4.db, there should be a way to edit the path, including the UUID, from within digiKam settings. If such an option is added, it should be done so that it is easy for the novice user to browse to the album/collection and have digiKam handle the UUID lookup and digikam4.db edit behind the scenes.

Copying a collection to a new disk is a standard user operation for moving, upgrading, restoring from backups. These operations should not break digiKam to the point where it cannot be fixed from within the application.
Comment 15 swatilodha27 2016-08-08 14:02:19 UTC
Is the file still valid using digiKam 5.1.0?
Users are requested to test and provide necessary updates.
Comment 16 Simon 2016-08-08 15:05:33 UTC
Yes this is still the same. There is no way to move a collection. The location of a collection is stored in the database, so it has to be manually adjusted in the database. Currently digikam sees that all files vanished, so deletes them in database and the only thing you can do is creating a new collection at the new location, obviously losing all information stored in the database. A solution within the current framework would be a function within digikam to move a collection.
In my opinion much better would be to take a different approach at storing collection locations: Keep them separate from the database either in the existing or a new configuration file. That means all the paths in the database should be relative to root and the path to root is stored in this configuration file. This allows for easily changing the location of a collection and even for two digikam instances to share a database (not simultaneously, of course, but if that is a worry, that should be solved by locking). A use case would be if data and database were shared over a network and both could be accessed on different computers.
Comment 17 caulier.gilles 2020-08-04 11:45:36 UTC
digiKam 7.0.0 stable release is now published:

https://www.digikam.org/news/2020-07-19-7.0.0_release_announcement/

We need a fresh feedback on this file using this version.

Best Regards

Gilles Caulier
Comment 18 caulier.gilles 2022-01-10 16:18:19 UTC
MAik,

No improvements done with last versions cannot help to solve this issue ?

Gilles
Comment 19 spiesant 2022-12-17 06:42:43 UTC
(In reply to Simon from comment #16)
> Yes this is still the same. There is no way to move a collection. The
> location of a collection is stored in the database, so it has to be manually
> adjusted in the database. Currently digikam sees that all files vanished, so
> deletes them in database and the only thing you can do is creating a new
> collection at the new location, obviously losing all information stored in
> the database. A solution within the current framework would be a function
> within digikam to move a collection.
> In my opinion much better would be to take a different approach at storing
> collection locations: Keep them separate from the database either in the
> existing or a new configuration file. That means all the paths in the
> database should be relative to root and the path to root is stored in this
> configuration file. This allows for easily changing the location of a
> collection and even for two digikam instances to share a database (not
> simultaneously, of course, but if that is a worry, that should be solved by
> locking). A use case would be if data and database were shared over a
> network and both could be accessed on different computers.

Has this been addressed yet? In particular, I'm interested in: "all the paths in the database should be relative to root and the path to root is stored in this configuration file" - for the purpose of sharing one database on a WIn+Linux dual boot machine, rather than needing to maintain double databases & double thumbnails for what is really the same photo collection. For other software, this is how it has always worked: I specify a base path, and as all other paths are relative, the same database can be used by both.
Comment 20 spiesant 2022-12-17 06:44:19 UTC
Edit: another common solution is to specify paths relative to the database location, i.e. ../../photos, which would also allow for the same.
Comment 21 Maik Qualmann 2022-12-17 06:55:21 UTC
In digiKam-8.0.0, additional base paths can be "attached" to a network collection in the collection settings. The respective base path that exists on the operating system is then used.

Maik
Comment 22 spiesant 2022-12-17 07:05:21 UTC
Ah, very cool!

So even if the images are technically stored in a shared partition of the internal local ssd, I should add it as "Collections on Network Shares"? (Meaning that if I have an existing Local Collection, I should remove that, & reestablish the same data as Collections on Network Shares?)
Comment 23 Maik Qualmann 2022-12-17 07:40:43 UTC
You can switch a local collection to a network collection in the collection settings without rescanning everything. A network collection can also be a local partition. The only difference is that with a network collection, the collection is identified by path and not by a drive UUID.

Maik
Comment 24 spiesant 2022-12-17 07:49:48 UTC
Cool - I didn't see that in v7 stable, perhaps it's a new option in v8.

I just found your other comment here https://bugs.kde.org/show_bug.cgi?id=456749#c4 - just curious, is the 8.0 feature simply a UI for appending a 2nd mountpoint query arg in the sqlite db? So I could effectively accomplish the exact same, while still using the v7 stable version? Or does v8 actually have other related fixes, so it's better to use v8 pre-release, even if it might be less stable?
Comment 25 Maik Qualmann 2022-12-17 09:00:17 UTC
Yes, the solution from Bug 456749 would also work with digiKam-7.x.x.

Maik
Comment 26 spiesant 2022-12-17 17:14:44 UTC
Apologies - I'm giving 8.0.0 a try, and I can't seem to find the option to move the existing collection from Local to Network. When adding a new Network collection from scratch, I do see the new "+" button, "Append a path to the collection." But I'm not sure how to "switch" my current Local connection to Network. If I remove Local & re-add the same path as Network, it does start a full re-scan (I backed up my previous db before trying).  I tried i.e. dragging that entry from "Local" to "Network" - just a guess - but that doesn't work. Could you specify where is the UI to switch a Local collection to Network?

Thanks again for all the replies
Comment 27 spiesant 2022-12-17 17:24:22 UTC
Sorry, nevermind, I found it. I thought the "refresh" icon was for refresh (it's the same icon as when you right-click an album & select "Refresh" - turns out that on this dialog, the icon means "switch location."

Got it & done, thanks!
Comment 28 caulier.gilles 2023-04-20 14:47:00 UTC
Following Maik comment #2@ and later, i considerate this file as solved now. I close it.