Bug 261277 - Windows and Linux Frontend using the same database
Summary: Windows and Linux Frontend using the same database
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Database-Multiusers (show other bugs)
Version: 1.7.0
Platform: Gentoo Packages Linux
: NOR wishlist
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-12-26 09:56 UTC by Tobias Kaminsky
Modified: 2017-07-25 16:49 UTC (History)
7 users (show)

See Also:
Latest Commit:
Version Fixed In: 2.3.0


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Tobias Kaminsky 2010-12-26 09:56:12 UTC
Version:           unspecified (using KDE 4.5.90) 
OS:                Linux

I have one database and three frontends. Frontend#2 is on Windows.
Therefore I have two entries in AlbumRoots:
4 fotos 0 3 networkshareid:?mountpath=%2Fmnt%2Fraid%2Ffotos /
5 fotos 0 3 networkshareid:?mountpath=Y:%2F /

All albums are now stored twice:
372     4     /unsortiert/Neuer Ordner     2010-07-11     NULL     NULL
    NULL
605     5     /unsortiert/Neuer Ordner     2010-07-11     NULL     NULL
    NULL

As you can see 4/5 is the AlbumRoots.id.
The problem is that, if I add a tag to one image on Windows the image is
related to album.id=605.
Linux does not recognize the tag as the same image is stored in
album.id=372.

(I have posted this on the mailing-list already, but I think a Bug-Report is better as one can apply patches, etc.)

Thank you
Tobi

Reproducible: Always
Comment 1 Anw 2011-02-07 21:02:41 UTC
Same here : I used digikam on Linux, with mysql backend.
When trying to join windows, I'm getting the following error : 
"Your locale has changed since this album was last opened. Old locale: UTF-8, new locale: System. If you have recently changed your locale, you need not be concerned. Please note that if you switched to a locale that does not support some of the filenames in your collection, these files may no longer be found in the collection. If you are sure that you want to continue, click 'Yes'. Otherwise, click 'No' and correct your locale setting before restarting digiKam."

I didn't try clicking "yes" to not ruing my existing albums... It would be really great if we could share same albums betweetn linux and windows.
Comment 2 Marcel Wiesweg 2011-02-19 19:05:15 UTC
Git commit 73fcc4ffb58dfdadd9bcec35487515c1eaad8ca5 by Marcel Wiesweg.
Committed on 19/02/2011 at 19:03.
Pushed by mwiesweg into branch 'development/2.0'.

Support testing of multiple mount paths for a network share

CCBUG: 261277

M  +19   -11   libs/database/collectionmanager.cpp     

http://commits.kde.org/digikam/73fcc4ffb58dfdadd9bcec35487515c1eaad8ca5
Comment 3 Marcel Wiesweg 2011-02-19 19:08:41 UTC
The idea is, as discussed on the mailing list, to allow to have multiple mountpath query items in the URL:
networkshareid:?mountpath=%2Fmnt%2Fraid%2Ffotos?mountpath=Y:%2F

No UI atm.
I did not test it, missing a good cross-platform testing environment. So any feedback is welcome.
Comment 4 Frank Hommes 2011-03-31 00:02:15 UTC
Since it fits better here I copied the content of a mail to the mailinglist:

At the moment (sorry, just using 1.9) the complete path is stored for
all files in mysql database.
I'm using mysql because there are more people with access to my
pictures. The path within the base directory is always the same but the
absolut path in the filesystem is sometimes different.
Example:
/home/freanki/win/pics/2010/freanki/testing/testpic.jpg
/home/directory/pics/2010/freanki/testing/testpic.jpg
z:\pics\2010\freanki\testing\testpic.jpg

This could be resolved with a basepath (stored local or with different
base path in mysql) and relativ path.

1 base path:/home/freanki/win/
2 base path:/home/directory/
3 base path:z:\ (with the addition that all / have to be converted to \
since it's windows)

And all pictures would be stored like this:
pics/2010/freanki/testing/testpic.jpg

This would also save some bandwith and storage...
Comment 5 Tobias Kaminsky 2011-10-19 08:11:19 UTC
(In reply to comment #3)
> The idea is, as discussed on the mailing list, to allow to have multiple
> mountpath query items in the URL:
> networkshareid:?mountpath=%2Fmnt%2Fraid%2Ffotos?mountpath=Y:%2F
> 
> No UI atm.
> I did not test it, missing a good cross-platform testing environment. So any
> feedback is welcome.

Hi,

I have tried this with two linux machines with digikam-2.2.0:
Computer A: /mnt/raid/fotos/ (local RAID-system)
Computer B: /mnt/mediacenter/fotos (mounted via nfs)

This is my entry in AlbumRoots:
networkshareid:?mountpath=%2Fmnt%2Fraid%2Ffotos?mountpath=%2Fmnt%2Fmediacenter%2Ffotos

But this does not work. No computer shows this album, but networkshareid:?mountpath=%2Fmnt%2Fraid%2Ffotos is working on Computer A.

What is wrong?

Thank you!
Tobi
Comment 6 Marcel Wiesweg 2011-10-19 16:55:11 UTC
Is the album root type set to network? It's the fourth field in the table and should be 3
Comment 7 Tobias Kaminsky 2011-10-19 17:55:08 UTC
(In reply to comment #6)
> Is the album root type set to network? It's the fourth field in the table and
> should be 3

id 	label 	status 	type 	identifier 	specificPath
1 	fotos	0 	3 	networkshareid:?mountpath=%2Fmnt%2Fraid%2Ffotos?mo...	/
2 	test	0 	3 	networkshareid:?mountpath=%2Fmnt%2Fraid%2Ffotos	/

Yes, it is. id=1 does not work, but id=2 does work.
Comment 8 Tobias Kaminsky 2011-10-20 14:38:18 UTC
return url.allQueryItemValues("mountpath"); (collectionmanager.cpp:460)
does not work properly.

I have changed this to 

return url.toString().split("?mountpath=",  QString::SkipEmptyParts);

and it seems to work.
Comment 9 caulier.gilles 2011-10-28 17:56:20 UTC
Ananta,

This fix from #8 is valid for you under Windows ?

Gilles Caulier
Comment 10 Ananta Palani 2011-10-29 10:32:32 UTC
Does this fix only apply with mysql back-end or with sqlite as well?
Comment 11 Tobias Kaminsky 2011-10-29 10:37:58 UTC
(In reply to comment #10)
> Does this fix only apply with mysql back-end or with sqlite as well?

I have only tried it with mysql.
Dabase Settings also say:
Note: a remote file system, such as NFS, cannot be used here.

So I think that sqlite is only for local usage.
If you want to use digikam on multiple computer, you should setup one pc as master with all pictures and a mysql-server on it.
Then the other machines connect directly to this mysql server and can mount via samba/nfs the directory with the albums.

Tobi
Comment 12 caulier.gilles 2011-10-29 11:41:10 UTC
Tobias,

You can use share collection with sqlite, but DB file must still on local computer and cannot be shared as well.

Mysql is more powerful and versatile of course.

Gilles Caulier
Comment 13 Marcel Wiesweg 2011-10-29 12:36:45 UTC
Sorry, the URL formatting I gave is wrong: The second query item is appended with a "&", the "?" is only appended once before all query items.
Interestingly, QUrl does not percent-encode the slashes at all:

"networkshareid:?mountpath=/mnt/test1&mountpath=/mnt/test2"

This test works:

    QUrl url;
    url.setScheme("networkshareid");
    url.addQueryItem("mountpath", "/mnt/test1");
    url.addQueryItem("mountpath", "/mnt/test2");
    QByteArray urlString = url.toEncoded();
    QUrl url2 = QUrl::fromEncoded(urlString);
    qDebug() << url2.toEncoded() 
             << url.toString();
    qDebug() << url2.allQueryItemValues("mountpath");

Output

"networkshareid:?mountpath=/mnt/test1&mountpath=/mnt/test2" "networkshareid:?mountpath=/mnt/test1&mountpath=/mnt/test2" 
("/mnt/test1", "/mnt/test2")
Comment 14 Tobias Kaminsky 2011-10-29 12:55:08 UTC
Great.
Is this also in Digikam 2.2.0 for Windows?
If so, then I will try it with this setup:
Linux (Master)
Linux: connects via NFS
Windows: connects via Samba

Thank you
Tobi
Comment 15 Tobias Kaminsky 2011-10-30 14:35:29 UTC
(In reply to comment #14)
> Great.
> Is this also in Digikam 2.2.0 for Windows?
> If so, then I will try it with this setup:
> Linux (Master)
> Linux: connects via NFS
> Windows: connects via Samba
> 
> Thank you
> Tobi

It works without a problem, but on windows it says that the old locale is utf-8 and the new is "system". But as far as I see there are no problems.

Tobi
Comment 16 Ananta Palani 2011-11-02 15:30:02 UTC
(In reply to comment #14)
> Is this also in Digikam 2.2.0 for Windows?

Only the changes made prior to comment #4 are included in 2.2.0 for Windows.


(In reply to comment #9)
> This fix from #8 is valid for you under Windows ?

I will try recompiling soon and let you know. It looks like it would work just fine.


(In reply to comment #13)
Marcel, what is the expected behavior if digiKam can access multiple mountpaths for a single networkshareid? Will it ignore all the other mountpaths after the first working mountpath it finds?

Also, how is the behavior of digiKam different with 'Local collections' and 'Collections on Network Shares' if I am able to add a local collection to either? In other words, why is there a distinction between the two at all? Couldn't there be just be a list of 'collections' and for each collection we have can have multiple mountpaths (i.e. replace 'Local collections' and 'Collections on Network Shares' with a single 'Collections')? Maybe in Linux the behavior is different.

-Ananta
Comment 17 Ananta Palani 2011-11-02 15:33:26 UTC
(In reply to comment #13)
Also, will a sigle thumbnail be shared for the same image between all mountpoints? (ex. 3 mount points, but only 1 thumbnail created).
Comment 18 Marcel Wiesweg 2011-11-02 21:02:50 UTC
> Marcel, what is the expected behavior if digiKam can access multiple mountpaths
> for a single networkshareid? Will it ignore all the other mountpaths after the
> first working mountpath it finds?

Yes

> Also, how is the behavior of digiKam different with 'Local collections' and
> 'Collections on Network Shares' if I am able to add a local collection to
> either? In other words, why is there a distinction between the two at all?
> Couldn't there be just be a list of 'collections' and for each collection we
> have can have multiple mountpaths (i.e. replace 'Local collections' and
> 'Collections on Network Shares' with a single 'Collections')? Maybe in Linux
> the behavior is different.

There is at the moment no good way for us to detect if a specific path is a mounted network share (there's statfs on Linux, something different on Windows, neither implemented, no cross-platform KDE libs support).

So for network collections, we just see if the specified folder is empty, then the collection is unavailable. It's really accessed by mount path only. Local collections are tied to a storage medium (harddisk partition)

> Also, will a sigle thumbnail be shared for the same image between all
> mountpoints? (ex. 3 mount points, but only 1 thumbnail created).

Yes, because thumbnails are preferably referenced by a file hash
Comment 19 Marcel Wiesweg 2011-11-05 13:43:41 UTC
Ok, backend solution seems to work. Not sure if and how this should be presented in a GUI.
Comment 20 julien.t43+kde 2012-10-10 15:45:55 UTC
Just as a follow-up of the following thread http://mail.kde.org/pipermail/digikam-users/2012-September/016862.html
I tried today to share my collections (Linux, external disk, fat32 partition - recently switched from ext4 to fat32 for sharing as sadly it is still the only "universal" filesystem recognized by nearly any system) with a macos macbook.

Constraints: because of fat32, I had to move my db files to alternative locations (instead of Images/) as thumbnails db is far above 2GB as both db have to stay together. It seems there is no way currently to make it splitted (by size or by period of time yearly/monthly/...)

Need
* change db path in ~/.kde/share/config/digikamrc or ~/Library/Preferences/KDE/share/config/digikamrc (both Database Name and Database Name Thumbnails: if the second is different from the first, digikam will rewrite it with the same previous pah)
* change root path of Images in database. for sqlite
$ sqlite3 ~/path/to/digikam4.db
SQLite version 3.7.14 2012-09-03 15:42:36
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .headers on
sqlite> select * from AlbumRoots;
id|label|status|type|identifier|specificPath
1|Images|0|1|volumeid:?uuid=1e369250-3d4f-40b3-b65b-7797264c712b|/path1/to/Images
sqlite> UPDATE AlbumRoots SET identifier='volumeid:?path=/new/path2/to/Images', specificPath='/' WHERE id=1

after I started digikam, rebuild thumbnails and it seems working fine.
Still hope performances will not be too bad with fat32 and not too many problems in case of crash (fat32 nothing against journaling and else of others)

Other reference
http://comments.gmane.org/gmane.comp.kde.digikam.user/18828
https://bugs.kde.org/show_bug.cgi?id=175923
Comment 21 caulier.gilles 2012-10-10 16:02:46 UTC
NTFS 3G is fully support by Linux in RW mode and also under OSX last version or at least you need to install linux driver adapted for OSX (it's free)

NTFS is journalized and better than FAT 32 about long file naming support...

Gilles Caulier
Comment 22 julien.t43+kde 2012-10-11 04:31:51 UTC
Sorry but, for most OS, it's "fully supported" with additional drivers.
As currently a traveler, I would say that in many situations, you only rely on default install and can't/don't want to install applications and especially drivers.

Linux is the only OS which almost support everything. Most unix rely on fuse/ntfs-3g and only have native read support (and experimental/not recommended write). See MacOS, *BSD
http://en.wikipedia.org/wiki/Comparison_of_BSD_operating_systems#Technical_information
http://www.openbsd.org/cgi-bin/man.cgi?query=mount_ntfs
http://forums.macrumors.com/showthread.php?t=1308755
http://en.wikipedia.org/wiki/NTFS#Interoperability

And if additional components are possible, Ext2/3 would be probably a better choice.
http://en.wikipedia.org/wiki/Ext4#Compatibility_with_Windows_and_Macintosh

For now, I would say that there is a noticeable performance hit using fat32 for images
Comment 23 Tex 2013-03-15 14:39:10 UTC
I think this is a great but unfortunately probably little known solution (a front end interface would be nice).  However in real life, it's not very useful for me because of this related bug:

https://bugs.kde.org/show_bug.cgi?id=302131

It's too bad, because digikam has the potential to be a multiplatform NAS client to let the whole family (or business) share pictures and tags throughout the house.  All the real work is done.