SUMMARY If working with large collections, which are containing independent content, the only way to change a collection is, to remove one and add the other, then to rebuild the database ... This takes time and is almost impossible with large collections. STEPS TO REPRODUCE 1. Remove a collection from the settings dialog. 2. Add anothe one. 3. Rebuild the database. OBSERVED RESULT First at all, there is no description, how to rebuild the database ... or at least I didn't find it. With a large amount of pictures (100'000 or more) this takes days, even on a strong machine with 24 cores (5GHz), 128 GB of RAM, SSDs ... This has to be done everytime, one wants to work with a different collection. EXPECTED RESULT I just would like to unload a collection (or disable it ...) and load another one, including another database. (Similar to calibre ebook management, which allows just to select another database ...) The benefit would be: - smaller databases and image libraries - higher efficiency - higher speed to find and compare something SOFTWARE/OS VERSIONS Windows: Windows 10 Pro, 22H2, Build 19045.5247 macOS: (available in the Info Center app, or by running `kinfo` in a terminal window) Linux/KDE Plasma: KDE Plasma Version: KDE Frameworks Version: Qt Version: ADDITIONAL INFORMATION
A speed improvement on rebuilding the database can be done by installing the MariaDB server. The improvement is about factor 100 (means 100 times faster ;) - but is still to slow. It requires still over 1 hour to collect all the local images on my system. Then I'm not sure ... are all the tags and settings lost, I've done before? (I guess, I have to save it in the images to prevent this ... ???) What about faces data and similarities, groups?
Hi Daniel, I'm trying to understand why you need to unload a collection to load a new one. Are the collections on different drives? What's the reason for unloading a collection? Cheers, Mike
Just to compare: Loading the local images (~150'000) took ~1:30 (h:min). Loading images from network (~5'300) took ~2:10. So it's important to have the ability to load/unload a database, just only to prevent from such a waste of time.
(In reply to Michael Miller from comment #2) > Hi Daniel, > I'm trying to understand why you need to unload a collection to load a new > one. Are the collections on different drives? What's the reason for > unloading a collection? > > Cheers, > Mike Hi Mike One reason is the work on different projects. It's important for me to separate the projects to prevent from mixed data. As examples: Face recognization data must not cross the projects boundaries. Also different projects may have the same tags, but on a search only tags of a specific project have to be used. Checking for similarities must not cross the the projects boundaries. One thing is the work on image libraries of different clients ... it's almost like that. Im working on image libraries of my wife, on my own photo libraries, on a stable diffusion image library and on some voluntary web-projects. So I've got different sources - my own with a large amount of pictures on different local drives, the others on USB-drives or on the network with more or less speed (between 20MB/s and 1GB/s).
As you may have noticed, I've created two databases now: - 1 for a drive - 1 for a network drive I did this the following way: 1. I've created a base directory for settings called ".database" ... 2. then a subdirectory called "local", 3. and a subdirectory called "network" Remark: The base directory could be anywhere, maybe on "AppData" ... or just in the user profile ... Now I've created settings for the database for each subdirectory and did let the build-up start and complete. I now only need to select the subdirectory for a database in the settings to change the database. The database will then be reloaded from this subdirectory, even the settings for the collections will be restored. This works verywell! Best Regards, Daniel
> Hi Mike > > One reason is the work on different projects. It's important for me to > separate the projects to prevent from mixed data. > As examples: > Face recognization data must not cross the projects boundaries. > Also different projects may have the same tags, but on a search only tags of > a specific project have to be used. > Checking for similarities must not cross the the projects boundaries. > > One thing is the work on image libraries of different clients ... it's > almost like that. > Im working on image libraries of my wife, on my own photo libraries, on a > stable diffusion image library and on some voluntary web-projects. So I've > got different sources - my own with a large amount of pictures on different > local drives, the others on USB-drives or on the network with more or less > speed (between 20MB/s and 1GB/s). Hi Daniel, Thanks. I think I understand why you want to unload the collection to get segmentation between projects. One idea is to use different DBs for each collection. Instead of re-using the same digiKam DB, SQLite makes it easy to have multiple DBs. This would eliminate the need to rescan the collection whenever you swapped them. You could create a "digiKamDB" folder on your local drive, and then create subfolders for each instance of the digiKam DBs that correspond to a collection. You could also create DBs on the same media as the collection, but performance may suffer depending on the speed of the media. In Preferences->Database you can select a different digiKam DB. Keep in mind that if you use SQLite, you'll need to store the DBs in different folders for each collection. To swap DBs, you would select a new database in Preferences->Database, and then restart digiKam. I know it's not an ideal solution, but it's something you can do now while the rest of the team and I consider better options. It will definitely be faster than rescanning large collections, and it will give you the collection segmentation you're looking for. One caveat is that when you remount the removable storage it needs to be in exactly the same path it was before. Cheers, Mike
(In reply to Daniel from comment #5) > As you may have noticed, I've created two databases now: > - 1 for a drive > - 1 for a network drive > > I did this the following way: > 1. I've created a base directory for settings called ".database" ... > 2. then a subdirectory called "local", > 3. and a subdirectory called "network" > > Remark: The base directory could be anywhere, maybe on "AppData" ... or just > in the user profile ... > > Now I've created settings for the database for each subdirectory and did let > the build-up start and complete. > I now only need to select the subdirectory for a database in the settings to > change the database. > The database will then be reloaded from this subdirectory, even the settings > for the collections will be restored. > > This works verywell! > > Best Regards, Daniel Looks like we had the same idea at the same time! Let me know if there is anything else I can do to help. Cheers, Mike
So this description could be forwarded to the maintainer of the help documentation. Maybe it could be adapted to the database settings section ... Best Regards, Daniel
Hi Mike (In reply to Michael Miller from comment #7) > ... > Looks like we had the same idea at the same time! > > Let me know if there is anything else I can do to help. > > Cheers, > Mike Indeed ... It's fun. Best Regards
(In reply to Daniel from comment #8) > So this description could be forwarded to the maintainer of the help > documentation. > Maybe it could be adapted to the database settings section ... > > Best Regards, Daniel It just so happens that I'm working on updating the Face Engine and Autotags documentation for 8.6.0 right now. I'll add this to my list of documentation to update. Cheers, Mike
digiKam has command line parameters. You can specify a different directory for SQLite where the databases are located. Or better still, the config parameter, with this you can use different configurations and databases. Simply create different shortcuts on the desktop and specify the appropriate parameters. Maik
(In reply to Maik Qualmann from comment #11) > digiKam has command line parameters. You can specify a different directory > for SQLite where the databases are located. Or better still, the config > parameter, with this you can use different configurations and databases. > Simply create different shortcuts on the desktop and specify the appropriate > parameters. > > Maik Thank you, Maik This improves the usability. In the same command file I can check, whether the image libraries on removable drives do exist and whether the drive is mounted, otherwise I can throw a message instead of starting digiKam. Best Regards.
(In reply to Daniel from comment #12) > (In reply to Maik Qualmann from comment #11) > > digiKam has command line parameters. You can specify a different directory > > for SQLite where the databases are located. Or better still, the config > > parameter, with this you can use different configurations and databases. > > ... First I didn't realize, that the --database parameter is for SQLite only (like you wrote ... sometimes people are blind ...), but with the --config it works for mySQL too. I've copied the digikamrc file from AppData\local to the database directory and started with this copy of the configfile there. There are just some questions for the first time, digiKam is starting with it, but the next time everthing works fine. Many thanks and best Regards
You can also start with an empty config file, but the file must exist, then digiKam runs the first setup assistant and you can configure completely new. Maik
Seems that there is a problem on using Mysql internal. If I'm starting digiKam it will always use the last opened configuration, independently I'm using the config of a different directory, where I first did another config and database. Say, I've made a directory stucture like: \.dk \.dk\local \.dk\archive Then I met an emty file for each directory with the same name, like local_rc or archive_rc I've reated shortcuts with 1. E:\Programme\digiKam\digikam.exe --config E:\.dk\lokal\local_rc 2. E:\Programme\digiKam\digikam.exe --config E:\.dk\archive\archive_rc After starting digiKam I had to configure it, then it created a directory .mysql.digikam in each of the .dk subdirectories. Last I've used the shortcut 2. (archive_rc) Then I've started shortcut 1. (archive_rc) - but the collections from the last session were loaded, instead of the other ones. Opening settings it displayed the wrong collections for config 1. (local_rc). After correction and the long, long waiting time, even in config 2. the collections were altered, after restarting digikam with this config (archive_rc).
MariaDB Version 11.6 (x64)
> Last I've used the shortcut 2. (archive_rc) > Then I've started shortcut 1. (archive_rc) - but the collections from the > last session were loaded, instead of the other ones. > Opening settings it displayed the wrong collections for config 1. (local_rc). Mistyped: Last I've used the shortcut 2. (archive_rc) Then I've started shortcut 1. (local_rc) - but the collections from the last session were loaded, instead of the other ones. Opening settings it displayed the wrong collections for config 1. (local_rc).
I have recreated this under Linux and cannot reproduce any problem. Perhaps you started the next digiKam session too "quickly" and the internal server had not yet shut down. We use port 3307 for the internal MySQL server if it is free. Unfortunately, there is no connection via a WinSocket or shared memory with Qt under Windows. Please also use the current pre-release version of digiKam-8.6.0. Other versions may not be able to terminate completely and get stuck in the task manager. https://files.kde.org/digikam/ Maik
I've tried to migrate the database from Mysql Internal to Mysql Server. The copy process was sucessful, but the database setting did not change to Mysql Server. After a manual change in the settings, the config file archive_rc did still contain the old settings and was not updated. So I've stopped digiKam, removed the .mysql.digikam directory and restarted it again. Then I was able to do the correct settings and the file archive_rc was updated. I did let digiKam newly build the second config and database from scratch (local_rc). Now everthing works pretty well. If I'm starting digiKam with any configuration, the correct collections will be loaded. There's just one thing, I've realized: In Task-Manager background processes were left open for digiKam and ExifTool Perl Launcher after termination of the software, as many as I've started it before (1, 2, 3, 4, ...) without termination.
Well I'm able now to start digiKam in multiple instances with different databases. About the background processes in Task-Manager I found, that this has to do with the system latency. In fact, there are only just as many processes open, as instances of the software.
So for me this is solved. I don't see any differences in speed between Mysql Internal and Server ... and Server seems to be more reliable. I also can define databases with different names, so there is more freedom. Even the config seems to be easier, because it's files can be laid into the same directory. Only for new databases I've got a little bit more to do, because the databases have to be created first and will not be created automatically.
Thanks for help!