Version: 0.10.0 (using KDE 4.2.2) OS: Linux Installed from: Ubuntu Packages With the new multiple collections feature in Digikam 0.10.0, I was able to add quite a few DVDs to my digikam collections. However, I can only list the pictures on them if I insert the DVD _before_ starting digikam. This is not very useful, since I might not know where to find the picture before inserting the DVD. It would be great if all the search options were available for offline collections. Maybe Digikam could store the metadata (including thumbnails) in the local db so they can be used to perform searches, and then prompt the user to insert the removable media to see the pictures (e.g. "Please insert "DVD3-2009" to see this picture").
We have all info in the local db, but not the thumbnails. The freedesktop standard that is used for storing thumbnails is not caring for removable media, it stores by file path only. Though for now storing by filepath may be enough for a wide range of use cases. So yes this is possible. When inserting a DVD digikam should recognize it even when running, but maybe something is broken there. That is a bug.
Maybe some visual indication for albums and/or individual images would be helpful when they are not physically present (eg. some red border etc.). Also any action which would require the real image need to be either disabled or recognized (eg, moving, renaming, editing etc.) And yes, this would be a great feature....
Another solution could be creation of some .hidden hierarchy inside of main root album where thumbnails for off-line images would be stored. Max sized thumbnail in JPG format would be ca. 5kB so it wouldn't be hard on disk space. #2 Arnd - red border is very strong signal. New Qt4 List Model View Marcel is working on could use small overlayed CD icon (borrowed from FotoStation and probably other software supporting off-line collections).
It will be interresting to know how other pro photo-management store thumbnails, especially with remote media. there use database ? gilles
FotoStation uses database with thumbnails embedded in it. Same for ACDSee as far as I know. The same also for non-pro apps like XnView.
In these database, do you know which file format is use to store thumbs ? JPEG, or PNG ? Which DB file size ? I think if we store thumb in DB, it will become huge speedly... Marcel, what do you think about ? Perhaps we can create a second DB only dedicated to store thumbs and forget now ~./thumbnails. I think to have a better support of removable media is very important for the future. This will solve backup issues... Gilles
I suppose they use JPEG. FotoStation doesn't even supports PNG. Don't remember FotoStation database size and related numbers but for my home machine: .thumbnails/large - 13000 almost all of them are coming from digiKam - total size: 1.1GB. After compression to JPEG quality 80, subsampling 2x1 - 147MB. There is quality difference, especially after upscaling, but there are gains. Note however that my collection isn't very big. When we go into 50,000 - 100,000 collections db could become monstrous. This rings true for comparable size of FotoStation db - they recommend 2GB of RAM... Current digiKam database with for those images: 17MB
If we choose this path, let's use a second database for this. There is no need to bloat the main database, which contains valuable data and should be easy to backup. We should also critically reconsider what are the gains of a thumnail database file versus simple storage as files in the file system (be it according to the current spec or at a different place)
To Marcel, I'm agree to separate DB : one for thumbs, another one for metadata. The gain will be to use digiKam with remote media. Of course we lost compatibility with opendesktop paper, but it's really important here ? Gwenview and Dolphin/konqueror already support it. digiKam work out of simple desktop image management scope now. For me it's not a problem, we are over : photo-management. With a dedicated thumbs DB, we can only save one thumbnail size by item : 256x256. No need to store another size as with OpenDesktop paper. As Mik said, we can use JPEG with no compression value (to have the best image quality) or better, another one based on wavelet compression to optimize DB size. Candidate are : JPEG2000 : GPL : slow (compression and decompression) DJvu Photo : GPL : not yet tested. OpenEXR : GPL : very slow (unadapted here) HDPhoto : not GPL : Fast, but pattented by M$. PGF : GPL : Not tested but very promissing : http://en.wikipedia.org/wiki/Progressive_Graphics_File The advantage to use wavelet instead JPEG : a better compression ratio than JPEG : 5-10 ! Gilles
I gave examples for `convert -quality 80` which is quite lossy compression. With transformation I really don't see differences between JPG and PNG `convert -quality 90 -sampling-format 1x1` result is reduction of size from 1.1GB -> 247MB.
Yes, JPEG with 80 compression is fine, but... using Wavelet compression will decrease drastically thumbs data size. It's very important. Of course, compression and decompression must be fast, at least as JPEG I will make some tests with PGF format. It sound great... Gilles
I found a comparison paper about PGF image format : http://www.xeraina.ch/download/pgf_facts.pdf Incredible no ? I think we need to use this format to host thumbs in digiKam : space optimization a and fast load/saving. Gilles
Marcel, What's about to use a DB to host thumbnails cache ? As we use multi-threading with current thumbs interface, if we use libsqlite to host binary blob of thumbs will be compatible ? In other words, it safe to use libsqlite with separate threads ? Look there : http://www.sqlite.org/faq.html#q6 Gilles
For Informations: PGF can host customized meta-data as byte-array. It's hosted just after header of file and before image data. There are methods to set and get user data. There is no no data size restriction. We can host what we want in this blob, as Exif, IPTC or XMP (preferred), or all if necessary. This is want mean that it's safe to use this format for photo purpose too... Gilles
The Qt SQL implementation requires that a database connection is not used across threads; therefore we create one connection per thread. It requires (at least I think so) not to access a database concurrently from two threads; so we use a global mutex (by creating DatabaseAccess() each time). So we dont require any special thread safety from sqlite.
Created attachment 34097 [details] PGF compression tests Some test results with PGF format using pgf command line tool: * Original image: -rwxr-xr-x 1 gilles gilles 17485555 2008-12-29 10:49 pict1999.png * Original converted to PGF with wavelets lossless compression : -rw-r--r-- 1 gilles gilles 13806596 2009-05-29 10:26 pict1999.pgf Encoding time (encoding, writing PGF): 4.25 s Total time (reading source, conversion, writing destination): 6.2 s Compression ratio: 1.26646 * Original converted to PGF with wavelets compression level 10 (no visible artifacts): -rw-r--r-- 1 gilles gilles 949812 2009-05-29 10:51 pict1999-q10.pgf Encoding time (encoding, writing PGF): 0.93 s Total time (reading source, conversion, writing destination): 2.94 s Compression ratio: 18.4095 * Thumbnail 256x256: -rw-r--r-- 1 gilles gilles 366042 2009-05-29 10:28 pict1999-256.png * Thumbnail converted to PGF with wavelets lossless compression : -rw-r--r-- 1 gilles gilles 106942 2009-05-29 10:28 pict1999-256.pgf Encoding time (encoding, writing PGF): 0.13 s Total time (reading source, conversion, writing destination): 0.21 s Compression ratio: 3.42281 * Thumbnail converted to PGF with wavelets compression level 10 (no visible artifacts): -rw-r--r-- 1 gilles gilles 22682 2009-05-29 10:45 pict1999-256-q10.pgf Encoding time (encoding, writing PGF): 0.05 s Total time (reading source, conversion, writing destination): 0.13 s Compression ratio: 16.138
Final output size and encoding time correlate with R^2=0.98 (though it's not linear, the plot is exponential). Is disk I/O the limiting factor? Then PGF is better because it creates so much smaller files.
It's logarithmic, I mean
>Is disk I/O the limiting factor? Then PGF is better because it creates so much >smaller files. Well, here we don't care about I/O files acess to HDD, because data will be hosted in one DB file. Of course db file will be smaller than PNG or JPEG format. I have also compared PGF compression 10 with JPEG quality 80 with the same file. PGF is 30-40 % smaller... Gilles
I was always proponent of internal thumbs database but those numbers doesn't sound as good as they seem in first moment. 22kb for thumbnail means for modest collection of 15000 (yes, it is modest as you probably know) means 330MB which will be loaded into memory just by opening of digiKam. If someone shoots more and longer, have family and manages photos for them, and last but not least uses digiKam for management of his (semi)professional workflow 100000 is easy to achieve. In this case dadatabase can reach 2GB of data!
Yes, I agree. This is why I asked if this is only used for offline / remote collections. But plans are to use it for local collections as well, which means that you might end up with twice the thumbnails data, once in .thumbnails folder and then again in the thumbs.db.
@ Mikolaj: Calculating for 100,000 is ok. I do tests with 30,000 because I dont have more. If you open a database file I assume not all of this is loaded into memory! That would not work at all. So the question here is: What is the most efficient storage for 10,000s of files of the size 10k-30k. The offered solutions are: a) in filesystem, assumed standard Linux ext3/ext4 b) as BLOBs in sqlite I have done some superficial research, came up with these pages: http://stackoverflow.com/questions/757493/sqlite-blob-or-file-system-for-images http://stackoverflow.com/questions/325126/storing-a-small-number-of-images-blob-or-fs http://stackoverflow.com/questions/3748/storing-images-in-db-yea-or-nay all of this is from the web development field and has some different priorities, I did not really get a clear answer.
I read somewhere that a sqlite db is not loaded to memory at all, only a very small part for managing purposes. This is why sqlite is quite slow when writing to the database. It creates a journal on the harddisk to keep track of changes. When running ext4 with barriers, you can see how much this technology can slow down digiKam. So a 2GB database will definitely not waste 2GB of RAM, this would be insane :D But even MySQL will never hold a whole database in memory, only the most used one (I guess, not the big database expert). Andi
From a performance view, I don't think we will gain more speed here, since sqlite needs to write twice to disk on each transaction, once for the journal, and then again for the actual data. It is not cached like in a real SQL server, so speed gains in reading and writing should not be visible I guess. Who knows, it might be even slower, but this can only be seen when we actually use it. Sure it could be packed into one big transaction, but I guess this will waste memory and kill performance. As I said before, I would think some hybrid would be better, store all remote thumbs in the database, the local ones in the file system. But this makes code more complicated again. Andi
I also would like to see this functionality, but allow me to explain a scenario and this may help bring the problem into focus. I have Albums that are stored on a network share to allow for backups and redundant disks. I use a laptop to do my image editing. When the laptop is connected to the network, I have full access to my images. If I am away from network, I can't do any image editing nor can I even browse the albums because the albums are networked albums. Because I shoot in raw, when I edit an image I start with the raw file, convert it, edit it, and save it as a .png in a different album. I like the idea above of storing thumbnails that can be viewed offline, but let's take it one step further. I would also like to be able to mark images for offline use then work on them while not connected to the network. Here is how I envision this working: 1. While connected to the network, mark an image for offline use. This would probably make a copy on the local drive somewhere. 2. Disconnect from the network and go sit under a shady tree with my laptop. 3. Navigate back to the image marked for offline use and start editing it. 4. Save as a .png to my "developed images" album that is also offline 5. Plug back into the network 6. Digikam knows about the new image I added to the Developed album and copies it to the Network share. I know that's a pretty tall order. The resynchronization is not as important as having specific images available to me, but it sure would be nifty! Thanks for your consideration!
*** Bug 133638 has been marked as a duplicate of this bug. ***
So... after all comments, there's nothing more since June 2009. We have the possibility to add CD and DVD albums to digikam album collection, but I can't make a search in these if CD/DVD is not inserted. What's on with this wish? It will be add as a feature in the next digikam version?
Marcel, Since we use a database to host collections item thumbs, this feature must be easily done now. Right ? Gilles
Yes, we can show offline collections. What is missing is UI handling: If you open such an image in the editor, the user must be told that the image is offline. Etc. for all editing tools.
Thanks Marcel, It must be not to hard to implement Gilles
his entry is also relevant indirectly of bug #114539
Git commit 3daab2fc8a17e20d68a69c319ab65edcca3fc876 by Marcel Wiesweg. Committed on 15/11/2014 at 17:14. Pushed by mwiesweg into branch 'master'. Allow to use the notification tooltip with some more safety also from other sources than overlays M +13 -0 libs/widgets/itemview/dcategorizedview.cpp M +1 -0 libs/widgets/itemview/dcategorizedview.h http://commits.kde.org/digikam/3daab2fc8a17e20d68a69c319ab65edcca3fc876 diff --git a/libs/widgets/itemview/dcategorizedview.cpp b/libs/widgets/itemview/dcategorizedview.cpp index 1f1b406..d98069b 100644 --- a/libs/widgets/itemview/dcategorizedview.cpp +++ b/libs/widgets/itemview/dcategorizedview.cpp @@ -769,6 +769,11 @@ void DCategorizedView::contextMenuEvent(QContextMenuEvent* event) } } +void DCategorizedView::leaveEvent(QEvent*) +{ + hideIndexNotification(); +} + void DCategorizedView::mousePressEvent(QMouseEvent* event) { userInteraction(); @@ -835,6 +840,14 @@ void DCategorizedView::mouseMoveEvent(QMouseEvent* event) unsetCursor(); } + if (d->notificationToolTip && d->notificationToolTip->isVisible()) + { + if (!d->notificationToolTip->rect().adjusted(-50, -50, 50, 50).contains(event->pos())) + { + hideIndexNotification(); + } + } + DigikamKCategorizedView::mouseMoveEvent(event); d->delegate->mouseMoved(event, indexVisualRect, index); diff --git a/libs/widgets/itemview/dcategorizedview.h b/libs/widgets/itemview/dcategorizedview.h index f07fc3f..31d1adb 100644 --- a/libs/widgets/itemview/dcategorizedview.h +++ b/libs/widgets/itemview/dcategorizedview.h @@ -143,6 +143,7 @@ protected: // reimplemented from parent class void contextMenuEvent(QContextMenuEvent* event); void keyPressEvent(QKeyEvent* event); + void leaveEvent(QEvent* event); void mouseMoveEvent(QMouseEvent* event); void mousePressEvent(QMouseEvent* event); void mouseReleaseEvent(QMouseEvent* event);
Git commit 6c9241b209855a40a731b1918eff0c5d557f6e94 by Marcel Wiesweg. Committed on 15/11/2014 at 17:16. Pushed by mwiesweg into branch 'master'. Support listing of images on not-available collections M +3 -1 kioslave/digikamalbums.cpp M +1 -0 kioslave/digikamdates.cpp M +1 -0 kioslave/digikammapimages.cpp M +1 -0 kioslave/digikamsearch.cpp M +2 -0 kioslave/digikamtags.cpp M +10 -0 libs/database/imageinfo.cpp M +7 -0 libs/database/imageinfo.h M +25 -4 libs/models/imagealbummodel.cpp M +2 -0 libs/models/imagealbummodel.h http://commits.kde.org/digikam/6c9241b209855a40a731b1918eff0c5d557f6e94
Git commit 44ed4eacf9c9439c3b892f9a3a65c9231dd88e28 by Marcel Wiesweg. Committed on 15/11/2014 at 17:16. Pushed by mwiesweg into branch 'master'. Enable showing albums on collections which are not available (removable media) There is currently no option in the UI to switch this off, though there is a simple method in album manager to toggle this setting M +120 -21 app/album/albummanager.cpp M +6 -0 app/album/albummanager.h http://commits.kde.org/digikam/44ed4eacf9c9439c3b892f9a3a65c9231dd88e28
Git commit 8fa0304a3144cbf9da29a734622fa6f501c4a1b6 by Marcel Wiesweg. Committed on 15/11/2014 at 17:17. Pushed by mwiesweg into branch 'master'. If the user tries to open an image which is not availabe, display a tooltip to explain the situation M +5 -0 app/items/imagecategorizedview.cpp M +2 -0 app/items/imagecategorizedview.h M +14 -5 app/views/digikamview.cpp http://commits.kde.org/digikam/8fa0304a3144cbf9da29a734622fa6f501c4a1b6
Git commit 5ef8b08f7cbe18b1cfd4b0629020361414966353 by Marcel Wiesweg. Committed on 16/11/2014 at 15:35. Pushed by mwiesweg into branch 'master'. Chnge the thumbnail loading API to be based on ThumbnailIdentifier M +28 -3 libs/threadimageio/loadingdescription.cpp M +8 -6 libs/threadimageio/loadingdescription.h M +54 -24 libs/threadimageio/thumbnailcreator.cpp M +13 -58 libs/threadimageio/thumbnailcreator.h M +60 -56 libs/threadimageio/thumbnailloadthread.cpp M +20 -19 libs/threadimageio/thumbnailloadthread.h M +4 -4 libs/threadimageio/thumbnailtask.cpp http://commits.kde.org/digikam/5ef8b08f7cbe18b1cfd4b0629020361414966353
Git commit 0e9549a3822ef30809dc8a7e91393765f51d3ba7 by Marcel Wiesweg. Committed on 16/11/2014 at 16:18. Pushed by mwiesweg into branch 'master'. The final fixes to make thumbnails for offline collections work M +12 -2 libs/models/imagethumbnailmodel.cpp M +6 -1 libs/threadimageio/thumbnailcreator.cpp http://commits.kde.org/digikam/0e9549a3822ef30809dc8a7e91393765f51d3ba7
For next 4.6.0, Marcel has commit a lots of fix to fix this entry. The last missing point explained in comment #29 is now fixed by recent commits, by using tooltips about unavailable items. Gilles Caulier