Oops.  I meant, of course, "and then continue thumbnailing the other images in the directory."

*** Bug 140616 has been marked as a duplicate of this bug. ***

I'd like to point out that bugs 137320, 138927, 140615 are all similar in the sense that in each case the reporter would like to have *background* (re)creation of thumbnails.

Yes, but there a some differences:
* 140615: create all MISSING thumbnails for WHOLE album or ONLY A FOLDER (without blocking)
* 137320: create all MISSING thumbnails for CURRENT FOLDER WHILE SHOWING it (without blocking)
* 138927: REBUILD ALL thumbnails WITHOUT BLOCKING

*** This bug has been confirmed by popular vote. ***

Created attachment 23811 [details]
GUI part; adds variable + checkbox

This little piece of code adds the variable «thumbnailLoadAll» for the
settings. There is a new checkbox in the settings page, too.

The variable is boolean and false by default.

To bring back part of the discussion on the mailing list to this bug, 
a description of Gilles:
"""
- a class named PixmapManager handle thumbs image for each icon item.
- a class named AlbumIconView manage view of icon from main
interface.

in AlbumIconView, we have a method named nextItemToThumbnail() which
is called by PixmapManager::slotComplete() when a thumb have been
found in cache (or regenerate in background if necessary by a
kio-slave)

When you select a new album or if you scrool icon view, PixmapManager
will call AlbumIconView::nextItemToThumbnail() to get the next icon
instance to thumbnailize.

Now if you look how AlbumIconView::nextItemToThumbnail() method work,
you will see than the first and the last _visible_ items are used to
limit regeneration of thumbs...

If you want to change the behaviours of thumbnails genration done in
background the code from AlbumIconView::nextItemToThumbnail() need to
be fixed to handle the first and the last iconview item item from
current album, not the visible ones.

But take a care : regenerate all icon thumbs in background take a
while, especially with slow computer. Of course a kioslave is a
separate process which run outside of digiKam but we don't have any
control to set the priority (as a separate thread)

So, for huge album and slow computer (using a single processor
especially) thumb generation will slow down digiKam. This is why the
current thumbs generation behaviours have been implemented to only
generate current visible items.

This is my proposal :

- added a new option in "Album Settings" page from Setup dialog, into
"Interface Options". A checkbox is enough here. The new option must
be disable by default.
- patch AlbumSettings class to manage this new option.
- patch AlbumIconView::nextItemToThumbnail() to handle this option
(thrue an AlbumSettings method) to change behaviours accordingly
(only visible items or all items)
"""

Just a comment on the possible performance:
What happens, if one displays a huge album (e.g. all images from 2007),
where the thumbnails have not been generated?
Possible problems:
- moving to a different place (with the slider) in the same view:
  would those images thumbnails only be displayed after all other ones
  have been generated?
  (Or are thumbnails for images currently in view generated first?)
- if one then moves to a different view (e.g. 2006),
  will the 2007 thumbnails be still generated in the background?
  (I.e., is there any way to stop the thumbnail generation when
   changing views, so that full priority goes to the currently wanted
   thumbnails?)

Now how to address the issue:
In principle 
    IconItem *fItem = firstItem();
    IconItem *lItem = lastItem();
should technically work  in AlbumIconView::nextItemToThumbnail(), 
but this doesn't.

Marcel, your help is needed here!

We're not asking for Digikam to generate thumbnails for every image in
the database--besides, that option already exists in the menu.

We are saying that Digikam should generate thumbnails for every image
in the CURRENT VIEW, not just the ones that are CURRENTLY VISIBLE.  If
I click on a tag that has 100 images in it, and can view 10 images at
a time with the thumbnail size I have set, and leave Digikam sitting
there for an hour, when I come back and scroll down, the thumbnails
for the other 90 images should have already been generated.  But
currently, Digikam won't generate thumbnails for them until I have
scrolled them into view.  This makes it impossible to visually search
for an image unless you've already seen it!

To see exactly the behavior that we are asking for, try GwenView.
Open a folder in the Browse Mode and watch as GwenView generates
thumbnails for all the images in the folder.  If I scroll down to the
bottom, GwenView will generate thumbnails for those images at the
bottom, and then return to generating thumbnails for the rest of the
images in the folder.  GwenView always generates thumbnails for the
currently-visible images first, and continues with the rest in the
background.  This is the way Digikam should work.

(Maybe you could look at GwenView's source to see how it does it.  :)

Adam, yes, that's precisely how it should work. And the attempt discussed
in #c7 should in principle achieve this; however
there is a currently unsolved issue ... 

After looking at the source I have these answers to you:

 - moving to a different place (with the slider) in the same view: 
   would those images thumbnails only be displayed after all other ones 
   have been generated? 
   (Or are thumbnails for images currently in view generated first?) 
 - if one then moves to a different view (e.g. 2006), 
   will the 2007 thumbnails be still generated in the background? 
   (I.e., is there any way to stop the thumbnail generation when 
    changing views, so that full priority goes to the currently wanted 
    thumbnails?) 

It seems that from PixmapManager, there is only one ThumbnailJob active for one thumbnail at any time. The next thumbnail to generate is determined by nextItemToThumbnail, which always is a visible item. This means there is a waiting time of at most the currently active thumbnail creation when moving to a new view, until the first now-visible item is processed.

There are mechanisms in ThumbnailJob to support multiple Urls in one job, but it seems to me these are not used.

Also note that PixmapManager stores the retrieved _pixmaps_ in a cache of size 100. This means that pixmaps should not be loaded extensively, as they are not directly transferred to the view, but the loading is:
paint -> load -> in pixmap cache? no-> load from disk -> store in pixmap cache
paint -> load -> retrieve from pixmap cache
so the process relies on having all visible pixmaps in the cache.

Then, thumbnail generation is a two step process:
1) Original image -> thumbnail on disk
2) Thumbnail on disk -> image and pixmap in memory
(these steps are optionally merged on first creation)
What takes more time is 1), the second step is relatively fast, especially if the thumbnail files are still in the OS disk cache. Step 2) is still noticeable; but, you can do the maths better than me, even keeping around 100 pixmaps in memory is in the lower two-digit megabyte range as to memory usage (on the x.org process, for pixmaps, I think)

Marcel, thanks a lot for the detailed explanation!
So from my point of view, there are two things 
a) automatically pre-generate the thumbnails on disk for all images 
   in the current album.
   This process should stop, when changing to a different view.
b) maybe pre-load the next line (or next two lines) into the pixmap cache.

Here b) should only give a slight improvement, as a) is the crucial point.

So my question is: 
What kind of infrastructure could be used to achieve a)?
If nothing is there for this, maybe it is better to deal with
this issue in the KDE4 branch?

For b) I thought that the above mentioned:
    IconItem *fItem = firstItem();
    IconItem *lItem = lastItem();
should technically work  in AlbumIconView::nextItemToThumbnail(),
but this doesn't. Maybe this should be replaced by
     IconItem *fItem = findFirstVisibleItem(r);
     IconItem *lItem = findLastVisibleItem(r) + 10;
                               // metacode, but you get the idea
(Maybe you addressed already in your reply, why this can't work ...)

Best, Arnd

Thanks Marcel for this indeep review. This code is not clear for me too (:=)))

Arnd, i recommend you to put few explanation as comments in code to remember it.

I'm agree with point a) and b). Performance tests are require to check which is the best way. Perhaps an option in setup to customize can be a solution... but this can be confuse for end users. 

Marcel, it will be fine to have a new panel in setup dedicated for thumbnails settings:

- thread priority (there are few reports in B.K.O about)
- number of items in cache memory.
- etc... 

...and certainly others one. we need to review all Thumbnails entries from B.K.O :

http://bugs.kde.org/buglist.cgi?product=digikam&component=Thumbnails&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED

Your viewpoint ?

Gilles Caulier

> b) maybe pre-load the next line (or next two lines) into the pixmap cache.


This is moment for user input ;)

One line is definitely too small.

1) Lest advanced users are scrolling only by one line but it allows them 
to scan those images quickly and if they are not satisfied they go to 
next line. Will be caching mechanism quick enough to load rows fast?

2) More advanced users use scrolling whell with more flourish which 
usually is equal to ca. half of screen of new images. Number of new rows 
depends on size of thumbnails, but it should be 2-3 rows.

3) Most advanced users jump by PageUp/PageDown or scroll wheel equivalent.

So, preloading of one row will satisfy only one group of users and not 
in all scenarios. Globally, they make the most numerous group of 
software users but due to geekiness factor they are probably weakly 
represented among Linux/KDE/digiKam users.

The best preloading, which will satisfy all users is preloading of whole 
neighbouring screen of thumbnails.

Don't know if this is possible but: in ImageMagick is switch -size. 
According to docs it tells libjpeg [sic!] to load only limited number of 
bytes to satisfy users request. It is used to speed up execution and to 
lessen memory usage. Wording suggests it is libjpeg feature so it can be 
used by other programs as well. Main problem is this applies to libjpeg 
and thumbnails in .thumbnails are PNGs... Quick scan of libpng 
documentation shows possibility to read images in chunks (interlacing). 
Maybe this could speed some things? Just random thought of 
non-programmer.

I am suggesting this because in many cases when browsing Album I don't 
have to see 100% perfect thumbnail. I see green blobs, not for me, I am 
looking for blue blobs (sea), go next screen.

Several versions ago digiKam has similar mechanism with auxiliary 
database of bastardized images. Due to improved preloading mechanisms 
this was scrapped but maybe this is time to investigate similar solution?

----------------------------------------------------
Wiosna w Muzeum Narodowym w Krakowie
Zobacz wi

#11:
For KDE3, to achieve a), you could start a separate ThumbnailJob with all URLs from the current view. I dont see a way to make it have less priority than the main thumbnail job then though.
In KDE4, we can of course have a thread, with a low priority then.
For b): Yes this can work. Just make sure that after loading the next row/rows/screen of thumbnails, the number of currently visible + preloaded items does not exceed the cache size (For cases of large monitors and small pixmap size; in the case of small pixmaps and thus more visible and preloaded pixmaps, the number of cached pixmaps can of course be increased, a smaller pixmap taking less memory)

#12:
These are advanced options, for the usual users it should work out of the box, but if there is real user request why not. Perhaps sliders trading CPU and memory usage vs. performance?

#13:
The progressive reading feature applies when the data itself comes in chunks over a network; and interlaced reading applies if the thumbs are interlaced, which they must not be according to the thumbnail spec.

With current KDE4 implementation, duing the integration of thumbbar with preview mode, all thumbnails of current album are loaded in background.

http://digikam3rdparty.free.fr/Screenshots/FullColorThemeSupport/thumbbarineditor.png

In fact this is a side effect, not a real wish from me. But i cannot see any latency time with thumbnail rendering in icon view.

Is you can compile and test KDE4 version of digiKam, can you report your feedback ?

Thanks in advance

Gilles Caulier

Sounds great (can't test due to lack of KDE4 env.):
Questions:  
- Can the thumbbar in the preview mode be switched off
  (and if yes, will the loading in the background still work?).
- What happens if one changes to a different album (i.e. is the loading 
  in the background stopped?)?
- What happens if one moves to a different place in the current album.
  Will the visible ones at the new place be generated first/with priority?
In any case, these are additional improvements/aspects, it seems
you killed another 100+ votes wish!

- Can the thumbbar in the preview mode be switched off

no yet. Just cached.

  (and if yes, will the loading in the background still work?).

- What happens if one changes to a different album (i.e. is the loading
  in the background stopped?)?

yes, loading is stopped and restated using new place.

- What happens if one moves to a different place in the current album.
  Will the visible ones at the new place be generated first/with priority?

Good question. Not yet tested...

Gilles

Thumbnails in the thumbnail thread are currently simply appended to the list of a single thread.
Note that in KDE4 parts of the above discussion does not apply, we can do more what we like.

We could e.g. use a precreation thread that creates thumbnails on disk, but does not care for pixmaps (thread->setPixmapRequested(false)) as a first step.
On album change, the old process can be stopped by thread->stopLoading(QString()), and all new thumbnail tasks can be appended as a whole.
For this solution to make sense, the thumbar should also apply the load-when-needed principle.

In the icon view, loading is also more straightforward: When an item is painted, it requests its pixmap from the thread. If the pixmap is cached, it is returned and painted. Else, a loading operation is initiated. AlbumIconView will then receive a signal in slotThumbnailLoaded, find out the affected item, and cause a repaint of this item.
Introducing pixmap preloading here for the next row/screen is another point. We need to think about finding the correct place for a hook for starting the preloading.

Marcel, Arnd,

In #15, i'm talk about a side effect of thumbbar included with preview effect.

After to have tested this implementation i can said than the result are catastrophic using simple processor computer. the thread which compute thumbs in background from thumbbar take all CPU and digiKam is not suitable.

Under a double core, this is not the same. the thread is forked to another CPU and digiKam is suitable.

So, i will fix the current implementation.

But this side effect is interesing to hack the proposal given by Marcel in #18

Gilles

Isn't this fixed now? I just tested it with Digikam 1.8 and it seemed to keep on thumbnailing all the images of a directory, even the ones not currently visible.

Ricardo: This should be fine / is even done as per feature request, but the thread must have low priority. I'm not sure about the status in 1.8, and of course I have a double core for testing only. But I did not see any complaints in the last months... Note that Gilles' comment is from 2008!

*** Bug 188029 has been marked as a duplicate of this bug. ***

Since digiKam use a database to store thumbnail, showing icon view items is very fast and faster than storing thumbnail using .thumbnails cache directory. I think this feature don't need to be implemented.

I compared digiKam with Gwenview on my i7-3Ghz-16Gb-SSD using a large album including more than 150 24Mpx JPEG.

digiKam is very fast. It take 2 seconds to regenerate all thumbnails and cache items in database. Navigate to this album with cache show items immediately : delay is very short.

Gwenview take around 5s to generate thumbnails. Cache is done in .thumbnails, not in database. Navigate to this album with cache show items immediately : delay is very short as digiKam.

Gilles Caulier