Bug 382361

Here Digikam (on Windows workstation) is configured to store its core, thumb and face database in a MySQL server (on another host running Linux), and to store pictures in a network share (a CIFS share hosted by a Samba service running on the same host the mysql server runs). The Digikam configuration is stored in local workstation's drive (see bug #382358). So, only database and pictures file system is currently used through the network.

```
 ___________________                             ___________________
|                   |    ___________________    |                   |
|      server       |   (                   )   |    workstation    |
|___________________|   ( 10 Gbit/s network )   |___________________|
|  _______________  |   (                   )   |  _______________  |
| [ MySQL service ]←―――――――――――――――――――――――――――――→[               ] |
|  ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾  |   (                   )   | [    digiKam    ] |
|  _______________  |   (                   )   | [    client     ] |
| [ Samba service ]←―――――――――――――――――――――――――――――→[_______________] |
|  ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾  [   (___________________)   |                   |
|___________________|                           |___________________|

```

There is more than 70 000 pictures stored in that Samba share. I recently run the tool to find (but not to recognizes) faces. It ended with almost ≈ 40 000 entries in the “Unknown” people tag.

I noticed that if I open an album or a tag with a few pictures, the network usage grows a bit while the thumbnails download, then when the thumbnails are loaded, the network usage goes down to 0.

But if I open the “Unknown” people tag view with 40 000 entries, the thumbnails load for visible pictures but never goes down. On the server side I see a straight 50 MBit/s rate on Outgoing (from server to network) and a 1 MBit/s rate on Incoming. At the same time on the Workstation side I see the same rate (but 50 MBit/s on Incoming from network to workstation and 1 MBit/s on Outgoing, of course).

At the same time, `tcpdump` reports me an endless rate of MySQL connections coming from the workstation and going to the workstation, and `iotop` is showing me multiple processes continuously eating 30% of IO. In this case, samba does not make any IO at this time (the thumbnails are probably already generated, so, no need to load full files).

So, If I just open a large collection, and let the digiKam window opened and go away doing nothing, digiKam seems to cause a continous MySQL/IO/network (which is like being endless due to the large amount of thumbnails to load).

I suppose digiKam is downloading thumbnails for the complete list of pictures available on the current view, because when I open a view with only small number of pictures, the MySQL/IO/network usage goes down once all the thumbnails are loaded.

Even if both workstation and server are living on a fast 10Gbit/s network (which is more a 2.5GBit/s network due to some Windows network inefficiency), this behavior seems to produce small annoying freezes in digiKam. By the way, 2.5 GBit/s sounds to be large enough for a 50Mbit/s network usage. The annoyances probably comes from the useless disk IO on the server, the useless MySQL usage, and perhaps from the useless network interruptions involved making troubles with priority of other requests.

For example these very annoying micro freeze comes each time I apply a people tag to a picture face. On normal usage, even with every “save metadata to picture” option being activated, this kind of action is instantaneous: the tag is applied, the picture is saved, and the face disappear from "Unknown" tag list. 

But with this kind of large Unknown people tag list, with background useless network rate, useless IO usage and useless MySQL queries, when tagging someone, users have to suffer a ~1sec wait to be able to apply a people tag on another face, even when all “save metadata to picture” options are disabled.

The configuration is solid: The server has two Intel Xeon E5430 CPU (2×4 cores) with 32Gb of ram and RAID10 made from 4× 7200rpm drive with 64Mb cache each one, running Debian jessie with latest kernel from backports. The Workstation has an Intel E3-1270 v5 CPU with 32 GB/ram and SSD RAID1 (which is only involved for config file in that setup), running Windows 10. The server and the workstation both have a 10GB/s interface and are connected via a 10GB/s switch. The MySQL server was tuned to fit the digiKam queries and uses a ramdisk as tmp disk (so, even when doing disk cache, it's still on ram). The server is a bit old (the ram speed is probably a kind of bottleneck in some scenario), so I expect some slower RAM IO than more modern servers can do, but it's still not an anemic one. The same way I know my server's SATA controller is a bit limited, but that RAID10 setup is still able to read 240MB/s sequentially and to read 4×100MB/s at the same time.

So, it looks like those micro-freezes comes from digiKam endlessly querying and downloading thumbnails. Of course, this useless network, disk IO and CPU usage is able to slow down everyone else on the network.

If it's not implemented, it would be very cool to get digiKam only loading thumbnails from visible pictures. Pre-downloading thumbnails for one or two next screen would be very ok to get smooth scrolling, but no more.

Also, the thumbnails download must prevent querying one thumbnail at a time. I explain: We can imagine digiKam downloading the thumbnails for the 24 pictures currently displayed on the screen, then, to get smooth scrolling, downloading the 24 next thumbnails, then the 24 previous thumbnails, then 24 next-next thumbnails.

But digiKam must NOT download the only one next thumbnail needed because only one picture was removed from the current viewed collection. When the user is setting people tags on pictures within “Unknown” face list, once the picture is tagged, the picture disappear from that “Unknown” list.

If digiKam implements a “download missing invisible thumbnail up to N each time preloaded invisible thumbnail number < N” algorithm, it would mean digiKam would query a thumbnail and download it every time the user is tagging someone in a picture, triggering annoying MySQL queries and thumbnail download/generation each one a people tag is applied. In this case, it would be better to chunk the thumbnail query/download, so the micro freeze only comes from time to time, not everytime. It would be OK for an use to wait some time after having tagged 10/20 pictures than after each picture.

Also, since people will probably tag different people on the picture list (having to manually select another label etc.) it would probably make that chunked thumbnail download less visible, because of the higher chance of doing it while user is doing something else.