Bug 382361

Summary: Continuous net usage, server CPU and IO usage while doing nothing but keeping a collection opened [network share & mysql db]
Product: [Applications] digikam Reporter: Thomas Debesse <dev>
Component: Database-MysqlAssignee: Digikam Developers <digikam-bugs-null>
Status: RESOLVED FIXED    
Severity: normal CC: caulier.gilles, codestruct, david.varnes, KDE, metzpinguin
Priority: NOR Keywords: triaged
Version: 5.6.0   
Target Milestone: ---   
Platform: Microsoft Windows   
OS: Microsoft Windows   
Latest Commit: Version Fixed In: 8.2.0

Description Thomas Debesse 2017-07-15 06:04:56 UTC
Here Digikam (on Windows workstation) is configured to store its core, thumb and face database in a MySQL server (on another host running Linux), and to store pictures in a network share (a CIFS share hosted by a Samba service running on the same host the mysql server runs). The Digikam configuration is stored in local workstation's drive (see bug #382358). So, only database and pictures file system is currently used through the network.

```
 ___________________                             ___________________
|                   |    ___________________    |                   |
|      server       |   (                   )   |    workstation    |
|___________________|   ( 10 Gbit/s network )   |___________________|
|  _______________  |   (                   )   |  _______________  |
| [ MySQL service ]←―――――――――――――――――――――――――――――→[               ] |
|  ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾  |   (                   )   | [    digiKam    ] |
|  _______________  |   (                   )   | [    client     ] |
| [ Samba service ]←―――――――――――――――――――――――――――――→[_______________] |
|  ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾  [   (___________________)   |                   |
|___________________|                           |___________________|

```

There is more than 70 000 pictures stored in that Samba share. I recently run the tool to find (but not to recognizes) faces. It ended with almost ≈ 40 000 entries in the “Unknown” people tag.

I noticed that if I open an album or a tag with a few pictures, the network usage grows a bit while the thumbnails download, then when the thumbnails are loaded, the network usage goes down to 0.

But if I open the “Unknown” people tag view with 40 000 entries, the thumbnails load for visible pictures but never goes down. On the server side I see a straight 50 MBit/s rate on Outgoing (from server to network) and a 1 MBit/s rate on Incoming. At the same time on the Workstation side I see the same rate (but 50 MBit/s on Incoming from network to workstation and 1 MBit/s on Outgoing, of course).

At the same time, `tcpdump` reports me an endless rate of MySQL connections coming from the workstation and going to the workstation, and `iotop` is showing me multiple processes continuously eating 30% of IO. In this case, samba does not make any IO at this time (the thumbnails are probably already generated, so, no need to load full files).

So, If I just open a large collection, and let the digiKam window opened and go away doing nothing, digiKam seems to cause a continous MySQL/IO/network (which is like being endless due to the large amount of thumbnails to load).

I suppose digiKam is downloading thumbnails for the complete list of pictures available on the current view, because when I open a view with only small number of pictures, the MySQL/IO/network usage goes down once all the thumbnails are loaded.

Even if both workstation and server are living on a fast 10Gbit/s network (which is more a 2.5GBit/s network due to some Windows network inefficiency), this behavior seems to produce small annoying freezes in digiKam. By the way, 2.5 GBit/s sounds to be large enough for a 50Mbit/s network usage. The annoyances probably comes from the useless disk IO on the server, the useless MySQL usage, and perhaps from the useless network interruptions involved making troubles with priority of other requests.

For example these very annoying micro freeze comes each time I apply a people tag to a picture face. On normal usage, even with every “save metadata to picture” option being activated, this kind of action is instantaneous: the tag is applied, the picture is saved, and the face disappear from "Unknown" tag list. 

But with this kind of large Unknown people tag list, with background useless network rate, useless IO usage and useless MySQL queries, when tagging someone, users have to suffer a ~1sec wait to be able to apply a people tag on another face, even when all “save metadata to picture” options are disabled.

The configuration is solid: The server has two Intel Xeon E5430 CPU (2×4 cores) with 32Gb of ram and RAID10 made from 4× 7200rpm drive with 64Mb cache each one, running Debian jessie with latest kernel from backports. The Workstation has an Intel E3-1270 v5 CPU with 32 GB/ram and SSD RAID1 (which is only involved for config file in that setup), running Windows 10. The server and the workstation both have a 10GB/s interface and are connected via a 10GB/s switch. The MySQL server was tuned to fit the digiKam queries and uses a ramdisk as tmp disk (so, even when doing disk cache, it's still on ram). The server is a bit old (the ram speed is probably a kind of bottleneck in some scenario), so I expect some slower RAM IO than more modern servers can do, but it's still not an anemic one. The same way I know my server's SATA controller is a bit limited, but that RAID10 setup is still able to read 240MB/s sequentially and to read 4×100MB/s at the same time.

So, it looks like those micro-freezes comes from digiKam endlessly querying and downloading thumbnails. Of course, this useless network, disk IO and CPU usage is able to slow down everyone else on the network.

If it's not implemented, it would be very cool to get digiKam only loading thumbnails from visible pictures. Pre-downloading thumbnails for one or two next screen would be very ok to get smooth scrolling, but no more.

Also, the thumbnails download must prevent querying one thumbnail at a time. I explain: We can imagine digiKam downloading the thumbnails for the 24 pictures currently displayed on the screen, then, to get smooth scrolling, downloading the 24 next thumbnails, then the 24 previous thumbnails, then 24 next-next thumbnails.

But digiKam must NOT download the only one next thumbnail needed because only one picture was removed from the current viewed collection. When the user is setting people tags on pictures within “Unknown” face list, once the picture is tagged, the picture disappear from that “Unknown” list.

If digiKam implements a “download missing invisible thumbnail up to N each time preloaded invisible thumbnail number < N” algorithm, it would mean digiKam would query a thumbnail and download it every time the user is tagging someone in a picture, triggering annoying MySQL queries and thumbnail download/generation each one a people tag is applied. In this case, it would be better to chunk the thumbnail query/download, so the micro freeze only comes from time to time, not everytime. It would be OK for an use to wait some time after having tagged 10/20 pictures than after each picture.

Also, since people will probably tag different people on the picture list (having to manually select another label etc.) it would probably make that chunked thumbnail download less visible, because of the higher chance of doing it while user is doing something else.
Comment 1 Gregor Mi 2018-03-25 18:04:13 UTC
Thanks for the detailed bug report. Some time has passed by now. Please confirm that your report is still valid.
Comment 2 Thomas Debesse 2018-03-26 01:03:25 UTC
I'm stuck in digiKam 5.7 because of a database mismatch when trying to upgrade to digiKam 5.8, so I can't attempt to reproduce the bug on new versions. Also users completely purged the list of unlabelled faces to avoid the issue (it does not mean the bug is not an issue for them, it means they had to not use the face discovery feature at all, but they need it).

This is the kind of bug that is hard to fix by mistake so if it was fixed, someone would probably knows it.
Comment 3 Maik Qualmann 2018-03-26 06:08:08 UTC
Because of the DB Update problem look here: Bug 388977

Maik
Comment 4 Andrew Crouthamel 2018-09-28 03:25:29 UTC
Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least 15 days. Please provide the requested information as soon as possible and set the bug status as REPORTED. Due to regular bug tracker maintenance, if the bug is still in NEEDSINFO status with no change in 30 days, the bug will be closed as RESOLVED > WORKSFORME due to lack of needed information.

For more information about our bug triaging procedures please read the wiki located here: https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

If you have already provided the requested information, please set the bug status as REPORTED so that the KDE team knows that the bug is ready to be confirmed.

Thank you for helping us make KDE software even better for everyone!
Comment 5 Andrew Crouthamel 2018-10-29 02:03:33 UTC
Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least 30 days. The bug is now closed as RESOLVED > WORKSFORME due to lack of needed information.

For more information about our bug triaging procedures please read the wiki located here: https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

Thank you for helping us make KDE software even better for everyone!
Comment 6 caulier.gilles 2020-08-04 08:11:57 UTC
digiKam 7.0.0 stable release is now published:

https://www.digikam.org/news/2020-07-19-7.0.0_release_announcement/

We need a fresh feedback on this file using this version.

Best Regards

Gilles Caulier
Comment 7 caulier.gilles 2022-01-10 11:34:45 UTC
Hi Maik,

File #388977 is now fixed. What' sthe goal for this entry ?

Gilles
Comment 8 Thomas Debesse 2022-01-10 13:07:47 UTC
#388977 was an issue preventing me to update Digikam and then preventing me to check if the bug was reproduceable in newer versions of Digikam. Now that #388977 is said to be fixed, it would have been possible to update Digikam but unfortunately I don't work anymore for the Digikam users reproducing the issues at the time, so I lost access to the testbed.

I had described in first post the architecture reproducing it (basically a workstation working with networked database on mySQL and networked file system on Samba), but I'm not available to reproduce the setup from scratch for now.
Comment 9 caulier.gilles 2023-04-22 06:25:02 UTC
Hi Maik,

This is not the expected goal to prevent to reproduce the situation where digiKam become unresponsive with the remote collection by fixing bug #388977 ? As this entry cannot be reproduced, it can be closed now ?

Gilles
Comment 10 caulier.gilles 2023-10-15 12:51:43 UTC
Closing following comment #8...