Bug 467711 - Time latency introduced while applying changes in Sqlite database under Windows.
Summary: Time latency introduced while applying changes in Sqlite database under Windows.
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Database-Sqlite (other bugs)
Version First Reported In: 7.10.0
Platform: Microsoft Windows Microsoft Windows
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-03-23 09:32 UTC by Daimonicon
Modified: 2024-03-29 11:10 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed/Implemented In: 8.4.0
Sentry Crash Report:


Attachments
no workload showing (345.79 KB, image/gif)
2023-03-23 09:32 UTC, Daimonicon
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Daimonicon 2023-03-23 09:32:45 UTC
Created attachment 157526 [details]
no workload showing

It would be great if the status would show that something is still happening in the background. The Windows egg timer for work load shows that digiKam is busy and sometimes even "No response", but the process status says "No process". It would be nice to be able to see this better.
Comment 1 Maik Qualmann 2023-03-23 10:52:24 UTC
How many images did you apply the tag change to? And how many images are in the current view?

Maik
Comment 2 Daimonicon 2023-03-23 11:09:33 UTC
(In reply to Maik Qualmann from comment #1)
> How many images did you apply the tag change to? And how many images are in
> the current view?
> 
> Maik

When I noticed it, it was 5-15 pictures, but sometimes I have considerably more, up to 100, but then I usually expect it to take longer. But often the process status doesn't say anything, but the GUI simply doesn't react for a while.

If the creation takes a while, that's fine, but a visible status would be helpful.
Comment 3 Maik Qualmann 2023-03-23 11:38:42 UTC
If it took longer to apply the tags, the progress bar would activate. A wait cursor could only occur if there are several thousand images in the current album, since the tag status then has to be read again from the DB and this sequential action is relatively slow with MySQL.

Along with their other bug report, digiKam is showing strange behavior, so we need a DebugView log as described here for Windows:

https://www.digikam.org/contribute/

Maik
Comment 4 Daimonicon 2023-03-23 12:32:04 UTC
(In reply to Maik Qualmann from comment #3)
> If it took longer to apply the tags, the progress bar would activate. A wait
> cursor could only occur if there are several thousand images in the current
> album, since the tag status then has to be read again from the DB and this
> sequential action is relatively slow with MySQL.
> 
> Along with their other bug report, digiKam is showing strange behavior, so
> we need a DebugView log as described here for Windows:
> 
> https://www.digikam.org/contribute/
> 
> Maik

I wanted to follow the steps that lead to the behaviour or error but this time digiKam crashed completely.

I'll try again with DebugViewer afterwards.

I am attaching a log that will last until the programme hangs. It has not crashed but has been "no responding" for more than 10 minutes.
Comment 5 Daimonicon 2023-03-23 12:36:27 UTC
Attachments doesn't work cause of file size

Link: https://gist.github.com/Daimonicon/1d70ab6c4692a43c19fee366ef14b680
Comment 6 Daimonicon 2023-03-23 14:00:02 UTC
I created another log this time it took several minutes but the app didn't hang - estimate 20-30 minutes I can't figure out from the log time.

139 images with colour markers. I added a user marker to the beginning and end of the process in the log [UserMarker].

https://gist.github.com/Daimonicon/5d38d31e1f7eb5e7fc9ce89db51c2f63
Comment 7 Daimonicon 2023-03-23 15:28:22 UTC
I may have found the problem, but I'm not 100% sure. The DB contains a total of about 3500 entries, for the last 100-150 I had the idea that it would not be bad to shrink the size of the pictures without quality lost (App: JPEGmini, www.jpegmini.com).

Assigning a new colour tag to 167 images that have not been shrinked - happens almost instantly.

Assigning a new colour tag to 139 images that have been shrinked - freezes the digiKam permanently or for a very long time.

Question: Can such an image shrinker "damage" the images in such a way that they are no longer really usable for photo management?

I trusted the programme because it was developed by photo professionals.
Comment 8 Maik Qualmann 2023-03-23 20:54:35 UTC
Your database is constantly locked for the maximum wait time of 10 seconds and then discards the job. Normally, this wait time is used to wait for the other thread that is currently performing a write (SQLite can only write one at a time). I can't say at the moment where the problem is here. However, your database may now be extremely inconsistent. This also explains the behavior in the other bugreort on rating.

It can't be related to the shrinking of the images.

Maik
Comment 9 Maik Qualmann 2023-03-23 21:04:50 UTC
I see in your log: Using 24 CPU core to run thread
We have seen strange behavior with machines that have such a high number of CPU cores.
I'll take a look at the relevant code again in the next few days.

Maik
Comment 10 Daimonicon 2023-03-24 11:36:26 UTC
Thanks in any case for your time and effort.

It's not because the images have been shrinked in size ? To be on the safe side, I had removed the ~200 shrinked images and replaced them with completely new ones. Since then, there have been no more app freezes, no more unwanted ratings after cropping the size of the pictures, no more waiting for minutes when assigning a tag.

I think the log looks very different too ? I'll link a log from (UTC time: 03/23/2023 21:51:36) to (UTC time: 03/24/2023 00:36:00). In this time there was only once a previously seen behaviour and that was because I had overlooked an image that JPEGmini had changed and resulted in an already known behaviour. During this time, I had done all as before by recreating the files.

Before I had the idea of running a log, I had deactivated the xmp file creation for better performance and moved the four .db files to SSD. However, the error can still be reproduced on my page when I bring a file changed by JPEGmini into play. However, the app reacts faster and more reliably.

https://gist.github.com/Daimonicon/665944c8cd58a5d106cb965df4704a39
Comment 11 caulier.gilles 2023-11-24 05:04:00 UTC
Daimonicon,

The next digiKam 8.2.0 for Windows is now build natively under Windows using VCPKG+MSVC toolchain instead to be cross compiled to Linux. It's a big change which will close plenty of file as cross compilation cannot emulate all Windows low level features provided by Windows API.

So i recommend to test the Windows 10 and later digiKam 8.2.0 pre-release installer available here :

https://files.kde.org/digikam/

Best regards

Gilles Caulier