Bug 477853

Summary: digiKam 8.3.0 slow startup on Windows 10 operating system
Product: [Applications] digikam Reporter: Peter <benedekppeter>
Component: Bundle-WindowsAssignee: Digikam Developers <digikam-bugs-null>
Status: RESOLVED FIXED    
Severity: normal CC: caulier.gilles, hannesjaagura, ivar.ekman, ivorwjones, joachim.kroemer, Lothar-Mueller, metzpinguin, nonobio
Priority: HI    
Version: 8.3.0   
Target Milestone: ---   
Platform: Microsoft Windows   
OS: Microsoft Windows   
Latest Commit: Version Fixed In: 8.3.0
Sentry Crash Report:
Attachments: Debug View log
digiKam 8.2.0 start and 8.3.0 start in one video (only 2 min 14 sec)
Marble dialog
2023.12.03. DebugLog
This log was created while digiKam was running when I clicked on the Configure digiKam menu item.
attachment-1354632-0.html
Versions currently available in Windows
Qt6Core.dll
20231215_debugview_log
A log the after "configure digiKam..." opening/closing/opening
A log the after "configure digiKam..." opening/closing/opening
digiKam-8.3.0-20231224T080451-Win64.exe
"Find new items" is completed
I disabled the "Find new items" and restarted digiKam.
A log on the same computer, in digiKam 8.2.
digiKam 8.3 log

Description Peter 2023-12-01 17:11:45 UTC
Created attachment 163711 [details]
Debug View log

Preventive thread: https://bugs.kde.org/show_bug.cgi?id=477565#add_comment
Note: In my opinion, there is no connection, so I opened a new thread
Debug View is attached

"Tested in: digiKam-8.2.0-20231128T125958-Win64.exe
Slow startup on Windows. "Loading Tools...": It just waits and waits and waits and after about 2 minutes it starts
Antivirus: Windows Defender"

I turned off the antivirus, but the problem did not go away

On the same computer, previous digikam versions start in 15 seconds

SOFTWARE/OS VERSIONS
Windows: 10
Comment 1 Peter 2023-12-01 17:14:40 UTC
The wait occurs between lines 395 and 396
Comment 2 caulier.gilles 2023-12-01 21:29:05 UTC
00000393	16.04066658	[7048] digikam.general: Finish Main Thread	
00000394	16.04195786	[7048] digikam.general: Finish Main Thread	
00000395	16.04323387	[7048] digikam.general: Finish Main Thread	
00000396	101.44808960	[7048] digikam.general: One job is done	
00000397	101.45188141	[7048] digikam.general: Finish Main Thread	
00000398	101.57834625	[7048] digikam.general: Finish Main Thread	
00000399	102.62887573	[7048] digikam.general: scan mode: ScanDeferredFiles	
00000400	103.17636108	[7048] digikam.general: total scan value :  204674	
00000401	103.24131775	[7048] digikam.general: total scan value :  215858	

Around 80 seconds, probably due to scan process
Comment 3 Peter 2023-12-02 07:43:12 UTC
Created attachment 163745 [details]
digiKam 8.2.0 start and 8.3.0 start in one video (only 2 min 14 sec)
Comment 4 caulier.gilles 2023-12-02 09:42:26 UTC
Maik,

I can see in you video that the collections are huge.

It's possible that albums/tags search completer need to be populated at startup with GUI blocking.
If yes, the Q is why it's blocking under windows ? Here under Linux my collections are giant and i cannot reproduce the problem...

Another possibility is the Marble blank dialog visible at startup when one geo-location tab is visible in the GUI.

Peter, please hide the geo-location view in digiKam, close and restart to see if the time latency still present...

Gilles Caulier
Comment 5 Peter 2023-12-02 10:07:53 UTC
"hide the geo-location view in digiKam"
Maik, please: where do i find this setting?

Configure digikam -- Views -- Icons -- Show geolocation indicator (I have it is disabled)
Comment 6 Peter 2023-12-02 10:24:55 UTC
I tryed:
I disabled the "Scan for new items at startup" function in the Settings -- Miscellaneous -- Behaviour.
Result: A blank white window for long minutes (maybe "Marble blank dialog").
After 2 minutes it disappeared and digiKam started.
Comment 7 Peter 2023-12-02 10:25:28 UTC
Created attachment 163754 [details]
Marble dialog
Comment 8 Peter 2023-12-02 10:36:31 UTC
(In reply to Peter from comment #6)
> I tryed:
> I disabled the "Scan for new items at startup" function in the Settings --
> Miscellaneous -- Behaviour.
> Result: A blank white window for long minutes (maybe "Marble blank dialog").
> After 2 minutes it disappeared and digiKam started.

Settings -- Configure digiKam...
I enabled the "Scan for new items at startup" function in the Settings -- Miscellaneous -- Behaviour.
Result: A blank white window for long minutes (maybe "Marble blank dialog").
After 2 minutes it disappeared and digiKam worked.

OK, now again:

Settings -- Configure digiKam...
Result: A blank white window for long minutes (window titlebar: Configure digiKam).
After 2 minutes  the contents of the settings panel appeared
Comment 9 Peter 2023-12-02 10:44:44 UTC
In fact, this will also do it:
Setting -- Configure digiKam -- OK
...and again
Setting -- Configure digiKam

Result: digiKam waits for minutes, the "Configure digiKam" window is blank and white.
Comment 10 caulier.gilles 2023-12-02 12:06:02 UTC
Peter,

The blank dialog problem from Marble is know:

https://bugs.kde.org/show_bug.cgi?id=452185

I already tried to fix this in Marble code (that i ported to Qt6), but dialog still here.

To hide geolocation view, there is no settings. Just change the active tab from right and left sidebar. On the left you have the map search, and on the right the map view. Just change the current active tabs at left and right side and restart digiKam.

Gilles Caulier
Comment 11 Peter 2023-12-02 12:39:54 UTC
(In reply to caulier.gilles from comment #10)
> Peter,
> 
> The blank dialog problem from Marble is know:
> 
> https://bugs.kde.org/show_bug.cgi?id=452185
> 
> I already tried to fix this in Marble code (that i ported to Qt6), but
> dialog still here.
> 
> To hide geolocation view, there is no settings. Just change the active tab
> from right and left sidebar. On the left you have the map search, and on the
> right the map view. Just change the current active tabs at left and right
> side and restart digiKam.
> 
> Gilles Caulier

-- I did it. I hide geolocation view (I never use these anyway)
-- Close digiKam
-- Restart digiKam
Unfortunately, it didn't solve the problem

"The blank dialog problem from Marble is know:"
Ok. In this case, this error ticket can be closed.
I'm sorry if I wasted your time, but thanks for addressing the issue!
Comment 12 caulier.gilles 2023-12-02 15:41:50 UTC
>-- I did it. I hide geolocation view (I never use these anyway)
>-- Close digiKam
>-- Restart digiKam
>Unfortunately, it didn't solve the problem

>"The blank dialog problem from Marble is know:"
>Ok. In this case, this error ticket can be closed.
>I'm sorry if I wasted your time, but thanks for addressing the issue!

Sorry i don't understand which ticket can be closed ? The problem is not solved at all as i can see...

Gilles Caulier
Comment 13 Peter 2023-12-02 15:49:59 UTC
I thought this ticket (477853) was a duplicate, that's why I wrote that it can be closed.
Comment 14 caulier.gilles 2023-12-02 15:56:25 UTC
But it's not a duplicate, as the origin of the time latency at startup is not clearly identified.
As you have already write in this file, the Marble blank dialog is not the really problem. 

Gilles Caulier
Comment 15 Maik Qualmann 2023-12-03 09:30:43 UTC
Git commit c4f76622403f36f0543b2d6ca7996354eb45e98a by Maik Qualmann.
Committed on 03/12/2023 at 10:29.
Pushed by mqualmann into branch 'master'.

more test debug output for ActionThreadBase

M  +4    -4    core/libs/threads/actionthreadbase.cpp

https://invent.kde.org/graphics/digikam/-/commit/c4f76622403f36f0543b2d6ca7996354eb45e98a
Comment 16 caulier.gilles 2023-12-03 10:31:02 UTC
Peter, 

The 8.3.0 pre-release installer is updated online with the last changes from Mail to add more debug trace in debugview at startup.

Please install file and report: https://files.kde.org/digikam/

Best
Gilles caulier
Comment 17 caulier.gilles 2023-12-03 10:35:56 UTC
Maik, 

For info in the "legacy" sub directory, i upload the cross-compiled version of pre-release of digiKam, if we needs to compare with the native version:

https://files.kde.org/digikam/legacy/

Gilles
Comment 18 Peter 2023-12-03 12:19:41 UTC
(In reply to caulier.gilles from comment #16)
> Peter, 
> 
> The 8.3.0 pre-release installer is updated online with the last changes from
> Mail to add more debug trace in debugview at startup.
> 
> Please install file and report: https://files.kde.org/digikam/
> 
> Best
> Gilles caulier

Hi Gilles,
New log file is uploaded
Regards,
Peter
Comment 19 Peter 2023-12-03 12:20:12 UTC
Created attachment 163811 [details]
2023.12.03. DebugLog
Comment 20 Peter 2023-12-03 12:38:49 UTC
Created attachment 163812 [details]
This log was created while digiKam was running when I clicked on the Configure digiKam menu item.
Comment 21 Maik Qualmann 2023-12-03 12:49:10 UTC
Git commit 11883c4309073cd7f0236b58453b60dcc3e45dd0 by Maik Qualmann.
Committed on 03/12/2023 at 13:48.
Pushed by mqualmann into branch 'master'.

reset a commit to the refresh timers

M  +8    -2    core/libs/album/manager/albummanager_album.cpp
M  +13   -3    core/libs/album/manager/albummanager_collection.cpp
M  +4    -1    core/libs/album/manager/albummanager_salbum.cpp
M  +8    -2    core/libs/album/manager/albummanager_talbum.cpp

https://invent.kde.org/graphics/digikam/-/commit/11883c4309073cd7f0236b58453b60dcc3e45dd0
Comment 22 IvorJ 2023-12-04 20:03:37 UTC
It's also far slower than 8.1.0 when assigning geolocations - presumably also a database operation
Comment 23 Peter 2023-12-05 07:34:07 UTC
I don't know how important this information is...
-- The newly released 8.2.0 also contains this bug
-- Versions of 8.2.0 released at the beginning of November (weekly snapshots) did not yet contain this bug
Comment 24 Maik Qualmann 2023-12-05 08:43:03 UTC
The new digiKam-8.2.0 version is based on Qt6 and is created with the Microsoft compiler. Test the current digiKam-8.2.0 version, which was created as before:

https://download.kde.org/stable/digikam/8.2.0/digiKam-8.2.0-Qt5-Win64.exe

Does this version run normally again for you?

Maik
Comment 25 IvorJ 2023-12-05 09:10:32 UTC
Created attachment 163896 [details]
attachment-1354632-0.html

Yes, it's fine

On Tue, 5 Dec 2023 at 08:43, Maik Qualmann <bugzilla_noreply@kde.org> wrote:

> https://bugs.kde.org/show_bug.cgi?id=477853
>
> Maik Qualmann <metzpinguin@gmail.com> changed:
>
>            What    |Removed                     |Added
>
> ----------------------------------------------------------------------------
>                  CC|                            |metzpinguin@gmail.com
>
> --- Comment #24 from Maik Qualmann <metzpinguin@gmail.com> ---
> The new digiKam-8.2.0 version is based on Qt6 and is created with the
> Microsoft
> compiler. Test the current digiKam-8.2.0 version, which was created as
> before:
>
> https://download.kde.org/stable/digikam/8.2.0/digiKam-8.2.0-Qt5-Win64.exe
>
> Does this version run normally again for you?
>
> Maik
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
Comment 26 Maik Qualmann 2023-12-05 10:54:43 UTC
Please compare it again with this current version:

https://files.kde.org/digikam/digiKam-8.3.0-20231205T055434-Win64.exe

Maik
Comment 27 Peter 2023-12-05 10:56:25 UTC
> Does this version run normally again for you?

Yes! This works very well!
Comment 28 Peter 2023-12-05 11:06:41 UTC
(In reply to Maik Qualmann from comment #26)
> Please compare it again with this current version:
> 
> https://files.kde.org/digikam/digiKam-8.3.0-20231205T055434-Win64.exe
> 
> Maik

Unfortunately, there is no change in this. It starts very slowly.

Additional information:
Settings -- Configure digikam... -- OK
and again
Settings -- Configure digikam... -- (After a very long wait, the contents of the window appear.)
Comment 29 Maik Qualmann 2023-12-06 07:09:15 UTC
*** Bug 478075 has been marked as a duplicate of this bug. ***
Comment 30 Maik Qualmann 2023-12-06 07:09:31 UTC
*** Bug 478131 has been marked as a duplicate of this bug. ***
Comment 31 Maik Qualmann 2023-12-06 07:23:28 UTC
As a note to everyone for whom our current digiKam version, created with Qt6 and  Microsoft Komplier, is too slow, here is a version that was created as before:

https://download.kde.org/stable/digikam/8.2.0/digiKam-8.2.0-Qt5-Win64.exe

I have now tested both versions here with all database backends on a Windows 10 machine. At the moment "only" with a collection of 50,000 images but with a lot of albums. But I have no difference when starting, performing database operations or opening the configuration settings.

I therefore suspect that it may be related to runtime libraries that users have previously installed. I therefore ask you to install/update the "Visual C++ Redistributable for Visual Studio 2015".

Maik
Comment 32 caulier.gilles 2023-12-06 08:05:25 UTC
>I therefore suspect that it may be related to runtime libraries that users have previously installed. I therefore ask you to install/update the "Visual C++ Redistributable for Visual Studio 2015".

Hi Maik,

Why the MSVC 2015 ? digiKam is currently compiled with MSVC 2022. When i bundle the installer i plug the redistributable dll from the MSVC installed from my computer of course:

https://invent.kde.org/graphics/digikam/-/blob/master/project/bundles/vcpkg/04-build-installer.sh?ref_type=heads#L305

I miss something here ?

I also installed the current 8.3.0 on my Windows 10 computer in my office where no MSVC is installed, and it work properly. The MXE installer was installed before also and have been uninstalled when updating the application.

Gilles
Comment 33 caulier.gilles 2023-12-06 08:09:38 UTC
Another point about redistributable Dlls from M$ (that i never understand why these files are not included as well in Windows update) is the performance and architecture support:

- arm | intel
- single thread | multithreads (openmp)

In digiKam, i put the intel/multithreads versions only.

Gilles
Comment 34 caulier.gilles 2023-12-06 08:14:52 UTC
MSVC redistributable notice :

https://learn.microsoft.com/en-us/cpp/windows/redistributing-visual-cpp-files?view=msvc-170

Gilles
Comment 35 caulier.gilles 2023-12-06 08:25:28 UTC
This is the list of redistributable dlls :

https://learn.microsoft.com/en-us/cpp/windows/determining-which-dlls-to-redistribute?view=msvc-170

Included in digiKam bundle are Intel 64 bits files with non debug symbols:

- vcomp140.dll (missing in 8.2.0, included in 8.3.0)
- vcruntime140.dll + vcruntime140_1.dll
- msvcp140.dll + msvcp140_1.dll + msvcp140_2.dll + msvcp140_atomic_wait.dll + msvcp140_codecvt_ids.dll
- concrt140.dll 

Do i need to put more files in the bundle ?

Gilles
Comment 36 Maik Qualmann 2023-12-06 09:06:44 UTC
Hmm, why 2015 good question, it's the version, if I see it correctly, that also applies to 2022.
I think that our DLLs in the digiKam path will not be used if they are already installed in the system. So we would probably have to change the library search path. I know from other Windows software that they include the Microsoft DLL installer and run it beforehand.

But this may not be the reason why the digiKam version is so slow for some users. And there is another reason...

Maik
Comment 37 caulier.gilles 2023-12-06 09:14:19 UTC
By experience, the dlls search path priority under Windows is local application path first and after system path. This allows to mix different versions of dlls without conflict.

Gilles
Comment 38 Maik Qualmann 2023-12-06 09:26:10 UTC
Afterwards, the application path is only in the 7th position, but before the system directory.

https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order

Maik
Comment 39 Peter 2023-12-06 09:54:46 UTC
(In reply to Maik Qualmann from comment #31)
> As a note to everyone for whom our current digiKam version, created with Qt6
> and  Microsoft Komplier, is too slow, here is a version that was created as
> before:
> 
> https://download.kde.org/stable/digikam/8.2.0/digiKam-8.2.0-Qt5-Win64.exe
> 
> I have now tested both versions here with all database backends on a Windows
> 10 machine. At the moment "only" with a collection of 50,000 images but with
> a lot of albums. But I have no difference when starting, performing database
> operations or opening the configuration settings.
> 
> I therefore suspect that it may be related to runtime libraries that users
> have previously installed. I therefore ask you to install/update the "Visual
> C++ Redistributable for Visual Studio 2015".
> 
> Maik

Hi Maik.
I installed Visual C++ Redistributable for Visual Studio a new version (14.38.33130).
Unfortunately, it didn't solve the problem.
Comment 40 Peter 2023-12-06 10:01:11 UTC
Created attachment 163935 [details]
Versions currently available in Windows
Comment 41 nonobio 2023-12-08 10:01:46 UTC
Hi,

I'm not sure if it is the same issue but I have a lot of time latency with the" Digikam not responding" message since I installed Digikam 8.3.0, not at startup but when I use Digikam.
Comment 42 Peter 2023-12-08 10:09:28 UTC
(In reply to nonobio from comment #41)
> Hi,
> 
> I'm not sure if it is the same issue but I have a lot of time latency with
> the" Digikam not responding" message since I installed Digikam 8.3.0, not at
> startup but when I use Digikam.

The problem occurs in different ways. I mentioned the Configure window earlier.
Comment 43 nonobio 2023-12-08 10:19:49 UTC
The Configure Digikam window seems to open fine for me.
Comment 44 Peter 2023-12-09 09:57:36 UTC
(In reply to Maik Qualmann from comment #26)
> Please compare it again with this current version:
> 
> https://files.kde.org/digikam/digiKam-8.3.0-20231205T055434-Win64.exe
> 
> Maik

Hi Maik,
I downloaded digiKam-8.3.0-Qt5-20231207T072634-Win64.exe version from https://files.kde.org/digikam/legacy/
This works fine for me!
Comment 45 caulier.gilles 2023-12-10 22:12:33 UTC
*** Bug 478373 has been marked as a duplicate of this bug. ***
Comment 46 Maik Qualmann 2023-12-11 11:52:41 UTC
*** Bug 478396 has been marked as a duplicate of this bug. ***
Comment 47 caulier.gilles 2023-12-11 17:58:48 UTC
MAik,

With Qt6, under Linux, i can reproduce a time latency at startup. It's happen when QtAV try to init the audio output on the system.

Typically, We don't needs QtAV at all with Qt6, even if videoslideshow is the last part using few QtAV API. This last one must be ported to another solution, as a direct ffmpeg API calls, or the ffmpeg CLI tool, or future QtMultimedia API (not existing in Qt6, but present in Qt5...)

Gilles
Comment 48 Maik Qualmann 2023-12-11 18:23:01 UTC
When I look at the DebugVew logs, it seems to be something related to the DatesJob. There are these bug reports and slow QDateTime comparisons. 
Note the post from KPhotAlbum:

https://bugreports.qt.io/browse/QTBUG-75585
https://bugreports.qt.io/browse/QTBUG-41714

Maik
Comment 49 caulier.gilles 2023-12-13 19:51:10 UTC
Hi Peter,

Today i made an important change in the Windows native version based on Qt6. I dropped the internal legacy video player based on QtAV framework.

I would to know if the current 8.3.0 Windows pre-release installer improve the situation about this report.

Thanks in advance.

Best

Gilles Caulier
Comment 50 Peter 2023-12-13 20:26:51 UTC
(In reply to caulier.gilles from comment #49)
> Hi Peter,
> 
> Today i made an important change in the Windows native version based on Qt6.
> I dropped the internal legacy video player based on QtAV framework.
> 
> I would to know if the current 8.3.0 Windows pre-release installer improve
> the situation about this report.
> 
> Thanks in advance.
> 
> Best
> 
> Gilles Caulier

Hi Gilles,

I tested digiKam-8.3.0-20231213T161333-Win64.exe
Unfortunately, this version also starts slowly. The problem was not solved for me.
If this version doesn't include the fix (dropped QtAV framework), I'll test again later.
Comment 51 Peter 2023-12-13 20:38:06 UTC
I took a screenshot in process explorer while my digiKam is not responding.
digikam seems to closing Qt6Core threads. When you're done, digiKam will start working again.

Maybe this info will help...
Comment 52 Peter 2023-12-13 20:38:33 UTC
Created attachment 164149 [details]
Qt6Core.dll
Comment 53 Peter 2023-12-13 21:11:13 UTC
(In reply to Peter from comment #52)
> Created attachment 164149 [details]
> Qt6Core.dll

Here is a video of the entire start process in process explorer
https://youtu.be/ocgNrsmGUiw
Comment 54 Peter 2023-12-13 21:40:07 UTC
(In reply to Peter from comment #53)
> (In reply to Peter from comment #52)
> > Created attachment 164149 [details]
> > Qt6Core.dll
> 
> Here is a video of the entire start process in process explorer
> https://youtu.be/ocgNrsmGUiw

But the Crash status of Qt6WebEngineCore.dll might be interesting...
Comment 55 caulier.gilles 2023-12-14 03:58:15 UTC
Maik,

Currently the Qt version used by digiKam through VCPKG under Windows is the 6.6.0. The last 6.6.1 is out in VCPKG. Perhaps this can fix the problem.

Another improvement can be the massive update of all KDE 6 frameworks.

Both requires to recompile all from scratch on my Win10 VM of course....

Gilles
Comment 56 Maik Qualmann 2023-12-14 07:11:04 UTC
@Gilles, can you please create a new Windows bundle, I want to exclude the DatesJob or not.

@Peter, as a test you can sort your albums and items by name/path instead of by date. Does it make a difference?

https://invent.kde.org/graphics/digikam/-/commit/d5c951e7791873832f2f2b9d748c94036c152c3c

Maik
Comment 57 caulier.gilles 2023-12-14 07:21:22 UTC
Maik, sure this evening I think
Comment 58 Maik Qualmann 2023-12-14 19:53:29 UTC
Git commit 13b7cb786389e7ac05fc86533f7b498ed335b05e by Maik Qualmann.
Committed on 14/12/2023 at 20:52.
Pushed by mqualmann into branch 'master'.

add a timer to each Action Thread job

M  +9    -5    core/libs/threads/actionthreadbase.cpp
M  +5    -0    core/libs/threads/actionthreadbase.h

https://invent.kde.org/graphics/digikam/-/commit/13b7cb786389e7ac05fc86533f7b498ed335b05e
Comment 59 caulier.gilles 2023-12-14 22:06:44 UTC
Maik,

New Windows bundle is online

Gilles
Comment 60 Peter 2023-12-15 06:33:50 UTC
(In reply to Maik Qualmann from comment #56)
> @Gilles, can you please create a new Windows bundle, I want to exclude the
> DatesJob or not.
> 
> @Peter, as a test you can sort your albums and items by name/path instead of
> by date. Does it make a difference?
> 
> https://invent.kde.org/graphics/digikam/-/commit/
> d5c951e7791873832f2f2b9d748c94036c152c3c
> 
> Maik

Hi Maik,
Tested in digiKam-8.3.0-20231214T211706-Win64.exe
Unfortunately it doesn't work for me (tested on two Windows 10 pro 22H2)

Settings:
View -- Sort Albums -- By Folder
View -- Sort Items -- By Name
Comment 61 Maik Qualmann 2023-12-15 06:54:37 UTC
The commit doesn't fix the problem either. He adds timers to try to identify which process is causing the slowdown. So now I need a new DebugView Log where the problem occurs.

Maik
Comment 62 caulier.gilles 2023-12-15 07:04:32 UTC
Maik,

Depending of the next debugview results and the my free time while this week end, i will plan this week end a major update of the Win10 VM to rebuild VCPKG from scratch. This will include a bump of Qt6.6.0 to 6.6.1 and all KF6 components.

Let's me hears if it can be a good time for this.

Gilles
Comment 63 Peter 2023-12-15 07:27:19 UTC
(In reply to caulier.gilles from comment #62)
> Maik,
> 
> Depending of the next debugview results and the my free time while this week
> end, i will plan this week end a major update of the Win10 VM to rebuild
> VCPKG from scratch. This will include a bump of Qt6.6.0 to 6.6.1 and all KF6
> components.
> 
> Let's me hears if it can be a good time for this.
> 
> Gilles

Yes, I will have time to test.
Comment 64 Peter 2023-12-15 07:28:27 UTC
Created attachment 164178 [details]
20231215_debugview_log
Comment 65 Peter 2023-12-15 07:37:18 UTC
Created attachment 164179 [details]
A log the after "configure digiKam..." opening/closing/opening
Comment 66 Peter 2023-12-15 07:42:38 UTC
Created attachment 164180 [details]
A log the after "configure digiKam..." opening/closing/opening
Comment 67 Peter 2023-12-15 07:49:23 UTC
(In reply to Peter from comment #65)
> Created attachment 164179 [details]
> A log the after "configure digiKam..." opening/closing/opening

Oops, the first 'A log the after "configure digiKam..." opening/closing/opening' is the same as 20231215_debugview_log (sorry)
Comment 68 Maik Qualmann 2023-12-15 09:07:17 UTC
I think this confirms my suspicion.
About 9 seconds to collect the date information and count. This still happens in the JobThread. But 95 seconds to create the date map in the GuiThread is extreme (AlbumManager::slotDatesJobData).
Qt recommends using UTC if a lot of QDateTime comparisons are necessary, the problem arises from converting time zones and daylight saving time etc. Maybe we can also move something to the JobThread.

00000501	19.40586090	[6100] digikam.general: Dates DB Job time: 8830	
00000502	105.08583832	[6100] digikam.general: Dates create time: 94511	
00000503	105.12340546	[6100] digikam.general: One job is done Digikam::DatesJob(0x22a4c3b10f0) time: 94543

Maik
Comment 69 Maik Qualmann 2023-12-15 09:15:30 UTC
I think we can solve the biggest timing problem if we disable sorting while adding date items to the date list view. So that the list view is not sorted again every time an item is added.

Maik
Comment 70 caulier.gilles 2023-12-15 09:31:33 UTC
But one Q is : why this time latency do not appears in Qt5 version ?

Gilles
Comment 71 Peter 2023-12-20 07:34:41 UTC
(In reply to caulier.gilles from comment #62)
> Maik,
> 
> Depending of the next debugview results and the my free time while this week
> end, i will plan this week end a major update of the Win10 VM to rebuild
> VCPKG from scratch. This will include a bump of Qt6.6.0 to 6.6.1 and all KF6
> components.
> 
> Let's me hears if it can be a good time for this.
> 
> Gilles

HI Gilles,
Were you able to solve this over the weekend?

Regards,
Peter
Comment 72 Maik Qualmann 2023-12-20 07:38:02 UTC
@Peter, do you have a “full” trash with lots of files in your collections?

Maik
Comment 73 Peter 2023-12-20 07:40:21 UTC
(In reply to Maik Qualmann from comment #72)
> @Peter, do you have a “full” trash with lots of files in your collections?
> 
> Maik

Yes!
Comment 74 caulier.gilles 2023-12-20 07:46:20 UTC
>> Maik,
>> 
>> Depending of the next debugview results and the my free time while this week
>> end, i will plan this week end a major update of the Win10 VM to rebuild
>> VCPKG from scratch. This will include a bump of Qt6.6.0 to 6.6.1 and all KF6
>> components.
>> 
>> Let's me hears if it can be a good time for this.
>> 
>> Gilles

>HI Gilles,
>Were you able to solve this over the weekend?

Hi Peter,

There is no problem here, just a possibility to update internal framework to last version, in case of... As Maik apply some fixes to digiKam codes, i will delay this task probably while Christmas holiday...

Gilles
Comment 75 Peter 2023-12-20 07:50:49 UTC
(In reply to Peter from comment #73)
> (In reply to Maik Qualmann from comment #72)
> > @Peter, do you have a “full” trash with lots of files in your collections?
> > 
> > Maik
> 
> Yes!

The collection is owned by a small museum. The collection is currently being cleaned, so I rarely empty the bin.
Current bin sizes (two collections): 73GB and 365GB
Comment 76 Maik Qualmann 2023-12-20 08:51:24 UTC
For the trash problem, check for bug 478722.
Before you delete the trash, wait for the next digiKam test version.
A test here shows a slowdown of 1.3 seconds when calculating the size of the trash with my slow network collection, even with 700 files in the trash. And the calculation is called in the QTreeView model with every refresh.

Maik
Comment 77 Maik Qualmann 2023-12-24 07:01:10 UTC
Git commit f637ec4c07185389b18872150511f55b3ba7e389 by Maik Qualmann.
Committed on 24/12/2023 at 08:00.
Pushed by mqualmann into branch 'master'.

for a test force raster widgets

M  +6    -0    core/app/main/main.cpp

https://invent.kde.org/graphics/digikam/-/commit/f637ec4c07185389b18872150511f55b3ba7e389
Comment 78 caulier.gilles 2023-12-24 09:20:15 UTC
Note : digiKam 8.3.0 pre-release Windows installer updated at usual place with last Maik changes and ready for testing...
Comment 79 Peter 2023-12-24 10:44:54 UTC
Created attachment 164419 [details]
digiKam-8.3.0-20231224T080451-Win64.exe

Tested: digiKam-8.3.0-20231224T080451-Win64.exe
Unfortunately, the situation is not better. :-(
Log attached.
Comment 80 Maik Qualmann 2023-12-24 11:02:40 UTC
What surprises me about your logs is that you have activated the initial scan:

 16.33045959   [7800] digikam.general: scan mode: ScanDeferredFiles	
 18.13652802   [7800] digikam.general: total scan value :  202908	
102.61582184  [7800] digikam.general: total scan value :  214066

But the debug output never shows how long the initial scan took. Do you close digiKam before the initial scan is finished, or do we have a problem here?

Maik
Comment 81 Peter 2023-12-24 11:12:37 UTC
(In reply to Maik Qualmann from comment #80)
> What surprises me about your logs is that you have activated the initial
> scan:
> 
>  16.33045959   [7800] digikam.general: scan mode: ScanDeferredFiles	
>  18.13652802   [7800] digikam.general: total scan value :  202908	
> 102.61582184  [7800] digikam.general: total scan value :  214066
> 
> But the debug output never shows how long the initial scan took. Do you
> close digiKam before the initial scan is finished, or do we have a problem
> here?
> 
> Maik

I stopped the scan after the digiKam became accessible. I'm making a new test.
Comment 82 Peter 2023-12-24 11:23:23 UTC
Created attachment 164420 [details]
"Find new items" is completed
Comment 83 Peter 2023-12-24 11:45:09 UTC
Created attachment 164421 [details]
I disabled the "Find new items" and restarted digiKam.
Comment 84 Peter 2023-12-24 11:51:59 UTC
Created attachment 164422 [details]
A log on the same computer, in digiKam 8.2.
Comment 85 Maik Qualmann 2023-12-24 11:57:44 UTC
I'm sure this Qt bug is the cause as it only affects Windows. What time zone do you live in?

https://bugreports.qt.io/browse/QTBUG-120285

Maik
Comment 86 Peter 2023-12-24 12:06:12 UTC
(In reply to Maik Qualmann from comment #85)
> I'm sure this Qt bug is the cause as it only affects Windows. What time zone
> do you live in?
> 
> https://bugreports.qt.io/browse/QTBUG-120285
> 
> Maik

I live in Central European Time (Hungary)
Comment 87 caulier.gilles 2023-12-24 13:39:48 UTC
Maik,

Even if the Qt bug is not yet fixed in 6.6.1, perhaps it's the good moment to rebuild the VCPKG installation on my Windows 10 VM ? This will update Qt to 6.6.1 and KDE frameworks. What do you think about ?

Note : Qt 6.6.2 is planed in January :

https://wiki.qt.io/Qt_6.6_Release

Gilles
Comment 88 Maik Qualmann 2023-12-24 14:03:37 UTC
Gilles,

I think not much will change at the moment when switching to Qt-6.6.1 and the current KF6.
I suspect it will take hours to compile, if you have the time feel free to do so.

But I'm afraid that the bug won't be fixed in Qt-6.6.2 either. We may have to switch completely to UTC internally.

Maik
Comment 89 caulier.gilles 2023-12-24 14:41:54 UTC
Hi Maik,

First, have an happy Christmas time with your family.

Here with my i9 32 cores + 64 Gb, it take 4 hours. QtWebEngine is the most consuming part to build. with MSVC, 2 hours to build the no debug, and 2 other ones to build the debug. MSVC build both separately, it's different than Linux G++ which mix debug symbols in same targets.

Note : QtWebEngine requires at least 24Gb for 8 cores. It's exponential: 12 cores 32Gb, 16 cores 48Gb, 20 cores 64Gb. So i let 8 cores to prevent broken compilations due to lack of RAM. Funny...

Ok i will stay with Qt 6.0.0 for the moment, to be able to quickly provides a new version if you switch DB time to UTC.

Gilles.
Comment 90 Maik Qualmann 2023-12-25 08:36:26 UTC
Git commit 19be2e0a54364bc5bf23a446e1f84bc0d187fa22 by Maik Qualmann.
Committed on 25/12/2023 at 09:35.
Pushed by mqualmann into branch 'master'.

for a test use UTC for the DatesJob

M  +1    -1    core/libs/database/dbjobs/dbjob.cpp

https://invent.kde.org/graphics/digikam/-/commit/19be2e0a54364bc5bf23a446e1f84bc0d187fa22
Comment 91 caulier.gilles 2023-12-25 09:52:43 UTC
*** Bug 478988 has been marked as a duplicate of this bug. ***
Comment 92 caulier.gilles 2023-12-25 10:37:40 UTC
Peter, 

The new 8.3.0 Windows installer is updated at usual place.

Have an happy Christmas time.

Gilles Caulier
Comment 93 Peter 2023-12-25 10:41:01 UTC
(In reply to caulier.gilles from comment #92)
> Peter, 
> 
> The new 8.3.0 Windows installer is updated at usual place.
> 
> Have an happy Christmas time.
> 
> Gilles Caulier

Thanks Gilles,
I'll test it right away.
Merry Christmas and all the best!
Comment 94 Peter 2023-12-25 10:53:24 UTC
Excellent work! digiKam is now working fast again!

Thank You Maik and Gilles!
Comment 95 Maik Qualmann 2023-12-25 11:43:42 UTC
@Peter, can you post a debug log to see how the DatesJob time changes?
This is also just a test, the date display may differ in the Dates View depending on whether the UTC date <-> local date falls on a different day.

Maik
Comment 96 Peter 2023-12-25 12:38:10 UTC
Created attachment 164438 [details]
digiKam 8.3 log
Comment 97 Maik Qualmann 2023-12-25 18:49:37 UTC
Git commit 0c41c8c23198ef7232c03d16e65349002f60f25b by Maik Qualmann.
Committed on 25/12/2023 at 19:48.
Pushed by mqualmann into branch 'master'.

use UTC representation for the creation and modification time of items

M  +1    -1    NEWS
M  +11   -4    core/libs/database/collection/collectionscanner_scan.cpp
M  +22   -7    core/libs/database/coredb/coredb.cpp
M  +2    -1    core/libs/database/dbjobs/dbjob.cpp
M  +2    -0    core/libs/database/item/containers/iteminfo_properties.cpp
M  +2    -0    core/libs/database/item/scanner/itemscanner.cpp
M  +15   -6    core/libs/database/similaritydb/similaritydb.cpp
M  +1    -0    core/libs/database/thumbsdb/thumbsdb.cpp
M  +4    -0    core/libs/metadataengine/engine/metaengine_exif.cpp
M  +20   -0    core/libs/metadataengine/engine/metaengine_item.cpp
M  +1    -0    core/libs/metadataengine/engine/metaengine_xmp.cpp

https://invent.kde.org/graphics/digikam/-/commit/0c41c8c23198ef7232c03d16e65349002f60f25b
Comment 98 Maik Qualmann 2023-12-25 18:56:04 UTC
This is a relatively big change, I'm sure I may have missed something. It will be further stabilized over the next few weeks.
We don't need to convert the database and can even work with a mix of local time and UTC time.

Maik
Comment 99 caulier.gilles 2023-12-25 19:05:59 UTC
Hi Maik,

I will rebuild the Windows installer tomorrow morning.

What's about the duplicates bugs ? Perhaps we can forward the information about these changes to see the side-effects.

Best
Gilles
Comment 100 Maik Qualmann 2023-12-25 22:15:49 UTC
I always thought that all duplicate bugs also receive the messages?

Maik
Comment 101 caulier.gilles 2023-12-25 22:37:52 UTC
I'm not sure that root file messages are forwarded to all peoples in all duplicates... I will post a message in all files tomorrow  when next installer will be ready to test.
Comment 102 Peter 2023-12-26 14:47:15 UTC
(In reply to caulier.gilles from comment #101)
> I'm not sure that root file messages are forwarded to all peoples in all
> duplicates... I will post a message in all files tomorrow  when next
> installer will be ready to test.

Tested: digiKam-8.3.0-20231226T105122-Win64.exe

Works well.
Comment 103 ivar.ekman 2023-12-29 06:33:27 UTC
digiKam-8.3.0-20231228T113341-Win64-debug.exe is still fails to start identically as version 8.2.0 for me on Windows 10 with the splash stating "Loading tools..." This happens even with a fresh library without any files. Last lines in the logs:

[10612] digikam.general: Finish Main Thread
[10612] digikam.geoiface: ----
[10612] digikam.general: Using  4  CPU core to run threads
[10612] digikam.general: Action Thread run 1 new jobs
[10612] digikam.general: One job is done  Digikam::TagsJob(0x18dfce29410)  time: 44
[10612] digikam.general: Finish Main Thread
Comment 104 Maik Qualmann 2023-12-29 06:59:54 UTC
This is not the same problem, your date job only requires 44 ms. Open a new bug report and post a complete debug log from start to finish.

Maik
Comment 105 Peter 2023-12-29 07:46:49 UTC
(In reply to ivar.ekman from comment #103)
> digiKam-8.3.0-20231228T113341-Win64-debug.exe is still fails to start
> identically as version 8.2.0 for me on Windows 10 with the splash stating
> "Loading tools..." This happens even with a fresh library without any files.
> Last lines in the logs:
> 
> [10612] digikam.general: Finish Main Thread
> [10612] digikam.geoiface: ----
> [10612] digikam.general: Using  4  CPU core to run threads
> [10612] digikam.general: Action Thread run 1 new jobs
> [10612] digikam.general: One job is done  Digikam::TagsJob(0x18dfce29410) 
> time: 44
> [10612] digikam.general: Finish Main Thread

Tested the digiKam-8.3.0-20231228T113341-Win64.exe too.
I have no problem here
Comment 106 ivar.ekman 2023-12-29 10:03:50 UTC
Ok, so this must be different then. I created bug 479148 with full my log.
Comment 107 Maik Qualmann 2024-01-05 11:38:55 UTC
Qt has closed the bug (https://bugreports.qt.io/browse/QTBUG-120285) for version 6.6.2. Qt added a cache for the timezone. I'll test the performance, maybe we'll go back to a local datetime internally.

Maik
Comment 108 caulier.gilles 2024-01-05 12:17:09 UTC
Qt 6.6.2 is planed for the 17 january:

https://wiki.qt.io/Qt_6.6_Release

As usual Qt team delay a little bit the release date. Also, Qt 6.6.2 must be updated to VCPKG. This will delay again the usage of this version under Windows.

So, the possible update in the Windows 10 VM used to compile digiKam can be expected around the 15 February i think...

Gilles