Bug 474655

Summary: Still reading files after exiting digiKam.
Product: [Applications] digikam Reporter: Kane <shikai.xu>
Component: Database-ThumbsAssignee: Digikam Developers <digikam-bugs-null>
Status: REPORTED ---    
Severity: normal CC: caulier.gilles, metzpinguin
Priority: NOR    
Version: 8.2.0   
Target Milestone: ---   
Platform: macOS (DMG)   
OS: macOS   
Latest Commit: Version Fixed In:
Sentry Crash Report:

Description Kane 2023-09-18 14:44:17 UTC
SUMMARY
***
I have tons of photos (over 600GB) over SMB server, they've all been added in my Digikam collections. I'm using SQLITE db in local SSD as suggested. I'm using Wifi 5G and the siginal is just good, not perfect to get the Wifi upmost speed.

By ensure "No active process" from the status bar and then I hit cmd+Q to exist Digikam, the program is still active in background and reading photos on the SMB drive.
I check my SMB server status (`sudo smbstatus`) rapidaly/frequently and I can see my photo folder are being used/scaned, I don't have any other program accessing the folder so it's absolutely Digikam doing it. And this lasts for serveral minutes.
***


STEPS TO REPRODUCE
1.  Prepare tons of photos and add them into Digikam collections. Prepare a not ver fast enviroment for data trasmission.
2.  Open Digikam, wait until the bottom status bar showing "No active process"
3.  Quit Digikam
4.  Check process monitor in local machine, check smbstatus in smb server

OBSERVED RESULT
in local machine, Digikam still running in background
in SMB server, photo folders are being read
and this lasts for a very long time

EXPECTED RESULT
when user closes the program, and the main window disappeared, process should be end soon
if it's still scanning photos, user should be notified and whatever Digikam is doing can be stoppable

SOFTWARE/OS VERSIONS
macOS: 10.15.7

ADDITIONAL INFORMATION
Comment 1 Maik Qualmann 2023-09-18 15:20:30 UTC
This is normal, thumbnails are created. There is no display in the progress manager. Only when all thumbnails have been created will there no longer be a process that accesses the SMB share. The thumbnail preloading process is always triggered when you open an album whose thumbnails have not yet been created.

Maik
Comment 2 Maik Qualmann 2023-09-18 15:22:48 UTC
You can also have all thumbnails created in the maintenance tool, but this will take some time with such a large collection. Otherwise they are created on the fly.

Maik
Comment 3 Maik Qualmann 2023-09-18 15:37:02 UTC
After closing digiKam, the thumbnail process must of course stop. If this is not the case under macOS, we need a log from the terminal.

Maik
Comment 4 Maik Qualmann 2023-09-18 15:41:01 UTC
Please test the pre-release version digiKam-8.2.0, the problem should be fixed there.

https://files.kde.org/digikam/

Maik
Comment 5 Kane 2023-09-19 00:18:25 UTC
Hi Maik,

Thanks for looking into this.
Before I try the 8.2 pre-release verison, I want to check that if I run it, will it make some setting/config file in system folder incompatible if I go back and run 8.1?

Also, could you please instruct me where can I find/enable the log you mentioned in macOS?

Thanks,
Kane
Comment 6 Maik Qualmann 2023-09-19 06:06:09 UTC
No, there are no database changes. There is a change that is slightly bigger. We create a digikam.uuid file in the collection (in the .dtrash folder) to identify the collection based on it. For this purpose, the root entry of the collection in the database is also expanded to include this UUID. This means that there will no longer be a problem under macOS after a major update if Apple changes the partition UUID. This change in digiKam-8.2.0 is backwards compatible with older digiKam versions.
Of course, it can't hurt to make a backup of the SQLite files, which can be done quickly.

Maik
Comment 7 Kane 2023-09-19 20:43:29 UTC
Sure I'll backup my db files. But I was asking some extra files in system folders about software configurations(e.g, ~/Library/Application Support/... ), or there isn't any?
And, about the log, do you know how can I capture it?

Thanks,
Kane
Comment 8 Kane 2023-09-21 08:48:36 UTC
Hi Maik,

I've encounter another issue, but I believe the cause is same so I'm not creating new ticket but posting it here. Below are the steps:

1, prepare a not-very-fast environment, like mine, photo files are in SMB server, connected by 5G wifi
2, add some big collection/photos. on my scenario it's a 600G+ photos, already added into Digikam collections
2.1, when I say "already added", I mean if I open Digikam, do nothing, and exit Digikam, the process will end soon after main window disappears, this is normal.("Scan for new item at starup" option is disabled)
3, add another collection over SMB, the new collection/folder contains no files, or only 1 file

here's the problem:
4, After adding, Digikam's status is indicating "Scan for new items", and this lasts a long time. However, my new collection is empty or just 1 file, shouldn't take that much of time
5, I couldn't wait and then I clicked "stop icon button", and "Aborting..." shows for some seconds, then, another session of "Scan for new items" started, also it lasts for a very long time
6, I still couldn't wait, so I clicked "stop icon button" again, "Aborting..." shows for some seconds, and then status bar indicated "No active process"
7, Now, the new collestion is not listed in "Collection" tab
8, Now I close Digikam, but the process remains. I check `smbstatus` on my SMB server, folders of old collections are being read
9, I terminated Digikam process and opened Digikam again, my new collection is still not listed. I guess it's due to the "scan" wasn't done earlier

In summary, 
1, Why are there two sessions of "Scan for new items"? (and one "hidden session" even after I close Digikam)
2, Could it display more info, like "Scan for new items (path/file)"?
3, Why it needs to scan existing folder when I just add a new one. This doesn't make sense. If everytime I just add a new collection/folder and it needs hours to re-scan old files, it's too much time consuming :-(
4, Hopefully the process should be terminated right after I quit the program

Thanks,
Kane
Comment 9 Maik Qualmann 2023-09-21 09:21:44 UTC
The behavior has already been reported, but is intentional. When you add a single collection, all collections are scanned. For digiKam to work, the entire collection must be scanned. It makes no sense to constantly interrupt digiKam in the scanning process. With a slow network connection it may take days, but before then you will not be able to work with digiKam and your images.

Maik
Comment 10 Maik Qualmann 2023-09-21 09:25:40 UTC
Please note the recommended requirements for the collection types. For a network collection it is Gigabit Ethernet.

https://docs.digikam.org/en/setup_application/database_settings.html#database-type-criteria

Maik
Comment 11 Kane 2023-09-21 09:32:27 UTC
(In reply to Maik Qualmann from comment #10)
> Please note the recommended requirements for the collection types. For a
> network collection it is Gigabit Ethernet.
> 
> https://docs.digikam.org/en/setup_application/database_settings.
> html#database-type-criteria
> 
> Maik

Again, thanks Maik, for looking into my issue.
The document you provided is about db, my db is in my local ssd. I'm only talking about my photo files.

I'm still confused why it is intentionaly to scan all collections when adding a new one?
So, each time I add a new collection, I have to wait overnight? (I used ethernet cable connection earlier when adding my 600G+ photos, and still it took overnight)
Comment 12 Maik Qualmann 2023-09-21 09:38:09 UTC
These recommended settings also apply to the collection itself.

Maik
Comment 13 caulier.gilles 2023-10-12 14:52:25 UTC
@Kane,

What's about this file using current 8.2.0 MacOS PKG bundle ? It's
reproducible ?

https://files.kde.org/digikam/

Note: bundle is now based on Qt 5.15.11 and KDE framework 5.110.

Thanks in advance

Gilles Caulier
Comment 14 Kane 2023-10-13 01:26:01 UTC
(In reply to caulier.gilles from comment #13)
> @Kane,
> 
> What's about this file using current 8.2.0 MacOS PKG bundle ? It's
> reproducible ?
> 
> https://files.kde.org/digikam/
> 
> Note: bundle is now based on Qt 5.15.11 and KDE framework 5.110.
> 
> Thanks in advance
> 
> Gilles Caulier

hi Gilles,

Thanks for looking into this, I was thinking this bug was abandoned as Maik said the storage requirment for db is also for collections...

I'll try the latest 8.2 and do some tests and update you later :-)
Comment 15 Kane 2023-10-13 15:28:20 UTC
hi Gilles,

just tried the latest version:
Build date: 2023/10/6 08:06 (target: Debug)
Revision: de06fd28e3055db07ae6491c7310049a104965de

and what I described in comment #8 it's now still the same.

I myself do findout a workaround:
disconnect the smb drive of the unnecessary collections while I'm doing adding/removing other collections. 

Thanks,
Kane
Comment 16 caulier.gilles 2024-09-17 13:31:51 UTC
Maik, 

What's the status of this file ?

Gilles