SUMMARY I did not have this problem previously, but suddenly do. Either it cropped up because the size of my image collection has grown substantially, from importing a lot of files from a previous storage location, or it came with the upgrade to 7.2.0. Both of these changes happened at the same time. Here is the current issue. When I import a photo from a memory card in digiKam, even a single photo, it does not appear in the digiKam album where I imported it. It does copy on the filesystem because I can manually browse to it. If I select "Open in File Manager" from digiKam, the photo is there. It does not show up in the album in digiKam right away... and I can't figure out how long it takes to finally appear. If I right click on the album and choose refresh... it does not populate with the new item. It does eventually, but not when I am actually trying to work with the new file. This behavior occurs 100% of the time, and is making digiKam almost useless as an import and edit tool for new images. I import, and then have to manually find it in the filesystem to open it. I suspect it has to do with "Find new items" and how that is processing. It's almost as if it is scanning the entire folder structure, which is massive, vs just scanning the album I just downloaded to or clicked "refresh" for. I assume, again, it relates to my large volume of photos because I only experienced this problem after I decided to use digiKam as my primary management and import solution and imported all of my old files. STEPS TO REPRODUCE 1. Import from Media Card 2. Attempt to find in Album 3. OBSERVED RESULT Newly downloaded/imported file is not shown EXPECTED RESULT Expect to see new file that I just imported. SOFTWARE/OS VERSIONS Windows: macOS: Linux/KDE Plasma: (available in About System) KDE Plasma Version: KDE Frameworks Version: Qt Version: ADDITIONAL INFORMATION
Which kind of database type did you use ? Sqlite ? Mysql interanm ? Mysql or MariaDb server ? Where is hosted your database ? Which settings do you use exactly ? Can you provide a Debuview trace while digiKam is running ? Look details in contribute page for details : https://www.digikam.org/contribute/ Gilles Caulier
If you select the album into which you have just imported images, will "hidden files" be displayed in the status bar at the bottom? Maik
Created attachment 137233 [details] digiKam Debug.LOG Database is SQLite. It is hosted on my main system drive, which is an SSD. (C:\Users\vaart\OneDrive\Pictures\) My single Collection (huge) is hosted on a NAS (system hosting digiKam and NAS are both hard connected to router (non-wifi) Performance from NAS is reasonably quick under normal circumstances. DebugView log attached. Some notes:: I rebooted the system, and started debug and then digiKam off the fresh reboot. I did not have any other programs running. Find New Items task started upon opening digiKam, and it was nearly 10 minutes before it actually showed status above 0%. I let it run until completed, which took approximately 20 minutes to complete. I did open up the settings dialog during this process to get the answers for where the database was hosted, in case you see that in the debug log. Once there were no active processes, I attached a memory card and imported a single raw photo into an album.(Z:/Photos/2021/Raw/Astro/) Find new items scan started. My photo finally appeared in the album after 6 minutes. The find new times progress remained at 0% even after my photo appeared in the album, and remained at 0% for about 12 minutes until disappearing. I didn't notice the progress change, I just saw the process status disappear. I hope this information helps. Thanks for helping me figure this out. I am new to digiKam. I tried to ask this in the users list, but I got no response. On Wed, Mar 31, 2021 at 9:24 PM <bugzilla_noreply@kde.org> wrote: > https://bugs.kde.org/show_bug.cgi?id=435202 > > caulier.gilles@gmail.com changed: > > What |Removed |Added > > ---------------------------------------------------------------------------- > CC| |caulier.gilles@gmail.com > > --- Comment #1 from caulier.gilles@gmail.com --- > Which kind of database type did you use ? > > Sqlite ? Mysql interanm ? Mysql or MariaDb server ? > > Where is hosted your database ? Which settings do you use exactly ? > > Can you provide a Debuview trace while digiKam is running ? Look details in > contribute page for details : > > https://www.digikam.org/contribute/ > > Gilles Caulier > > -- > You are receiving this mail because: > You reported the bug.
Please post the full debugview trace for a better investigation...
Created attachment 137234 [details] attachment-10424-0.html I attached the debug capture log to my last response... what more do you need? I don't understand. On Thu, Apr 1, 2021 at 9:18 AM <bugzilla_noreply@kde.org> wrote: > https://bugs.kde.org/show_bug.cgi?id=435202 > > --- Comment #4 from caulier.gilles@gmail.com --- > Please post the full debugview trace for a better investigation... > > -- > You are receiving this mail because: > You reported the bug.
Oups, i do not the the file attachments... Sorry. I do not see a dysfunction in traces... Gilles Caulier
How many images or sub-folders are in the path of "Z:\Photos\2021\Raw\Astro\"? It takes almost 6 minutes from the start of the scan until the first images are scanned. This is really strange. There should be a large number of folders or the drive is very slow. Maik
I have been developing a folder cache patch for the collection scanner for a long time. I will introduce it in digiKam-7.3.0. We'll see if it brings the hoped-for improvement for network drives. Maik
Created attachment 137239 [details] attachment-18069-0.html There were only FIVE images in that folder when I did the import of the image... It's not scanning for 6 minutes... it hangs for 6 minutes and then does the scan. Same if I scan the entire Collection, which does take a little while... it hangs for 5-6 minutes before it starts scanning. On Thu, Apr 1, 2021 at 12:00 PM Maik Qualmann <bugzilla_noreply@kde.org> wrote: > https://bugs.kde.org/show_bug.cgi?id=435202 > > --- Comment #7 from Maik Qualmann <metzpinguin@gmail.com> --- > How many images or sub-folders are in the path of > "Z:\Photos\2021\Raw\Astro\"? > > It takes almost 6 minutes from the start of the scan until the first > images are > scanned. This is really strange. There should be a large number of folders > or > the drive is very slow. > > Maik > > -- > You are receiving this mail because: > You reported the bug.
Created attachment 137241 [details] image.png I ran a refresh, and while the progress is hung at 0%, here is what I am seeing from Process Monitor: There are TONS of these failed attempts: CreateFile C:\Windows\CSC\v2.0.6\namespace\192.168.2.100 NAME NOT FOUND over and over and over... then a legitimate file creation within my NAS folder structure. I think this is the key to the issue. 192.168.2.100 is the IP for my NAS, which is mounted at Z: Successful file writes look like this: \\192.168.2.100\STORAGE\Photos\Drobo-Robocopy\cpoPhoto\Aperture\Aperture 3 Library.aplibrary\Previews\2011\06\20\20110620-223517 [image: image.png] [image: image.png] On Thu, Apr 1, 2021 at 12:59 PM Chris Poldervaart <vaarticus@gmail.com> wrote: > There were only FIVE images in that folder when I did the import of > the image... It's not scanning for 6 minutes... it hangs for 6 minutes and > then does the scan. > > Same if I scan the entire Collection, which does take a little while... it > hangs for 5-6 minutes before it starts scanning. > > On Thu, Apr 1, 2021 at 12:00 PM Maik Qualmann <bugzilla_noreply@kde.org> > wrote: > >> https://bugs.kde.org/show_bug.cgi?id=435202 >> >> --- Comment #7 from Maik Qualmann <metzpinguin@gmail.com> --- >> How many images or sub-folders are in the path of >> "Z:\Photos\2021\Raw\Astro\"? >> >> It takes almost 6 minutes from the start of the scan until the first >> images are >> scanned. This is really strange. There should be a large number of >> folders or >> the drive is very slow. >> >> Maik >> >> -- >> You are receiving this mail because: >> You reported the bug. > >
Created attachment 137242 [details] image.png
The "NAME NOT FOUND" messages come from Windows. The Windows offline cache ("C:\Windows\CSC") searches for files here. I think digiKam "hangs" when accessing files over the Windows file API. The offline cache can be deactivated under Windows. Maik
Created attachment 137244 [details] attachment-22851-0.html I have changed the mapped drive letter location for the collection folders to the explicit network mapping via IP address. I am currently rebuilding the database (re-scanning) and there are far fewer instances of CSC write attempts. I'll report back if that changes anything once that process is complete. On Thu, Apr 1, 2021 at 1:42 PM Maik Qualmann <bugzilla_noreply@kde.org> wrote: > https://bugs.kde.org/show_bug.cgi?id=435202 > > --- Comment #12 from Maik Qualmann <metzpinguin@gmail.com> --- > The "NAME NOT FOUND" messages come from Windows. The Windows offline cache > ("C:\Windows\CSC") searches for files here. I think digiKam "hangs" when > accessing files over the Windows file API. The offline cache can be > deactivated > under Windows. > > Maik > > -- > You are receiving this mail because: > You reported the bug.
Created attachment 137248 [details] attachment-27279-0.html BTW.... Enable Offline Files was already disabled. On Thu, Apr 1, 2021 at 2:25 PM Chris Poldervaart <vaarticus@gmail.com> wrote: > I have changed the mapped drive letter location for the collection folders > to the explicit network mapping via IP address. I am currently rebuilding > the database (re-scanning) and there are far fewer instances of CSC write > attempts. I'll report back if that changes anything once that process is > complete. > > > On Thu, Apr 1, 2021 at 1:42 PM Maik Qualmann <bugzilla_noreply@kde.org> > wrote: > >> https://bugs.kde.org/show_bug.cgi?id=435202 >> >> --- Comment #12 from Maik Qualmann <metzpinguin@gmail.com> --- >> The "NAME NOT FOUND" messages come from Windows. The Windows offline cache >> ("C:\Windows\CSC") searches for files here. I think digiKam "hangs" when >> accessing files over the Windows file API. The offline cache can be >> deactivated >> under Windows. >> >> Maik >> >> -- >> You are receiving this mail because: >> You reported the bug. > >
Created attachment 137254 [details] attachment-27290-0.html After rebuilding the library with the explicit IP address network location, the same behavior exists. It is still attempting a ridiculous amount of CSC namespace create file operations, which fail. I do not have offline file caching enabled. Is there anything else you can think of to prevent digiKam from behaving this way? On Thu, Apr 1, 2021 at 4:00 PM Chris Poldervaart <vaarticus@gmail.com> wrote: > BTW.... Enable Offline Files was already disabled. > > On Thu, Apr 1, 2021 at 2:25 PM Chris Poldervaart <vaarticus@gmail.com> > wrote: > >> I have changed the mapped drive letter location for the collection >> folders to the explicit network mapping via IP address. I am currently >> rebuilding the database (re-scanning) and there are far fewer instances of >> CSC write attempts. I'll report back if that changes anything once that >> process is complete. >> >> >> On Thu, Apr 1, 2021 at 1:42 PM Maik Qualmann <bugzilla_noreply@kde.org> >> wrote: >> >>> https://bugs.kde.org/show_bug.cgi?id=435202 >>> >>> --- Comment #12 from Maik Qualmann <metzpinguin@gmail.com> --- >>> The "NAME NOT FOUND" messages come from Windows. The Windows offline >>> cache >>> ("C:\Windows\CSC") searches for files here. I think digiKam "hangs" when >>> accessing files over the Windows file API. The offline cache can be >>> deactivated >>> under Windows. >>> >>> Maik >>> >>> -- >>> You are receiving this mail because: >>> You reported the bug. >> >>
There are indications that it could be a Qt bug. Can you test, that your drive is not addressed under a drive letter, but via the UNC path, i.e. \\SERVER\. You don't need to rebuild the digiKam collection either. You can simply change the path to UNC in the collection settings in digiKam with the refresh icon (next to the trash icon). Maik
Created attachment 137262 [details] image.png Perhaps I wasn't very clear in my last few responses, but that is exactly what I changed, which made no impact. See attached photo. [image: image.png] On Fri, Apr 2, 2021 at 2:43 AM Maik Qualmann <bugzilla_noreply@kde.org> wrote: > https://bugs.kde.org/show_bug.cgi?id=435202 > > --- Comment #16 from Maik Qualmann <metzpinguin@gmail.com> --- > There are indications that it could be a Qt bug. Can you test, that your > drive > is not addressed under a drive letter, but via the UNC path, i.e. > \\SERVER\. > You don't need to rebuild the digiKam collection either. You can simply > change > the path to UNC in the collection settings in digiKam with the refresh icon > (next to the trash icon). > > Maik > > -- > You are receiving this mail because: > You reported the bug.
Created attachment 137270 [details] image.png I also ran into a new issue after pointing to the direct UNC path... it seems digiKam can't create new albums now. See photo. I'm not sure if this is all related somehow. [image: image.png] On Fri, Apr 2, 2021 at 6:10 AM Chris Poldervaart <vaarticus@gmail.com> wrote: > Perhaps I wasn't very clear in my last few responses, but that is exactly > what I changed, which made no impact. See attached photo. > > [image: image.png] > > On Fri, Apr 2, 2021 at 2:43 AM Maik Qualmann <bugzilla_noreply@kde.org> > wrote: > >> https://bugs.kde.org/show_bug.cgi?id=435202 >> >> --- Comment #16 from Maik Qualmann <metzpinguin@gmail.com> --- >> There are indications that it could be a Qt bug. Can you test, that your >> drive >> is not addressed under a drive letter, but via the UNC path, i.e. >> \\SERVER\. >> You don't need to rebuild the digiKam collection either. You can simply >> change >> the path to UNC in the collection settings in digiKam with the refresh >> icon >> (next to the trash icon). >> >> Maik >> >> -- >> You are receiving this mail because: >> You reported the bug. > >
Created attachment 137271 [details] image.png
No, in general there is no problem when creating a new album on UNC path collections. I have just tested it again with the current Windows version. However, the problem is sometimes reported to us, the cause is probably that the folder is locked by another program and digiKam cannot therefore create a new album. I looked at the code again when I imported it. After copying, we start a scanning process delayed by a few milliseconds. However, this only scans the folder in which the image was downloaded. The blocking of 6 minutes cannot be explained within digiKam, it has to be done outside. Are you sure that your network drive is not faulty? Maik
Created attachment 137276 [details] attachment-3146-0.html I can't create a new album from any location using digiKam. Trying to create within NAS Photos/2021 I get: Failed to create directory 'file:///2021/New Album' If I try and create it in the root (NAS Photos) I get: Failed to create directory 'file:////New Album' I am concerned the translation to the UNC share isn't working properly. The NAS is brand new, with brand new hard drives. (this was built 2 weeks ago for this purpose) I have gigabit connecttions. Everything else seems to connect and transfer files fine. I will do some more playing. On Fri, Apr 2, 2021 at 10:09 AM Maik Qualmann <bugzilla_noreply@kde.org> wrote: > https://bugs.kde.org/show_bug.cgi?id=435202 > > --- Comment #20 from Maik Qualmann <metzpinguin@gmail.com> --- > No, in general there is no problem when creating a new album on UNC path > collections. I have just tested it again with the current Windows version. > However, the problem is sometimes reported to us, the cause is probably > that > the folder is locked by another program and digiKam cannot therefore > create a > new album. > > I looked at the code again when I imported it. After copying, we start a > scanning process delayed by a few milliseconds. However, this only scans > the > folder in which the image was downloaded. The blocking of 6 minutes cannot > be > explained within digiKam, it has to be done outside. Are you sure that your > network drive is not faulty? > > Maik > > -- > You are receiving this mail because: > You reported the bug.
Created attachment 137279 [details] attachment-10653-0.html Interesting. Restarting digiKam solved the directory creation issue. No other changes. On Fri, Apr 2, 2021 at 11:05 AM Chris Poldervaart <vaarticus@gmail.com> wrote: > I can't create a new album from any location using digiKam. > > Trying to create within NAS Photos/2021 > > I get: Failed to create directory 'file:///2021/New Album' > > If I try and create it in the root (NAS Photos) I get: > Failed to create directory 'file:////New Album' > > I am concerned the translation to the UNC share isn't working properly. > > The NAS is brand new, with brand new hard drives. (this was built 2 weeks > ago for this purpose) I have gigabit connecttions. > > Everything else seems to connect and transfer files fine. > > I will do some more playing. > > > > On Fri, Apr 2, 2021 at 10:09 AM Maik Qualmann <bugzilla_noreply@kde.org> > wrote: > >> https://bugs.kde.org/show_bug.cgi?id=435202 >> >> --- Comment #20 from Maik Qualmann <metzpinguin@gmail.com> --- >> No, in general there is no problem when creating a new album on UNC path >> collections. I have just tested it again with the current Windows version. >> However, the problem is sometimes reported to us, the cause is probably >> that >> the folder is locked by another program and digiKam cannot therefore >> create a >> new album. >> >> I looked at the code again when I imported it. After copying, we start a >> scanning process delayed by a few milliseconds. However, this only scans >> the >> folder in which the image was downloaded. The blocking of 6 minutes >> cannot be >> explained within digiKam, it has to be done outside. Are you sure that >> your >> network drive is not faulty? >> >> Maik >> >> -- >> You are receiving this mail because: >> You reported the bug. > >
Created attachment 137288 [details] attachment-23048-0.html Okay. I've got some good information for you. 1. I was able to finally totally disable CSC. I had to make some registry changes: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\CSC *Start* DWORD *4* = Disable HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\CscService *Start* DWORD *4* = Disable Unfortunately, that didn't solve the problem. It did stop all of the failed attempts to query the CSC folder. But the behavior in digiKam persisted. 2. I think this will help! I recorded a video of the import process of a single file into a small folder. I also had Process Monitor and DebugView running. When digiKam is idle, there are periodic calls to some networking related registry keys. Pretty constant actually, but not really the problem. Then, when I import an image, it appears in DebugView that it starts a task to scan the folder it was imported to, however, as you can see in Process Monitor... it starts a scan of the entire collection! It takes quite a while before it finally gets to the folder that has the import, and then it clears that task. The entire time it is scanning the collection, there are no updates in DebugView, so that's why it looks like it is hung... but digiKam is doing something. Here is the video file link: digiKam test video - YouTube <https://www.youtube.com/watch?v=UW1r21ZAf-g> I assume this is unexpected behavior? I hope this helps. Chris On Fri, Apr 2, 2021 at 2:13 PM Chris Poldervaart <vaarticus@gmail.com> wrote: > Interesting. Restarting digiKam solved the directory creation issue. No > other changes. > > On Fri, Apr 2, 2021 at 11:05 AM Chris Poldervaart <vaarticus@gmail.com> > wrote: > >> I can't create a new album from any location using digiKam. >> >> Trying to create within NAS Photos/2021 >> >> I get: Failed to create directory 'file:///2021/New Album' >> >> If I try and create it in the root (NAS Photos) I get: >> Failed to create directory 'file:////New Album' >> >> I am concerned the translation to the UNC share isn't working properly. >> >> The NAS is brand new, with brand new hard drives. (this was built 2 >> weeks ago for this purpose) I have gigabit connecttions. >> >> Everything else seems to connect and transfer files fine. >> >> I will do some more playing. >> >> >> >> On Fri, Apr 2, 2021 at 10:09 AM Maik Qualmann <bugzilla_noreply@kde.org> >> wrote: >> >>> https://bugs.kde.org/show_bug.cgi?id=435202 >>> >>> --- Comment #20 from Maik Qualmann <metzpinguin@gmail.com> --- >>> No, in general there is no problem when creating a new album on UNC path >>> collections. I have just tested it again with the current Windows >>> version. >>> However, the problem is sometimes reported to us, the cause is probably >>> that >>> the folder is locked by another program and digiKam cannot therefore >>> create a >>> new album. >>> >>> I looked at the code again when I imported it. After copying, we start a >>> scanning process delayed by a few milliseconds. However, this only scans >>> the >>> folder in which the image was downloaded. The blocking of 6 minutes >>> cannot be >>> explained within digiKam, it has to be done outside. Are you sure that >>> your >>> network drive is not faulty? >>> >>> Maik >>> >>> -- >>> You are receiving this mail because: >>> You reported the bug. >> >>
Git commit 6741b9e2ebac47e8916113e92f8dea543fea43c5 by Maik Qualmann. Committed on 04/04/2021 at 11:15. Pushed by mqualmann into branch 'master'. no partial collection scan, only scan the imported file M +2 -2 core/utilities/import/main/importui.cpp https://invent.kde.org/graphics/digikam/commit/6741b9e2ebac47e8916113e92f8dea543fea43c5
Git commit 1ebdda5af47bd263a701da7b5893488d3072e24b by Maik Qualmann. Committed on 04/04/2021 at 11:24. Pushed by mqualmann into branch 'master'. ItemInfo is now up to date when we want to rotate M +1 -4 core/utilities/import/main/importui.cpp https://invent.kde.org/graphics/digikam/commit/1ebdda5af47bd263a701da7b5893488d3072e24b
Git commit f59f30aa7a828bd3dbc6c7c71a2341c43db9c4ce by Maik Qualmann. Committed on 04/04/2021 at 11:44. Pushed by mqualmann into branch 'master'. the new items finder is no longer necessary FIXED-IN: 7.3.0 M +2 -1 NEWS M +0 -21 core/utilities/import/main/importui.cpp M +0 -1 core/utilities/import/main/importui_p.h https://invent.kde.org/graphics/digikam/commit/f59f30aa7a828bd3dbc6c7c71a2341c43db9c4ce
Created attachment 137345 [details] attachment-3227-0.html I downloaded the 7.3.0 weekly build and can confirm that the import process is greatly improved! Thanks! On Sun, Apr 4, 2021 at 7:45 AM Maik Qualmann <bugzilla_noreply@kde.org> wrote: > https://bugs.kde.org/show_bug.cgi?id=435202 > > Maik Qualmann <metzpinguin@gmail.com> changed: > > What |Removed |Added > > ---------------------------------------------------------------------------- > Status|REPORTED |RESOLVED > Resolution|--- |FIXED > Version Fixed In| |7.3.0 > Latest Commit| | > https://invent.kde.org/grap > | > |hics/digikam/commit/f59f30a > | > |a7a828bd3dbc6c7c71a2341c43d > | |b9c4ce > > --- Comment #26 from Maik Qualmann <metzpinguin@gmail.com> --- > Git commit f59f30aa7a828bd3dbc6c7c71a2341c43db9c4ce by Maik Qualmann. > Committed on 04/04/2021 at 11:44. > Pushed by mqualmann into branch 'master'. > > the new items finder is no longer necessary > FIXED-IN: 7.3.0 > > M +2 -1 NEWS > M +0 -21 core/utilities/import/main/importui.cpp > M +0 -1 core/utilities/import/main/importui_p.h > > > https://invent.kde.org/graphics/digikam/commit/f59f30aa7a828bd3dbc6c7c71a2341c43db9c4ce > > -- > You are receiving this mail because: > You reported the bug.