Bug 377849

The albums that disappear are on network drives to digiKam open when the network connection is lost, or when an error. (Eg when the computer wakes up from standby, the network connection, and thus the network drive is not immediately available).
Once the network drive is accessible again, digiKam will import the whole collection again again.
Depending on the scope of the collections of images this may take several hours.

Why Digikam does not recognize the image collection has already been read on the network drive.

Which kind of database do you use ? mysql or sqlite ? Remote of local ?

i used mysql (MariaDB)
the database is local.

First I used sqlite as a database.
Not even the captions and keywords were saved.

Then I found on digikam.org this article -> https://www.digikam.org/node/219
".... filesystems (check this FAQ entry). But, network ...." http://www.sqlite.org/faq.html#q5
And i have read the sqlite can not deal with network paths.
Then I switched to mysql.

Now the entire assignments (labels and keywords) are saved and are not lost any more, but the problem remains with the loading of the picture folders.

The problem occurs only to whom digiKam is opened and then the network connection breaks. If the connection to the network from the beginning is missing and then opens digiKam is all still there and the images are correctly marked as unavailable.

Is there a solution or workaround for this problem?
Is this behavior known for the Windows installation?
Does this happen with other users?

*** Bug 380971 has been marked as a duplicate of this bug. ***

The albums of network folders will still disappear if the network connection is lost.

I am also affected by this bug in Windows 10 (64 bit) using Digikam 5.8.0. As soon as the connection is list, all albums disappear and need to be scanned again from the beginning. 

I described this problem in detail in this thread: http://digikam.1695700.n4.nabble.com/All-albums-are-removed-if-there-is-a-network-disconnection-td4704993.html

My pictures are stored in a network shared folder (and the collection is configured as such), but the database (SQLite) is stored locally. This behavior occurs regardless of the "scan on startup" and "clean database options". I tried to reproduce it in Linux (also Digikam 5.8.0) but a network interruption did not affect existing albums, so it seems to be a windows only problem.

I think the problem is the QFileSystemWatcher. Actually, a well thought-out idea and facility in digiKam, a file that is modified or deleted outside, is updated right in the database. If the connection is lost on Windows, digiKam is notified that there was a change in the directory and can not find any images anymore and remove them from the database. It actually seems that under Linux the QFileSystemWatcher does not send a message, even if a removable medium is removed hard.

My 3 ideas:

1. We remove QFileSystemWatcher kommplett under Windows, which also gives it further problems and we have installed workarounds (locked directories)

2. We only add new items or update changed ones, but do not delete any.

3. We test before we delete items or directories from the database, if a hidden file in the root path is still readable.

Maik

I can not reproduce the problem here right now under a virtual Windows VM. If I disconnect the network connection to the host computer, the images will not be deleted in the database. A hard disconnect I can not reproduce with the virtual machine. I'll test it at work tomorrow.

Maik

*** Bug 383711 has been marked as a duplicate of this bug. ***

*** Bug 392088 has been marked as a duplicate of this bug. ***

What is the current status about this?
I am having the same problem.

I launched digiKam yesterday afternoon at home and my network collection wasn't there. It started the scan (which took the whole night) and in the morning the collection looked good.
Then I put my notebook to standby, drove to work and now my collection is gone again :(

The main cause is probably missing Windows support in the Solid component. DigiKam will not be notified that the drive / collection is no longer available. The QFileSystemWatcher then triggers the scan of the collection and removes all albums. I'm going to release a special Windows version on my GDrive this evening with a workaround for testing. It would be nice if you would test it and my guess would be confirmed.

Maik

Maik ,

> The main cause is probably missing Windows support in the Solid component.

This is exactly the main problem.

Solutions :

1/ Implement the missing code in Solid interface for Windows
2/ Fix the code in Solid for Windows if it's exist.

It's not simple...

Gilles

Today I tested digiKam-5.9.0 on a real Windows machine. When I look at the messages in DebugView, it looks like Solid is working correctly, reporting removed removable media and network shares. Only, the QFileSytemWatcher is most faster as Solid. With digiKam-6.0.0 we have already delayed the external events of QFilesytemWatcher by 2 seconds. Maybe it will be enough. Another idea, we make the QFilesystemWatcher configurable. With an album selector everyone can enabled the shared monitored albums. The resources of the QFileSytemWatcher are limited, see Qt-Doc. Here on Linux, after about 30000-40000 albums, the kernel resources are exhausted and no more albums are added to the monitoring. I think that we generally monitor all albums is not a good idea.

Maik

For what it's worth, I used to have this problem frequently (see 383711) but the problem completely disappeared by itself. The only things that have changed in that time are:

- Synology NAS DSM version minor updates.
- Windows 10 feature updates.
- DigiKam updates.
- Homeplug Ethernet adapters upgraded.


Just now I deliberately forced the problem by removing the Ethernet cable from my PC, started the PC then opened DigiKam. All my photo thumbnails were present and there was no attempt to rescan. I restarted again with the Ethernet cable attached and everything was fine. No rescan.

Let me know if you need more info.

Also tried dismounting the network drive in Windows instead of removing the Ethernet cable. Again digiKam was unaffected.

The lines from the DebugView tool (download from Microsoft) would be interesting if they interrupt the network connection while digiKam is running.

Maik

I just managed to trigger the problem by messing around with the Homeplug adapter but I'm not sure of the sequence of events. 

Will try to isolate the trigger over the next few days and provide traces with and without the problem.

I can reproduce the problem at will now:

If I have...
1) digiKam open in thumbnail view after scanning the collection
2) then start DbgView.
3) then pull out Ethernet cable
4) then double click a thumbnail.
Then
5) I get Windows blue spinning wheel for about 15 seconds.
6) Then the album(s) is removed from the albums list and it returns to "Welcome to digiKam" page.


The bad news is that the log below does not show anything useful. The first entry 00000001 occurs at Step 6).


00000001	0.00000000	[10060] digikam.dimg: File  "P:/digiKamTest/Phone Dump/01.jpg"  does not exist	
00000002	0.01410834	[10060] digikam.dimg: File  "P:/digiKamTest/Phone Dump/01.jpg"  does not exist	
00000003	0.01456402	[10060] digikam.metaengine: Cannot load metadata from file   (Error # 9 :  P:/digiKamTest/Phone Dump/01.jpg: Failed to open the data source: No such file or directory (errno = 2)	
00000004	0.01519299	[10060] digikam.general: Detected change, triggering rescan of "P:/digiKamTest/Phone Dump"	
00000005	0.01639067	[10060] digikam.dimg: File  "P:/digiKamTest/Phone Dump/01.jpg"  does not exist	
00000006	0.01658720	[10060] digikam.dimg: "P:/digiKamTest/Phone Dump/01.jpg"  : Unknown image format !!!	
00000007	0.01665337	[10060] digikam.general: Cannot extract preview for  "P:/digiKamTest/Phone Dump/01.jpg"	
00000008	0.01682666	[10060] digikam.general: Cannot extract preview for  "P:/digiKamTest/Phone Dump/01.jpg"	
00000009	0.01932679	[10060] digikam.general: Stacked View Mode :  1	
00000010	0.52178597	[10060] digikam.database: Starting scan!	
00000011	0.53882611	[10060] digikam.database: Removed items: (1349, 1350, 1351, 1352, 1353, 1354, 1355, 1356, 1357, 1358, 1359, 1360, 1361, 1362, 1363, 1364, 1365, 1366, 1367, 1368, 1369, 1384, 1386, 1387, 1388, 1391, 1393, 1394, 1395, 1396, 1397, 1398, 1399, 1400, 1401, 1402, 1403, 1406, 1407, 1408, 1409, 1410, 1412, 1414, 1418, 1421, 1422, 1424, 1425, 1426, 1427, 1434, 1435, 1436, 1437, 1438, 1439, 1440, 1441, 1442, 1443, 1444, 1445, 1446, 1447, 1448, 1449, 1450, 1451, 1452, 1453, 1454, 1455, 1456, 1457, 1458, 1459, 1460, 1461, 1462, 1463, 1464, 1465, 1466, 1467, 1468, 1469, 1470, 1471, 1472, 1473, 1474, 1475, 1476, 1477, 1478, 1479, 1480, 1819, 2452, 2458, 2459, 2460, 2461, 2462, 2463, 2464, 2465, 2466, 2467, 2468, 2471, 2478, 2479, 2480, 2482, 2483, 2484, 2485, 2486, 2487, 2488, 2490, 2491, 2492, 2493, 2494, 2495, 2496, 2498, 2499, 2500, 2502, 2503, 2506, 2507, 2508, 2510, 2513, 2514, 2516, 2518, 2519, 2520, 2521, 2522, 2523, 2524, 2563, 2564, 2565, 2566, 2567, 2568, 2569, 2570, 2571, 2572, 2573, 2574, 2575, 2576, 2758, 3177, 3178, 3245, 3246, 3247) related items: ()	
00000012	0.56052071	[10060] digikam.database: Folder does not exist or is not readable:  "P:/digiKamTest/Phone Dump"	
00000013	0.64725453	[10060] digikam.geoiface: ----	
00000014	0.65683210	[10060] digikam.geoiface: ----	
00000015	0.65709949	[10060] digikam.general: Using  2  CPU core to run threads	
00000016	0.65866703	[10060] digikam.general: Stacked View Mode :  2	
00000017	0.65894783	[10060] digikam.general: Action Thread run  1  new jobs	
00000018	0.66701651	[10060] digikam.general: Data From DBJobsThread is null:  true	
00000019	0.66712683	[10060] digikam.general: Cancel Main Thread	
00000020	0.66768330	[10060] digikam.general: One job is done	
00000021	0.94223034	[10060] digikam.general: Using  2  CPU core to run threads	
00000022	0.95756429	[10060] digikam.geoiface: ----	
00000023	0.95942563	[10060] digikam.general: Action Thread run  1  new jobs	
00000024	0.96177846	[10060] digikam.general: Cancel Main Thread	
00000025	0.98371530	[10060] digikam.general: Data From DBJobsThread is null:  true	
00000026	0.98453760	[10060] digikam.general: Cancel Main Thread	
00000027	0.98461050	[10060] digikam.general: Cancel Main Thread	
00000028	4.25723648	[10060] digikam.general: Using  2  CPU core to run threads	
00000029	4.25854921	[10060] digikam.general: Action Thread run  1  new jobs	
00000030	4.26113892	[10060] digikam.general: One job is done	
00000031	4.26151943	[10060] digikam.general: Cancel Main Thread

Created attachment 112989 [details]
Power Saving Off

I have a refined way to trigger the issue:

1) digiKam open in thumbnail view after scanning the collection.
2) Start DbgView.
3) In Control Panel\Network and Internet\Network Connections, right click Ethernet adapter and click disable.
4) Without delay this time, the album(s) instantly disappear.

In the log below, the first entry is coincident with Steps 3/4.

It seems that "Starting Scan" and "Removed items" are premature actions if there is no verifiable network connection.

Regardless, I wonder if sufferers of this problem might have Ethernet adapter power saving options enabled. i.e. I do not have "Allow the computer to turn off this device to save power" enabled as shown in the screenshot. I cannot remember if this is the default or I switched this off manually which may explain why I did have the issue and now I don't have the issue.



======================================================

00000001	0.00000000	[2648] digikam.general: Detected change, triggering rescan of "P:/digiKamTest/Phone Dump"	
00000002	0.50904858	[2648] digikam.database: Starting scan!	
00000003	0.51694995	[2648] digikam.database: Removed items: (1349, 1350, 1351, 1352, 1353, 1354, 1355, 1356, 1357, 1358, 1359, 1360, 1361, 1362, 1363, 1364, 1365, 1366, 1367, 1368, 1369, 1384, 1386, 1387, 1388, 1391, 1393, 1394, 1395, 1396, 1397, 1398, 1399, 1400, 1401, 1402, 1403, 1406, 1407, 1408, 1409, 1410, 1412, 1414, 1418, 1421, 1422, 1424, 1425, 1426, 1427, 1434, 1435, 1436, 1437, 1438, 1439, 1440, 1441, 1442, 1443, 1444, 1445, 1446, 1447, 1448, 1449, 1450, 1451, 1452, 1453, 1454, 1455, 1456, 1457, 1458, 1459, 1460, 1461, 1462, 1463, 1464, 1465, 1466, 1467, 1468, 1469, 1470, 1471, 1472, 1473, 1474, 1475, 1476, 1477, 1478, 1479, 1480, 1819, 2452, 2458, 2459, 2460, 2461, 2462, 2463, 2464, 2465, 2466, 2467, 2468, 2471, 2478, 2479, 2480, 2482, 2483, 2484, 2485, 2486, 2487, 2488, 2490, 2491, 2492, 2493, 2494, 2495, 2496, 2498, 2499, 2500, 2502, 2503, 2506, 2507, 2508, 2510, 2513, 2514, 2516, 2518, 2519, 2520, 2521, 2522, 2523, 2524, 2563, 2564, 2565, 2566, 2567, 2568, 2569, 2570, 2571, 2572, 2573, 2574, 2575, 2576, 2758, 3177, 3178, 3245, 3246, 3247) related items: ()	
00000004	0.53222328	[2648] digikam.database: Folder does not exist or is not readable:  "P:/digiKamTest/Phone Dump"	
00000005	0.63326788	[2648] digikam.geoiface: ----	
00000006	0.63795233	[2648] digikam.geoiface: ----	
00000007	0.63811100	[2648] digikam.general: Using  2  CPU core to run threads	
00000008	0.63897473	[2648] digikam.general: Stacked View Mode :  2	
00000009	0.64360756	[2648] digikam.general: Action Thread run  1  new jobs	
00000010	0.75454795	[2648] digikam.general: Data From DBJobsThread is null:  true	
00000011	0.76419008	[2648] digikam.general: Cancel Main Thread	
00000012	0.77185392	[2648] digikam.general: One job is done	
00000013	0.78981644	[2648] digikam.geoiface: ----	
00000014	0.79946679	[2648] digikam.general: Cancel Main Thread	
00000015	0.81840372	[2648] digikam.general: Using  2  CPU core to run threads	
00000016	0.81938356	[2648] digikam.general: Action Thread run  1  new jobs	
00000017	0.82079357	[2648] digikam.general: Data From DBJobsThread is null:  true	
00000018	0.82084709	[2648] digikam.general: Cancel Main Thread	
00000019	0.82092232	[2648] digikam.general: One job is done	
00000020	0.82102710	[2648] digikam.general: Cancel Main Thread

Git commit 7e76a3cd872896e0e3532d43400ef549693ab43c by Maik Qualmann.
Committed on 01/06/2018 at 19:10.
Pushed by mqualmann into branch 'master'.

check if album root path exist before scanning

M  +6    -0    core/libs/database/collection/collectionscanner.cpp

https://commits.kde.org/digikam/7e76a3cd872896e0e3532d43400ef549693ab43c

This is a test version on my GDrive. The installer is compiled and created like the other snapshots under Linux and is virus-free. The file is smaller because I created it without debug symbols. Please test if the problem is resolved.


Installer Win64:
https://drive.google.com/open?id=17HQp7Bqwu0l5_h_2bGl48ZiODC_iGGWb

Checksum:
https://drive.google.com/open?id=1hDZwAbAkHjJevjD_u_wfGueV2HJ0U1Ou

Maik

The test version looks good Maik, thank you very much.

When the Ethernet adapter is disabled I instantly get:

[18216] digikam.general: Detected change, triggering rescan of "P:/digiKamTest/Phone Dump"
[18216] digikam.database: Starting scan!
[18216] digikam.database: The album root path does not exist - scan are canceled


In this condition double clicking the thumbnail does not open the image, as expected, but crucially the albums remain.

When this occurs, please can you add a dialogue message saying "The album root path does not exist - Check network status." or similar.

Raphael, IWBR, Weini, Does this solve your issue?

Git commit a97b4f780de9e9613ebea2af56d301ada4e7d153 by Maik Qualmann.
Committed on 02/06/2018 at 12:29.
Pushed by mqualmann into branch 'master'.

update collection locations if album root path not exists
FIXED-IN: 6.0.0

M  +2    -1    NEWS
M  +9    -2    core/libs/database/collection/collectionmanager.cpp
M  +0    -6    core/libs/database/collection/collectionscanner.cpp

https://commits.kde.org/digikam/a97b4f780de9e9613ebea2af56d301ada4e7d153

New digiKam-6.0.0 Windows test version from git/master.

Installer Win64:
https://drive.google.com/open?id=1M47NKpoVMhVKuLsqICdQ8ibmnBtEEttA

File       : digiKam-6.0.0-git-20180602T144349-Win64.exe
Size       : 164M
MD5 sum    : 93f646dac62f6e37fd0a78752a973646
SHA1 sum   : 82c162fa189d0d71841f15b97d32e3647ebcbdb5
SHA256 sum : 7b2309cb38f5dcdd3a9d2a6b228a870bdfe8178bf35693d8d96077486e3e10d8

Maik

Thanks. I installed the second version. When I double click on a thumbnail while the  network is disabled, I briefly see a note saying 

"The storage location of this image is currently not available"

It's only displayed for about 0.5s so not long enough to read it. Please make it linger for a longer time.

Due to the system settings, a double click is used to invoke the preview, which immediately makes the tool-tip disappear again. I will correct it. I know we once had a bug report (but can not find it) and the double click is standard on Windows. But would not a single click be better here?

Maik

Git commit 6d5c89784813321aa1280bbda4a3e8d809e96c88 by Maik Qualmann.
Committed on 03/06/2018 at 07:32.
Pushed by mqualmann into branch 'master'.

ignore mouse button release event for hide tool-tip

M  +0    -1    core/libs/widgets/itemview/itemviewtooltip.cpp

https://commits.kde.org/digikam/6d5c89784813321aa1280bbda4a3e8d809e96c88

Installer Win64:
https://drive.google.com/open?id=17V3GMrYAnYeVOPaBlZq_FF5uau4KD7wr

File       : digiKam-6.0.0-git-20180603T094233-Win64.exe
Size       : 164M
MD5 sum    : 7c237ffc896a32f49d9d5dcb9141bef7
SHA1 sum   : 9db911fdc5822da6ed730b42616362013718618f
SHA256 sum : bb022f3b6a2997d94269c2886d8c14ce69db489d9476e5d7d1281e6190f6306c

Maik

That version works well, many thanks.

I think showing the warning message with a single click would be fine too. Is that what you meant?

*** Bug 399594 has been marked as a duplicate of this bug. ***

*** Bug 400669 has been marked as a duplicate of this bug. ***

*** Bug 401338 has been marked as a duplicate of this bug. ***