SUMMARY *** NOTE: If you are reporting a crash, please try to attach a backtrace with debug symbols. See https://community.kde.org/Guidelines_and_HOWTOs/Debugging/How_to_create_useful_crash_reports *** STEPS TO REPRODUCE 1. Have a collection with a moderately large number of images. In my case 70,000. Problems started arouond 65,000 2. Selecting a tag in the left pane would sometimes show files that appeared to not have that tag. 3. Changing tags would occasionally corrupt the file. OBSERVED RESULT Corupted data. EXPECTED RESULT Since this appears it might be a database limitation, a suggestion would be to have a message at startup and image add about changing to MariaDB if the setup is for SQL Lite and > 50,000 images in the collection. SOFTWARE/OS VERSIONS Windows: 11 macOS: Linux/KDE Plasma: (available in About System) KDE Plasma Version: KDE Frameworks Version: Qt Version: ADDITIONAL INFORMATION I am currently setup on MariaDB and trying to see if that has fixed the problem. I have around 20k corrupted files now.
What do you means about "around 20k corrupted files" ?
Corrupt files are not caused by the database used. As Gilles already writes, what do you mean by "corrupt", files no longer readable or just missing metadata. We have had users using SQLite with over 200,000 images. Definitely check your drives for errors. I also work a little in this area and SSD hard drives can also have very ugly data area errors. Maik
(In reply to caulier.gilles from comment #1) > What do you means about "around 20k corrupted files" ? All tags stripped, and in some images, face tags were stripped. The images themselves ere ok (except for a couple that seem to have been destroyed by random crashes - but that is a different problem). Its like the SQL Lite database got overwhelmed (too many entries? Too Large a dataset?) and started getting lost. Any change to one of the files where the database was confused would strip all of the tags. SInce I have switched to the MariaDB from SQL Lite, I have stopped seeing this, and I have stopped seeing the warning signs of this (an occasional image that showed on the right hand pane without tags, but when one was added, they all were magically reappeared.). So, not exactly a corrupted file, but still data loss.
(In reply to jm7@acm.org from comment #0) > SUMMARY > *** > NOTE: If you are reporting a crash, please try to attach a backtrace with > debug symbols. > See > https://community.kde.org/Guidelines_and_HOWTOs/Debugging/ > How_to_create_useful_crash_reports > *** > > > STEPS TO REPRODUCE > 1. Have a collection with a moderately large number of images. In my case > 70,000. Problems started arouond 65,000 > 2. Selecting a tag in the left pane would sometimes show files that appeared > to not have that tag. > 3. Changing tags would occasionally corrupt the file. > > OBSERVED RESULT > Corupted data. > > EXPECTED RESULT > Since this appears it might be a database limitation, a suggestion would be > to have a message at startup and image add about changing to MariaDB if the > setup is for SQL Lite and > 50,000 images in the collection. > > SOFTWARE/OS VERSIONS > Windows: 11 > macOS: > Linux/KDE Plasma: > (available in About System) > KDE Plasma Version: > KDE Frameworks Version: > Qt Version: > > ADDITIONAL INFORMATION > I am currently setup on MariaDB and trying to see if that has fixed the > problem. I have around 20k corrupted files now. No, not a crash. Silent data loss.
(In reply to Maik Qualmann from comment #2) > Corrupt files are not caused by the database used. As Gilles already writes, > what do you mean by "corrupt", files no longer readable or just missing > metadata. > We have had users using SQLite with over 200,000 images. > Definitely check your drives for errors. I also work a little in this area > and SSD hard drives can also have very ugly data area errors. > > Maik I switched to Maria DB, and the problem stopped getting worse. The computer is not that old, and the first thing I did was check the drives. The database is one drive and the images are on another, removeable, drive. THe main drive has 1TB, and the removeable has 4 TB of space. The iamge set is somewhere around 800K and growing. Having all the tags and some face tags stripped from tens of thousands of files is a setback.
(In reply to Maik Qualmann from comment #2) > Corrupt files are not caused by the database used. As Gilles already writes, > what do you mean by "corrupt", files no longer readable or just missing > metadata. > We have had users using SQLite with over 200,000 images. > Definitely check your drives for errors. I also work a little in this area > and SSD hard drives can also have very ugly data area errors. > > Maik But the Symptoms I was seeing seem to be caused by the database. 0 bad sectors on either drive. I've checked. Maria DB seems to have fixed the problem, but only going forward, the data loss seems to be permanent (not in the files, not in the database). First symptom? Select a tag in the left frame. Select images in the Thumbnails or Preview. Notice that some show no tags in the right frame, but when adding a tag, all the others miraculously appear. Later symptoms involve actual data loss. I;ve switched to MariaDB. I've now made a couple hundred thousand changes to the files (the easy ones). No data loss so far. Question. Are those with 200K images in SQL Lite on Windows? If not, the data may not apply. I'm using windows.
Created attachment 165111 [details] Clear tag error. Ok. Now the problem has started with Maria DB. This is the first problem I have seen. I did have a hang yesterday (mouse and keyboard completely unresponsive for a half an hour - which prompted a hard power reset). This is the first symptom I was seeing with SQL Lite. If I have to, I will drop DigiKam and find another solution. Loss of data is NEVER acceptable.
Created attachment 165112 [details] after adding a tag. to the file with the problem. I added a tag to the image with the problem, and all the tags magically reappeared. I have no idea how the first one is even possible. I don't know how adding a tag corrects the problem, and I don't know how this eventually leads to all tag information for thousands of files being deleted. But, I really don't want to go through this again. How do I fix this on my end, and how can the code base be changed to prevent this nonsense.
Well, I don't know exactly where the error lies in your screenshots. For example, because the tag "14: Crop & Color" no longer has any images? I think you have a completely different problem, if your mouse and keyboard freezes, there is a system error (hardware or software). And that has relatively little to do with digiKam; rather, digiKam is affected by it. Maik
Hi, I second the Maik viewpoint here. This is the first time that i ear about this kind of major dysfunction. Can you imagine that a long time project as digiKam (23 years now) can be faced with similar problems ? We have a large users base, under Windows too, and i never seen important and faster lost of data as your. Check your system (hardware + software) first. Gilles Caulier
The computer freezing is certainly a hardware/software problem outside of digiKam. The tag loss can be explained with this Bug 481630, so I'll close here now. Maik