| Summary: | Out of memory during face tagging | ||
|---|---|---|---|
| Product: | [Applications] digikam | Reporter: | lesf |
| Component: | Faces-Detection | Assignee: | Digikam Developers <digikam-bugs-null> |
| Status: | RESOLVED FIXED | ||
| Severity: | crash | CC: | caulier.gilles, mario.frank, metzpinguin |
| Priority: | NOR | ||
| Version: | 5.6.0 | ||
| Target Milestone: | --- | ||
| Platform: | Arch Linux | ||
| OS: | Linux | ||
| Latest Commit: | Version Fixed In: | 5.8.0 | |
| Sentry Crash Report: | |||
|
Description
lesf
2017-01-14 10:54:16 UTC
Please try the Linux Universal AppImage bundle instead system based package. There is nothing to install. Run it as well. http://download.kde.org/stable/digikam/ Face tagging use OpenCV in background. I suspect that system based OpenCV is badly packaged (a puzzle). The AppImage include OpenCV with the minimum component enabled. Gilles Caulier Tried the AppImage, unfortunately the same effect. It seems OK if only a few (3, 4) persons are tagged. As soon as there were too many (different) persons tagged in the current session, the system runs out of memory. I checked the system log, it contains a lot of following entries (with slightly different numbers): digikam.facedb: Checkout compressed histogram 10039 for identity 4 with size 9377 Downgraded to 5.3, to no avail. I had a closer look into the system log: it seems that for every new face, Digikam runs this "digikam.facedb: Checkout compressed histogram". Say, I'm currently tagging 'Alice'. I see the 'Checkout compressed histogram' only for the first time I tag 'Alice'. Tagging following 'Alice' faces don't trigger a checkout. Now, when I want to tag 'Bob', the 'Checkout compressed histogram' runs again, eats up all my memory, and Digikam gets killed. So, work-around for me: tag all faces of one person, restart Digikam, tag another person, and so on. Not a very nice workflow :-/ Is your recognition.db still created with digiKam-4.xx? Please rename the recognition.db and try again. Actually this problem was fixed in the last versions of digiKam-4.xx. But it was necessary to create a new DB. Maik Yes, the recognition.db was created newly by Digikam 5.x (next to digikam4.db and the thumbnails) After activating swap digikam doesn't crash anymore -- it eats up most of the RAM, though.
I guess it doesn't allocate any more memory because OpenCV is miffed and throws exceptions for every face-tagged picture:
digikam.facesengine: Default exception from OpenCV
(In reply to lesf from comment #6) > After activating swap digikam doesn't crash anymore -- it eats up most of > the RAM, though. > I guess it doesn't allocate any more memory because OpenCV is miffed and > throws exceptions for every face-tagged picture: > digikam.facesengine: Default exception from OpenCV Can you test with the current AppImage again? I have done some changes in 5.5.0 which reduce the count of created threads. This may also have lead to out of memory problems. It is possible that your problem does not occur any more. The most current AppImages are always stored here by Gilles: https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM Unfortunately, this appimage doesn't work either. Still the same effect as I wrote in comment #2: tagging a few persons is OK, but after that, the system runs out of memory. Do you use Sqlite or Mysql as database ? Can you reproduce the problem using a fresh account and a fresh database (data can be corrupted with previous crashes) ? Gilles Caulier I use Sqlite database. I can reproduce the problem. Steps: - Created a new user account - Prepared a data set of ~58000 pictures. Stripped all metadata [exiftool -r -overwrite_original -P -all= DIR], as they contained face tags and regions - Started digikam (digikam-5.5.0-01-x86-64.appimage from 27.2.2017, Revision 7e3182fb6562c8bc01d689dcce9b99c7215b2ae9) - Run face detection with vanilla settings -- around 28000 faces were detected - In the 'unknown' tag, I selected groups of images (ignoring whether they were actual faces or faces from different persons) and assigned them to different 'persons'. Current status: 'Person A' ~ 10.000 tags 'Person B' ~ 10.000 tags 'Person C' ~ 4.500 tags 7 other persons with 20 to 400 tags - Now, selecting more images and assigning them to a person resulted in a forced kill by the kernel due to memory exhaustion. recognition.db is ~219MB The system has 8GB RAM, and no swap. My normal working set of usually running programs is ~1 GB RAM. new 5.6.0 pre-release as bundle is available here : https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM Please check if this problem still reproducible with these versions. Thanks in advance Gilles Caulier Issue still exists for 5.6.0 pre-release Setup: - 62800 pictures, JPG only - stripped metadata, re-compressed with ImageMagick to save disc space and exclude JPG issues (mogrify -strip -quality 50) Start digikam: cd /var/tmp/dkdebug ./digikam-5.6.0-01-x86-64.appimage --config /var/tmp/dkdebug/digikam-5.6.0.config Vanilla digikam setup Started face detection with default settings 2 x crashes (if the "Unknown" tag is selected while the detection is running) ~ 28900 faces (+ other) detected Created 3 persons Assigned ~10'000 faces to Person A, ~10'000 faces to Person B, ~5'000 faces to Person C by selecting the thumbnails and assigning a name Now, if a picture is opened and there are unassigned face tags in it, digikam crashes with out-of-memory if I set a name to an unassigned face digikam4.db: 28M recognition.db: 213M thumbnails-digikam.db: 1,7G Git commit 6a9fd9891a3ff341dc42fe36158cbf0101d513b6 by Maik Qualmann. Committed on 17/12/2017 at 19:20. Pushed by mqualmann into branch 'master'. delete threads from memory when they are finished Related: bug 375317, bug 321784, bug 325712, bug 328732, bug 330227, bug 331912, bug 344661, bug 345395, bug 350549, bug 381877, bug 338249, bug 329651, bug 329091, bug 387821, bug 381222 M +2 -1 NEWS M +25 -0 libs/database/dbjobs/dbjobsmanager.cpp M +35 -0 libs/iojobs/iojobsmanager.cpp https://commits.kde.org/digikam/6a9fd9891a3ff341dc42fe36158cbf0101d513b6 Following this commit: https://commits.kde.org/digikam/6a9fd9891a3ff341dc42fe36158cbf0101d513b6 ... the approach to fix this problem is under way and new digiKam 5.8.0 pre-release bundles will be compiled tonight to lets a chance to end-users to give a feedback about this fix before the 5.8.0 official release planed before Christmas 2017. The bundles will be available in 2 hours at this url: https://files.kde.org/digikam/ Please do not waste time to test if this file is always valid for next 5.8.0. Thanks in advance Gilles Caulier For digikam-5.8.0-20171227T061903-x86-64.appimage, I couldn't reproduce the issue with my test data set. => change status to resolved/fixed |