SUMMARY see $SUBJECT STEPS TO REPRODUCE 1. have database with >40.000 images 2. Select many (>30) albums with new images 3. start OBSERVED RESULT memory usage jumps to >1GB and keeps growing slowly to ~1.5GB after a few minutes EXPECTED RESULT Memory usage should not grow significantly. Using neural network enginge for detecting faces. Using KDE Neon 5.14.4, and most current appimage from files.kde.org.
Update: this also happens when using the other engine (not the deep learning engine), and the memory usage stays at >1GB after using face detection, and memory usage was <200MB before using face detection. So there is a leak somewhere, it seems.
Update 2: With appimage 20181208, memory usage also spikes, but not as high (up to ~900MB, instead of 1.5GB). Same database, same settings, etc. Also, with both engines, the speed for detection has become RREEAALLYY SSLLOOWW ... I was used to ten times this speed with Digikam 5.x. But 20181208 is still noticeably faster than 20181215, too.
This is really strange, as here, the face detection allocate memory, but there is no leak at end of process (Mageia 6). Same for AppImage 6.0.0-beta3. I suspect a different system configuration, especially for OpenCV. You talk about Depp Learning : it's face recognition, not detection. It's completely different. Which settings do you use exactly from face management dialog ? Gilles Caulier
Main dialog: - select "search again and merge results" in dropdown menu - select "detect and recognize faces" Options: - Albums: choose some albums (in my case, 10..56) - all keywords (default) - Parameters: 80% precision - Use all cores - Deep Learning or LBP algorighm (doesn't make a difference) All this on an i5-4750 with 8G RAM, KDE Neon 5.14.4 with all updates applied and Digikam appimage as mentioned before.
I am now using the appimage from 20181228 and the memory hole has even become worse. The digikam process now has 2GB resident memory without using face detection at all, just for having Digikam hang around in the background for half a day. When I restart it, it's back to ~200MB at the start. How can I further help debug this?
I should add I am using Digikam on a dual screen setup, one 1920x1080 FullHD monitor and one 2560x1440 QHD-Monitor to the left (secondary screen). Just in case this makes a difference ...
Well i use also, dual screen, and i cannot reproduce the memory consomption in all case. 2 gb is enormous and i use system indicators if a whole memory leak appears while debugging. Which host system do you use to run the AppImage ? Gilles Caulier
KDE Neon (based on Ubuntu 18.04) with all updates applied. 64bit. I have about 180GB of images and videos, ~33000 files. If you want we can debug this together, using Teamviewer or something.
With which program you determine the memory consumption? I can not confirm this at all. Here with the AppImage ~130MB after start and ~600-800MB after 30 minutes face detection. Maik
Basically, this started after Linux's OOM killer went on a rampage on my system a few times. I have 8GB of RAM. After the initial shock I started looking more closely and it *always* happened when digikam had been running for a while. Then I started the KDE system monitor and htop (RSS column) and watched the digikam processes (htop shows about 15 PIDs, all of which are called 'digikam'). Initial startup size is ~200MB (~2G virtual). When looking at a few images, it jumps to ~650MB (~4G virtual). When running the face detection on a few images (via context menu) RAM usage jumps to ~1.5G temporarily, then settles at ~1.2GB. I never experienced this on previous releases and never with my old Ubuntu 16.04 system. This is a new phenomenon on KDE Neon.
Anything I can enable using kdebugdialog5 to help track this?
I suspect something about OpenGL and video card driver. Why ? because Face management is relevant of OpenCV library which is... a big puzzle. I know already that OpenCL feature in OpenCV is problematic and can lock/crash digiKam. In AppImage i disable all OpenCV feature that we don't care. The goal : stability. Look the cmake configuration options passed to OpenCV : https://cgit.kde.org/digikam.git/tree/project/bundles/3rdparty/ext_opencv/CMakeLists.txt All that i can disable, especially the runtime computation optimisations, i do, as CUDA, DIRECTX, OPENCL. But for OpenGL, as i know, there is no option (QT_OPENGL is a different stuff in OpenCV). So, a good experience is to install a VM (as Virtualbox for ex), and to reinstall KDE Neon inside, an of course to disable OpenGL and 3D extension with emulated video card. It will be interested to see if the memory leak still here. Gilles Caulier
Will do. For now, I cleaned all databases, started Digikam (~160MB RSS), then looked at a single image (preview) and RSS went up to ~312MB. Just because of a single image. Look at another one, RSS jumps to 512MB. Very strange. I looked at the console output from before clicking on a thumbnail and after, and saw multiple "invalid memory allocation request" errors: Do you think this could be part of the reason? Digikam::DImg::load: "/home/jens/Bilder/Digikam/2018/2018.09.30/20180930_184649.jpg" : JPEG file identified Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile. Digikam::StackedView::setViewMode: Stacked View Mode : 1 Digikam::DImg::load: "/home/jens/Bilder/Digikam/2018/2018.09.30/20180930_161217.jpg" : JPEG file identified Digikam::MetaEngine::Private::printExiv2ExceptionError: Cannot load metadata from file /home/jens/Bilder/Digikam/2018/2018.09.30/20180930_161217.jpg (Error # 57 : invalid memory allocation request Digikam::DImg::load: "/home/jens/Bilder/Digikam/2018/2018.09.30/20180930_161217.jpg" : QIMAGE file identified Digikam::MetaEngine::Private::printExiv2ExceptionError: Cannot load metadata from file /home/jens/Bilder/Digikam/2018/2018.09.30/20180930_161217.jpg (Error # 57 : invalid memory allocation request Digikam::QImageLoader::load: Can not load " "/home/jens/Bilder/Digikam/2018/2018.09.30/20180930_161217.jpg" " using DImg::QImageLoader! Digikam::PreviewLoadingTask::execute: Cannot extract preview for "/home/jens/Bilder/Digikam/2018/2018.09.30/20180930_161217.jpg" Digikam::StackedView::setViewMode: Stacked View Mode : 1 Digikam::MetaEngine::Private::printExiv2ExceptionError: Cannot load metadata from file /home/jens/Bilder/Digikam/2018/2018.09.30/20180930_161217.jpg (Error # 57 : invalid memory allocation request Digikam::JPEGUtils::isJpegImage: mimetype = "" ext = "JPG" Digikam::ThumbnailCreator::createThumbnail: Trying to load Embedded preview with libraw Digikam::DRawDecoder::loadEmbeddedPreview: Failed to load embedded RAW preview Digikam::ThumbnailCreator::createThumbnail: Trying to load half preview with libraw Digikam::ThumbnailCreator::createThumbnail: Trying to load Embedded preview with Exiv2 Digikam::MetaEngine::Private::printExiv2ExceptionError: Cannot load metadata using Exiv2 (Error # 57 : invalid memory allocation request Digikam::DImg::load: "/home/jens/Bilder/Digikam/2018/2018.09.30/20180930_161217.jpg" : JPEG file identified Digikam::DImg::load: "/home/jens/Bilder/Digikam/2018/2018.09.30/20180930_161217.jpg" : QIMAGE file identified Digikam::QImageLoader::load: Can not load " "/home/jens/Bilder/Digikam/2018/2018.09.30/20180930_161217.jpg" " using DImg::QImageLoader! Digikam::ThumbnailCreator::createThumbnail: Trying to load video preview with FFmpeg Digikam::VideoDecoder::initialize: Could not open input file: "/home/jens/Bilder/Digikam/2018/2018.09.30/20180930_161217.jpg" Digikam::ThumbnailCreator::createThumbnail: Cannot create thumbnail for "/home/jens/Bilder/Digikam/2018/2018.09.30/20180930_161217.jpg" Digikam::ThumbnailCreator::load: Thumbnail is null for "/home/jens/Bilder/Digikam/2018/2018.09.30/20180930_161217.jpg"
The memory increases when looking at images is normal. We perform preloading, the next images are already loaded into a QCache. The memory error message from Exiv2 is rather strange. Maik
Can you provide an image that triggers the "invalid memory allocation request"? Maik
Yes, this is very strange than non of previewer cannot load a JPG thumbnail... it's it ? libjpeg, Exiv2, ffmpeg, qimage. none can load the image contents to render a small preview. This is definitively abnormal. Can these images can be loaded in image editor (F4) or previewed in AlbumView (F3) ? for ex this one : /home/jens/Bilder/Digikam/2018/2018.09.30/20180930_161217.jpg Gilles Caulier
20180930_161217.jpg is indeed broken - no thumbnail and no preview. I actually didn't notice that at all. So this image is probably not part of the problem. I installed Virtualbox, KDE Neon 18.04 and Digikam appimage and started Digikam with a new and empty user account, went through the wizard, accepted all defaults and then imported about 20 JPG images of ~5MB each. Results: - after startup, RSS=192MB - after showing the first image, RSS=256MB - after the second image, 285MB, after the third, 312MB - after browsing all images once, ~420MB - Running face detection on *just* this folder (context menu) makes RES go up to 550MB temporarily, then dropping back to ~360MB So it's less, but still a lot for an application that "just" displays some JPEG files. For KDE Neon, digikam 5.9 is available in the repos. Can I jump back and forth between 5.9 and 6.0 without destroying the database to check whether this makes any difference?
No. 6.0.0 database include new tables which are incompatible with 5.9.0. Try the same test with a fresh database and 5.9.0. Gilles
5.9 on new user account with clean KDE Neon installation: - after first startup, 370MB - more than 6.0-beta3, actually - after previewing images: 376MB, 400MB, ... up to 531MB - after detecting faces: 514MB Don't get me wrong. If Digikam temporarily needs 2GB of RAM to do some task, that's fine. But after that task the memory should be freed again, otherwise there's a bug.
After 3 weeks of work, i finally completed the compilation of AppImage using Qt 5.11.3 + QWebkit 5.212. New 6.1.0 pre-release AppImage bundle can be found here (64 bits only for the moment) : https://files.kde.org/digikam/ Please check if this bugzilla entry still valid. Thanks in advance Gilles Caulier
Jens, I found this opencv topic to disable opencl (GPU) usage with opencv at runtime. https://github.com/opencv/opencv/issues/9852 Typically, there is an env.variable to setup before to run digikam : OPENCV_OPENCL_DEVICE=disabled Can you check if this solve your problem ? Gilles Caulier
Problem is fixed with new 7.0.0-beta1 through this long story from this bug https://bugs.kde.org/show_bug.cgi?id=399923 You can test digiKam 7.0.0-beta1 with bundle available here: https://download.kde.org/unstable/digikam/ Don't hesitate to give us a fresh feedback about his entry. Thanks in advance Gilles Caulier