|
Description
Alberto
2013-08-22 10:10:58 UTC
*** Bug 277163 has been marked as a duplicate of this bug. *** Can you run digiKam into valgrind as explained here : http://www.digikam.org/contrib and report valgrind trace here for future investigations... Also we need to know which OpenCV version you use. Look into Help/Components Info dialog for details. Gilles Caulier LibOpenCV release was 2.4.5, I now updated it to 2.4.6 with the same results. Valgrind report will follow. Created attachment 81852 [details]
Valgrind report
Affects me as well. Arch linux, opencv 2.4.6.1 $ du -sh .kde4/share/apps/libkface/database/recognition.db 57M .kde4/share/apps/libkface/database/recognition.db Why is it so big? $ sqlite3 ~/.kde4/share/apps/libkface/database/recognition.db 'select count(*) from opencvlbphistograms;' 6915 I tried to delete recognition.db and run face recognition wizard with option "Clear and rebuildall training data", and after this operation I got recognition.db < 1MiB. $ sqlite3 ~/.kde4/share/apps/libkface/database/recognition.db 'select count(*) from opencvlbphistograms;' 116 And I have exactly 116 face tags. Original 57 MiB is compressible by xz down to 2%, recreated - down to 72%. Therefore I conclude 57MiB recognition.db had lots of duplicates. After recreating recognition.db digikam does not concume all RAM in face related opeartions anymore. Steps to reproduce growing recognition.db with duplicate records 1) re-launch Digikam, switch to people view, select face tag "Unknown". 2) Manually assign one face to specific human. 3) Wait 30seconds (it appears that there is "sleep" before DB update) Outcome: before operation you had N records in table "opencvlbphistograms", after operation number of entries in "opencvlbphistograms" becomes 2*N+1 Created attachment 82206 [details]
valgrind --tool=memcheck --leak-check=full --error-limit=no digikam 2> digikam.leaks.err
me too ...
with valgrind it took too long, so I canceled it and closed digikam
and killed it after another hour...
hope it still helps ^^
Hello,
Digikam 3.5 build from git. OpenCV 2.4.5 build from source.
Scanning a collection of 28000 pictures (jpg, raw ...) with digikam use in gdb - nothing else running except a top in terminal.
Digikam crash sometimes with this error :
what digikam was doing :
digikam(3834)/digikam (core) Digikam::DetectionWorker::process: Found 0 faces in "IMG_2389.CR2" QSize(2848, 4272) QSize(2848, 4272)
digikam(3834)/digikam (core) Digikam::FacePipeline::Private::checkFinished: Check for finish: 51 packages, 0 infos to filter, hasFinished() false
digikam(3834)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
digikam(3834)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation => 8
digikam(3834)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 533 800 searchIncrement 1.1 grouping 3 flags 1 min size 26 26
digikam(3834)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
digikam(3834)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation => 8
digikam(3834)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(3834)/digikam (core) Digikam::DetectionWorker::process: Found 0 faces in "IMG_2390.CR2" QSize(2848, 4272) QSize(2848, 4272)
digikam(3834)/digikam (core) Digikam::FacePipeline::Private::checkFinished: Check for finish: 51 packages, 0 infos to filter, hasFinished() false
digikam(3834)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
digikam(3834)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation => 8
error :
------
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x94db5b40 (LWP 3876)]
0xb6efd89a in Digikam::DImg::DImg (this=0x94db5124, image=...)
at /home/eric/digikamgit/dk/core/libs/dimg/dimg.cpp:165
165 dptr[0] = qBlue(*sptr);
Cordialement,
Eric
*** This bug has been confirmed by popular vote. *** Hello,
Here some more info :
Reproducible : yes
How : restart computer (freeing memory, cache ..), then launch digikam in gdb (nothing else except a top in terminal), wait a few minutes while scanning pictures for faces (skip images already scan).
Res memory grow to 1.2 1.5 gb (60%) when scan is launched.
then swap is used, swap and load grow (especially with raw files). Then digikam crash with error behind :
dbg :
[New Thread 0x95db7b40 (LWP 2642)]
digikam(2423)/digikam (core) Digikam::FacePipeline::Private::checkFinished: Check for finish: 51 packages, 0 infos to filter, hasFinished() false
digikam(2423)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
digikam(2423)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation => 1
digikam(2423)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 800 533 searchIncrement 1.1 grouping 3 flags 1 min size 26 26
digikam(2423)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation => 1
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2423)/digikam (core) Digikam::DetectionWorker::process: Found 0 faces in "IMG_2467.CR2" QSize(4272, 2848) QSize(4272, 2848)
digikam(2423)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
digikam(2423)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation => 1
digikam(2423)/digikam (core) Digikam::FacePipeline::Private::checkFinished: Check for finish: 51 packages, 0 infos to filter, hasFinished() false
digikam(2423)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
digikam(2423)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation => 1
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 800 533 searchIncrement 1.1 grouping 3 flags 1 min size 26 26
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2423)/digikam (core) Digikam::DetectionWorker::process: Found 0 faces in "IMG_2468.CR2" QSize(4272, 2848) QSize(4272, 2848)
digikam(2423)/digikam (core) Digikam::FacePipeline::Private::checkFinished: Check for finish: 51 packages, 0 infos to filter, hasFinished() false
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 800 533 searchIncrement 1.1 grouping 3 flags 1 min size 26 26
digikam(2423)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
digikam(2423)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation => 1
digikam(2423)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
digikam(2423)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation => 1
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2423)/digikam (core) Digikam::DetectionWorker::process: Found 0 faces in "IMG_2469.CR2" QSize(4272, 2848) QSize(4272, 2848)
digikam(2423)/digikam (core) Digikam::FacePipeline::Private::checkFinished: Check for finish: 51 packages, 0 infos to filter, hasFinished() false
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 800 533 searchIncrement 1.1 grouping 3 flags 1 min size 26 26
digikam(2423)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
digikam(2423)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation => 1
error :
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x94db5b40 (LWP 2502)]
0xb6efd89a in Digikam::DImg::DImg (this=0x94db5124, image=...)
at /home/eric/digikamgit/dk/core/libs/dimg/dimg.cpp:165
165 dptr[0] = qBlue(*sptr);
(gdb) backtrace :
#0 0xb6efd89a in Digikam::DImg::DImg (this=0x94db5124, image=...)
at /home/eric/digikamgit/dk/core/libs/dimg/dimg.cpp:165
#1 0xb70d5d7d in Digikam::PreviewLoadingTask::execute (this=0xaca8d38)
at /home/eric/digikamgit/dk/core/libs/threadimageio/previewtask.cpp:288
#2 0xb70c5205 in Digikam::LoadSaveThread::run (this=0xb033c38)
at /home/eric/digikamgit/dk/core/libs/threadimageio/loadsavethread.cpp:136
#3 0xb70ecd58 in Digikam::DynamicThread::DynamicThreadPriv::run (
this=0xb033cd0)
at /home/eric/digikamgit/dk/core/libs/threads/dynamicthread.cpp:186
#4 0xb53d446b in QThreadPoolThread::run (this=0x96bc2a0)
at concurrent/qthreadpool.cpp:107
#5 0xb53e1eb0 in QThreadPrivate::start (arg=0x96bc2a0)
at thread/qthread_unix.cpp:307
#6 0xb36c1d4c in start_thread (arg=0x94db5b40) at pthread_create.c:308
#7 0xb510fdde in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Created attachment 82594 [details]
Crash after full fill the memory - valgrind log
View line 4197 -> no crash but halt (about 1 hour) and fill the memory then continue.
View line 5835 -> crash
(In reply to comment #11) > Created attachment 82594 [details] > Crash after full fill the memory - valgrind log > View line 4197 -> no crash but halt (about 1 hour - before the first warning) and fill the memory then continue. View line 5835 -> crash Created attachment 82602 [details]
Crash after full fill the memory - valgrind log - run 2
Hello,
an other run in valgrind with the same result.
Hang from digikam in line 4189 and continue with a warning (always to the same picture):
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave (QRect(32,224 217x217) )
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(32,224 217x217) using cascade 0
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 297 297 searchIncrement 1.1 grouping 3 flags 0 min size 130 130
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(32,224 217x217) using cascade 1
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 297 297 searchIncrement 1.1 grouping 3 flags 0 min size 130 130
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(32,224 217x217) using cascade 2
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(32,224 217x217) using cascade 3
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 297 297 searchIncrement 1.1 grouping 3 flags 0 min size 130 130
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(32,224 217x217) using cascade 4
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 297 297 searchIncrement 1.1 grouping 3 flags 0 min size 130 130
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(32,224 217x217) using cascade 5
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: feature QRectF(0,0 0.6x0.6) QRect(32,224 217x217) QRect(32,224 130x130)
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 130 130 searchIncrement 1.1 grouping 2 flags 0 min size 22 22
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(32,224 217x217) using cascade 6
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: feature QRectF(0.4,0 0.6x0.6) QRect(32,224 217x217) QRect(119,224 130x130)
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 130 130 searchIncrement 1.1 grouping 2 flags 0 min size 22 22
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(32,224 217x217) using cascade 7
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: feature QRectF(0.2,0.25 0.6x0.6) QRect(32,224 217x217) QRect(75,278 130x130)
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 130 130 searchIncrement 1.1 grouping 2 flags 0 min size 22 22
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(32,224 217x217) using cascade 8
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: feature QRectF(0.1,0.4 0.8x0.6) QRect(32,224 217x217) QRect(54,311 174x130)
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 174 130 searchIncrement 1.1 grouping 2 flags 0 min size 0 0
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2423)/digikam (core) Digikam::DetectionWorker::process: Found 0 faces in "IMG_7905.jpg" QSize(4272, 2848) QSize(4272, 2848)
<---HERE--> HANG FOR PAST AN HOUR and continue-
==2423== Warning: set address range perms: large range [0x49dc6028, 0x5a827018) (undefined)
==2423== Warning: set address range perms: large range [0x49dc6018, 0x5a827028) (noaccess)
<---HERE-->
Crash to line 5839 :
digikam(2423)/digikam (core) Digikam::DetectionWorker::process: Found 0 faces in "IMG_2479.CR2" QSize(4272, 2848) QSize(4272, 2848)
digikam(2423)/digikam (core) Digikam::FacePipeline::Private::checkFinished: Check for finish: 51 packages, 0 infos to filter, hasFinished() false
digikam(2423)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 800 533 searchIncrement 1.1 grouping 3 flags 1 min size 26 26
digikam(2423)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
digikam(2423)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation => 1
<--CRASH-->
==2423== Thread 7:
==2423== Invalid read of size 4
==2423== at 0x4D7C89A: Digikam::DImg::DImg(QImage const&) (dimg.cpp:165)
==2423== by 0x4F54D7C: Digikam::PreviewLoadingTask::execute() (previewtask.cpp:288)
==2423== by 0x4F44204: Digikam::LoadSaveThread::run() (loadsavethread.cpp:136)
==2423== by 0x4F6BD57: Digikam::DynamicThread::DynamicThreadPriv::run() (dynamicthread.cpp:186)
==2423== by 0x6A1246A: QThreadPoolThread::run() (in /usr/lib/i386-linux-gnu/libQtCore.so.4.8.2)
==2423== by 0x6A1FEAF: QThreadPrivate::start(void*) (in /usr/lib/i386-linux-gnu/libQtCore.so.4.8.2)
==2423== by 0x96CCD4B: start_thread (pthread_create.c:308)
==2423== by 0x6F34DDD: clone (clone.S:130)
==2423== Address 0x0 is not stack'd, malloc'd or (recently) free'd
==2423==
KCrash: Application 'digikam' crashing...
Created attachment 82603 [details]
Crash after full fill the memory - valgrind log - run 2
Hello, I've continue to test. I have disable zram on my ubuntu kernel ( see here :https://bugs.launchpad.net/ubuntu/+source/linux-lts-raring/+bug/1217189) : I use Xubuntu 12.04 LTS with : Linux version 3.2.0-54-generic-pae (buildd@batsu) (gcc version 4. 6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #82-Ubuntu SMP Tue Sep 10 20:29:22 UTC 2013 (Ubuntu 3.2.0-54.82-generic-pae 3.2.50) I was having those errors : [ 1.864606] Adding 1922976k swap on /dev/zram0. Priority:100 extents:1 across:1922976k SS [ 30.459194] Buffer I/O error on device zram0, logical block 480744 [ 30.459234] Buffer I/O error on device zram0, logical block 480744 [ 30.459313] Buffer I/O error on device zram0, logical block 480744 [ 30.459349] Buffer I/O error on device zram0, logical block 480744 [ 30.459384] Buffer I/O error on device zram0, logical block 480744 in dmesg And Digikam doesn't full fill the memory nor Crash anymore : run a face scan to 100 % How should I test again : suppress face database file and rescan again ? more files to suppress ? Greatings, Eric Hello,
Have a few more tests.
I was having some problem with ZRAM, but the bug still occur when do a
scan with option : Clear unconfirmed results and rescan. (restart scan
for beginning except for result wich have been confirmed - I suppose).
Detection and recognition begin from root of the collection to the few 1
rst albums then crash with same error as my precedent posts :
GDB :
error :
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x94db5b40 (LWP 2502)]
0xb6efd89a in Digikam::DImg::DImg (this=0x94db5124, image=...)
at /home/eric/digikamgit/dk/core/libs/dimg/dimg.cpp:165
165 dptr[0] = qBlue(*sptr);
(gdb) backtrace :
#0 0xb6efd89a in Digikam::DImg::DImg (this=0x94db5124, image=...)
at /home/eric/digikamgit/dk/core/libs/dimg/dimg.cpp:165
#1 0xb70d5d7d in Digikam::PreviewLoadingTask::execute (this=0xaca8d38)
at /home/eric/digikamgit/dk/core/libs/threadimageio/previewtask.cpp:288
#2 0xb70c5205 in Digikam::LoadSaveThread::run (this=0xb033c38)
at /home/eric/digikamgit/dk/core/libs/threadimageio/loadsavethread.cpp:136
#3 0xb70ecd58 in Digikam::DynamicThread::DynamicThreadPriv::run (
this=0xb033cd0)
at /home/eric/digikamgit/dk/core/libs/threads/dynamicthread.cpp:186
#4 0xb53d446b in QThreadPoolThread::run (this=0x96bc2a0)
at concurrent/qthreadpool.cpp:107
#5 0xb53e1eb0 in QThreadPrivate::start (arg=0x96bc2a0)
at thread/qthread_unix.cpp:307
#6 0xb36c1d4c in start_thread (arg=0x94db5b40) at pthread_create.c:308
#7 0xb510fdde in clone ()
at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Valgrind :
digikam(2423)/digikam (core) Digikam::DMetadata::getIccProfile: Exif
color-space tag is sRGB. Using default sRGB ICC profile.
digikam(2423)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation:
Orientation =>
Exif.Image.Orientation => 1
<--CRASH-->
==2423== Thread 7:
==2423== Invalid read of size 4
==2423== at 0x4D7C89A: Digikam::DImg::DImg(QImage const&)
(dimg.cpp:165)
==2423== by 0x4F54D7C: Digikam::PreviewLoadingTask::execute()
(previewtask.cpp:288)
==2423== by 0x4F44204: Digikam::LoadSaveThread::run()
(loadsavethread.cpp:136)
==2423== by 0x4F6BD57:
Digikam::DynamicThread::DynamicThreadPriv::run()
(dynamicthread.cpp:186)
==2423== by 0x6A1246A: QThreadPoolThread::run() (in
/usr/lib/i386-linux-gnu/libQtCore.so.4.8.2)
==2423== by 0x6A1FEAF: QThreadPrivate::start(void*) (in
/usr/lib/i386-linux-gnu/libQtCore.so.4.8.2)
==2423== by 0x96CCD4B: start_thread (pthread_create.c:308)
==2423== by 0x6F34DDD: clone (clone.S:130)
==2423== Address 0x0 is not stack'd, malloc'd or (recently) free'd
==2423==
This is on digikam 3.5 build from git with opencv 2.4.5 built from
opencv repository.
Build date: Sep 30 2013 (target: debugfull)
Rev.: 04ec199e75e8a6bc01f29bac3a9476e3b72960e0
Components :
digiKam version 3.5.0
Exiv2 can write to Jp2: Yes
Exiv2 can write to Jpeg: Yes
Exiv2 can write to Pgf: Yes
Exiv2 can write to Png: Yes
Exiv2 can write to Tiff: Yes
Exiv2 supports XMP metadata: Yes
LibCImg: 130
LibEigen: 3.0.5
LibExiv2: 0.22
LibJPEG: 80
LibJasper: 1.900.1
LibKDE: 4.11.1
LibKExiv2: 2.3.1
LibKGeoMap: 2.0.0
LibKdcraw: 2.3.1
LibLCMS: 2040
LibLensFun: 0.2.7-0
LibPGF: 6.12.24 - external shared library
LibPNG: 1.2.46
LibQt: 4.8.2
LibRaw: 0.15.4
LibTIFF: LIBTIFF, Version 3.9.5 Copyright (c) 1988-1996 Sam Leffler
Copyright (c) 1991-1996 Silicon Graphics, Inc.
Marble Widget: 0.16.1 (stable version)
Parallelized PGF codec: No
Parallelized demosaicing: Yes
RawSpeed codec support: Yes
Database backend: QMYSQL
Database internal server: No
Kipi-Plugins: 3.5.0
LibGphoto2: 2.4.14
LibKface: 3.0.0
LibKipi: 2.1.0
LibOpenCV: 2.4.5
digiKam version 3.5.0
Images:
BMP: 5
GIF: 56
JP2: 16
JPG: 22730
PNG: 130
RAW-CR2: 3432
XCF: 209
total: 26578
:
:
Videos:
AVI: 29
MOV: 185
MP4: 43
MPEG: 2
WMV: 1
total: 260
:
:
Total Items: 26838
Albums: 2749
Tags: 492
Database backend: QMYSQL
Database internal server: No
Greatings,
Eric
Hello, Looking at this bug : https://bugs.kde.org/show_bug.cgi?id=277169 I look for 0 size files in album : find -type f -size 0 ./La creche/Kayak Fishing.flv Delete this files but he wasn't in the album. Look at the DB : SELECT * FROM digikam.Images where name like "%.flv"; But nothing : .flv are not collected. Greatings, Eric Created attachment 82701 [details]
New crash information added by DrKonqi
digikam (3.5.0) on KDE Platform 4.11.2 using Qt 4.8.2
- What I was doing when the application crashed: scanning faces RAW files - memory fill up and digikam crash.
I've updated libkdcraw from kde 12.04 backport repository.
-- Backtrace (Reduced):
#8 0xb48661df in __GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#9 0xb4869825 in __GI_abort () at abort.c:91
[...]
#14 0xb4beb5ee in QThreadPoolThread::run (this=0x99df7e0) at concurrent/qthreadpool.cpp:114
#15 0xb4bf8eb0 in QThreadPrivate::start (arg=0x99df7e0) at thread/qthread_unix.cpp:307
#16 0xb2c5cd4c in start_thread (arg=0x9b5c6b40) at pthread_create.c:308
Digikam components : digiKam version 3.5.0 Exiv2 can write to Jp2: Yes Exiv2 can write to Jpeg: Yes Exiv2 can write to Pgf: Yes Exiv2 can write to Png: Yes Exiv2 can write to Tiff: Yes Exiv2 supports XMP metadata: Yes LibCImg: 130 LibEigen: 3.0.5 LibExiv2: 0.22 LibJPEG: 80 LibJasper: 1.900.1 LibKDE: 4.11.2 LibKExiv2: 2.3.1 LibKGeoMap: 2.0.0 LibKdcraw: 2.3.1-2 from kubuntu backport repository LibLCMS: 2040 LibLensFun: 0.2.7-0 LibPGF: 6.12.24 - external shared library LibPNG: 1.2.46 LibQt: 4.8.2 LibRaw: 0.15.3 LibTIFF: LIBTIFF, Version 3.9.5 Copyright (c) 1988-1996 Sam Leffler Copyright (c) 1991-1996 Silicon Graphics, Inc. Marble Widget: 0.16.1 (stable version) Parallelized PGF codec: No Parallelized demosaicing: No RawSpeed codec support: No Database backend: QMYSQL Database internal server: No Kipi-Plugins: 3.5.0 LibGphoto2: 2.4.14 LibKface: 3.0.0 LibKipi: 2.1.0 LibOpenCV: 2.4.5 Hello, After updates it's still the same error in Valgring : Thread 5: ==3494== Invalid read of size 4 ==3494== at 0x4D0489A: Digikam::DImg::DImg(QImage const&) (dimg.cpp:165) ==3494== by 0x4EDCD7C: Digikam::PreviewLoadingTask::execute() (previewtask.cpp:288) ==3494== by 0x4ECC204: Digikam::LoadSaveThread::run() (loadsavethread.cpp:136) ==3494== by 0x4EF3D57: Digikam::DynamicThread::DynamicThreadPriv::run() (dynamicthread.cpp:186) ==3494== by 0x699A46A: QThreadPoolThread::run() (in /usr/lib/i386-linux-gnu/libQtCore.so.4.8.2) ==3494== by 0x69A7EAF: QThreadPrivate::start(void*) (in /usr/lib/i386-linux-gnu/libQtCore.so.4.8.2) ==3494== by 0x98D5D4B: start_thread (pthread_create.c:308) ==3494== by 0x6EBCDDD: clone (clone.S:130) ==3494== Address 0x0 is not stack'd, malloc'd or (recently) free'd Note that those lines disappeared from logs (digikam in terminal, gdb, and valgrind) : digikam(2423)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation => 1 Another point is that face scan (finding face but without recognition) was running before : there was faces in unknown but they were not automatically tagged. So I think this is an issue from modifications since activation of face recognition or recognition.db and perhaps related to the camera RAW files (libkdcraw, libraw issue ?). Process for jpg files and raw files is different : no file generation for jpg. RAW files are from a canon EOS 1100D. Greatings, Eric Hello, Runing GDB with breakpoints ... I have those lines : Digikam::LoadSaveThread::run (this=0xa0f4c20) at /home/eric/digikamgit/dk/core/libs/threadimageio/loadsavethread.cpp:106 106 while (runningFlag()) (gdb) 139 } (gdb) Digikam::DynamicThread::DynamicThreadPriv::run (this=0x9d601d8) at /home/eric/digikamgit/dk/core/libs/threads/dynamicthread.cpp:193 193 if (emitSignals) (gdb) 198 transitionToInactive(); (gdb) 200 } (gdb) QThreadPoolThread::run (this=0x96aec70) at concurrent/qthreadpool.cpp:117 117 concurrent/qthreadpool.cpp: Aucun fichier ou dossier de ce type. between each (thread/picture) for face scan (RAW files). Is it usual or not ? PS : i'm able to locate qthreadpool.h but no qthreadpoll.cpp Hi, Typically, loadsavethread will load Raw image data in a separated thread to be able to parse image contents for face. qthreadpoll.cpp come from Qt4. GDB cannot found it because you don't have installed Qt source package on your computer. In simple way, just download Qt4 source code and look cpp file accordingly Gilles Caulier Created attachment 82746 [details]
Graph of physical (RSS) and virtual memory (VSZ)
Hello,
A graph (with bash script on ps and gnuplot) of the memory usage for digikam when scanning face of one album with option "clear unconfirmed results and rescan".
The increase is when the scan is run. Never end and digikam crash. The size of recognition.db doesn't move.
Thanks,
Eric
PS :
Created attachment 82747 [details]
Second Graph of physical (RSS) and virtual memory (VSZ)
Same conditions
Created attachment 82751 [details]
gdb log with break set to Digikam::DImg::DImg(QImage const&)
Created attachment 82752 [details]
Graph of physical (RSS) and virtual memory (VSZ) with break set to Digikam::DImg::DImg(QImage const&) in dgb
Hello,
I run digikam in dgb with a break point set to Digikam::DImg::DImg(QImage const&).
Clearly, when this is executed, no memory is freed :
1 --- first stair
(gdb) c
Continuing.
[Thread 0x9be24b40 (LWP 13898) exited]
digikam(8415)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
Breakpoint 2, Digikam::DImg::DImg (this=0xa2cf1124, image=...)
at /home/eric/digikamgit/dk/core/libs/dimg/dimg.cpp:140
140 : m_priv(new Private)
It's the "stair effect". Memory grow and doesn't go down.
2 -- second stair go down to same level before executed (1-- level)
When this is executed :
(gdb) c
Continuing.
digikam(8415)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
digikam(8415)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(8415)/digikam (core) Digikam::DetectionWorker::process: Found 0 faces in "IMG_2793.CR2" QSize(4272, 2848) QSize(4272, 2848)
digikam(8415)/digikam (core) Digikam::DatabaseWriter::process: Removing old entries ()
digikam(8415)/digikam (core) Digikam::FacePipeline::Private::checkFinished: Check for finish: 51 packages, 0 infos to filter, hasFinished() false
digikam(8415)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 800 533 searchIncrement 1.1 grouping 3 flags 1 min size 26 26
Breakpoint 2, Digikam::DImg::DImg (this=0x9be24124, image=...)
at /home/eric/digikamgit/dk/core/libs/dimg/dimg.cpp:140
140 : m_priv(new Private)
Memory grow and go down
3 --- downstairs - more memory than "lost" in 1-- is back
When this occure (only 2 times) :
(gdb) c
Continuing.
[New Thread 0xa36ceb40 (LWP 14797)]
digikam(8415)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(8415)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
digikam(8415)/digikam (core) Digikam::DetectionWorker::process: Found 0 faces in "IMG_2814.CR2" QSize(4272, 2848) QSize(4272, 2848)
digikam(8415)/digikam (core) Digikam::DatabaseWriter::process: Removing old entries ()
digikam(8415)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 800 533 searchIncrement 1.1 grouping 3 flags 1 min size 26 26
digikam(8415)/digikam (core) Digikam::FacePipeline::Private::checkFinished: Check for finish: 51 packages, 0 infos to filter, hasFinished() false
digikam(8415)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(8415)/digikam (core) Digikam::DetectionWorker::process: Found 0 faces in "IMG_2815.CR2" QSize(4272, 2848) QSize(4272, 2848)
digikam(8415)/digikam (core) Digikam::DatabaseWriter::process: Removing old entries ()
[New Thread 0xa2cf1b40 (LWP 14808)]
digikam(8415)/digikam (core) Digikam::FacePipeline::Private::checkFinished: Check for finish: 51 packages, 0 infos to filter, hasFinished() false
digikam(8415)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 800 533 searchIncrement 1.1 grouping 3 flags 1 min size 26 26
Breakpoint 2, Digikam::DImg::DImg (this=0x9be24124, image=...)
at /home/eric/digikamgit/dk/core/libs/dimg/dimg.cpp:140
140 : m_priv(new Private)
(gdb) c
Two pictures are proceed and memory go down more.
Those are the U on the graph.
So as step 1-- and 2-- occure more than step 3-- memory use go up and up, and digikam crash.
Greatings,
Eric
Ps : each stair on the graph is a continue on gdb, then wait and do another continue.
PS : look at this graph : https://bugs.kde.org/attachment.cgi?id=82752 gdb logs are here : https://bugs.kde.org/attachment.cgi?id=82751 Created attachment 82781 [details]
New crash information added by DrKonqi
digikam (3.5.0) on KDE Platform 4.11.2 using Qt 4.8.2
- What I was doing when the application crashed:
Scanning an album with option Clear unconfirmed results and rescan
-- Backtrace (Reduced):
#8 0xb48b61df in __GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#9 0xb48b9825 in __GI_abort () at abort.c:91
[...]
#14 0xb4c3b5ee in QThreadPoolThread::run (this=0x988a018) at concurrent/qthreadpool.cpp:114
#15 0xb4c48eb0 in QThreadPrivate::start (arg=0x988a018) at thread/qthread_unix.cpp:307
#16 0xb2cacd4c in start_thread (arg=0x9de19b40) at pthread_create.c:308
Created attachment 82819 [details]
New crash information added by DrKonqi
digikam (3.5.0) on KDE Platform 4.11.2 using Qt 4.8.2
- What I was doing when the application crashed:
Scanning a new folders with new pictures (RAW and JPG), with option skip already scanned image ( I try to scan this folder one time before and digikam crashed with same traces).
-- Backtrace (Reduced):
#8 0xb48331df in __GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#9 0xb4836825 in __GI_abort () at abort.c:91
[...]
#14 0xb4bb85ee in QThreadPoolThread::run (this=0xa9d3270) at concurrent/qthreadpool.cpp:114
#15 0xb4bc5eb0 in QThreadPrivate::start (arg=0xa9d3270) at thread/qthread_unix.cpp:307
#16 0xb2c29d4c in start_thread (arg=0x9ed97b40) at pthread_create.c:308
As Marcel said previously, there is a no catched exception when digiKam crash You backtrace is uncomplete because you don't get the exception properly. To do it, follow this tutorial and report relevant and suitable backtrace : http://techbase.kde.org/Development/Tutorials/Debugging/How_to_create_useful_crash_reports#Retrieving_a_backtrace_when_an_uncaught_exception_is_causing_a_crash Gilles Caulier Hello,
All the traces for the crash described in precedent post (scan faces in a new folder with option : skip already scanned pictures):
digikam
digikam(2380)/digikam (core) Digikam::KInotify::Private::open: Successfully opened connection to inotify: 15
digikam(2380)/digikam (core) Digikam::AlbumManager::setDatabase: DatabaseParameters: [ Type "QMYSQL", Name "digikam" (Thumbnails Name "digikam_thumb"); Host Name and Port: "192.168.1.2" 3306; Username and Password: "***", "****"]
QDBusConnection: name 'org.freedesktop.UDisks2' had owner '' but we thought it was ':1.45'
QSqlDatabasePrivate::removeDatabase: connection 'ConnectionTest' is still in use, all queries will cease to work.
digikam(2380)/digikam (core) Digikam::DatabaseConfigElementLoader::readConfig: Loading SQL code from config file "/usr/share/kde4/apps/digikam/database/dbconfig.xml"
digikam(2380)/digikam (core) Digikam::DatabaseConfigElementLoader::readConfig: false "1" 1 1
digikam(2380)/digikam (core) Digikam::SchemaUpdater::update: SchemaUpdater update
digikam(2380)/digikam (core) Digikam::SchemaUpdater::startUpdates: Have a database structure version 7
digikam(2380)/digikam (core) Digikam::SchemaUpdater::makeUpdates: makeUpdates 7 to 7
"/org/freedesktop/UDisks2/drives/SAMSUNG_HM160HC_S12TJDZB314313" : property "DeviceNumber" does not exist
"/org/freedesktop/UDisks2/drives/SAMSUNG_HM160HC_S12TJDZB314313" : property "Device" does not exist
digikam(2380)/digikam (core) Digikam::AlbumRootLocation::AlbumRootLocation: Creating new Location "/" uuid "networkshareid:?mountpath=%2Fmedia%2FDiskphotos"
digikam(2380)/digikam (core) Digikam::CollectionManager::updateLocations: location for "/media/Diskphotos" is available true
digikam(2380)/digikam (core) Digikam::KMemoryInfo::update: Platform identified : "LINUX"
digikam(2380)/digikam (core) Digikam::KMemoryInfo::bytes: TotalRam: 2110857216
digikam(2380)/digikam (core) Digikam::LoadingCache::setCacheSize: Allowing a cache size of 100 MB
digikam(2380)/digikam (core) Digikam::ThumbnailSchemaUpdater::startUpdates: Have a thumbnail database structure version "2"
digikam(2380)/digikam (core) Digikam::ThumbnailLoadThread::initializeThumbnailDatabase: Thumbnail db ready for use
digikam(2380)/digikam (core) Digikam::CollectionScanner::completeScan: Complete scan (file scanning deferred) took: 13646 msecs.
digikam(2380)/digikam (core) Digikam::IccSettings::Private::scanDirectories: ("/usr/share/color/icc", "/home/eric/.color/icc")
digikam(2380)/KFACE KFaceIface::SchemaUpdater::startUpdates: Have a database structure version "1"
digikam(2380)/KGEOMAP KGeoMap::KGeoMapWidget::setBackend: "setting backend marble"
digikam(2380)/KGEOMAP KGeoMap::KGeoMapWidget::setBackend: "setting backend marble"
digikam(2380)/digikam (core) Digikam::LensFunCameraSelector::populateLensCombo: variant: QVariant(Digikam::LensFunIface::DevicePtr, )
digikam(2380)/digikam (core) Digikam::LensFunCameraSelector::populateLensCombo: dev: Asahi Optical Co.,Ltd :: PENTAX Optio 430 :: 4.85
digikam(2380)/digikam (core) Digikam::LensFunIface::findCamera: Search for camera "" - "" ==> false
digikam(2380)/digikam (core) Digikam::LensFunIface::findLens: Search for lens "" ==> false
digikam(2380)/digikam (core) Digikam::CurvesWidget::updateData: updating data
digikam(2380)/digikam (core) Digikam::BlurFilter::cimgBlurImage: Radius: 0.1
digikam(2380)/digikam (core) Digikam::BlurFilter::cimgBlurImage: BlurFilter::Process Computation...
digikam(2380)/digikam (core) Digikam::BlurFilter::cimgBlurImage: Radius: 0.2
digikam(2380)/digikam (core) Digikam::BlurFilter::cimgBlurImage: BlurFilter::Process Computation...
digikam(2380)/digikam (core) Digikam::BlurFilter::cimgBlurImage: Radius: 0.5
digikam(2380)/digikam (core) Digikam::BlurFilter::cimgBlurImage: BlurFilter::Process Computation...
digikam(2380)/digikam (core) Digikam::BlurFilter::cimgBlurImage: Radius: 0.3
digikam(2380)/digikam (core) Digikam::BlurFilter::cimgBlurImage: BlurFilter::Process Computation...
digikam(2380)/digikam (core) Digikam::BlurFilter::cimgBlurImage: BlurFilter::Finalization...
digikam(2380)/digikam (core) Digikam::BlurFilter::cimgBlurImage: BlurFilter::Finalization...
digikam(2380)/digikam (core) Digikam::BlurFilter::cimgBlurImage: BlurFilter::Finalization...
digikam(2380)/digikam (core) Digikam::BlurFilter::cimgBlurImage: BlurFilter::Finalization...
digikam(2380)/digikam (core) Digikam::CurvesWidget::updateData: updating data
digikam(2380)/digikam (core) Digikam::AlbumFilterModel::setSearchTextSettings: new search text settings: "" : hasResult = false , validRows = 0
digikam(2380)/digikam (core) Digikam::WorkflowManager::load: Cannot open XML file to load Workflow
digikam(2380)/digikam (core) Digikam::ThemeManager::slotChangePalette: "" :: ""
digikam(2380)/digikam (core) Digikam::ThemeManager::slotChangePalette: "" :: ""
digikam(2380)/digikam (core) Digikam::KInotify::Private::open: Successfully opened connection to inotify: 66
digikam(2380)/KGEOMAP KGeoMap::BackendGoogleMaps::setMapType: "ROADMAP"
digikam(2380)/KGEOMAP KGeoMap::BackendGoogleMaps::setMapType: "ROADMAP"
digikam(2380)/KGEOMAP KGeoMap::KGeoMapWidget::setBackend: "setting backend marble"
digikam(2380)/KGEOMAP KGeoMap::BackendGoogleMaps::setMapType: "ROADMAP"
digikam(2380)/KGEOMAP KGeoMap::ItemMarkerTiler::slotSourceModelReset: ----
digikam(2380)/KGEOMAP KGeoMap::ItemMarkerTiler::slotSourceModelReset: ----
digikam(2380)/KGEOMAP KGeoMap::BackendGoogleMaps::setMapType: "ROADMAP"
digikam(2380)/KIPI (general) KIPIImageshackExportPlugin::Plugin_ImageshackExport::Plugin_ImageshackExport: Plugin_ImageshackExport plugin loaded
digikam(2380)/KIPI (general) KIPI::Plugin::Private::XMLParser::removeDisabledActions: Plugin action ' "batch_border_images" ' is disabled.
digikam(2380)/KIPI (general) KIPI::Plugin::Private::XMLParser::removeDisabledActions: Plugin action ' "batch_color_images" ' is disabled.
digikam(2380)/KIPI (general) KIPI::Plugin::Private::XMLParser::removeDisabledActions: Plugin action ' "batch_convert_images" ' is disabled.
digikam(2380)/KIPI (general) KIPI::Plugin::Private::XMLParser::removeDisabledActions: Plugin action ' "batch_filter_images" ' is disabled.
digikam(2380)/KIPI (general) KIPI::Plugin::Private::XMLParser::removeDisabledActions: Plugin action ' "batch_rename_images" ' is disabled.
digikam(2380)/KIPI (general) KIPI::Plugin::Private::XMLParser::removeDisabledActions: Plugin action ' "batch_recompress_images" ' is disabled.
digikam(2380)/KIPI (general) KIPIGalleryExportPlugin::Gallery::load: Reading data from kipirc file..
digikam(2380)/KIPI (general) KIPIJAlbumExportPlugin::JAlbum::load: Reading data from kipirc file..
digikam(2380)/KIPI (general) KIPIPhotoLayoutsEditor::Plugin_PhotoLayoutsEditor::Plugin_PhotoLayoutsEditor: Plugin_PhotoLayoutsEditor plugin loaded
digikam(2380)/KIPI (general) KIPI::Plugin::addAction: Action ' "batch_border_images" ' is disabled.
digikam(2380)/KIPI (general) KIPI::Plugin::addAction: Action ' "batch_color_images" ' is disabled.
digikam(2380)/KIPI (general) KIPI::Plugin::addAction: Action ' "batch_convert_images" ' is disabled.
digikam(2380)/KIPI (general) KIPI::Plugin::addAction: Action ' "batch_filter_images" ' is disabled.
digikam(2380)/KIPI (general) KIPI::Plugin::addAction: Action ' "batch_rename_images" ' is disabled.
digikam(2380)/KIPI (general) KIPI::Plugin::addAction: Action ' "batch_recompress_images" ' is disabled.
digikam(2380)/KIPI (general) KIPIPiwigoExportPlugin::Piwigo::load: Reading data from kipirc file..
digikam(2380)/KGEOMAP KGeoMap::ItemMarkerTiler::slotSourceModelReset: ----
digikam(2380)/KGEOMAP KGeoMap::ItemMarkerTiler::slotSourceModelReset: ----
digikam(2380)/digikam (editor plugins) DigikamColorImagePlugin::ImagePlugin_Color::ImagePlugin_Color: ImagePlugin_Color plugin loaded
digikam(2380)/digikam (core) Digikam::ImagePluginLoader::loadPluginsFromList: ImagePluginLoader: Loaded plugin "ImagePlugin_Color"
digikam(2380)/digikam (editor plugins) DigikamDecorateImagePlugin::ImagePlugin_Decorate::ImagePlugin_Decorate: ImagePlugin_Decorate plugin loaded
digikam(2380)/digikam (core) Digikam::ImagePluginLoader::loadPluginsFromList: ImagePluginLoader: Loaded plugin "ImagePlugin_Decorate"
digikam(2380)/digikam (editor plugins) DigikamEnhanceImagePlugin::ImagePlugin_Enhance::ImagePlugin_Enhance: ImagePlugin_Enhance plugin loaded
digikam(2380)/digikam (core) Digikam::ImagePluginLoader::loadPluginsFromList: ImagePluginLoader: Loaded plugin "ImagePlugin_Enhance"
digikam(2380)/digikam (editor plugins) DigikamFxFiltersImagePlugin::ImagePlugin_FxFilters::ImagePlugin_FxFilters: ImagePlugin_FxFilters plugin loaded
digikam(2380)/digikam (core) Digikam::ImagePluginLoader::loadPluginsFromList: ImagePluginLoader: Loaded plugin "ImagePlugin_FxFilters"
digikam(2380)/digikam (editor plugins) DigikamTransformImagePlugin::ImagePlugin_Transform::ImagePlugin_Transform: ImagePlugin_Transform plugin loaded
digikam(2380)/digikam (core) Digikam::ImagePluginLoader::loadPluginsFromList: ImagePluginLoader: Loaded plugin "ImagePlugin_Transform"
digikam(2380)/KGEOMAP KGeoMap::KGeoMapWidget::setBackend: "setting backend marble"
digikam(2380)/KGEOMAP KGeoMap::BackendGoogleMaps::setMapType: "ROADMAP"
digikam(2380)/KGEOMAP KGeoMap::KGeoMapWidget::setBackend: "setting backend marble"
digikam(2380)/KGEOMAP KGeoMap::BackendGoogleMaps::setMapType: "ROADMAP"
digikam(2380)/KGEOMAP KGeoMap::KGeoMapWidget::setBackend: "setting backend marble"
digikam(2380)/KGEOMAP KGeoMap::BackendGoogleMaps::setMapType: "ROADMAP"
digikam(2380)/digikam (core) Digikam::NewItemsFinder::slotStart: scan mode: ScanDeferredFiles
digikam(2380)/digikam (core) Digikam::DImg::load: "/media/Diskphotos/mormal octobre 2013/MVI_2868.MOV" : QIMAGE file identified
digikam(2380)/digikam (core) Digikam::QImageLoader::load: Can not load " "/media/Diskphotos/mormal octobre 2013/MVI_2868.MOV" " using DImg::QImageLoader!
digikam(2380)/digikam (core) Digikam::JPEGUtils::isJpegImage: mimetype = "" ext = "MOV"
digikam(2380)/digikam (core) Digikam::ThumbnailCreator::createThumbnail: Cannot create thumbnail for "/media/Diskphotos/mormal octobre 2013/MVI_2868.MOV"
digikam(2380)/digikam (core) Digikam::ThumbnailCreator::load: Thumbnail is null for "/media/Diskphotos/mormal octobre 2013/MVI_2868.MOV"
digikam(2380)/KGEOMAP KGeoMap::ItemMarkerTiler::slotSourceModelReset: ----
digikam(2380)/KGEOMAP KGeoMap::ItemMarkerTiler::slotSourceModelReset: ----
digikam(2380)/KGEOMAP KGeoMap::ItemMarkerTiler::slotSourceModelReset: ----
digikam(2380)/KGEOMAP KGeoMap::ItemMarkerTiler::slotSourceModelReset: ----
digikam(2380)/digikam (core) Digikam::NewItemsFinder::slotTotalFilesToScan: total scan value : 28165
digikam(2380)/digikam (core) Digikam::FaceScanDialog::doLoadState: "Face Detection Dialog"
digikam(2380)/digikam (core) Digikam::DImg::load: "/media/Diskphotos/archive materiel peche/aero technium/eclaté_galet-shimano_aero_technium_et_ultegra_modifiée.JPG" : JPEG file identified
digikam(2380)/digikam (core) Digikam::ImageScanner::prepareAddImage: Adding new item "/media/Diskphotos/archive materiel peche/aero technium/eclaté_galet-shimano_aero_technium_et_ultegra_modifiée.JPG"
digikam(2380)/digikam (core) Digikam::ImageScanner::scanTags: Pick Label found : 0
digikam(2380)/digikam (core) Digikam::ImageScanner::scanTags: Assigned Pick Label Tag : 12
digikam(2380)/digikam (core) Digikam::ImageScanner::scanTags: Color Label found : 0
digikam(2380)/digikam (core) Digikam::ImageScanner::scanTags: Assigned Color Label Tag : 2
digikam(2380)/digikam (core) Digikam::DMetadata::getImageHistory: Loading image history ""
digikam(2380)/digikam (core) Digikam::ImageScanner::commit: Scanning took 54 ms
digikam(2380)/digikam (core) Digikam::ImageScanner::~ImageScanner: Finishing took 18 ms
digikam(2380)/digikam (core) Digikam::FaceScanDialog::doSaveState: "Face Detection Dialog"
digikam(2380)/digikam (core) Digikam::FaceDetector::slotStart: Total is 124165
digikam(2380)/digikam (core) Digikam::FaceDetector::slotContinueAlbumListing: false false
digikam(2380)/digikam (core) Digikam::FaceDetector::slotContinueAlbumListing: false true
digikam(2380)/digikam (core) Digikam::FacePipeline::Private::checkFinished: Check for finish: 0 packages, 34 infos to filter, hasFinished() false
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
digikam(2380)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
(digikam:2380): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
digikam(2380)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
---- START of face scan ---
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 800 533 searchIncrement 1.1 grouping 3 flags 1 min size 26 26
digikam(2380)/digikam (core) Digikam::DImg::load: "/media/Diskphotos/vendee 2010/bretignolle juin 2010 immersion/lune_bretignolles.jpg" : JPEG file identified
digikam(2380)/digikam (core) Digikam::ImageScanner::prepareAddImage: Adding new item "/media/Diskphotos/vendee 2010/bretignolle juin 2010 immersion/lune_bretignolles.jpg"
digikam(2380)/digikam (core) Digikam::ImageScanner::scanTags: Pick Label found : 0
digikam(2380)/digikam (core) Digikam::ImageScanner::scanTags: Assigned Pick Label Tag : 12
digikam(2380)/digikam (core) Digikam::ImageScanner::scanTags: Color Label found : 0
digikam(2380)/digikam (core) Digikam::ImageScanner::scanTags: Assigned Color Label Tag : 2
digikam(2380)/digikam (core) Digikam::DMetadata::getImageHistory: Loading image history "<?xml version="1.0"?>
<history version="1">
<file uuid="3b6a2a35ffeb7720318bf7910e7f2265da8767f8944cc68cde090b7fe3abddd3" type="original">
<fileParams fileName="lune_bretignolles.JPG" filePath="/media/Diskphotos/vendee 2010/bretignolle juin 2010 immersion/" fileHash="da8767f8944cc68cde090b7fe3abddd3" fileSize="2159724" creationDate="2010-06-21T23:55:11"/>
</file>
</history>
"
digikam(2380)/digikam (core) Digikam::ImageScanner::commit: Scanning took 73 ms
digikam(2380)/digikam (core) Digikam::ImageScanner::~ImageScanner: Finishing took 41 ms
digikam(2380)/digikam (core) Digikam::CollectionScanner::completeHistoryScanning: items to tag (22180)
digikam(2380)/digikam (core) Digikam::ImageScanner::tagImageHistoryGraph: Graph with 2 vertices:
"{ Id: 22180 UUID: 3b6a2a... } -> { Id: 60303 UUID: ac2c01... }"
digikam(2380)/digikam (core) Digikam::ImageScanner::tagImageHistoryGraph: Image 60303 type QFlags(0x8)
digikam(2380)/digikam (core) Digikam::ImageScanner::tagImageHistoryGraph: Image 22180 type QFlags(0x1|0x8)
digikam(2380)/digikam (core) Digikam::CollectionScanner::completeScan: Complete scan took: 27180 msecs.
digikam(2380)/digikam (core) Digikam::DImg::load: "/media/Diskphotos/archive materiel peche/aero technium/eclaté_galet-shimano_aero_technium_et_ultegra_modifiee.jpg" : JPEG file identified
digikam(2380)/digikam (core) Digikam::ImageScanner::prepareAddImage: Adding new item "/media/Diskphotos/archive materiel peche/aero technium/eclaté_galet-shimano_aero_technium_et_ultegra_modifiee.jpg"
digikam(2380)/digikam (core) Digikam::ImageScanner::commit: Scanning took 40 ms
digikam(2380)/digikam (core) Digikam::ImageScanner::~ImageScanner: Finishing took 293 ms
digikam(2380)/digikam (core) Digikam::DImg::load: "/media/Diskphotos/vendee 2010/bretignolle juin 2010 immersion/lune_bretignolles.JPG" : JPEG file identified
digikam(2380)/digikam (core) Digikam::ImageScanner::prepareAddImage: Adding new item "/media/Diskphotos/vendee 2010/bretignolle juin 2010 immersion/lune_bretignolles.JPG"
digikam(2380)/digikam (core) Digikam::ImageScanner::scanFromIdenticalFile: Recognized "/media/Diskphotos/vendee 2010/bretignolle juin 2010 immersion/lune_bretignolles.JPG" as identical to item 22180
digikam(2380)/digikam (core) Digikam::ImageScanner::commit: Scanning took 25 ms
digikam(2380)/digikam (core) Digikam::ImageScanner::~ImageScanner: Finishing took 302 ms
digikam(2380)/digikam (core) Digikam::CollectionScanner::completeHistoryScanning: items to tag ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave (QRect(73,84 82x82) , QRect(66,132 93x93) )
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(73,84 82x82) using cascade 0
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 162 162 searchIncrement 1.1 grouping 3 flags 0 min size 49 49
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave (QRect(45,40 80x80) )
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(73,84 82x82) using cascade 1
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 162 162 searchIncrement 1.1 grouping 3 flags 0 min size 49 49
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave (QRect(39,40 82x82) )
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(73,84 82x82) using cascade 2
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(73,84 82x82) using cascade 3
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 162 162 searchIncrement 1.1 grouping 3 flags 0 min size 49 49
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(73,84 82x82) using cascade 4
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 162 162 searchIncrement 1.1 grouping 3 flags 0 min size 49 49
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(73,84 82x82) using cascade 5
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: feature QRectF(0,0 0.6x0.6) QRect(73,84 82x82) QRect(73,84 49x49)
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 49 49 searchIncrement 1.1 grouping 2 flags 0 min size 0 0
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(73,84 82x82) using cascade 6
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: feature QRectF(0.4,0 0.6x0.6) QRect(73,84 82x82) QRect(106,84 49x49)
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 49 49 searchIncrement 1.1 grouping 2 flags 0 min size 0 0
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(73,84 82x82) using cascade 7
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: feature QRectF(0.2,0.25 0.6x0.6) QRect(73,84 82x82) QRect(89,105 49x49)
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 49 49 searchIncrement 1.1 grouping 2 flags 0 min size 0 0
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(73,84 82x82) using cascade 8
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: feature QRectF(0.1,0.4 0.8x0.6) QRect(73,84 82x82) QRect(81,117 66x49)
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 66 49 searchIncrement 1.1 grouping 2 flags 0 min size 0 0
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(66,132 93x93) using cascade 0
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 173 173 searchIncrement 1.1 grouping 3 flags 0 min size 56 56
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave (QRect(36,35 97x97) )
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(66,132 93x93) using cascade 1
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 173 173 searchIncrement 1.1 grouping 3 flags 0 min size 56 56
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave (QRect(50,49 70x70) )
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(66,132 93x93) using cascade 2
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(66,132 93x93) using cascade 3
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 173 173 searchIncrement 1.1 grouping 3 flags 0 min size 56 56
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(66,132 93x93) using cascade 4
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 173 173 searchIncrement 1.1 grouping 3 flags 0 min size 56 56
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(66,132 93x93) using cascade 5
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: feature QRectF(0,0 0.6x0.6) QRect(66,132 93x93) QRect(66,132 56x56)
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 56 56 searchIncrement 1.1 grouping 2 flags 0 min size 0 0
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(66,132 93x93) using cascade 6
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: feature QRectF(0.4,0 0.6x0.6) QRect(66,132 93x93) QRect(103,132 56x56)
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 56 56 searchIncrement 1.1 grouping 2 flags 0 min size 0 0
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(66,132 93x93) using cascade 7
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: feature QRectF(0.2,0.25 0.6x0.6) QRect(66,132 93x93) QRect(85,155 56x56)
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 56 56 searchIncrement 1.1 grouping 2 flags 0 min size 0 0
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(66,132 93x93) using cascade 8
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: feature QRectF(0.1,0.4 0.8x0.6) QRect(66,132 93x93) QRect(75,169 74x56)
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 74 56 searchIncrement 1.1 grouping 2 flags 0 min size 0 0
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/digikam (core) Digikam::DetectionWorker::process: Found 2 faces in "IMG_2860.CR2" QSize(4272, 2848) QSize(4272, 2848)
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 800 533 searchIncrement 1.1 grouping 3 flags 1 min size 26 26
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave (QRect(16,257 53x53) )
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(16,257 53x53) using cascade 0
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 133 133 searchIncrement 1.1 grouping 3 flags 0 min size 32 32
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(16,257 53x53) using cascade 1
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 133 133 searchIncrement 1.1 grouping 3 flags 0 min size 32 32
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(16,257 53x53) using cascade 2
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(16,257 53x53) using cascade 3
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 133 133 searchIncrement 1.1 grouping 3 flags 0 min size 32 32
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(16,257 53x53) using cascade 4
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 133 133 searchIncrement 1.1 grouping 3 flags 0 min size 32 32
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(16,257 53x53) using cascade 5
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: feature QRectF(0,0 0.6x0.6) QRect(16,257 53x53) QRect(16,257 32x32)
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 32 32 searchIncrement 1.1 grouping 2 flags 0 min size 0 0
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(16,257 53x53) using cascade 6
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: feature QRectF(0.4,0 0.6x0.6) QRect(16,257 53x53) QRect(37,257 32x32)
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 32 32 searchIncrement 1.1 grouping 2 flags 0 min size 0 0
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(16,257 53x53) using cascade 7
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: feature QRectF(0.2,0.25 0.6x0.6) QRect(16,257 53x53) QRect(27,270 32x32)
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 32 32 searchIncrement 1.1 grouping 2 flags 0 min size 0 0
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: Verifying face QRect(16,257 53x53) using cascade 8
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::verifyFace: feature QRectF(0.1,0.4 0.8x0.6) QRect(16,257 53x53) QRect(21,278 42x32)
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale: image size 42 32 searchIncrement 1.1 grouping 2 flags 0 min size 0 0
digikam(2380)/KFACE KFaceIface::OpenCVFaceDetector::cascadeResult: detectMultiScale gave ()
digikam(2380)/digikam (core) Digikam::DetectionWorker::process: Found 0 faces in "IMG_2861.CR2" QSize(4272, 2848) QSize(4272, 2848)
-- big HALT (long time, fill the memory and crash)
OpenCV Error: Insufficient memory (Failed to allocate 262144 bytes) in OutOfMemoryError, file /home/eric/opencv-2-4-5-SH/OpenCV/opencv-2.4.5/modules/core/src/alloc.cpp, line 52
digikam(2380)/KFACE: cv::Exception: /home/eric/opencv-2-4-5-SH/OpenCV/opencv-2.4.5/modules/core/src/alloc.cpp:52: error: (-4) Failed to allocate 262144 bytes in function OutOfMemoryError
Qt has caught an exception thrown from an event handler. Throwing
exceptions from an event handler is not supported in Qt. You must
reimplement QApplication::notify() and catch all exceptions there.
Qt Concurrent has caught an exception thrown from a worker thread.
This is not supported, exceptions thrown in worker threads must be
caught before control returns to Qt Concurrent.
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
KCrash: Application 'digikam' crashing...
KCrash: Attempting to start /usr/lib/kde4/libexec/drkonqi from kdeinit
sock_file=/home/eric/.kde/socket-eric-laptop/kdeinit4__0
QSocketNotifier: Invalid socket 13 and type 'Read', disabling...
QSocketNotifier: Invalid socket 16 and type 'Read', disabling...
QSocketNotifier: Invalid socket 22 and type 'Read', disabling...
digikam: Fatal IO error 9 (Mauvais descripteur de fichier) on X server :0.0.
[1]+ Stoppé digikam
eric@eric-laptop:~$ QSocketNotifier: Invalid socket 17 and type 'Read', disabling...
[1]+ Termine 253 digikam
(In reply to comment #32) > As Marcel said previously, there is a no catched exception when digiKam crash > > You backtrace is uncomplete because you don't get the exception properly. > > To do it, follow this tutorial and report relevant and suitable backtrace : > > http://techbase.kde.org/Development/Tutorials/Debugging/ > How_to_create_useful_crash_reports#Retrieving_a_backtrace_when_an_uncaught_ex > ception_is_causing_a_crash > > Gilles Caulier Ok Gilles, I will try. Thank you. Created attachment 82823 [details] Crash after full fill the memory when scanning a new folder with new pictures Hello, attach the gdb logs as explain here : http://techbase.kde.org/Development/Tutorials/Debugging/How_to_create_useful_crash_reports set break point two "to __GI_raise" and breakpoint three to "__GI_abort". What I was doing : scanning a new folder with new pictures with option : skip already scanned. Thank you, Eric Created attachment 82826 [details]
Crash after full fill the memory - gdb log -
Crash after full fill the memory when scanning a folder
Hello,
Set break point two "to __GI_raise" and breakpoint three to "__GI_abort".
What I was doing : scanning a folder with pictures(PS1) with option : Clear unconfirmed results and rescan.
Thank you,
Eric
PS 1: I rescan the folder used for testing in precedent post ( not the one used this morning)
PS 2 : this message from opencv doesn't appear in gdb :
OpenCV Error: Insufficient memory (Failed to allocate 262144 bytes) in OutOfMemoryError, file /home/eric/opencv-2-4-5-SH/OpenCV/opencv-2.4.5/modules/core/src/alloc.cpp, line 52 digikam(2380)/KFACE: cv::Exception: /home/eric/opencv-2-4-5-SH/OpenCV/opencv-2.4.5/modules/core/src/alloc.cpp:52: error: (-4) Failed to allocate 262144 bytes in function OutOfMemoryError
Created attachment 82827 [details]
gdb full log - full fill memory - cont to SIGALRM -
- same folder as first test with option Clear unconfirmed results and rescan.
- breakpoint to "__GI_raise" and to "__GI_abort".
- "thread apply all bt" then "cont" in gdb until SIGALRM
Hello,
I have a new try with digikam 4.0 beta, compiled from git for eclipse and debug in eclipse+gdb.
I have made a test with two folders move from original source albums to the folder image then tag only one person in those folders (recognition.db about 86k). I use 2 sqlite db for digikam and digikam_thumb. I should do anything without problem. Memory usage remain stable.
Then I clone my working databases (mysql) into two new mysql databases : one for the thumb and the other for digikam, clone all the albums collection too. I configure digikam for using those clones dbs and the clone collection. Do a learning pass with facetagging tools wich was a succes (as in precedent tests - digikam 3.3, 3.4, 3.5) and do a new face scan in the same folders that precedent test : memory usage grow, then digikam crash with segfault at :
digikam [5359] [cores: 0]
Thread [63] 12305 [core: 0] (Suspended : Signal : SIGSEGV:Segmentation fault)
Digikam::DImg::DImg() at dimg.cpp:165 0xe3e148
Digikam::PreviewLoadingTask::execute() at previewtask.cpp:288 0x104d2ac
Digikam::LoadSaveThread::run() at loadsavethread.cpp:136 0x1038e58
Digikam::DynamicThread::DynamicThreadPriv::run() at dynamicthread.cpp:186 0x106c11e
QThreadPoolThread::run() at qthreadpool.cpp:107 0x30f346b
QThreadPrivate::start() at qthread_unix.cpp:307 0x3100eb0
start_thread() at pthread_create.c:308 0x495ad4c
clone() at clone.S:130 0x3614dde
Thread [60] 12011 [core: 0] (Suspended : Container)
__kernel_vsyscall() at 0x132416
__GI___poll() at poll.c:87 0x3606690
g_poll() at 0x516aa3b
0x515d06e
g_main_context_iteration() at 0x515d1c1
QEventDispatcherGlib::processEvents() at qeventdispatcher_glib.cpp:424 0x3248d87
QEventLoop::processEvents() at qeventloop.cpp:149 0x32146ad
QEventLoop::exec() at qeventloop.cpp:204 0x3214949
Digikam::WorkerObjectRunnable::run() at threadmanager.cpp:196 0x1069a8e
QThreadPoolThread::run() at qthreadpool.cpp:107 0x30f346b
QThreadPrivate::start() at qthread_unix.cpp:307 0x3100eb0
start_thread() at pthread_create.c:308 0x495ad4c
clone() at clone.S:130 0x3614dde
Thread [59] 12010 [core: 0] (Suspended : Container)
__kernel_vsyscall() at 0x132416
__GI___poll() at poll.c:87 0x3606690
g_poll() at 0x516aa3b
0x515d06e
g_main_context_iteration() at 0x515d1c1
QEventDispatcherGlib::processEvents() at qeventdispatcher_glib.cpp:424 0x3248d87
QEventLoop::processEvents() at qeventloop.cpp:149 0x32146ad
QEventLoop::exec() at qeventloop.cpp:204 0x3214949
Digikam::WorkerObjectRunnable::run() at threadmanager.cpp:196 0x1069a8e
QThreadPoolThread::run() at qthreadpool.cpp:107 0x30f346b
QThreadPrivate::start() at qthread_unix.cpp:307 0x3100eb0
start_thread() at pthread_create.c:308 0x495ad4c
clone() at clone.S:130 0x3614dde
Thread [58] 12009 [core: 0] (Suspended : Container)
icvEvalHidHaarClassifier() at haar.cpp:812 0x3f5bd36
cvRunHaarClassifierCascadeSum() at haar.cpp:1Â 258 0x3f5d3ff
cvRunHaarClassifierCascade() at haar.cpp:1Â 274 0x3f5d60d
cv::HaarDetectObjects_ScaleCascade_Invoker::operator() at haar.cpp:1Â 463 0x3f5df5c
() at parallel.cpp:142 0x16dbbff
() at parallel.cpp:162 0x16dbcd2
tbb::interface6::internal::start_for<tbb::blocked_range<int>, {anonymous}::ProxyLoopBody, tbb::auto_partitioner>::run_body() at parallel_for.h:110 0x16dcc46
tbb::interface6::internal::partition_type_base<tbb::interface6::internal::auto_partition_type>::execute<tbb::interface6::internal::start_for<tbb::blocked_range<int>, {anonymous}::ProxyLoopBody, tbb::auto_partitioner>, tbb::blocked_range<int> >() at partitioner.h:265 0x16dc9b5
tbb::interface6::internal::start_for<tbb::blocked_range<int>, {anonymous}::ProxyLoopBody, tbb::auto_partitioner>::execute() at parallel_for.h:116 0x16dc500
tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all() at custom_scheduler.h:449 0x4e538c9
tbb::internal::generic_scheduler::local_spawn_root_and_wait() at scheduler.cpp:627 0x4e51ea0
tbb::internal::generic_scheduler::spawn_root_and_wait() at scheduler.h:535 0x4e52509
tbb::task::spawn_root_and_wait() at task.h:692 0x16db51b
tbb::interface6::internal::start_for<tbb::blocked_range<int>, {anonymous}::ProxyLoopBody, tbb::auto_partitioner>::run() at parallel_for.h:94 0x16dc239
tbb::parallel_for<tbb::blocked_range<int>, {anonymous}::ProxyLoopBody>() at parallel_for.h:162 0x16dc0cc
cv::parallel_for_() at parallel.cpp:238 0x16dbd78
cvHaarDetectObjectsForROC() at haar.cpp:1Â 720 0x3f5ff4c
cv::CascadeClassifier::detectMultiScale() at cascadedetect.cpp:1Â 109 0x3f77c32
cv::CascadeClassifier::detectMultiScale() at cascadedetect.cpp:1Â 204 0x3f7875e
KFaceIface::OpenCVFaceDetector::cascadeResult() at opencvfacedetector.cpp:419 0xa218ab
KFaceIface::OpenCVFaceDetector::detectFaces() at opencvfacedetector.cpp:699 0xa266ad
KFaceIface::FaceDetector::detectFaces() at facedetector.cpp:152 0xa1e712
Digikam::DetectionWorker::process() at facepipeline.cpp:482 0x834745b
Digikam::DetectionWorker::qt_static_metacall() at facepipeline_p.moc:318 0x8347cdc
QMetaCallEvent::placeMetaCall() at qobject.cpp:525 0x3228101
QObject::event() at qobject.cpp:1Â 195 0x323117b
Digikam::WorkerObject::event() at workerobject.cpp:160 0x106a762
notify_helper() at qapplication.cpp:4Â 556 0x242adf4
QApplicationPrivate::notify_helper() at qapplication.cpp:4Â 528 0x242adf4
QApplication::notify() at qapplication.cpp:4Â 285 0x243015d
KApplication::notify() at kapplication.cpp:311 0x2099ed1
QCoreApplication::notifyInternal() at qcoreapplication.cpp:915 0x3215e0e
sendEvent() at qcoreapplication.h:231 0x3219f68
QCoreApplicationPrivate::sendPostedEvents() at qcoreapplication.cpp:1Â 539 0x3219f68
QCoreApplication::sendPostedEvents() at qcoreapplication.cpp:1Â 432 0x321a29c
sendPostedEvents() at qcoreapplication.h:236 0x3248994
postEventSourceDispatch() at qeventdispatcher_glib.cpp:279 0x3248994
g_main_context_dispatch() at 0x515cd46
0x515d0e5
g_main_context_iteration() at 0x515d1c1
QEventDispatcherGlib::processEvents() at qeventdispatcher_glib.cpp:424 0x3248d87
QEventLoop::processEvents() at qeventloop.cpp:149 0x32146ad
QEventLoop::exec() at qeventloop.cpp:204 0x3214949
Digikam::WorkerObjectRunnable::run() at threadmanager.cpp:196 0x1069a8e
QThreadPoolThread::run() at qthreadpool.cpp:107 0x30f346b
QThreadPrivate::start() at qthread_unix.cpp:307 0x3100eb0
start_thread() at pthread_create.c:308 0x495ad4c
clone() at clone.S:130 0x3614dde
Thread [7] 5640 [core: 0] (Suspended : Container)
Thread [3] 5552 [core: 0] (Suspended : Container)
Thread [2] 5550 [core: 0] (Suspended : Container)
Thread [1] 5359 [core: 0] (Suspended : Container)
Please view attached variables
Thanks,
Eric
Created attachment 82913 [details]
Variables when SigFault at dimg.cpp ligne 165
Variables when digikam crash on dimg.cpp line 165 (scanning faces)
Hello great Digikam team, please fix this bug...this is a huge regression since last 2 versions : face recognition is not usable anymore :-(. Regards, Olivier The huge memory leak is already fixed. I miss libPGF memory corruption/leakto be fixed. I waiting libpgf feedback for that. Note: libpgf problems already exist in previous versions... Gilles Caulier Thank you Gilles for your quick feedback. Hope you'll be able to publish a release soon :-). Olivier *** Bug 327197 has been marked as a duplicate of this bug. *** *** Bug 325712 has been marked as a duplicate of this bug. *** *** Bug 329651 has been marked as a duplicate of this bug. *** (In reply to comment #41) > The huge memory leak is already fixed. I miss libPGF memory > corruption/leakto be fixed. I waiting libpgf feedback for that. > > Note: libpgf problems already exist in previous versions... > > Gilles Caulier Hi Gilles, I wish you all the best for you & Digikam in 2014 ! is libpgf bug fixed ? Regards Olivier Thanks for your 2014 hopes. libpgf is fixed into digiKam core, but stand alone library is not yet released with fixes. This is why libpgf still in digiKam core for the moment... Gilles Caulier Hi Gilles, you mean it is fixed in digikam core in current version (3.5.0) ? Or in a version to be published ? because I have this bug in version 3.5.0 (on archlinux). regards Olivier Hello, You must build digikam 3.5 with internal libpgf (given in digikam 3.5.0 sources). For this, you must remove the libpgf package from distribution then rebuild digikam. I will answer you more in detail tomorrow, but if i well remenber, i think remove lipgf package from dist is enougth. So remove libpgf, then make an make install for digikam. Regards, Eric *** Bug 331912 has been marked as a duplicate of this bug. *** This bug stil occurs in Digikam 4.11.3 within OpenSUSE 13.1, linux 3.11.10-7-desktop. I also get this bug in digiKam Version 4.0.0-beta3 (installed from the ppa at http://ppa.launchpad.net/msylwester/digikam-beta/ubuntu). I use this PPA because I don't believe that in this day and age I should have to compile the code myself. Why is there no "official" PPA for Digikam betas? Would this not help with testing? Is there plans to resolve this bug properly? It seems that face recognition is a major draw card for Digikam and the fact that it makes the program grind to a halt isn't a good look. Is there a workaround? With easily followed instructions? I am also still getting this in digikam 4.0.0.beta4 from msylwester's ppa. is it at all possible to compile the internal libpgf and install this on the system, or stick it in a ppa somewhere to save having to wait for an update to a package that has not been updated upstream in 3 years? I get error after error trying to compile digikam using the flaky instructions provided on the digikam website and just dont have the time to sort the issues out, though I do love digikam. This is a shame that one of the major features that digikam offers is completely unusable... I'm experiencing this bug in digikam 4.0.0.beta4 (compiled it myself) After killing my "unresponsive" (all changes to face-tags are ignored and undone) digikam I've found that my recognition.db was 11GB in size. After deleting (renaming) it the tagging now works brilliantly :) (In reply to comment #54) > I'm experiencing this bug in digikam 4.0.0.beta4 (compiled it myself) > After killing my "unresponsive" (all changes to face-tags are ignored and > undone) digikam I've found that my recognition.db was 11GB in size. After > deleting (renaming) it the tagging now works brilliantly :) Can anyone else confirm this works? And provide some clear "how to" steps if it does? (In reply to comment #41) > The huge memory leak is already fixed. I miss libPGF memory > corruption/leakto be fixed. I waiting libpgf feedback for that. > > Note: libpgf problems already exist in previous versions... > > Gilles Caulier @Gilles, I don't think the problem lies with libPGF. First based on the advise, I recompiled libPGF as was recommended. That did not help. Then, I recompiled Digikam (4.0 Beta 4) with internal libPGF support. It still loses memory at the same insane rate. digiKam version 4.0.0-beta4 Exiv2 can write to Jp2: Yes Exiv2 can write to Jpeg: Yes Exiv2 can write to Pgf: Yes Exiv2 can write to Png: Yes Exiv2 can write to Tiff: Yes Exiv2 supports XMP metadata: Yes LibCImg: 130 LibEigen: 3.2.1 LibExiv2: 0.23 LibJPEG: 80 LibJasper: 1.900.1 LibKDE: 4.12.4 LibKExiv2: 2.3.1 LibKGeoMap: 2.0.0 LibKdcraw: 2.3.1 LibLCMS: 2060 LibLensFun: 0.2.8-0 LibPGF: 6.13.45 - internal library LibPNG: 1.2.50 LibQt: 4.8.6 LibRaw: 0.15.3 LibTIFF: LIBTIFF, Version 4.0.3 Copyright (c) 1988-1996 Sam Leffler Copyright (c) 1991-1996 Silicon Graphics, Inc. Marble Widget: 0.16.5 (stable version) Parallelized PGF codec: No Parallelized demosaicing: No RawSpeed codec support: No Database backend: QSQLITE Kipi-Plugins: 4.0.0-beta4 LibGphoto2: 2.5.4 LibKface: 3.0.0 LibKipi: 2.1.0 LibOpenCV: 2.4.8 I can confirm that this bug has been fixed in libpgf upstream, and therefore Digikam. Of course though, the libpgf package still hasn't made it to the Debian/Ubuntu repo's as yet... The steps that I took to resolve this (on x64 Ubuntu 13.10, running digikam from the package maintained by msylwester's ppa), and I am trying to make this as accessible to the everyday user as possible: Firstly: Don't uninstall the official package if you installed digikam from a repository!! It is a dependency of digikam, so you want the system to think it is there still! As I am still a reasonably new user to compiling from source, I know no better way than to fool the system into thinking it has one version, when it has a newer... Get the package source from here: http://sourceforge.net/projects/libpgf/files/libpgf/6.14.12-latest/libpgf-src-6.14.12.tar.gz/download unpack it in a terminal using: tar -xzvf libpgf-src-6.14.12.tar.gz Get the required essentials for building packages if you havent already: sudo apt-get install build-essential checkinstall On top of build-essentials etc. you also need libtools and automake. Get those too: sudo apt-get install libtools automake Once you've got those, cd to the directory where libpgf extracted to, then run ./autogen.sh Once that has finished, you need to run ./configure For me, there seemed to be an error in the generated config file due to a malformed command. I kept getting the error: ".in'ig.status: error: cannot find input file: `Makefile". To resolve this, open the configure file using your favourite text editor, for me it's: gedit configure Search for a line that has "cat >$CONFIG_STATUS <<_ASEOF || as_write_fail=1" (the easiest way is just to search for ">$C" and it should come up. Add a space in between the > and $ so the line then reads: "cat > $CONFIG_STATUS <<_ASEOF || as_write_fail=1". Save and close the file and rerun: ./configure If all goes well, proceed as normal: make sudo make install There is one final step though. This technique puts the libraries in the wrong position for digikam. It puts them in the /usr/local/lib/, while digikam looks for them in /usr/lib/x86_64-linux-gnu/ but this is easily fixed by copying them to the correct location running: sudo cp -f /usr/local/lib/libpgf.so.6.0.11 /usr/lib/x86_64-linux-gnu/libpgf.so.6 && sudo cp -f /usr/local/lib/libpgf.so.6.0.11 /usr/lib/x86_64-linux-gnu/libpgf.so.6.0.7 && sudo cp -f /usr/local/lib/libpgf.so.6.0.11 /usr/lib/x86_64-linux-gnu/libpgf.so.6.0.11 This forces any other programs that may be looking for libpgf.so.6.0.7 to use libpgf.so.6.0.11 also. And for those others out there, yes I know I could have made symlinks, but every time I tried, the links were broken... I have no idea what I was doing wrong, but copying worked, so I'm sticking with it... Once I had done this, the first tag took forever still, but after the first went much, much quicker than previously. Good work libpgf team! Hope this helps a bit for some... Now just to wait for the official repos to catch up. :) (In reply to comment #57) > I can confirm that this bug has been fixed in libpgf upstream, and therefore > Digikam. Of course though, the libpgf package still hasn't made it to the > Debian/Ubuntu repo's as yet... > > The steps that I took to resolve this (on x64 Ubuntu 13.10, running digikam > from the package maintained by msylwester's ppa), and I am trying to make > this as accessible to the everyday user as possible: > > Firstly: Don't uninstall the official package if you installed digikam from > a repository!! It is a dependency of digikam, so you want the system to > think it is there still! As I am still a reasonably new user to compiling > from source, I know no better way than to fool the system into thinking it > has one version, when it has a newer... > > Get the package source from here: > http://sourceforge.net/projects/libpgf/files/libpgf/6.14.12-latest/libpgf- > src-6.14.12.tar.gz/download > > unpack it in a terminal using: > tar -xzvf libpgf-src-6.14.12.tar.gz > > Get the required essentials for building packages if you havent already: > sudo apt-get install build-essential checkinstall > > On top of build-essentials etc. you also need libtools and automake. Get > those too: > sudo apt-get install libtools automake > > Once you've got those, cd to the directory where libpgf extracted to, then > run > ./autogen.sh > > Once that has finished, you need to run > ./configure > > For me, there seemed to be an error in the generated config file due to a > malformed command. I kept getting the error: ".in'ig.status: error: cannot > find input file: `Makefile". To resolve this, open the configure file using > your favourite text editor, for me it's: > gedit configure > I saw this error too. From what it looks, the generated configure file as invalid characters in it. Given that this file is generated at runtime, it makes me curious on where lies the bug. > Search for a line that has "cat >$CONFIG_STATUS <<_ASEOF || as_write_fail=1" > (the easiest way is just to search for ">$C" and it should come up. Add a > space in between the > and $ so the line then reads: "cat > $CONFIG_STATUS > <<_ASEOF || as_write_fail=1". Save and close the file and rerun: > ./configure > This alone did not solve the problem. You still get errors like config.h.in not present etc... Open the configure script and look for many instances of the word "Makefile". It has some invalid characters in it. Remove them and then the configure scirpt runs fine.. But..... > If all goes well, proceed as normal: > make > sudo make install > > There is one final step though. This technique puts the libraries in the > wrong position for digikam. It puts them in the /usr/local/lib/, while > digikam looks for them in /usr/lib/x86_64-linux-gnu/ but this is easily > fixed by copying them to the correct location running: > sudo cp -f /usr/local/lib/libpgf.so.6.0.11 > /usr/lib/x86_64-linux-gnu/libpgf.so.6 && sudo cp -f > /usr/local/lib/libpgf.so.6.0.11 /usr/lib/x86_64-linux-gnu/libpgf.so.6.0.7 && > sudo cp -f /usr/local/lib/libpgf.so.6.0.11 > /usr/lib/x86_64-linux-gnu/libpgf.so.6.0.11 > > This forces any other programs that may be looking for libpgf.so.6.0.7 to > use libpgf.so.6.0.11 also. And for those others out there, yes I know I > could have made symlinks, but every time I tried, the links were broken... > I have no idea what I was doing wrong, but copying worked, so I'm sticking > with it... > > Once I had done this, the first tag took forever still, but after the first > went much, much quicker than previously. Good work libpgf team! > > Hope this helps a bit for some... Now just to wait for the official repos > to catch up. :) But... The problem remains the same. Digikam still leaks memory. Can you please double check if it has sorted the problem for you ? The other part where memory issue can be relevant of this file is OpenCV shared library. Please check which version is installed on your system.... Gilles Caulier (In reply to comment #59) > The other part where memory issue can be relevant of this file is OpenCV > shared library. Please check which version is installed on your system.... > > Gilles Caulier The version I have is: 2.4.8+dfsg1-2 Does it have known issues ? Hello, I don't know if openCV have an issue, but I was having a problem with it too. Should you look here to check some openCV compilation feature and your hardware (for me disable SSE3 feature - Intel pentium M): https://bugs.kde.org/show_bug.cgi?id=325309 Greatings, Eric > But... The problem remains the same. Digikam still leaks memory. Can you
> please double check if it has sorted the problem for you ?
I can confirm that for me digikam uses alot of memory, but no longer leaks memory. I have just tagged 200+ of my 31,000+ outstanding face tags (I have ALOT of portrait photography dating back to 2003) and the memory usage is consistantly between 780-850MB, with an average of 800MB.
Btw devs, should I post a link the the solution (for me) on the digikam website, as a comment to the beta4 release? Or should I let this flow downstream first?
17:00:58 rrs@zan:~$ dstat -am ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send| in out | int csw | used buff cach free 2 1 97 0 0 0|2179k 103k| 0 0 | 0 0 | 559 2702 |2513M 131M 1038M 3895M 11 2 87 0 0 0| 54M 0 | 0 0 | 0 0 |1111 1800 |2895M 131M 1091M 3459M 11 1 87 0 0 0| 53M 88k| 0 0 | 0 0 |1057 1501 |3272M 131M 1145M 3029M 11 2 87 0 0 0| 54M 1240k| 0 0 | 0 0 |1171 1436 |3659M 131M 1198M 2589M 11 2 87 0 0 0| 46M 0 | 428B 0 | 0 0 | 906 1316 |3988M 131M 1244M 2215M 12 1 87 0 0 0| 46M 0 | 0 0 | 0 0 | 948 1731 |4321M 131M 1290M 1835M 11 1 87 0 0 0| 38M 0 | 0 65B| 0 0 | 885 1872 |4593M 131M 1328M 1525M 12 1 87 0 0 0| 44M 0 | 0 0 | 0 0 | 929 1745 |4908M 131M 1372M 1166M 11 2 87 0 0 0| 46M 0 | 0 0 | 0 0 | 932 1386 |5236M 131M 1418M 792M 12 1 87 0 0 0| 47M 36k| 0 0 | 0 0 | 965 1349 |5575M 131M 1465M 406M 11 3 86 1 0 0| 49M 0 | 0 0 | 0 0 |1682 1666 |5927M 89.3M 1397M 163M 11 3 86 0 0 0| 48M 88k| 42B 60B| 0 88k| 12k 2628 |6266M 38.9M 1128M 144M 5 9 76 9 0 0| 11M 35M| 0 0 | 0 80M| 25k 3218 |7319M 328k 157M 100M 1 4 69 26 0 0|3872k 79M| 0 0 | 0 98M| 16k 938 |7332M 328k 142M 102M 1 3 73 22 0 0|3196k 86M| 428B 0 | 48k 83M| 12k 967 |7330M 328k 145M 102M 1 3 72 25 0 0| 976k 77M| 0 65B| 160k 84M| 13k 885 |7328M 328k 144M 104M 1 2 64 33 0 0|9024k 36M| 0 0 | 816k 25M|8423 1742 |7328M 328k 147M 101M 1 1 60 39 0 0|4856k 9916k| 0 0 |1080k 16M| 622 1476 |7327M 328k 148M 101M 0 3 45 52 0 0|1024k 63M| 0 0 | 188k 63M| 12k 920 |7322M 328k 146M 108M 1 2 74 23 0 0|1120k 56M| 0 0 | 152k 52M|7944 809 |7329M 328k 148M 100M 0 3 63 33 0 0|1256k 74M| 0 0 | 88k 79M| 12k 881 |7330M 328k 146M 100M 0 2 56 42 0 0| 404k 46M| 0 0 | 272k 57M|5198 995 |7324M 352k 148M 104M 0 1 55 44 0 0| 912k 34M| 0 0 | 908k 21M|4964 1115 |7326M 352k 148M 103M 0 1 50 48 0 0|2136k 13M| 0 0 | 680k 15M|5817 980 |7328M 352k 148M 101M 0 0 62 37 0 0|5072k 2048k| 0 0 |1212k 0 | 605 1207 |7327M 352k 149M 101M 0 1 64 35 0 0| 820k 30M| 0 65B| 808k 32M|4568 1209 |7320M 348k 148M 109M 1 1 78 20 0 0|1792k 19M| 0 0 | 688k 20M| 941 1638 |7326M 352k 148M 102M 0 2 54 44 0 0|1096k 32M| 0 0 | 496k 38M|4716 1035 |7324M 348k 149M 104M 0 1 64 34 0 0|5708k 14M| 0 0 |1940k 10M|2101 1601 |7331M 492k 145M 100M 0 1 47 51 0 0| 760k 37M| 0 0 | 608k 32M|1192 868 |7330M 492k 146M 101M 1 1 61 37 0 0|1416k 13M| 0 0 |1408k 20M|1587 1108 |7324M 492k 145M 107M ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send| in out | int csw | used buff cach free 0 0 53 46 0 0| 900k 2560k| 0 0 | 896k 0 | 865 999 |7325M 492k 145M 106M 1 1 58 41 0 0|1620k 8212k| 0 0 |1152k 6148k|3741 1193 |7330M 492k 146M 101M 0 2 67 32 0 0| 468k 58M| 0 0 | 296k 63M|1338 718 |7323M 492k 146M 108M 0 1 63 36 0 0| 520k 24M| 0 65B| 524k 23M|1973 780 |7324M 500k 146M 107M 0 1 63 36 0 0|1756k 25M| 0 0 |1372k 24M|7324 987 |7324M 516k 146M 106M 1 0 66 34 0 0|3056k 0 | 0 0 |1200k 0 |1026 1494 |7327M 516k 148M 102M 0 0 64 35 0 0|2104k 0 | 0 0 |1356k 0 | 643 1255 |7327M 516k 149M 101M 0 1 71 28 0 0|2464k 21M| 0 0 | 872k 15M|4744 1087 |7326M 516k 150M 101M 0 2 59 39 0 0|2260k 39M| 0 0 | 548k 36M| 10k 960 |7325M 504k 151M 101M 0 1 64 35 0 0|2796k 3584k| 0 0 |1500k 12M| 499 1133 |7320M 504k 151M 105M 1 0 63 36 0 0|3804k 0 | 0 0 |1340k 0 | 843 1453 |7322M 644k 153M 101M 0 7 53 40 0 0|3520k 0 |4363B 0 |1392k 0 |1212 1148 |7322M 644k 154M 101M 0 0 77 23 0 0|2148k 4132k| 0 0 |1528k 0 | 835 928 |7322M 548k 153M 101M 0 0 75 25 0 0|2940k 0 | 0 65B|1520k 0 | 528 964 |7323M 496k 153M 100M 0 0 69 31 0 0|3972k 4096B| 0 0 |1252k 0 | 565 1198 |7323M 496k 153M 101M 0 0 60 39 0 0|3296k 2048k| 0 0 |1060k 0 | 602 1403 |7324M 488k 147M 105M 0 0 74 25 0 0|3960k 0 | 0 0 |1200k 0 |1084 1312 |7325M 488k 150M 101M 0 0 75 25 0 0|1644k 2060k| 0 0 | 952k 0 | 942 1055 |7325M 496k 150M 101M 0 3 75 23 0 0|3216k 2048k| 42B 60B|1096k 104k|1210 1284 |5440M 376k 142M 1995M 0 3 48 49 0 0|5372k 0 | 0 0 |1380k 0 |1311 1561 | 514M 644k 140M 6922M 0 1 78 21 0 0|4812k 0 | 0 0 |1908k 0 |1319 2009 | 516M 652k 143M 6917M 0 0 81 19 0 0|4372k 0 | 0 0 |1500k 0 |1163 1518 | 518M 652k 146M 6913M 0 0 75 25 0 0|6104k 516k| 0 65B|1748k 0 |1324 1766 | 520M 928k 150M 6906M 0 0 74 26 0 0|5268k 0 | 0 212B|1796k 0 |1161 1576 | 522M 928k 153M 6901M 0 0 82 17 0 0|4004k 0 | 0 0 |1416k 0 |1069 1417 | 523M 928k 156M 6897M 0 1 87 12 0 0|5808k 32k| 0 0 |1956k 0 |1211 1875 | 525M 936k 159M 6892M 0 0 88 12 0 0|2220k 50M| 428B 0 | 384k 0 | 821 837 | 520M 936k 161M 6895M 1 0 88 11 0 0|4304k 6252k| 0 0 |1056k 0 | 517 1615 | 519M 936k 164M 6892M 1 0 88 10 0 0|3052k 0 | 0 0 |1108k 0 | 599 1660 | 520M 936k 166M 6890M^C @Giles, Do you have any idea why Digikam is reading disk so bad ? Right when the leak is triggered, you can see, it reads 500+ MiB of data and then similary writes it down again. I tried the fix in comments 57 & 58 and the problem still exists for me. I am running DK Version 3.5.0 Using KDE Development Platform 4.13.0 on Linux Mint 17 64 bit The problem occurs when manually adding a single face tag. Please let me know if I can help by providing logs or other info Thanks Mick Hi, We need a fresh feedback using 4.0.0 just released. Also, to have a log using valgrind about memory leak can help. Use this command in CLI, and report console trace to investiguate : "valgrind --tool=memcheck --leak-check=full --error-limit=no digikam" Gilles Caulier Created attachment 86795 [details]
digikam 4.0 still leaking memory
(In reply to comment #65) > Hi, > > We need a fresh feedback using 4.0.0 just released. > > Also, to have a log using valgrind about memory leak can help. Use this > command in CLI, and report console trace to investiguate : > > "valgrind --tool=memcheck --leak-check=full --error-limit=no digikam" > > Gilles Caulier Comment #66 has what you asked. This now is the pristine copy of Digikam as uploaded in Debian, i.e. the libPGF in use is external. Sorry for the long delay, I can confirm that his bug had regressed when I upgraded Ubuntu from 13.10 to 14.04 and from 4.0.0beta to stable. But for me, after deleting recognition.db it fixed the issue again, so I am starting to think there is more than meets the eye. The problem with libPGF is gone, but clearly one with libkface remains... Scott. I can confirm that nothing has changed for me with 4.0.0 - running face detection even on one single picture will make digikam fill 70GB memory and more (killed it after that). I'm using MariaDB as backend. What's about this memory leak, if SQlite DB is used instead ? Gilles Caulier (In reply to comment #70) > What's about this memory leak, if SQlite DB is used instead ? If I read it correctly, the valgrind log from comment #66 uses SQLite, so that doesn't seem to make a difference. It seems that everyone ignored my comments where I have investigated the issue a bit https://bugs.kde.org/show_bug.cgi?id=323888#c5 https://bugs.kde.org/show_bug.cgi?id=323888#c6 In a nutshell my findings show that in some cases recognition.db becomes two times larger, so if this is repeated several times recognition.db becomes unbearable for digikam. I have described steps to reproduce, please try to reproduce according to my guide. It takes less than 3 minutes. (In reply to comment #68) > Sorry for the long delay, > > I can confirm that his bug had regressed when I upgraded Ubuntu from 13.10 > to 14.04 and from 4.0.0beta to stable. But for me, after deleting > recognition.db it fixed the issue again, so I am starting to think there is > more than meets the eye. > > The problem with libPGF is gone, but clearly one with libkface remains... Thank you for sharing this informaion Scott. Based on your feedback I went and did the same. After removing the recognition.db file, it is working a charm. But I still don't know if we have covered all the cases where the bug was reproducible. I'll check my MariaDB instance for the face recognition DB size. However, will deleting the DB mean losing the face tags? (In reply to comment #74) > I'll check my MariaDB instance for the face recognition DB size. However, > will deleting the DB mean losing the face tags? recognition.db contains just "fingerprints" for auto face detection. It can be re-created according to your face tags after deletion. (In reply to comment #73) > (In reply to comment #68) > > Sorry for the long delay, > > > > I can confirm that his bug had regressed when I upgraded Ubuntu from 13.10 > > to 14.04 and from 4.0.0beta to stable. But for me, after deleting > > recognition.db it fixed the issue again, so I am starting to think there is > > more than meets the eye. > > > > The problem with libPGF is gone, but clearly one with libkface remains... > > > Thank you for sharing this informaion Scott. Based on your feedback I went > and did the same. After removing the recognition.db file, it is working a > charm. But I still don't know if we have covered all the cases where the bug > was reproducible. I totally agree. While in my specific case recompiling libPGF, putting it in the right places and removing recognition.db when the bug regressed works, it's clear that it did not for everyone. Either steps were not followed, or there is something more. But either way, it would be nice to know just what is corrupting recognition.db in the first place... On trying to repeat the corruption (I think it may be related to the detection and recognition option), i am now getting a crash when detecting, but tagging is still working. As for deleting face tags, as said before, that recognition.db is for "fingerprints" only. The tags and the face boxes themselves seem to be kept in digikam4.db, if not in the metadata itself (that is of course if the write metadata to images option is on!). For my piece of mind, and portability, I personally leave the metadata writing on... Git commit fa909aa96e7e41225551cd9e866618717f1ea546 by Gilles Caulier. Committed on 11/06/2014 at 21:06. Pushed by cgilles into branch 'master'. use a better C++ exception wrapper to handle non OpenCV exceptions (as pure C++ one) Related: bug 335624, bug 330342, bug 329873, bug 326742, bug 326586, bug 326585, bug 324774, bug 323361, bug 320812, bug 312440, bug 309027, bug 308645, bug 301611, bug 297558, bug 285517 M +4 -0 libkface/facedetector.cpp M +17 -2 libkface/recognitiondatabase.cpp http://commits.kde.org/libkface/fa909aa96e7e41225551cd9e866618717f1ea546 With next digiKam 4.1.0, i fixed libkface to handle all C++ exception (and not only OpenCV exception). So, at least, digiKam must crash lesser now. If you want to review this entry again, use current implementation from git/master, or wait next 4.1.0 release... Thanks to update your feedback Gilles Caulier Git commit 4662dd94102f8144bc65ce1cb66d6b6cb1d500fd by Gilles Caulier. Committed on 18/06/2014 at 14:22. Pushed by cgilles into branch 'master'. Libkface now depand of last stable OpenCV library version 2.4.9 This prevent internal crash int Cv:Algorithm that we cannot handle in libkface as exception. Now, Face Recognition do not crash but still report Exception at training operations, especially about wrong Cv:Matrix size : digikam(8673)/digikam (core) Digikam::DImg::load:"/mnt/data2/photos/GILLES/NEW/Adrien/2010-04-13/20100413_009.jpg" : JPEG file identified OpenCV Error: Assertion failed (0 <= _dims && _dims <= CV_MAX_DIM) in setSize, file /mnt/devel/opencv/modules/core/src/matrix.cpp, line 89 digikam(8673)/KFACE: cv::Exception training LBPH: /mnt/devel/opencv/modules/core/src/matrix.cpp:89: error: (-215) 0 <=_dims && _dims <= CV_MAX_DIM in function setSize It still a problem somwhere, but it's better than previous state. Related: bug 335624, bug 330342, bug 329873, bug 326742, bug 326586, bug 326585, bug 324774, bug 323361, bug 320812, bug 312440, bug 309027, bug 308645, bug 301611, bug 297558, bug 285517 M +1 -1 CMakeLists.txt http://commits.kde.org/libkface/4662dd94102f8144bc65ce1cb66d6b6cb1d500fd Git commit 1fff86f31e3bf47a2a2cfa6eaa98bb1bdf1a863b by Gilles Caulier. Committed on 24/06/2014 at 12:32. Pushed by cgilles into branch 'master'. add more test before to commit/checkout compressed histogram data in Face database, to prevent crashes, especially if data are corrupted from database. Related: bug 335624, bug 330342, bug 329873, bug 326742, bug 326586, bug 326585, bug 324774, bug 320812, bug 312440, bug 309027, bug 308645, bug 301611, bug 297558, bug 285517 M +62 -28 libkface/database/trainingdb.cpp http://commits.kde.org/libkface/1fff86f31e3bf47a2a2cfa6eaa98bb1bdf1a863b I have bad news for this entry. My investigations result sound to conclude that there are huge memory liberation performed outside digiKam database interface. I tested it with SQLite. It's not memory leak. allocated memory is liberated at end of digiKam session. But database transactions take memory again and again. There is no reason for that. This is true for main digiKam database, and of course with Face database which store image compressed histogram data processed by faces training process. I'm sure to not have seen it in the pass. I suspect a regression in libsqlite or Qt4-sqlite plugin. To resume, i suspect a UPSTREAM bug. Gilles Caulier Note : I suspect a similar problem with QSlite about main digiKAm database at scanning process, especially at first time run, when collection to parse is huge. I remember to see report about crash at start up when swap become active. Here i cannot reproduce this typical crash, because my computer use 16 Gb of RAM. It's of course reproducible with a VM using limited RAM. Gilles Caulier I think Alexander has a point in #5 #6 and #72 - when the recognition.db is deleted, everything works fine. So the real issue is that it must be verified that it cannot grow as large as it had for those of us who had been affected by this bug. In a VM, i can reproduce the memory leak from SQlite. I take a similar fingerprint with valgrind here. Look log from attachment "digikam 4.0 still leaking memory" : ==26673== 305,920 bytes in 239 blocks are possibly lost in loss record 120,501 of 120,538 ==26673== at 0x4C274A0: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==26673== by 0x21076F46: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21050AE9: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21059217: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x2105CEBA: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x2105D125: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x210569B0: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21083B35: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21083F1C: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21086102: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21086179: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21086257: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== ==26673== 1,332,480 bytes in 1,041 blocks are possibly lost in loss record 120,527 of 120,538 ==26673== at 0x4C274A0: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==26673== by 0x21076F46: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21050AE9: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21059217: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x2105CEBA: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x2105D125: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x210569B0: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21083B35: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21083F1C: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21086102: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21086179: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x2108B2AB: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== ==26673== 2,752,344 bytes in 43 blocks are possibly lost in loss record 120,533 of 120,538 ==26673== at 0x4C274A0: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==26673== by 0x21076F46: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21050AE9: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21059217: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21062FD1: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x210D7F9D: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x36ED985E: QSQLiteDriver::open(QString const&, QString const&, QString const&, QString const&, int, QString const&) (in /usr/lib/x86_64-linux-gnu/qt4/plugins/sqldrivers/libqsqlite.so) ==26673== by 0x4E44D60: QSqlDatabase::open() (in /usr/lib/x86_64-linux-gnu/libQtSql.so.4.8.6) ==26673== by 0x7516600: Digikam::DatabaseCoreBackendPrivate::open(QSqlDatabase&) (in /usr/lib/digikam/libdigikamcore.so.4.0.0) ==26673== by 0x7516BCA: Digikam::DatabaseCoreBackendPrivate::databaseForThread() (in /usr/lib/digikam/libdigikamcore.so.4.0.0) ==26673== by 0x7516E99: Digikam::DatabaseCoreBackend::getQuery() (in /usr/lib/digikam/libdigikamcore.so.4.0.0) ==26673== by 0x7519391: Digikam::DatabaseCoreBackend::prepareQuery(QString const&) (in /usr/lib/digikam/libdigikamcore.so.4.0.0) ==26673== ==26673== 4,497,920 bytes in 3,514 blocks are possibly lost in loss record 120,535 of 120,538 ==26673== at 0x4C274A0: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==26673== by 0x21076F46: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21050AE9: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21059217: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x2105CEBA: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x2105D125: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x210569B0: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21083B35: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x2108AA23: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x2108C55E: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x210B72E8: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x210B834E: sqlite3_step (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== ==26673== 305,920 bytes in 239 blocks are possibly lost in loss record 120,501 of 120,538 ==26673== at 0x4C274A0: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==26673== by 0x21076F46: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21050AE9: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21059217: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x2105CEBA: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x2105D125: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x210569B0: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21083B35: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21083F1C: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21086102: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21086179: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21086257: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== ==26673== 1,332,480 bytes in 1,041 blocks are possibly lost in loss record 120,527 of 120,538 ==26673== at 0x4C274A0: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==26673== by 0x21076F46: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21050AE9: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21059217: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x2105CEBA: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x2105D125: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x210569B0: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21083B35: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21083F1C: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21086102: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21086179: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x2108B2AB: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== ==26673== 2,752,344 bytes in 43 blocks are possibly lost in loss record 120,533 of 120,538 ==26673== at 0x4C274A0: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==26673== by 0x21076F46: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21050AE9: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21059217: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21062FD1: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x210D7F9D: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x36ED985E: QSQLiteDriver::open(QString const&, QString const&, QString const&, QString const&, int, QString const&) (in /usr/lib/x86_64-linux-gnu/qt4/plugins/sqldrivers/libqsqlite.so) ==26673== by 0x4E44D60: QSqlDatabase::open() (in /usr/lib/x86_64-linux-gnu/libQtSql.so.4.8.6) ==26673== by 0x7516600: Digikam::DatabaseCoreBackendPrivate::open(QSqlDatabase&) (in /usr/lib/digikam/libdigikamcore.so.4.0.0) ==26673== by 0x7516BCA: Digikam::DatabaseCoreBackendPrivate::databaseForThread() (in /usr/lib/digikam/libdigikamcore.so.4.0.0) ==26673== by 0x7516E99: Digikam::DatabaseCoreBackend::getQuery() (in /usr/lib/digikam/libdigikamcore.so.4.0.0) ==26673== by 0x7519391: Digikam::DatabaseCoreBackend::prepareQuery(QString const&) (in /usr/lib/digikam/libdigikamcore.so.4.0.0) ==26673== ==26673== 4,497,920 bytes in 3,514 blocks are possibly lost in loss record 120,535 of 120,538 ==26673== at 0x4C274A0: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==26673== by 0x21076F46: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21050AE9: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21059217: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x2105CEBA: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x2105D125: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x210569B0: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x21083B35: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x2108AA23: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x2108C55E: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x210B72E8: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== by 0x210B834E: sqlite3_step (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) ==26673== Memory leak from SQlite is just enormous... Gilles Caulier With next digiKam 4.1.0, a lots of improvements/fixes have be done around face management. Please give us a fresh feedback. Note i recommend to delete face recognition database to prevent dysfunction due to possible wrong data store in this container. Look where file is located in my computer : [gilles@localhost database]$ pwd /home/gilles/.kde4/share/apps/libkface/database [gilles@localhost database]$ ls -al total 397028 drwx------ 2 gilles gilles 4096 juin 24 14:22 ./ drwx------ 3 gilles gilles 4096 juin 18 19:08 ../ -rw-r--r-- 1 gilles gilles 406543360 juin 24 14:22 recognition.db [gilles@localhost database]$ I just check again digiKam 4.2.0 (current git/master implementation), and i can confirm the huge memory leak done by Qt Sqlite plugin when face are detected from image and registered to digiKam Database. Sound like there is a big problem in Qt Sqlite plugin. See end of valgrind backtrace below : ==7348== 34,560 bytes in 27 blocks are possibly lost in loss record 26,533 of 26,604 ==7348== at 0x4C266ED: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==7348== by 0x1F8569A6: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F832189: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F83A317: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F83CDCA: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F83D021: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F8378E8: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F862255: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F86250C: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F864701: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F8649EE: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F8660E4: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== ==7348== 44,800 bytes in 35 blocks are possibly lost in loss record 26,540 of 26,604 ==7348== at 0x4C266ED: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==7348== by 0x1F8569A6: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F832189: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F83A317: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F83CDCA: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F83D021: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F8378E8: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F862255: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F86250C: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F864701: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F864779: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F864867: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== ==7348== 64,008 bytes in 1 blocks are possibly lost in loss record 26,549 of 26,604 ==7348== at 0x4C266ED: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==7348== by 0x1F8569A6: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F832189: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F83A317: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F840AE1: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F8B42A6: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x2D04581E: ??? (in /usr/lib64/qt4/plugins/sqldrivers/libqsqlite.so) ==7348== by 0x4E42D60: QSqlDatabase::open() (in /usr/lib64/libQtSql.so.4.8.6) ==7348== by 0x7C179D6: Digikam::DatabaseCoreBackendPrivate::open(QSqlDatabase&) (databasecorebackend.cpp:214) ==7348== by 0x7C172C0: Digikam::DatabaseCoreBackendPrivate::databaseForThread() (databasecorebackend.cpp:120) ==7348== by 0x7C1A7A3: Digikam::DatabaseCoreBackend::open(Digikam::DatabaseParameters const&) (databasecorebackend.cpp:769) ==7348== by 0x81D2BD1: Digikam::DatabaseAccess::checkReadyForUse(Digikam::InitializationObserver*) (databaseaccess.cpp:286) ==7348== ==7348== 602,880 bytes in 471 blocks are possibly lost in loss record 26,596 of 26,604 ==7348== at 0x4C266ED: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==7348== by 0x1F8569A6: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F832189: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F83A317: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F83CDCA: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F83D021: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F8378E8: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F862255: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F86250C: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F864701: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F864779: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F866338: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== ==7348== 768,096 bytes in 12 blocks are possibly lost in loss record 26,600 of 26,604 ==7348== at 0x4C266ED: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==7348== by 0x1F8569A6: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F832189: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F83A317: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F840AE1: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F8B42A6: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x2D04581E: ??? (in /usr/lib64/qt4/plugins/sqldrivers/libqsqlite.so) ==7348== by 0x4E42D60: QSqlDatabase::open() (in /usr/lib64/libQtSql.so.4.8.6) ==7348== by 0x7C179D6: Digikam::DatabaseCoreBackendPrivate::open(QSqlDatabase&) (databasecorebackend.cpp:214) ==7348== by 0x7C172C0: Digikam::DatabaseCoreBackendPrivate::databaseForThread() (databasecorebackend.cpp:120) ==7348== by 0x7C1D2DD: Digikam::DatabaseCoreBackend::getQuery() (databasecorebackend.cpp:1512) ==7348== by 0x7C1CF12: Digikam::DatabaseCoreBackend::prepareQuery(QString const&) (databasecorebackend.cpp:1467) ==7348== ==7348== 2,291,200 bytes in 1,790 blocks are possibly lost in loss record 26,603 of 26,604 ==7348== at 0x4C266ED: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==7348== by 0x1F8569A6: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F832189: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F83A317: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F83CDCA: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F83D021: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F8378E8: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F862255: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F865B53: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F866B26: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F895911: ??? (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== by 0x1F89655E: sqlite3_step (in /usr/lib64/libsqlite3.so.0.8.6) ==7348== ==7348== LEAK SUMMARY: ==7348== definitely lost: 246,004 bytes in 1,710 blocks ==7348== indirectly lost: 584,784 bytes in 10,817 blocks ==7348== possibly lost: 4,004,231 bytes in 4,642 blocks ==7348== still reachable: 21,369,113 bytes in 64,209 blocks ==7348== suppressed: 0 bytes in 0 blocks ==7348== Reachable blocks (those to which a pointer was found) are not shown. ==7348== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==7348== ==7348== For counts of detected and suppressed errors, rerun with: -v ==7348== Use --track-origins=yes to see where uninitialised values come from ==7348== ERROR SUMMARY: 1474 errors from 1450 contexts (suppressed: 4 from 3) Gilles Caulier Note : Linux Mageia 4 64 bits, Qt 4.8.2, libsqlite 3.8.0.2 I forward this file to qt components for future investigations from Qt developers. Gilles Caulier Without to run whole digiKam with Face Management feature, you can reproduce memory leak with libkface "kfacegui" test program. When tool is started and closed imediatly, whitout too processs any image to detect or recognize faces, the face database is open by Qt sqlite plugin and valgrind report this trace about : ==15618== 476 (96 direct, 380 indirect) bytes in 1 blocks are definitely lost in loss record 635 of 697 ==15618== at 0x4C26BF5: operator new(unsigned long) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==15618== by 0x6218FA2: ??? (in /usr/lib64/libQtCore.so.4.8.6) ==15618== by 0x6213567: QFactoryLoader::updateDir(QString const&, QSettings&) (in /usr/lib64/libQtCore.so.4.8.6) ==15618== by 0x62154C6: QFactoryLoader::update() (in /usr/lib64/libQtCore.so.4.8.6) ==15618== by 0x62156E9: QFactoryLoader::QFactoryLoader(char const*, QString const&, Qt::CaseSensitivity) (in /usr/lib64/libQtCore.so.4.8.6) ==15618== by 0x8A94844: ??? (in /usr/lib64/libQtSql.so.4.8.6) ==15618== by 0x8A9551F: QSqlDatabase::drivers() (in /usr/lib64/libQtSql.so.4.8.6) ==15618== by 0x4E812B8: KFaceIface::DatabaseAccess::checkReadyForUse(KFaceIface::DatabaseAccessData*, KFaceIface::InitializationObserver*) (databaseaccess.cpp:201) ==15618== by 0x4E583C0: KFaceIface::RecognitionDatabase::Private::Private(QString const&) (recognitiondatabase.cpp:215) ==15618== by 0x4E5814F: KFaceIface::RecognitionDatabaseStaticPriv::database(QString const&) (recognitiondatabase.cpp:192) ==15618== by 0x4E588DE: KFaceIface::RecognitionDatabase::addDatabase(QString const&) (recognitiondatabase.cpp:276) ==15618== by 0x40A5DE: MainWindow::MainWindow(QWidget*) (mainwindow.cpp:159) Of course, it's not memory leak due to DB data registration as with digiKam face management, but it's also abnormal to leak memory just to open database file. libkface implementation is available here : https://projects.kde.org/projects/extragear/libs/libkface/repository Gilles Caulier I progressed well in this entry about investigations. Here under Mageia4, i use Qt 4.8.2 and libsqlite 3.8.0.2 I written a small CLI program to check digiKam database init with valgrind. Installing qt and sqlite debug package, i can identify now where memory is leak. In fact it's a know problem already reported in this entry : https://bugs.kde.org/show_bug.cgi?id=329697 ... where users complain that digiKam crash into libsqlite memory management functions. So now with valgrind, we can see where memory is leak on my computer : ==32332== 7,680 bytes in 6 blocks are possibly lost in loss record 2,049 of 2,066 ==32332== at 0x4C266ED: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==32332== by 0x1DDFF9A6: sqlite3MemMalloc (sqlite3.c:15739) ==32332== by 0x1DDDB189: mallocWithAlarm (sqlite3.c:19037) ==32332== by 0x1DDE3317: sqlite3Malloc (sqlite3.c:19070) ==32332== by 0x1DDE5DCA: pcache1Alloc (sqlite3.c:36875) ==32332== by 0x1DDE6021: pcache1Fetch (sqlite3.c:36959) ==32332== by 0x1DDE08E8: sqlite3PcacheFetch (sqlite3.c:36286) ==32332== by 0x1DE0B255: sqlite3PagerAcquire (sqlite3.c:43632) ==32332== by 0x1DE0B50C: btreeGetPage (sqlite3.c:51175) ==32332== by 0x1DE0D701: getAndInitPage (sqlite3.c:51230) ==32332== by 0x1DE0D9EE: moveToRoot (sqlite3.c:53987) ==32332== by 0x1DE0F0E4: sqlite3BtreeMovetoUnpacked (sqlite3.c:54204) ==32332== ==32332== 9,000 bytes in 45 blocks are possibly lost in loss record 2,056 of 2,066 ==32332== at 0x4C266ED: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==32332== by 0x1DDFF9A6: sqlite3MemMalloc (sqlite3.c:15739) ==32332== by 0x1DDDB189: mallocWithAlarm (sqlite3.c:19037) ==32332== by 0x1DDE3317: sqlite3Malloc (sqlite3.c:19070) ==32332== by 0x1DDE3450: sqlite3DbMallocRaw (sqlite3.c:19406) ==32332== by 0x1DDF06DB: exprDup (sqlite3.c:76262) ==32332== by 0x1DE2EEA3: sqlite3Parser (sqlite3.c:76350) ==32332== by 0x1DE34459: sqlite3RunParser (sqlite3.c:115140) ==32332== by 0x1DE34A81: sqlite3Prepare (sqlite3.c:96189) ==32332== by 0x1DE34D64: sqlite3LockAndPrepare (sqlite3.c:96280) ==32332== by 0x1DE34DF4: sqlite3_prepare (sqlite3.c:96344) ==32332== by 0x1DE34EAE: sqlite3InitCallback (sqlite3.c:95658) ==32332== ==32332== 9,564 (1,920 direct, 7,644 indirect) bytes in 20 blocks are definitely lost in loss record 2,058 of 2,066 ==32332== at 0x4C26BF5: operator new(unsigned long) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==32332== by 0x5C39FA2: ??? (in /usr/lib64/libQtCore.so.4.8.6) ==32332== by 0x5C34567: QFactoryLoader::updateDir(QString const&, QSettings&) (in /usr/lib64/libQtCore.so.4.8.6) ==32332== by 0x5C364C6: QFactoryLoader::update() (in /usr/lib64/libQtCore.so.4.8.6) ==32332== by 0x5C366E9: QFactoryLoader::QFactoryLoader(char const*, QString const&, Qt::CaseSensitivity) (in /usr/lib64/libQtCore.so.4.8.6) ==32332== by 0x6456754: ??? (in /usr/lib64/libQtGui.so.4.8.6) ==32332== by 0x6456BBF: ??? (in /usr/lib64/libQtGui.so.4.8.6) ==32332== by 0x6458891: ??? (in /usr/lib64/libQtGui.so.4.8.6) ==32332== by 0x6459DE7: QImageReader::read(QImage*) (in /usr/lib64/libQtGui.so.4.8.6) ==32332== by 0x6459FB3: QImageReader::read() (in /usr/lib64/libQtGui.so.4.8.6) ==32332== by 0x644E5D8: operator>>(QDataStream&, QImage&) (in /usr/lib64/libQtGui.so.4.8.6) ==32332== by 0x64676D8: operator>>(QDataStream&, QPixmap&) (in /usr/lib64/libQtGui.so.4.8.6) ==32332== ==32332== 15,288 bytes in 39 blocks are possibly lost in loss record 2,059 of 2,066 ==32332== at 0x4C266ED: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==32332== by 0x1DDFF9A6: sqlite3MemMalloc (sqlite3.c:15739) ==32332== by 0x1DDDB189: mallocWithAlarm (sqlite3.c:19037) ==32332== by 0x1DDE3317: sqlite3Malloc (sqlite3.c:19070) ==32332== by 0x1DDE3450: sqlite3DbMallocRaw (sqlite3.c:19406) ==32332== by 0x1DE31C06: sqlite3Parser (sqlite3.c:83487) ==32332== by 0x1DE34459: sqlite3RunParser (sqlite3.c:115140) ==32332== by 0x1DE34A81: sqlite3Prepare (sqlite3.c:96189) ==32332== by 0x1DE34D64: sqlite3LockAndPrepare (sqlite3.c:96280) ==32332== by 0x1DE34DF4: sqlite3_prepare (sqlite3.c:96344) ==32332== by 0x1DE34EAE: sqlite3InitCallback (sqlite3.c:95658) ==32332== by 0x1DE35258: sqlite3_exec (sqlite3.c:92433) ==32332== ==32332== 38,400 bytes in 30 blocks are possibly lost in loss record 2,063 of 2,066 ==32332== at 0x4C266ED: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==32332== by 0x1DDFF9A6: sqlite3MemMalloc (sqlite3.c:15739) ==32332== by 0x1DDDB189: mallocWithAlarm (sqlite3.c:19037) ==32332== by 0x1DDE3317: sqlite3Malloc (sqlite3.c:19070) ==32332== by 0x1DDE5DCA: pcache1Alloc (sqlite3.c:36875) ==32332== by 0x1DDE6021: pcache1Fetch (sqlite3.c:36959) ==32332== by 0x1DDE08E8: sqlite3PcacheFetch (sqlite3.c:36286) ==32332== by 0x1DE0B255: sqlite3PagerAcquire (sqlite3.c:43632) ==32332== by 0x1DE0B50C: btreeGetPage (sqlite3.c:51175) ==32332== by 0x1DE0D701: getAndInitPage (sqlite3.c:51230) ==32332== by 0x1DE0D779: moveToChild (sqlite3.c:53869) ==32332== by 0x1DE0D867: moveToLeftmost (sqlite3.c:54049) ==32332== ==32332== 64,008 bytes in 1 blocks are possibly lost in loss record 2,064 of 2,066 ==32332== at 0x4C266ED: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==32332== by 0x1DDFF9A6: sqlite3MemMalloc (sqlite3.c:15739) ==32332== by 0x1DDDB189: mallocWithAlarm (sqlite3.c:19037) ==32332== by 0x1DDE3317: sqlite3Malloc (sqlite3.c:19070) ==32332== by 0x1DDE9AE1: setupLookaside.part.235 (sqlite3.c:116154) ==32332== by 0x1DE5D2A6: openDatabase (sqlite3.c:93720) ==32332== by 0x2BA5A81E: ??? (in /usr/lib64/qt4/plugins/sqldrivers/libqsqlite.so) ==32332== by 0x6E8CD60: QSqlDatabase::open() (in /usr/lib64/libQtSql.so.4.8.6) ==32332== by 0x5173AA8: Digikam::DatabaseCoreBackendPrivate::open(QSqlDatabase&) (databasecorebackend.cpp:214) ==32332== by 0x51733D0: Digikam::DatabaseCoreBackendPrivate::databaseForThread() (databasecorebackend.cpp:120) ==32332== by 0x5176861: Digikam::DatabaseCoreBackend::open(Digikam::DatabaseParameters const&) (databasecorebackend.cpp:769) ==32332== by 0x572FBD1: Digikam::DatabaseAccess::checkReadyForUse(Digikam::InitializationObserver*) (databaseaccess.cpp:286) ==32332== ==32332== 64,008 bytes in 1 blocks are possibly lost in loss record 2,065 of 2,066 ==32332== at 0x4C266ED: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==32332== by 0x1DDFF9A6: sqlite3MemMalloc (sqlite3.c:15739) ==32332== by 0x1DDDB189: mallocWithAlarm (sqlite3.c:19037) ==32332== by 0x1DDE3317: sqlite3Malloc (sqlite3.c:19070) ==32332== by 0x1DDE9AE1: setupLookaside.part.235 (sqlite3.c:116154) ==32332== by 0x1DE5D2A6: openDatabase (sqlite3.c:93720) ==32332== by 0x2BA5A81E: ??? (in /usr/lib64/qt4/plugins/sqldrivers/libqsqlite.so) ==32332== by 0x6E8CD60: QSqlDatabase::open() (in /usr/lib64/libQtSql.so.4.8.6) ==32332== by 0x5173AA8: Digikam::DatabaseCoreBackendPrivate::open(QSqlDatabase&) (databasecorebackend.cpp:214) ==32332== by 0x51733D0: Digikam::DatabaseCoreBackendPrivate::databaseForThread() (databasecorebackend.cpp:120) ==32332== by 0x517939B: Digikam::DatabaseCoreBackend::getQuery() (databasecorebackend.cpp:1512) ==32332== by 0x5178FD0: Digikam::DatabaseCoreBackend::prepareQuery(QString const&) (databasecorebackend.cpp:1467) ==32332== ==32332== 64,008 bytes in 1 blocks are possibly lost in loss record 2,066 of 2,066 ==32332== at 0x4C266ED: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==32332== by 0x1DDFF9A6: sqlite3MemMalloc (sqlite3.c:15739) ==32332== by 0x1DDDB189: mallocWithAlarm (sqlite3.c:19037) ==32332== by 0x1DDE3317: sqlite3Malloc (sqlite3.c:19070) ==32332== by 0x1DDE9AE1: setupLookaside.part.235 (sqlite3.c:116154) ==32332== by 0x1DE5D2A6: openDatabase (sqlite3.c:93720) ==32332== by 0x2BA5A81E: ??? (in /usr/lib64/qt4/plugins/sqldrivers/libqsqlite.so) ==32332== by 0x6E8CD60: QSqlDatabase::open() (in /usr/lib64/libQtSql.so.4.8.6) ==32332== by 0x5173AA8: Digikam::DatabaseCoreBackendPrivate::open(QSqlDatabase&) (databasecorebackend.cpp:214) ==32332== by 0x51733D0: Digikam::DatabaseCoreBackendPrivate::databaseForThread() (databasecorebackend.cpp:120) ==32332== by 0x5176861: Digikam::DatabaseCoreBackend::open(Digikam::DatabaseParameters const&) (databasecorebackend.cpp:769) ==32332== by 0x5181716: Digikam::ThumbnailDatabaseAccess::checkReadyForUse(Digikam::InitializationObserver*) (thumbnaildatabaseaccess.cpp:216) ==32332== ==32332== LEAK SUMMARY: ==32332== definitely lost: 8,000 bytes in 690 blocks ==32332== indirectly lost: 9,358 bytes in 81 blocks ==32332== possibly lost: 391,691 bytes in 1,652 blocks ==32332== still reachable: 336,972 bytes in 4,708 blocks ==32332== suppressed: 0 bytes in 0 blocks ==32332== Reachable blocks (those to which a pointer was found) are not shown. ==32332== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==32332== ==32332== For counts of detected and suppressed errors, rerun with: -v ==32332== ERROR SUMMARY: 957 errors from 957 contexts (suppressed: 2 from 2) [gilles@localhost tests]$ So i suspect this entry to be fully relevant of Sqlite implementation as it's have already reported as UPSTREAM #329697 Note the difference : #329697 is about libsqlite:: sqlite3MemCompare() when this file is about libsqlite::sqlite3MemMalloc(). #329697 have been fixed with a patch release of libsqlite has it's explained in this Ubuntu report : https://bugs.launchpad.net/ubuntu/+source/sqlite3/+bug/1317449 - Packagers and Users must update libsqlite to last stable 3.8.5 (http://www.sqlite.org/news.html) - Users must try to run FaceManagement again and look if memory leak still present or not. - If yes, this file must be reported as UPSTREAM to SQLite bug tracker. Giles Caulier For information, the problem still present with digikam 4.1.0 and sqlite 3.8.5. if it really is fixed upstream, can you link the commit with the patch that fixes this issue if there is one? Nico, I never said that problem is already fixed as UPSTREAM fixes I closed this file as UPSTREAM because i suspect that problem is located in libsqlite. This problem must be reported to SQlite bugzilla to be analysed by Sqlite team. It can be a problem also in Qt sqlite plugin, but i'm not sure... Gilles Caulier oh, ok - I just wondered about the status change without any further information. Actually, I resolved the problem by deleting my ~700MB face recognition DB for now. I kept the file in case it helps for debugging. Since I'm still using the same SQLite version, I suspect either the huge file size (and in turn any indexes etc.) or something went wrong during an update (?) I can details more some technicals stuff here : - Face detection store region of image with face in main digiKam database. This point mangle memory. - Face recognition is another algorithm which store face histograms in a dedicated database (the famous file that you have deleted). Recognition with previous version crash digiKam, but do not mangle memory. This is due to wrong storage of histogram data in database blob, and some incompatibility with previous OpenCV version which compute histogram. Removing this database will prevent crash. With 4.2.0, histograms generated will be done in a more secure way. I also add more protection in code to prevent wrong context. So, this entry is about face detection only. Gilles Caulier From the bug title and description in the first entry, I guess this bug is actually about recognition.db and face recognition. IIRC I had the crash problem you mentioned with the upgrade to 4.1.0 and re-created the DB with the appropriate GUI-Entry at that time. Now with 4.2.0, face detection works (al always) but whenever I gave an unknown face a name, digikam kept growing in memory size until it was killed. Re-creating the DB from the GUI did result in the same error - but deleting the DB file manually worked. *** Bug 338249 has been marked as a duplicate of this bug. *** *** Bug 335013 has been marked as a duplicate of this bug. *** *** This bug has been confirmed by popular vote. *** Git commit f6bcec9d07f0f6530835498370a6042f9b1e1675 by Marcel Wiesweg. Committed on 14/11/2014 at 09:56. Pushed by mwiesweg into branch 'master'. Some fixes in face progress calculation M +1 -0 utilities/facemanagement/facepipeline.cpp M +2 -0 utilities/facemanagement/facepipeline.h M +31 -14 utilities/maintenance/facedetector.cpp http://commits.kde.org/digikam/f6bcec9d07f0f6530835498370a6042f9b1e1675 diff --git a/utilities/facemanagement/facepipeline.cpp b/utilities/facemanagement/facepipeline.cpp index a655385..d512a92 100644 --- a/utilities/facemanagement/facepipeline.cpp +++ b/utilities/facemanagement/facepipeline.cpp @@ -1171,6 +1171,7 @@ void FacePipeline::Private::send(FacePipelineExtendedPackage::Ptr package) { start(); ++totalPackagesAdded; + emit(q->processing(*package)); if (senderFlowControl(package)) { diff --git a/utilities/facemanagement/facepipeline.h b/utilities/facemanagement/facepipeline.h index 4b381ca..45e733a 100644 --- a/utilities/facemanagement/facepipeline.h +++ b/utilities/facemanagement/facepipeline.h @@ -292,6 +292,8 @@ Q_SIGNALS: /// Emitted when processing has started void started(const QString& message); + /// Emitted when one package begins processing + void processing(const FacePipelinePackage& package); /// Emitted when one package has finished processing void processed(const FacePipelinePackage& package); void progressValueChanged(float progress); diff --git a/utilities/maintenance/facedetector.cpp b/utilities/maintenance/facedetector.cpp index 7bd8c0e..222652b 100644 --- a/utilities/maintenance/facedetector.cpp +++ b/utilities/maintenance/facedetector.cpp @@ -96,18 +96,27 @@ class FaceDetector::Private public: Private() + : benchmark(false), + total(0), + progressValue(0), + currentProgressChunk(0), + currentScheduled(0), + currentFinished(0) { - benchmark = false; - total = 0; } - bool benchmark; + bool benchmark; - int total; + int total; - AlbumPointerList<> albumTodoList; - ImageInfoJob albumListing; - FacePipeline pipeline; + AlbumPointerList<> albumTodoList; + ImageInfoJob albumListing; + FacePipeline pipeline; + QMap<Album*,double> relativeProgressValue; + double progressValue; + double currentProgressChunk; + int currentScheduled; + int currentFinished; }; FaceDetector::FaceDetector(const FaceScanSettings& settings, ProgressItem* const parent) @@ -275,25 +284,33 @@ void FaceDetector::slotStart() QApplication::restoreOverrideCursor(); } - d->total = 0; - + // first, we use the relativeProgressValue map to store absolute counts foreach(Album* const album, d->albumTodoList) { if (album->type() == Album::PHYSICAL) { - d->total += palbumCounts.value(album->id()); + d->relativeProgressValue[album] = palbumCounts.value(album->id()); } else // this is possibly broken of course because we do not know if images have multiple tags, // but there's no better solution without expensive operation { - d->total += talbumCounts.value(album->id()); + d->relativeProgressValue[album] = talbumCounts.value(album->id()); } } - - kDebug() << "Total is" << d->total; - + // second, calculate (approximate) overall sum + d->total = 0; + foreach (double count, d->relativeProgressValue) + { + d->total += (int)count; + } d->total = qMax(1, d->total); + kDebug() << "Total is" << d->total; + // third, break absolute to relative values + for (QMap<Album*,double>::iterator it = d->relativeProgressValue.begin(); it != d->relativeProgressValue.end(); ++it) + { + it.value() /= double(d->total); + } setUsesBusyIndicator(false); setTotalItems(d->total); Git commit 59f0bdb6b2709171354418ed297d4aa1d667ebad by Marcel Wiesweg. Committed on 15/11/2014 at 13:47. Pushed by mwiesweg into branch 'master'. Add database cleanup calls to have a clean shutdown of SQLite data at application termination M +3 -0 app/main/main.cpp M +17 -3 tests/testdatabase.cpp http://commits.kde.org/digikam/59f0bdb6b2709171354418ed297d4aa1d667ebad diff --git a/app/main/main.cpp b/app/main/main.cpp index 3564d23..85fcbc7 100644 --- a/app/main/main.cpp +++ b/app/main/main.cpp @@ -62,6 +62,7 @@ #include "databaseparameters.h" #include "digikamapp.h" #include "scancontroller.h" +#include "thumbnaildatabaseaccess.h" #include "version.h" using namespace Digikam; @@ -235,6 +236,8 @@ int main(int argc, char* argv[]) int ret = app.exec(); + DatabaseAccess::cleanUpDatabase(); + ThumbnailDatabaseAccess::cleanUpDatabase(); KExiv2Iface::KExiv2::cleanupExiv2(); return ret; diff --git a/tests/testdatabase.cpp b/tests/testdatabase.cpp index 10c14ad..0930029 100644 --- a/tests/testdatabase.cpp +++ b/tests/testdatabase.cpp @@ -30,6 +30,7 @@ #include <QSqlDatabase> #include <QDBusConnection> #include <QString> +#include <QTimer> // KDE includes @@ -47,6 +48,7 @@ #include "databaseparameters.h" #include "scancontroller.h" #include "setup.h" +#include "thumbnaildatabaseaccess.h" #include "version.h" namespace Digikam @@ -76,16 +78,28 @@ int main(int argc, char** argv) KCmdLineArgs::init(argc, argv, &aboutData); KApplication app; - DatabaseParameters params = DatabaseParameters::parametersFromConfig(KGlobal::config()); + DatabaseParameters params; + params.databaseType = DatabaseParameters::SQLiteDatabaseType(); + params.setDatabasePath(QDir::currentPath() + "/digikam-test.db"); + params.setThumbsDatabasePath(QDir::currentPath() + "/digikam-thumbs-test.db"); + params.legacyAndDefaultChecks(); QDBusConnection::sessionBus().registerService("org.kde.digikam.startup-" + QString::number(QCoreApplication::instance()->applicationPid())); // initialize database - bool b = AlbumManager::instance()->setDatabase(params, false); + bool b = AlbumManager::instance()->setDatabase(params, false, "/media/fotos/Digikam Sample/"); kDebug() << "Database initialization done: " << b; - + + QTimer::singleShot(500, &app, SLOT(quit())); + app.exec(); + + ScanController::instance()->shutDown(); + + DatabaseAccess::cleanUpDatabase(); + ThumbnailDatabaseAccess::cleanUpDatabase(); + return 0; } Git commit b9f8dbfe470609ef31c5442cb2a4c97e02344233 by Marcel Wiesweg. Committed on 15/11/2014 at 13:45. Pushed by mwiesweg into branch 'master'. Rewrite per-thread database connection cleanup Use QThreadStorage of a per-thread DatabaseThreadData object which is destroyed when the thread finishes. M +103 -113 libs/database/core/databasecorebackend.cpp M +0 -5 libs/database/core/databasecorebackend.h M +22 -12 libs/database/core/databasecorebackend_p.h http://commits.kde.org/digikam/b9f8dbfe470609ef31c5442cb2a4c97e02344233 diff --git a/libs/database/core/databasecorebackend.cpp b/libs/database/core/databasecorebackend.cpp index c553bb1..76a958a 100644 --- a/libs/database/core/databasecorebackend.cpp +++ b/libs/database/core/databasecorebackend.cpp @@ -79,22 +79,64 @@ public: } }; +DatabaseThreadData::DatabaseThreadData() + : valid(0), + transactionCount(0) +{ +} + +DatabaseThreadData::~DatabaseThreadData() +{ + if (transactionCount) + { + kDebug() << "WARNING !!! Transaction count is" << transactionCount << "when destroying database!!!"; + } + closeDatabase(); +} + +void DatabaseThreadData::closeDatabase() +{ + QString connectionToRemove; + if (database.isOpen()) + { + connectionToRemove = database.connectionName(); + } + + // Destroy object + database = QSqlDatabase(); + + valid = 0; + transactionCount = 0; + lastError = QSqlError(); + + // Remove connection + if (!connectionToRemove.isNull()) + { + QSqlDatabase::removeDatabase(connectionToRemove); + } +} + DatabaseCoreBackendPrivate::DatabaseCoreBackendPrivate(DatabaseCoreBackend* const backend) - : q(backend) + : currentValidity(0), + isInTransaction(false), + status(DatabaseCoreBackend::Unavailable), + lock(0), + operationStatus(DatabaseCoreBackend::ExecuteNormal), + errorLockOperationStatus(DatabaseCoreBackend::ExecuteNormal), + errorHandler(0), + q(backend) { - status = DatabaseCoreBackend::Unavailable; - isInTransaction = false; - operationStatus = DatabaseCoreBackend::ExecuteNormal; - errorHandler = 0; - lock = 0; - errorLockOperationStatus = DatabaseCoreBackend::ExecuteNormal; } -void DatabaseCoreBackendPrivate::init(const QString& name, DatabaseLocking* const l) +DatabaseCoreBackendPrivate::~DatabaseCoreBackendPrivate() { - QObject::connect(QCoreApplication::instance(), SIGNAL(aboutToQuit()), - q, SLOT(slotMainThreadFinished())); + // Must be shut down from the main thread. + // Clean up the QThreadStorage. It deletes any stored data. + threadDataStorage.setLocalData(0); +} +void DatabaseCoreBackendPrivate::init(const QString& name, DatabaseLocking* const l) +{ backendName = name; lock = l; @@ -110,82 +152,43 @@ void DatabaseCoreBackendPrivate::init(const QString& name, DatabaseLocking* cons // finishing of the thread. QSqlDatabase DatabaseCoreBackendPrivate::databaseForThread() { - QThread* const thread = QThread::currentThread(); - QSqlDatabase db = threadDatabases[thread]; - int isValid = databasesValid[thread]; - - if (!isValid || !db.isOpen()) + DatabaseThreadData* threadData = 0; + if (!threadDataStorage.hasLocalData()) { - // need to open a db for thread - bool success = open(db); - - if (!success) - { - kDebug() << "Error while opening the database. Details: [" << db.lastError() << "]"; - } - - QObject::connect(thread, SIGNAL(finished()), - q, SLOT(slotThreadFinished())); + threadData = new DatabaseThreadData; + threadDataStorage.setLocalData(threadData); } - -#ifdef DATABASCOREBACKEND_DEBUG else { - kDebug() << "Database ["<< connectionName(thread) <<"] already open for thread ["<< thread <<"]."; + threadData = threadDataStorage.localData(); } -#endif - - return db; -} - -void DatabaseCoreBackendPrivate::closeDatabaseForThread() -{ - QThread* const thread = QThread::currentThread(); + // do we need to reopen the database because parameter changed and validity was increased? + if (threadData->valid && threadData->valid < currentValidity) + { + threadData->closeDatabase(); + } - // scope, so that db is destructed when calling removeDatabase + if (!threadData->valid || !threadData->database.isOpen()) { - QSqlDatabase db = threadDatabases[thread]; + threadData->database = createDatabaseConnection(); - if (db.isValid()) + if (threadData->database.open()) + { + threadData->valid = currentValidity; + } + else { - db.close(); + kDebug() << "Error while opening the database. Error was" << threadData->database.lastError(); } } - threadDatabases.remove(thread); - databaseErrors.remove(thread); - databasesValid[thread] = 0; - transactionCount.remove(thread); - QSqlDatabase::removeDatabase(connectionName(thread)); -} - -QSqlError DatabaseCoreBackendPrivate::databaseErrorForThread() -{ - QThread* const thread = QThread::currentThread(); - return databaseErrors[thread]; -} - -void DatabaseCoreBackendPrivate::setDatabaseErrorForThread(const QSqlError& lastError) -{ - QThread* const thread = QThread::currentThread(); - databaseErrors.insert(thread, lastError); -} - -QString DatabaseCoreBackendPrivate::connectionName(QThread* const thread) -{ - return backendName + QString::number((quintptr)thread); + return threadData->database; } -bool DatabaseCoreBackendPrivate::open(QSqlDatabase& db) +QSqlDatabase DatabaseCoreBackendPrivate::createDatabaseConnection() { - if (db.isValid()) - { - db.close(); - } - - QThread* const thread = QThread::currentThread(); - db = QSqlDatabase::addDatabase(parameters.databaseType, connectionName(thread)); + QSqlDatabase db = QSqlDatabase::addDatabase(parameters.databaseType, connectionName()); QString connectOptions = parameters.connectOptions; if (parameters.isSQLite()) @@ -211,46 +214,47 @@ bool DatabaseCoreBackendPrivate::open(QSqlDatabase& db) db.setUserName(parameters.userName); db.setPassword(parameters.password); - bool success = db.open(); + return db; +} - if (success==false) +void DatabaseCoreBackendPrivate::closeDatabaseForThread() +{ + if (threadDataStorage.hasLocalData()) { - kDebug() << "Error while opening the database. Error was <" << db.lastError() << ">"; + threadDataStorage.localData()->closeDatabase(); } - - threadDatabases[thread] = db; - databasesValid[thread] = 1; - transactionCount[thread] = 0; - - return success; } -bool DatabaseCoreBackendPrivate::incrementTransactionCount() +QSqlError DatabaseCoreBackendPrivate::databaseErrorForThread() { - QThread* const thread = QThread::currentThread(); - return (!transactionCount[thread]++); + if (threadDataStorage.hasLocalData()) + { + return threadDataStorage.localData()->lastError; + } + return QSqlError(); } -bool DatabaseCoreBackendPrivate::decrementTransactionCount() +void DatabaseCoreBackendPrivate::setDatabaseErrorForThread(const QSqlError& lastError) { - QThread* const thread = QThread::currentThread(); - return (!--transactionCount[thread]); + if (threadDataStorage.hasLocalData()) + { + threadDataStorage.localData()->lastError = lastError; + } } -bool DatabaseCoreBackendPrivate::isInTransactionInOtherThread() const +QString DatabaseCoreBackendPrivate::connectionName() { - QThread* const thread = QThread::currentThread(); - QHash<QThread*, int>::const_iterator it; + return backendName + QString::number((quintptr)QThread::currentThread()); +} - for (it = transactionCount.constBegin(); it != transactionCount.constEnd(); ++it) - { - if (it.key() != thread && it.value()) - { - return true; - } - } +bool DatabaseCoreBackendPrivate::incrementTransactionCount() +{ + return (!threadDataStorage.localData()->transactionCount++); +} - return false; +bool DatabaseCoreBackendPrivate::decrementTransactionCount() +{ + return (!--threadDataStorage.localData()->transactionCount); } bool DatabaseCoreBackendPrivate::isInMainThread() const @@ -740,18 +744,6 @@ void DatabaseCoreBackend::setDatabaseErrorHandler(DatabaseErrorHandler* const ha d->errorHandler = handler; } -void DatabaseCoreBackend::slotThreadFinished() -{ - Q_D(DatabaseCoreBackend); - d->closeDatabaseForThread(); -} - -void DatabaseCoreBackend::slotMainThreadFinished() -{ - Q_D(DatabaseCoreBackend); - d->closeDatabaseForThread(); -} - bool DatabaseCoreBackend::isCompatible(const DatabaseParameters& parameters) { return QSqlDatabase::drivers().contains(parameters.databaseType); @@ -761,10 +753,8 @@ bool DatabaseCoreBackend::open(const DatabaseParameters& parameters) { Q_D(DatabaseCoreBackend); d->parameters = parameters; - - // Force possibly opened thread dbs to re-open with new parameters. - // They are not accessible from this thread! - d->databasesValid.clear(); + // This will make possibly opened thread dbs reload at next access + d->currentValidity++; int retries = 0; @@ -1634,7 +1624,7 @@ DatabaseCoreBackend::QueryState DatabaseCoreBackend::commitTransaction() bool DatabaseCoreBackend::isInTransaction() const { Q_D(const DatabaseCoreBackend); - return d->isInTransactionInOtherThread(); + return d->isInTransaction; } void DatabaseCoreBackend::rollbackTransaction() diff --git a/libs/database/core/databasecorebackend.h b/libs/database/core/databasecorebackend.h index 41e47da..fa4dba4 100644 --- a/libs/database/core/databasecorebackend.h +++ b/libs/database/core/databasecorebackend.h @@ -472,11 +472,6 @@ public: LastInsertId */ -private Q_SLOTS: - - void slotThreadFinished(); - void slotMainThreadFinished(); - protected: DatabaseCoreBackendPrivate* const d_ptr; diff --git a/libs/database/core/databasecorebackend_p.h b/libs/database/core/databasecorebackend_p.h index 2078509..ff3a3fa 100644 --- a/libs/database/core/databasecorebackend_p.h +++ b/libs/database/core/databasecorebackend_p.h @@ -29,6 +29,7 @@ #include <QHash> #include <QSqlDatabase> #include <QThread> +#include <QThreadStorage> #include <QWaitCondition> // Local includes @@ -40,25 +41,38 @@ namespace Digikam { +class DatabaseThreadData +{ +public: + + DatabaseThreadData(); + ~DatabaseThreadData(); + + void closeDatabase(); + + QSqlDatabase database; + int valid; + int transactionCount; + QSqlError lastError; +}; + class DIGIKAM_EXPORT DatabaseCoreBackendPrivate : public DatabaseErrorAnswer { public: explicit DatabaseCoreBackendPrivate(DatabaseCoreBackend* const backend); - virtual ~DatabaseCoreBackendPrivate() - { - } + virtual ~DatabaseCoreBackendPrivate(); void init(const QString& connectionName, DatabaseLocking* const locking); - QString connectionName(QThread* const thread); + QString connectionName(); QSqlDatabase databaseForThread(); QSqlError databaseErrorForThread(); void setDatabaseErrorForThread(const QSqlError& lastError); + QSqlDatabase createDatabaseConnection(); void closeDatabaseForThread(); - bool open(QSqlDatabase& db); bool incrementTransactionCount(); bool decrementTransactionCount(); bool isInTransactionInOtherThread() const; @@ -88,14 +102,10 @@ public: public: - // this is always accessed in mutex context, no need for QThreadStorage - QHash<QThread*, QSqlDatabase> threadDatabases; - // this is not only db.isValid(), but also "parameters changed, need to reopen" - QHash<QThread*, int> databasesValid; - // for recursive transactions - QHash<QThread*, int> transactionCount; + QThreadStorage<DatabaseThreadData*> threadDataStorage; - QHash<QThread*, QSqlError> databaseErrors; + // This compares to DatabaseThreadData's valid. If currentValidity is increased and > valid, the db is marked as invalid + int currentValidity; bool isInTransaction; Git commit bab8c3f159930a3b0b31d09b25b9eb42ab1c3c62 by Marcel Wiesweg. Committed on 15/11/2014 at 14:45. Pushed by mwiesweg into branch 'master'. Backport all core db changes regarding thread clean up from main digikam Back and forward port code polish and style changes M +153 -150 libkface/database/databasecorebackend.cpp M +15 -12 libkface/database/databasecorebackend.h M +35 -25 libkface/database/databasecorebackend_p.h M +1 -0 libkface/database/databaseparameters.h http://commits.kde.org/libkface/bab8c3f159930a3b0b31d09b25b9eb42ab1c3c62 diff --git a/libkface/database/databasecorebackend.cpp b/libkface/database/databasecorebackend.cpp index 784e9b7..2193ce2 100644 --- a/libkface/database/databasecorebackend.cpp +++ b/libkface/database/databasecorebackend.cpp @@ -21,12 +21,6 @@ * * ============================================================ */ -/* -#ifndef DATABASCOREBACKEND_DEBUG -#define DATABASCOREBACKEND_DEBUG -#endif -*/ - #include "databasecorebackend.moc" #include "databasecorebackend_p.h" @@ -51,8 +45,8 @@ // Local includes -#include "schemaupdater.h" #include "dbactiontype.h" +#include "schemaupdater.h" namespace KFaceIface { @@ -102,22 +96,65 @@ DatabaseCoreBackendPrivate::ErrorLocker::ErrorLocker(DatabaseCoreBackendPrivate* // ----------------------------------------------------------------------------------------- + +DatabaseThreadData::DatabaseThreadData() + : valid(0), + transactionCount(0) +{ +} + +DatabaseThreadData::~DatabaseThreadData() +{ + if (transactionCount) + { + kDebug() << "WARNING !!! Transaction count is" << transactionCount << "when destroying database!!!"; + } + closeDatabase(); +} + +void DatabaseThreadData::closeDatabase() +{ + QString connectionToRemove; + if (database.isOpen()) + { + connectionToRemove = database.connectionName(); + } + + // Destroy object + database = QSqlDatabase(); + + valid = 0; + transactionCount = 0; + lastError = QSqlError(); + + // Remove connection + if (!connectionToRemove.isNull()) + { + QSqlDatabase::removeDatabase(connectionToRemove); + } +} + DatabaseCoreBackendPrivate::DatabaseCoreBackendPrivate(DatabaseCoreBackend* const backend) - : q(backend) + : currentValidity(0), + isInTransaction(false), + status(DatabaseCoreBackend::Unavailable), + lock(0), + operationStatus(DatabaseCoreBackend::ExecuteNormal), + errorLockOperationStatus(DatabaseCoreBackend::ExecuteNormal), + errorHandler(0), + q(backend) { - status = DatabaseCoreBackend::Unavailable; - isInTransaction = false; - operationStatus = DatabaseCoreBackend::ExecuteNormal; - errorLockOperationStatus = DatabaseCoreBackend::ExecuteNormal; - errorHandler = 0; - lock = 0; } -void DatabaseCoreBackendPrivate::init(const QString& name, DatabaseLocking* const l) +DatabaseCoreBackendPrivate::~DatabaseCoreBackendPrivate() { - QObject::connect(QCoreApplication::instance(), SIGNAL(aboutToQuit()), - q, SLOT(slotMainThreadFinished())); + // Must be shut down from the main thread. + // Clean up the QThreadStorage. It deletes any stored data. + threadDataStorage.setLocalData(0); +} +void DatabaseCoreBackendPrivate::init(const QString& name, DatabaseLocking* const l) +{ backendName = name; lock = l; @@ -133,84 +170,44 @@ void DatabaseCoreBackendPrivate::init(const QString& name, DatabaseLocking* cons // finishing of the thread. QSqlDatabase DatabaseCoreBackendPrivate::databaseForThread() { - QThread* const thread = QThread::currentThread(); - QSqlDatabase db = threadDatabases[thread]; - int isValid = databasesValid[thread]; - - if (!isValid || !db.isOpen()) + DatabaseThreadData* threadData = 0; + if (!threadDataStorage.hasLocalData()) { - // need to open a db for thread - bool success = open(db); - - if (!success) - { - kDebug() << "Error while opening the database. Details: [" << db.lastError() << "]"; - } - - QObject::connect(thread, SIGNAL(finished()), - q, SLOT(slotThreadFinished())); + threadData = new DatabaseThreadData; + threadDataStorage.setLocalData(threadData); } - -#ifdef DATABASCOREBACKEND_DEBUG else { - kDebug() << "Database ["<< connectionName(thread) <<"] already open for thread ["<< thread <<"]."; + threadData = threadDataStorage.localData(); } -#endif - - return db; -} - -void DatabaseCoreBackendPrivate::closeDatabaseForThread() -{ - QThread* const thread = QThread::currentThread(); + // do we need to reopen the database because parameter changed and validity was increased? + if (threadData->valid && threadData->valid < currentValidity) + { + threadData->closeDatabase(); + } - // scope, so that db is destructed when calling removeDatabase + if (!threadData->valid || !threadData->database.isOpen()) { - QSqlDatabase db = threadDatabases[thread]; + threadData->database = createDatabaseConnection(); - if (db.isValid()) + if (threadData->database.open()) { - db.close(); + threadData->valid = currentValidity; + } + else + { + kDebug() << "Error while opening the database. Error was" << threadData->database.lastError(); } } - threadDatabases.remove(thread); - databaseErrors.remove(thread); - databasesValid[thread] = 0; - transactionCount.remove(thread); - QSqlDatabase::removeDatabase(connectionName(thread)); -} - -QSqlError DatabaseCoreBackendPrivate::databaseErrorForThread() -{ - QThread* const thread = QThread::currentThread(); - return databaseErrors[thread]; -} - -void DatabaseCoreBackendPrivate::setDatabaseErrorForThread(const QSqlError& lastError) -{ - QThread* const thread = QThread::currentThread(); - databaseErrors.insert(thread, lastError); -} - -QString DatabaseCoreBackendPrivate::connectionName(QThread* const thread) -{ - return backendName + QString::number((quintptr)thread); + return threadData->database; } -bool DatabaseCoreBackendPrivate::open(QSqlDatabase& db) +QSqlDatabase DatabaseCoreBackendPrivate::createDatabaseConnection() { - if (db.isValid()) - { - db.close(); - } - - QThread* const thread = QThread::currentThread(); - db = QSqlDatabase::addDatabase(parameters.databaseType, connectionName(thread)); - - QString connectOptions;// = parameters.connectOptions; + QSqlDatabase db = QSqlDatabase::addDatabase(parameters.databaseType, connectionName()); + QString connectOptions = parameters.connectOptions; if (parameters.isSQLite()) { @@ -230,55 +227,48 @@ bool DatabaseCoreBackendPrivate::open(QSqlDatabase& db) db.setDatabaseName(parameters.databaseName); db.setConnectOptions(connectOptions); - /*db.setHostName(parameters.hostName); - db.setPort(parameters.port); - db.setUserName(parameters.userName); - db.setPassword(parameters.password);*/ - bool success = db.open(); + return db; +} - if (success) +void DatabaseCoreBackendPrivate::closeDatabaseForThread() +{ + if (threadDataStorage.hasLocalData()) { - db.exec("PRAGMA synchronous=1;"); + threadDataStorage.localData()->closeDatabase(); } - else +} + +QSqlError DatabaseCoreBackendPrivate::databaseErrorForThread() +{ + if (threadDataStorage.hasLocalData()) { - kDebug() << "Error while opening the database. Error was <" << db.lastError() << ">"; + return threadDataStorage.localData()->lastError; } - - threadDatabases[thread] = db; - databasesValid[thread] = 1; - transactionCount[thread] = 0; - - return success; + return QSqlError(); } -bool DatabaseCoreBackendPrivate::incrementTransactionCount() +void DatabaseCoreBackendPrivate::setDatabaseErrorForThread(const QSqlError& lastError) { - QThread* const thread = QThread::currentThread(); - return !transactionCount[thread]++; + if (threadDataStorage.hasLocalData()) + { + threadDataStorage.localData()->lastError = lastError; + } } -bool DatabaseCoreBackendPrivate::decrementTransactionCount() +QString DatabaseCoreBackendPrivate::connectionName() { - QThread* const thread = QThread::currentThread(); - return !--transactionCount[thread]; + return backendName + QString::number((quintptr)QThread::currentThread()); } -bool DatabaseCoreBackendPrivate::isInTransactionInOtherThread() const +bool DatabaseCoreBackendPrivate::incrementTransactionCount() { - QThread* const thread = QThread::currentThread(); - QHash<QThread*, int>::const_iterator it; - - for (it = transactionCount.constBegin(); it != transactionCount.constEnd(); ++it) - { - if (it.key() != thread && it.value()) - { - return true; - } - } + return (!threadDataStorage.localData()->transactionCount++); +} - return false; +bool DatabaseCoreBackendPrivate::decrementTransactionCount() +{ + return (!--threadDataStorage.localData()->transactionCount); } bool DatabaseCoreBackendPrivate::isInMainThread() const @@ -295,7 +285,7 @@ bool DatabaseCoreBackendPrivate::isInUIThread() const return false; } - return QThread::currentThread() == app->thread(); + return (QThread::currentThread() == app->thread()); } bool DatabaseCoreBackendPrivate::reconnectOnError() const @@ -325,7 +315,8 @@ bool DatabaseCoreBackendPrivate::isConnectionError(const SqlQuery& query) const return false; } - return (query.lastError().type() == QSqlError::ConnectionError || query.lastError().number()==2006); + return query.lastError().type() == QSqlError::ConnectionError || + query.lastError().number() == 2006; } bool DatabaseCoreBackendPrivate::needToConsultUserForError(const SqlQuery&) const @@ -336,12 +327,15 @@ bool DatabaseCoreBackendPrivate::needToConsultUserForError(const SqlQuery&) cons bool DatabaseCoreBackendPrivate::needToHandleWithErrorHandler(const SqlQuery& query) const { - return isConnectionError(query) || needToConsultUserForError(query); + return (isConnectionError(query) || needToConsultUserForError(query)); } bool DatabaseCoreBackendPrivate::checkRetrySQLiteLockError(int retries) { - kDebug() << "Database is locked. Waited" << retries*10; + if (!(retries % 25)) + { + kDebug() << "Database is locked. Waited" << retries*10; + } const int uiMaxRetries = 50; const int maxRetries = 1000; @@ -476,7 +470,7 @@ bool DatabaseCoreBackendPrivate::handleWithErrorHandler(const SqlQuery* const qu } else { - //TODO check if it's better to use an own error handler for kio slaves. + // TODO check if it's better to use an own error handler for kio slaves. // But for now, close only the database in the hope, that the next // access will be successful. closeDatabaseForThread(); @@ -755,18 +749,6 @@ void DatabaseCoreBackend::setDatabaseErrorHandler(DatabaseErrorHandler* const ha d->errorHandler = handler; } -void DatabaseCoreBackend::slotThreadFinished() -{ - Q_D(DatabaseCoreBackend); - d->closeDatabaseForThread(); -} - -void DatabaseCoreBackend::slotMainThreadFinished() -{ - Q_D(DatabaseCoreBackend); - d->closeDatabaseForThread(); -} - bool DatabaseCoreBackend::isCompatible(const DatabaseParameters& parameters) { return QSqlDatabase::drivers().contains(parameters.databaseType); @@ -776,10 +758,8 @@ bool DatabaseCoreBackend::open(const DatabaseParameters& parameters) { Q_D(DatabaseCoreBackend); d->parameters = parameters; - - // Force possibly opened thread dbs to re-open with new parameters. - // They are not accessible from this thread! - d->databasesValid.clear(); + // This will make possibly opened thread dbs reload at next access + d->currentValidity++; int retries = 0; @@ -913,7 +893,7 @@ DatabaseCoreBackend::QueryState DatabaseCoreBackend::handleQueryResult(SqlQuery& return DatabaseCoreBackend::NoErrors; } -// ---------------------------------------------------------------------------------------------------- +// ------------------------------------------------------------------------------------- DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(const QString& sql, QList<QVariant>* const values, QVariant* const lastInsertId) { @@ -922,52 +902,54 @@ DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(const QString& sql, } DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(const QString& sql, const QVariant& boundValue1, - QList<QVariant>* const values, QVariant* const lastInsertId) + QList<QVariant>* const values, QVariant* const lastInsertId) { SqlQuery query = execQuery(sql, boundValue1); return handleQueryResult(query, values, lastInsertId); } DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(const QString& sql, - const QVariant& boundValue1, const QVariant& boundValue2, - QList<QVariant>* const values, QVariant* const lastInsertId) + const QVariant& boundValue1, const QVariant& boundValue2, + QList<QVariant>* const values, QVariant* const lastInsertId) { SqlQuery query = execQuery(sql, boundValue1, boundValue2); return handleQueryResult(query, values, lastInsertId); } DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(const QString& sql, - const QVariant& boundValue1, const QVariant& boundValue2, - const QVariant& boundValue3, QList<QVariant>* const values, - QVariant* const lastInsertId) + const QVariant& boundValue1, const QVariant& boundValue2, + const QVariant& boundValue3, QList<QVariant>* const values, + QVariant* const lastInsertId) { SqlQuery query = execQuery(sql, boundValue1, boundValue2, boundValue3); return handleQueryResult(query, values, lastInsertId); } DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(const QString& sql, - const QVariant& boundValue1, const QVariant& boundValue2, - const QVariant& boundValue3, const QVariant& boundValue4, - QList<QVariant>* const values, QVariant* const lastInsertId) + const QVariant& boundValue1, const QVariant& boundValue2, + const QVariant& boundValue3, const QVariant& boundValue4, + QList<QVariant>* const values, QVariant* const lastInsertId) { SqlQuery query = execQuery(sql, boundValue1, boundValue2, boundValue3, boundValue4); return handleQueryResult(query, values, lastInsertId); } DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(const QString& sql, const QList<QVariant>& boundValues, - QList<QVariant>* const values, QVariant* const lastInsertId) + QList<QVariant>* const values, QVariant* const lastInsertId) { SqlQuery query = execQuery(sql, boundValues); return handleQueryResult(query, values, lastInsertId); } DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(const QString& sql, const QMap<QString, QVariant>& bindingMap, - QList<QVariant>* const values, QVariant* const lastInsertId) + QList<QVariant>* const values, QVariant* const lastInsertId) { SqlQuery query = execQuery(sql, bindingMap); return handleQueryResult(query, values, lastInsertId); } +// ------------------------------------------------------------------------------------- + DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(SqlQuery& preparedQuery, QList<QVariant>* const values, QVariant* const lastInsertId) { exec(preparedQuery); @@ -1014,6 +996,8 @@ DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(SqlQuery& preparedQ return handleQueryResult(preparedQuery, values, lastInsertId); } +// ------------------------------------------------------------------------------------- + SqlQuery DatabaseCoreBackend::execQuery(const QString& sql, const QVariant& boundValue1) { SqlQuery query = prepareQuery(sql); @@ -1066,6 +1050,8 @@ SqlQuery DatabaseCoreBackend::execQuery(const QString& sql) return query; } +// ------------------------------------------------------------------------------------- + void DatabaseCoreBackend::execQuery(SqlQuery& query, const QVariant& boundValue1) { query.bindValue(0, boundValue1); @@ -1111,6 +1097,8 @@ void DatabaseCoreBackend::execQuery(SqlQuery& query, const QList<QVariant>& boun exec(query); } +// ------------------------------------------------------------------------------------- + SqlQuery DatabaseCoreBackend::execQuery(const QString& sql, const QMap<QString, QVariant>& bindingMap) { QString preparedString = sql; @@ -1252,7 +1240,7 @@ SqlQuery DatabaseCoreBackend::execQuery(const QString& sql, const QMap<QString, SqlQuery query = prepareQuery(preparedString); - for (int i=0; i<valuesToBind.size(); ++i) + for (int i=0; i < valuesToBind.size(); ++i) { query.bindValue(i, valuesToBind.at(i)); } @@ -1262,12 +1250,12 @@ SqlQuery DatabaseCoreBackend::execQuery(const QString& sql, const QMap<QString, } DatabaseCoreBackend::QueryState DatabaseCoreBackend::execUpsertDBAction(const DatabaseAction& action, const QVariant& id, - const QStringList fieldNames, const QList<QVariant>& values) + const QStringList fieldNames, const QList<QVariant>& values) { QMap<QString, QVariant> parameters; QMap<QString, QVariant> fieldValueMap; - for (int i=0; i<fieldNames.size(); ++i) + for (int i = 0; i < fieldNames.size(); ++i) { fieldValueMap.insert(fieldNames.at(i), values.at(i)); } @@ -1285,7 +1273,7 @@ DatabaseCoreBackend::QueryState DatabaseCoreBackend::execUpsertDBAction(const Da } DatabaseCoreBackend::QueryState DatabaseCoreBackend::execUpsertDBAction(const QString& action, const QVariant& id, - const QStringList fieldNames, const QList<QVariant>& values) + const QStringList fieldNames, const QList<QVariant>& values) { return execUpsertDBAction(getDBAction(action), id, fieldNames, values); } @@ -1516,6 +1504,7 @@ SqlQuery DatabaseCoreBackend::copyQuery(const SqlQuery& old) #endif query.prepare(old.lastQuery()); query.setForwardOnly(old.isForwardOnly()); + // only for positional binding QList<QVariant> boundValues = old.boundValues().values(); @@ -1635,7 +1624,7 @@ DatabaseCoreBackend::QueryState DatabaseCoreBackend::commitTransaction() bool DatabaseCoreBackend::isInTransaction() const { Q_D(const DatabaseCoreBackend); - return d->isInTransactionInOtherThread(); + return d->isInTransaction; } void DatabaseCoreBackend::rollbackTransaction() @@ -1663,4 +1652,18 @@ QString DatabaseCoreBackend::lastError() return d->databaseForThread().lastError().text(); } +int DatabaseCoreBackend::maximumBoundValues() const +{ + Q_D(const DatabaseCoreBackend); + + if (d->parameters.isSQLite()) + { + return 999; // SQLITE_MAX_VARIABLE_NUMBER + } + else + { + return 65535; // MySQL + } +} + } // namespace KFaceIface diff --git a/libkface/database/databasecorebackend.h b/libkface/database/databasecorebackend.h index 9ce859a..a1beac3 100644 --- a/libkface/database/databasecorebackend.h +++ b/libkface/database/databasecorebackend.h @@ -44,6 +44,7 @@ namespace KFaceIface { class DatabaseCoreBackendPrivate; +class DatabaseErrorHandler; class SchemaUpdater; class DatabaseLocking @@ -147,17 +148,18 @@ public: */ void close(); - // ----------------------------------------------------------- +public: class QueryState { public: - QueryState() : value(DatabaseCoreBackend::NoErrors) + QueryState() + : value(DatabaseCoreBackend::NoErrors) { } - QueryState(QueryStateEnum value) + QueryState(const QueryStateEnum value) : value(value) { } @@ -177,6 +179,8 @@ public: QueryStateEnum value; }; +public: + /** * Returns the current status of the database backend */ @@ -275,10 +279,8 @@ public: const QVariant& boundValue1, const QVariant& boundValue2, const QVariant& boundValue3, const QVariant& boundValue4, QList<QVariant>* const values = 0, QVariant* const lastInsertId = 0); - QueryState execSql(const QString& sql, - const QList<QVariant>& boundValues, - QList<QVariant>* const values = 0, - QVariant* const lastInsertId = 0); + QueryState execSql(const QString& sql, const QList<QVariant>& boundValues, + QList<QVariant>* const values = 0, QVariant* const lastInsertId = 0); QueryState execSql(SqlQuery& preparedQuery, QList<QVariant>* const values = 0, QVariant* const lastInsertId = 0); QueryState execSql(SqlQuery& preparedQuery, const QVariant& boundValue1, @@ -438,6 +440,12 @@ public: */ QSqlError lastSQLError(); + /** + * Returns the maximum number of bound parameters allowed per query. + * This value depends on the database engine. + */ + int maximumBoundValues() const; + /* Qt SQL driver supported features SQLITE3: @@ -462,11 +470,6 @@ public: LastInsertId */ -private Q_SLOTS: - - void slotThreadFinished(); - void slotMainThreadFinished(); - protected: DatabaseCoreBackendPrivate* const d_ptr; diff --git a/libkface/database/databasecorebackend_p.h b/libkface/database/databasecorebackend_p.h index c39b33f..27faaa5 100644 --- a/libkface/database/databasecorebackend_p.h +++ b/libkface/database/databasecorebackend_p.h @@ -29,6 +29,7 @@ #include <QHash> #include <QSqlDatabase> #include <QThread> +#include <QThreadStorage> #include <QWaitCondition> // Local includes @@ -38,40 +39,54 @@ namespace KFaceIface { +class DatabaseThreadData +{ +public: + + DatabaseThreadData(); + ~DatabaseThreadData(); + + void closeDatabase(); + + QSqlDatabase database; + int valid; + int transactionCount; + QSqlError lastError; +}; + class DatabaseCoreBackendPrivate : public DatabaseErrorAnswer { public: - DatabaseCoreBackendPrivate(DatabaseCoreBackend* const backend); - virtual ~DatabaseCoreBackendPrivate() {} + explicit DatabaseCoreBackendPrivate(DatabaseCoreBackend* const backend); + virtual ~DatabaseCoreBackendPrivate(); void init(const QString& connectionName, DatabaseLocking* const locking); - QString connectionName(QThread* const thread); + QString connectionName(); QSqlDatabase databaseForThread(); QSqlError databaseErrorForThread(); void setDatabaseErrorForThread(const QSqlError& lastError); + QSqlDatabase createDatabaseConnection(); void closeDatabaseForThread(); - bool open(QSqlDatabase& db); bool incrementTransactionCount(); bool decrementTransactionCount(); - bool isInTransactionInOtherThread() const; bool isInMainThread() const; - bool isInUIThread() const; + bool isInUIThread() const; - bool reconnectOnError() const; - bool isSQLiteLockError(const SqlQuery& query) const; + bool reconnectOnError() const; + bool isSQLiteLockError(const SqlQuery& query) const; bool isSQLiteLockTransactionError(const QSqlError& lastError) const; - bool checkRetrySQLiteLockError(int retries); - bool isConnectionError(const SqlQuery& query) const; - bool needToConsultUserForError(const SqlQuery& query) const; - bool needToHandleWithErrorHandler(const SqlQuery& query) const; - void debugOutputFailedQuery(const QSqlQuery& query) const; - void debugOutputFailedTransaction(const QSqlError& error) const; + bool isConnectionError(const SqlQuery& query) const; + bool needToConsultUserForError(const SqlQuery& query) const; + bool needToHandleWithErrorHandler(const SqlQuery& query) const; + void debugOutputFailedQuery(const QSqlQuery& query) const; + void debugOutputFailedTransaction(const QSqlError& error) const; + bool checkRetrySQLiteLockError(int retries); bool checkOperationStatus(); bool handleWithErrorHandler(const SqlQuery* const query); void setQueryOperationFlag(DatabaseCoreBackend::QueryOperationStatus status); @@ -80,19 +95,14 @@ public: // called by DatabaseErrorHandler, implementing DatabaseErrorAnswer virtual void connectionErrorContinueQueries(); virtual void connectionErrorAbortQueries(); - virtual void transactionFinished(); public: - // this is always accessed in mutex context, no need for QThreadStorage - QHash<QThread*, QSqlDatabase> threadDatabases; - // this is not only db.isValid(), but also "parameters changed, need to reopen" - QHash<QThread*, int> databasesValid; - // for recursive transactions - QHash<QThread*, int> transactionCount; + QThreadStorage<DatabaseThreadData*> threadDataStorage; - QHash<QThread*, QSqlError> databaseErrors; + // This compares to DatabaseThreadData's valid. If currentValidity is increased and > valid, the db is marked as invalid + int currentValidity; bool isInTransaction; @@ -121,7 +131,7 @@ public : { public: - AbstractUnlocker(DatabaseCoreBackendPrivate* const d); + explicit AbstractUnlocker(DatabaseCoreBackendPrivate* const d); ~AbstractUnlocker(); void finishAcquire(); @@ -157,7 +167,7 @@ public : { public: - ErrorLocker(DatabaseCoreBackendPrivate* const d); + explicit ErrorLocker(DatabaseCoreBackendPrivate* const d); void wait(); }; @@ -167,7 +177,7 @@ public : { public: - BusyWaiter(DatabaseCoreBackendPrivate* const d); + explicit BusyWaiter(DatabaseCoreBackendPrivate* const d); }; public : diff --git a/libkface/database/databaseparameters.h b/libkface/database/databaseparameters.h index 655b87a..7cf84ec 100644 --- a/libkface/database/databaseparameters.h +++ b/libkface/database/databaseparameters.h @@ -52,6 +52,7 @@ public: QString databaseType; QString databaseName; + QString connectOptions; bool operator==(const DatabaseParameters& other) const; bool operator!=(const DatabaseParameters& other) const; Git commit 0fb2b35c650b20f7c208bb5b07bd701fd4d09d1f by Marcel Wiesweg. Committed on 15/11/2014 at 14:46. Pushed by mwiesweg into branch 'master'. Back and forward port code polish and style changes from libkface db backend. The files have now only the needed minor differences M +140 -136 libs/database/core/databasecorebackend.cpp M +6 -9 libs/database/core/databasecorebackend.h M +4 -5 libs/database/core/databasecorebackend_p.h http://commits.kde.org/digikam/0fb2b35c650b20f7c208bb5b07bd701fd4d09d1f diff --git a/libs/database/core/databasecorebackend.cpp b/libs/database/core/databasecorebackend.cpp index 76a958a..480f437 100644 --- a/libs/database/core/databasecorebackend.cpp +++ b/libs/database/core/databasecorebackend.cpp @@ -6,7 +6,7 @@ * Date : 2007-04-15 * Description : Abstract database backend * - * Copyright (C) 2007-2010 by Marcel Wiesweg <marcel dot wiesweg at gmx dot de> + * Copyright (C) 2007-2012 by Marcel Wiesweg <marcel dot wiesweg at gmx dot de> * * This program is free software; you can redistribute it * and/or modify it under the terms of the GNU General @@ -42,22 +42,23 @@ // KDE includes #include <kdebug.h> -#include <kglobal.h> // Local includes -#include "thumbnailschemaupdater.h" #include "dbactiontype.h" +#include "thumbnailschemaupdater.h" -//#define DATABASCOREBACKEND_DEBUG 1 namespace Digikam { DatabaseLocking::DatabaseLocking() - : mutex(QMutex::Recursive), lockCount(0) // create a recursive mutex + : mutex(QMutex::Recursive), + lockCount(0) // create a recursive mutex { } +// ----------------------------------------------------------------------------------------- + // For whatever reason, these methods are "static protected" class sotoSleep : public QThread { @@ -79,6 +80,23 @@ public: } }; +// ----------------------------------------------------------------------------------------- + +DatabaseCoreBackendPrivate::BusyWaiter::BusyWaiter(DatabaseCoreBackendPrivate* const d) + : AbstractWaitingUnlocker(d, &d->busyWaitMutex, &d->busyWaitCondVar) +{ +} + +// ----------------------------------------------------------------------------------------- + +DatabaseCoreBackendPrivate::ErrorLocker::ErrorLocker(DatabaseCoreBackendPrivate* const d) + : AbstractWaitingUnlocker(d, &d->errorLockMutex, &d->errorLockCondVar) +{ +} + +// ----------------------------------------------------------------------------------------- + + DatabaseThreadData::DatabaseThreadData() : valid(0), transactionCount(0) @@ -288,7 +306,7 @@ bool DatabaseCoreBackendPrivate::isSQLiteLockError(const SqlQuery& query) const bool DatabaseCoreBackendPrivate::isSQLiteLockTransactionError(const QSqlError& lastError) const { return parameters.isSQLite() && - lastError.type() == QSqlError::TransactionError && + lastError.type() == QSqlError::TransactionError && lastError.databaseText() == QLatin1String("database is locked"); // wouldnt it be great if they gave us the database error number... } @@ -356,74 +374,6 @@ void DatabaseCoreBackendPrivate::debugOutputFailedTransaction(const QSqlError& e << error.number() << error.type(); } - -DatabaseCoreBackendPrivate::AbstractUnlocker::AbstractUnlocker(DatabaseCoreBackendPrivate* const d) - : count(0), d(d) -{ - // Why two mutexes? The main mutex is recursive and won't work with a condvar. - - // acquire lock - d->lock->mutex.lock(); - // store lock count - count = d->lock->lockCount; - // set lock count to 0 - d->lock->lockCount = 0; - - // unlock - for (int i=0; i<count; ++i) - { - d->lock->mutex.unlock(); - } -} - -void DatabaseCoreBackendPrivate::AbstractUnlocker::finishAcquire() -{ - // drop lock acquired in first line. Main mutex is now free. - // We maintain lock order (first main mutex, second error lock mutex) - // but we drop main mutex lock for waiting on the cond var. - d->lock->mutex.unlock(); -} - -DatabaseCoreBackendPrivate::AbstractUnlocker::~AbstractUnlocker() -{ - // lock main mutex as often as it was locked before - for (int i=0; i<count; ++i) - { - d->lock->mutex.lock(); - } - - // update lock count - d->lock->lockCount += count; -} - -DatabaseCoreBackendPrivate::AbstractWaitingUnlocker::AbstractWaitingUnlocker(DatabaseCoreBackendPrivate* const d, - QMutex* const mutex, QWaitCondition* const condVar) - : AbstractUnlocker(d), mutex(mutex), condVar(condVar) -{ - // Why two mutexes? The main mutex is recursive and won't work with a condvar. - // lock condvar mutex (lock only if main mutex is locked) - mutex->lock(); - - finishAcquire(); -} - -DatabaseCoreBackendPrivate::AbstractWaitingUnlocker::~AbstractWaitingUnlocker() -{ - // unlock condvar mutex. Both mutexes are now free. - mutex->unlock(); - // now base class destructor is executed, reallocating main mutex -} - -bool DatabaseCoreBackendPrivate::AbstractWaitingUnlocker::wait(unsigned long time) -{ - return condVar->wait(mutex, time); -} - -DatabaseCoreBackendPrivate::BusyWaiter::BusyWaiter(DatabaseCoreBackendPrivate* const d) - : AbstractWaitingUnlocker(d, &d->busyWaitMutex, &d->busyWaitCondVar) -{ -} - void DatabaseCoreBackendPrivate::transactionFinished() { // wakes up any BusyWaiter waiting on the busyWaitCondVar. @@ -431,39 +381,18 @@ void DatabaseCoreBackendPrivate::transactionFinished() busyWaitCondVar.wakeOne(); } -DatabaseCoreBackendPrivate::ErrorLocker::ErrorLocker(DatabaseCoreBackendPrivate* const d) - : AbstractWaitingUnlocker(d, &d->errorLockMutex, &d->errorLockCondVar) -{ -} - -/** This suspends the current thread if the query status as - * set by setFlag() is Wait and until the thread is woken with wakeAll(). - * The DatabaseAccess mutex will be unlocked while waiting. - */ -void DatabaseCoreBackendPrivate::ErrorLocker::wait() -{ - // we use a copy of the flag under lock of the errorLockMutex to be able to check it here - while (d->errorLockOperationStatus == DatabaseCoreBackend::Wait) - { - wait(); - } -} - -/** Set the wait flag to queryStatus. Typically, call this with Wait. - */ +/** Set the wait flag to queryStatus. Typically, call this with Wait. */ void DatabaseCoreBackendPrivate::setQueryOperationFlag(DatabaseCoreBackend::QueryOperationStatus status) { // Enforce lock order (first main mutex, second error lock mutex) QMutexLocker l(&errorLockMutex); - // this change must be done under errorLockMutex lock errorLockOperationStatus = status; operationStatus = status; } /** Set the wait flag to queryStatus and wake all waiting threads. - * Typically, call wakeAll with status ExecuteNormal or AbortQueries. - */ + * Typically, call wakeAll with status ExecuteNormal or AbortQueries. */ void DatabaseCoreBackendPrivate::queryOperationWakeAll(DatabaseCoreBackend::QueryOperationStatus status) { QMutexLocker l(&errorLockMutex); @@ -530,7 +459,7 @@ bool DatabaseCoreBackendPrivate::handleWithErrorHandler(const SqlQuery* const qu } else { - kError() << "Failed to invoke DatabaseErrorHandler. Aborting all queries."; + kWarning() << "Failed to invoke DatabaseErrorHandler. Aborting all queries."; operationStatus = DatabaseCoreBackend::AbortQueries; } @@ -568,7 +497,88 @@ void DatabaseCoreBackendPrivate::connectionErrorAbortQueries() queryOperationWakeAll(DatabaseCoreBackend::AbortQueries); } -// ----------------------------------------------------------------------------------------------- +// ----------------------------------------------------------------------------------------- + +DatabaseCoreBackendPrivate::AbstractUnlocker::AbstractUnlocker(DatabaseCoreBackendPrivate* const d) + : count(0), d(d) +{ + // Why two mutexes? The main mutex is recursive and won't work with a condvar. + + // acquire lock + d->lock->mutex.lock(); + // store lock count + count = d->lock->lockCount; + // set lock count to 0 + d->lock->lockCount = 0; + + // unlock + for (int i=0; i<count; ++i) + { + d->lock->mutex.unlock(); + } +} + +void DatabaseCoreBackendPrivate::AbstractUnlocker::finishAcquire() +{ + // drop lock acquired in first line. Main mutex is now free. + // We maintain lock order (first main mutex, second error lock mutex) + // but we drop main mutex lock for waiting on the cond var. + d->lock->mutex.unlock(); +} + +DatabaseCoreBackendPrivate::AbstractUnlocker::~AbstractUnlocker() +{ + // lock main mutex as often as it was locked before + for (int i=0; i<count; ++i) + { + d->lock->mutex.lock(); + } + + // update lock count + d->lock->lockCount += count; +} + +// ----------------------------------------------------------------------------------------- + +DatabaseCoreBackendPrivate::AbstractWaitingUnlocker::AbstractWaitingUnlocker(DatabaseCoreBackendPrivate* const d, + QMutex* const mutex, QWaitCondition* const condVar) + : AbstractUnlocker(d), mutex(mutex), condVar(condVar) +{ + // Why two mutexes? The main mutex is recursive and won't work with a condvar. + // lock condvar mutex (lock only if main mutex is locked) + mutex->lock(); + + finishAcquire(); +} + +DatabaseCoreBackendPrivate::AbstractWaitingUnlocker::~AbstractWaitingUnlocker() +{ + // unlock condvar mutex. Both mutexes are now free. + mutex->unlock(); + // now base class destructor is executed, reallocating main mutex +} + +bool DatabaseCoreBackendPrivate::AbstractWaitingUnlocker::wait(unsigned long time) +{ + return condVar->wait(mutex, time); +} + +// ----------------------------------------------------------------------------------------- + +/** This suspends the current thread if the query status as + * set by setFlag() is Wait and until the thread is woken with wakeAll(). + * The DatabaseAccess mutex will be unlocked while waiting. + */ +void DatabaseCoreBackendPrivate::ErrorLocker::wait() +{ + // we use a copy of the flag under lock of the errorLockMutex to be able to check it here + while (d->errorLockOperationStatus == DatabaseCoreBackend::Wait) + { + wait(); + } +} + +// ----------------------------------------------------------------------------------------- DatabaseCoreBackend::DatabaseCoreBackend(const QString& backendName, DatabaseLocking* const locking) : d_ptr(new DatabaseCoreBackendPrivate(this)) @@ -602,33 +612,32 @@ DatabaseAction DatabaseCoreBackend::getDBAction(const QString& actionName) const if (action.name.isNull()) { - kError() << "No DB action defined for" << actionName << "! Implementation missing for this database type."; + kWarning() << "No DB action defined for" << actionName << "! Implementation missing for this database type."; } return action; } DatabaseCoreBackend::QueryState DatabaseCoreBackend::execDBAction(const DatabaseAction& action, QList<QVariant>* const values, - QVariant* const lastInsertId) + QVariant* const lastInsertId) { return execDBAction(action, QMap<QString, QVariant>(), values, lastInsertId); } DatabaseCoreBackend::QueryState DatabaseCoreBackend::execDBAction(const QString& action, QList<QVariant>* const values, - QVariant* const lastInsertId) + QVariant* const lastInsertId) { return execDBAction(getDBAction(action), QMap<QString, QVariant>(), values, lastInsertId); } DatabaseCoreBackend::QueryState DatabaseCoreBackend::execDBAction(const QString& action, const QMap<QString, QVariant>& bindingMap, - QList<QVariant>* const values, QVariant* const lastInsertId) + QList<QVariant>* const values, QVariant* const lastInsertId) { return execDBAction(getDBAction(action), bindingMap, values, lastInsertId); } -DatabaseCoreBackend::QueryState DatabaseCoreBackend::execDBAction(const DatabaseAction& action, const QMap<QString, - QVariant>& bindingMap, QList<QVariant>* const values, - QVariant* const lastInsertId) +DatabaseCoreBackend::QueryState DatabaseCoreBackend::execDBAction(const DatabaseAction& action, const QMap<QString, QVariant>& bindingMap, + QList<QVariant>* const values, QVariant* const lastInsertId) { Q_D(DatabaseCoreBackend); @@ -637,7 +646,7 @@ DatabaseCoreBackend::QueryState DatabaseCoreBackend::execDBAction(const Database if (action.name.isNull()) { - kError() << "Attempt to execute null action"; + kWarning() << "Attempt to execute null action"; return DatabaseCoreBackend::SQLError; } @@ -726,8 +735,8 @@ QSqlQuery DatabaseCoreBackend::execDBActionQuery(const DatabaseAction& action, c if (result.lastError().isValid() && result.lastError().number()) { - kDebug() << "Error while executing DBAction ["<< action.name - <<"] Statement ["<<actionElement.statement<<"] Errornr. [" << result.lastError() << "]"; + kDebug() << "Error while executing DBAction [" << action.name + << "] Statement [" << actionElement.statement << "] Errornr. [" << result.lastError() << "]"; break; } } @@ -822,7 +831,7 @@ DatabaseCoreBackend::Status DatabaseCoreBackend::status() const } /* -bool DatabaseCoreBackend::execSql(const QString& sql, QStringList* values) +bool DatabaseCoreBackend::execSql(const QString& sql, QStringList* const values) { QSqlQuery query = execQuery(sql); @@ -865,8 +874,7 @@ QList<QVariant> DatabaseCoreBackend::readToList(SqlQuery& query) return list; } -DatabaseCoreBackend::QueryState DatabaseCoreBackend::handleQueryResult(SqlQuery& query, QList<QVariant>* const values, - QVariant* const lastInsertId) +DatabaseCoreBackend::QueryState DatabaseCoreBackend::handleQueryResult(SqlQuery& query, QList<QVariant>* const values, QVariant* const lastInsertId) { if (!query.isActive()) { @@ -891,57 +899,54 @@ DatabaseCoreBackend::QueryState DatabaseCoreBackend::handleQueryResult(SqlQuery& // ------------------------------------------------------------------------------------- -DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(const QString& sql, QList<QVariant>* const values, - QVariant* const lastInsertId) +DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(const QString& sql, QList<QVariant>* const values, QVariant* const lastInsertId) { SqlQuery query = execQuery(sql); return handleQueryResult(query, values, lastInsertId); } DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(const QString& sql, const QVariant& boundValue1, - QList<QVariant>* values, QVariant* lastInsertId) + QList<QVariant>* const values, QVariant* const lastInsertId) { SqlQuery query = execQuery(sql, boundValue1); return handleQueryResult(query, values, lastInsertId); } DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(const QString& sql, - const QVariant& boundValue1, const QVariant& boundValue2, - QList<QVariant>* const values, QVariant* const lastInsertId) + const QVariant& boundValue1, const QVariant& boundValue2, + QList<QVariant>* const values, QVariant* const lastInsertId) { SqlQuery query = execQuery(sql, boundValue1, boundValue2); return handleQueryResult(query, values, lastInsertId); } DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(const QString& sql, - const QVariant& boundValue1, const QVariant& boundValue2, - const QVariant& boundValue3, QList<QVariant>* const values, - QVariant* const lastInsertId) + const QVariant& boundValue1, const QVariant& boundValue2, + const QVariant& boundValue3, QList<QVariant>* const values, + QVariant* const lastInsertId) { SqlQuery query = execQuery(sql, boundValue1, boundValue2, boundValue3); return handleQueryResult(query, values, lastInsertId); } DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(const QString& sql, - const QVariant& boundValue1, const QVariant& boundValue2, - const QVariant& boundValue3, const QVariant& boundValue4, - QList<QVariant>* const values, QVariant* const lastInsertId) + const QVariant& boundValue1, const QVariant& boundValue2, + const QVariant& boundValue3, const QVariant& boundValue4, + QList<QVariant>* const values, QVariant* const lastInsertId) { SqlQuery query = execQuery(sql, boundValue1, boundValue2, boundValue3, boundValue4); return handleQueryResult(query, values, lastInsertId); } DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(const QString& sql, const QList<QVariant>& boundValues, - QList<QVariant>* const values, QVariant* const lastInsertId) + QList<QVariant>* const values, QVariant* const lastInsertId) { SqlQuery query = execQuery(sql, boundValues); return handleQueryResult(query, values, lastInsertId); } -// ------------------------------------------------------------------------------------- - DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(const QString& sql, const QMap<QString, QVariant>& bindingMap, - QList<QVariant>* const values, QVariant* const lastInsertId) + QList<QVariant>* const values, QVariant* const lastInsertId) { SqlQuery query = execQuery(sql, bindingMap); return handleQueryResult(query, values, lastInsertId); @@ -949,8 +954,7 @@ DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(const QString& sql, // ------------------------------------------------------------------------------------- -DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(SqlQuery& preparedQuery, - QList<QVariant>* const values, QVariant* const lastInsertId) +DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(SqlQuery& preparedQuery, QList<QVariant>* const values, QVariant* const lastInsertId) { exec(preparedQuery); return handleQueryResult(preparedQuery, values, lastInsertId); @@ -996,7 +1000,7 @@ DatabaseCoreBackend::QueryState DatabaseCoreBackend::execSql(SqlQuery& preparedQ return handleQueryResult(preparedQuery, values, lastInsertId); } -// ---------------------------------------------------------------------------------------- +// ------------------------------------------------------------------------------------- SqlQuery DatabaseCoreBackend::execQuery(const QString& sql, const QVariant& boundValue1) { @@ -1119,9 +1123,9 @@ SqlQuery DatabaseCoreBackend::execQuery(const QString& sql, const QMap<QString, if (!bindingMap.contains(namedPlaceholder)) { - kError() << "Missing place holder" << namedPlaceholder - << "in binding map. The following values are defined for this action:" - << bindingMap.keys() <<". This is a setup error!"; + kWarning() << "Missing place holder" << namedPlaceholder + << "in binding map. The following values are defined for this action:" + << bindingMap.keys() <<". This is a setup error!"; //TODO What should we do here? How can we cancel that action? } @@ -1240,7 +1244,7 @@ SqlQuery DatabaseCoreBackend::execQuery(const QString& sql, const QMap<QString, SqlQuery query = prepareQuery(preparedString); - for (int i=0; i<valuesToBind.size(); ++i) + for (int i=0; i < valuesToBind.size(); ++i) { query.bindValue(i, valuesToBind.at(i)); } @@ -1250,7 +1254,7 @@ SqlQuery DatabaseCoreBackend::execQuery(const QString& sql, const QMap<QString, } DatabaseCoreBackend::QueryState DatabaseCoreBackend::execUpsertDBAction(const DatabaseAction& action, const QVariant& id, - const QStringList fieldNames, const QList<QVariant>& values) + const QStringList fieldNames, const QList<QVariant>& values) { QMap<QString, QVariant> parameters; QMap<QString, QVariant> fieldValueMap; @@ -1266,7 +1270,7 @@ DatabaseCoreBackend::QueryState DatabaseCoreBackend::execUpsertDBAction(const Da parameters.insert(":id", id); parameters.insert(":fieldValueList", qVariantFromValue(fieldValueList)); - parameters.insert(":fieldList", qVariantFromValue (fieldList)); + parameters.insert(":fieldList", qVariantFromValue(fieldList)); parameters.insert(":valueList", qVariantFromValue(valueList)); return execDBAction(action, parameters); diff --git a/libs/database/core/databasecorebackend.h b/libs/database/core/databasecorebackend.h index fa4dba4..ea8d558 100644 --- a/libs/database/core/databasecorebackend.h +++ b/libs/database/core/databasecorebackend.h @@ -42,15 +42,12 @@ namespace Digikam { -class ThumbnailSchemaUpdater; -class DatabaseErrorHandler; class DatabaseCoreBackendPrivate; - -// ------------------------------------------------------------------------------------------------------------ +class DatabaseErrorHandler; +class ThumbnailSchemaUpdater; class DIGIKAM_EXPORT DatabaseLocking { - public: DatabaseLocking(); @@ -61,7 +58,7 @@ public: int lockCount; }; -// ------------------------------------------------------------------------------------------------------------ +// ----------------------------------------------------------------- class DIGIKAM_EXPORT DatabaseCoreBackend : public QObject { @@ -202,7 +199,7 @@ public: * Add a DatabaseErrorHandler. This object must be created in the main thread. * If a database error occurs, this object can handle problem solving and user interaction. */ - void setDatabaseErrorHandler(DatabaseErrorHandler* handler); + void setDatabaseErrorHandler(DatabaseErrorHandler* const handler); /** * Return config read from XML, @@ -265,10 +262,10 @@ public: * Executes the SQL statement, and write the returned data into the values list. * If you are not interested in the returned data, set values to 0. * Methods are provided for up to four bound values (positional binding), or for a list of bound values. - * If you want the last inserted id (and your query is suitable), sett lastInsertId to the address of a QVariant. + * If you want the last inserted id (and your query is suitable), set lastInsertId to the address of a QVariant. * Additionally, methods are provided for prepared statements. */ - QueryState execSql(const QString& sql, QList<QVariant>* values = 0, QVariant* const lastInsertId = 0); + QueryState execSql(const QString& sql, QList<QVariant>* const values = 0, QVariant* const lastInsertId = 0); QueryState execSql(const QString& sql, const QVariant& boundValue1, QList<QVariant>* const values = 0, QVariant* const lastInsertId = 0); QueryState execSql(const QString& sql, diff --git a/libs/database/core/databasecorebackend_p.h b/libs/database/core/databasecorebackend_p.h index ff3a3fa..1f60dd7 100644 --- a/libs/database/core/databasecorebackend_p.h +++ b/libs/database/core/databasecorebackend_p.h @@ -75,7 +75,6 @@ public: void closeDatabaseForThread(); bool incrementTransactionCount(); bool decrementTransactionCount(); - bool isInTransactionInOtherThread() const; bool isInMainThread() const; bool isInUIThread() const; @@ -145,10 +144,10 @@ public : DatabaseCoreBackendPrivate* const d; }; - // ---------------------------------------------------------------------- - friend class AbstractUnlocker; + // ------------------------------------------------------------------ + class AbstractWaitingUnlocker : public AbstractUnlocker { public: @@ -164,7 +163,7 @@ public : QWaitCondition* const condVar; }; - // ---------------------------------------------------------------------- + // ------------------------------------------------------------------ class ErrorLocker : public AbstractWaitingUnlocker { @@ -174,7 +173,7 @@ public : void wait(); }; - // ---------------------------------------------------------------------- + // ------------------------------------------------------------------ class BusyWaiter : public AbstractWaitingUnlocker { Next digiKam release 4.6.0 will include several important commits from Marcel to try to fix this memory leak with SQlite database Gilles Caulier Hello guys, Very nice libre software. Great job. Every time I play face tagging, my digiKam 4.8 release became extremely unresponsive and slow. What about the work in progress of this bug? Jean-Martial I think bug is fixed with current implementation from git/master (next 4.9.0). See bug #338176 for details. I close this file now. Don't hesitate to re-open if necessary... Gilles Caulier Just upgraded to DK4.9.0 on Mint and the problem is still there. I select People > Unknown and select a name for one of the unknowns and when I select a name the memory usage starts to climb. Closing DigiKam does not clear the problem as a DigiKam process is still running and must be killed to stop the memory runaway. Have you updated libkface? Maik When I looked I had libkface2 version 1.0~digikam4.4.0-trusty~ppa2 libkface3 version 1.0~digikam4.9.0-trusty~ppa1 installed. I uninstalled libkface2 and rebooted but the problem is still the same. Is there anything else I can do? Not sure that you have last libkface version installed. At least, 3.5.0 ID must be visible in list given on Help/Components Info dialog. Gilles Caulier LibKface 3.5.0 is in the list. The full list is - digiKam version 4.9.0 CPU cores: 8 Demosaic GPL2 pack support: Unknown Demosaic GPL3 pack support: Unknown Exiv2 can write to Jp2: Yes Exiv2 can write to Jpeg: Yes Exiv2 can write to Pgf: Yes Exiv2 can write to Png: Yes Exiv2 can write to Tiff: Yes Exiv2 supports XMP metadata: Yes LibCImg: 130 LibEigen: 3.2.0 LibExiv2: 0.24 LibJPEG: 80 LibJasper: 1.900.1 LibKDE: 4.13.3 LibKExiv2: 2.4.0 LibKGeoMap: 3.1.0 LibKdcraw: 2.4.2 LibLCMS: 2050 LibLensFun: 0.2.8-0 LibLqr support: yes LibPGF: 6.12.24 LibPNG: 1.2.50 LibQt: 4.8.6 LibRaw: 0.15.4 LibTIFF: LIBTIFF, Version 4.0.3 Copyright (c) 1988-1996 Sam Leffler Copyright (c) 1991-1996 Silicon Graphics, Inc. Marble Widget: 0.18.3 (stable release) Parallelised demosaicing: Unknown RawSpeed codec support: Unknown Baloo support: no Database backend: QSQLITE Kdepimlibs support: Yes Kipi-Plugins: 4.9.0 LibGphoto2: 2.5.3.1 LibKface: 3.5.0 LibKipi: 2.2.0 LibOpenCV: 2.4.9 Sqlite2 support: Yes It's also possible that your face database file is corrupted due to previous dysfunctions. The idea is to rename file as .old and to re-try to scan face with a fresh DB file. file is located in your home dir at ~/.kde4/share/apps/libkface/database/recognition.db Gilles Caulier Just renamed the database, tried it again and it works!!!!! Thank you so much for your help and for fixing this problem. DK is brilliant and this was the only thing I have found which did not work. Thank you Mick |