Bug 326525

Summary: Severe memory leak during maintainance scan for rebuild thumbnails
Product: [Applications] digikam Reporter: Jim Shipman <JimShip>
Component: Maintenance-ThumbsAssignee: Digikam Developers <digikam-bugs-null>
Status: RESOLVED FIXED    
Severity: critical CC: anantapalani, caulier.gilles, dario.vischi, e.longuemare, marcel.wiesweg, matteo.fortini, nucleo, rdieter, regi.hops, rschweizer, vamp898
Priority: NOR    
Version: 3.5.0   
Target Milestone: ---   
Platform: Fedora RPMs   
OS: Linux   
Latest Commit: Version Fixed In: 4.0.0
Sentry Crash Report:
Attachments: digikam_thumbnail_rebuild.log
New crash information added by DrKonqi
Build traces for digikam with external libpgf
Build traces for digikam with inrenal libpgf
Patch out OpenMP #pragma s

Description Jim Shipman 2013-10-23 19:38:37 UTC 
Tools->Maintainance->Rebuild Thumbnails (scan changed) generates large memory leak.
Fedora 19 Gome, X86_64 w/ 12GB Memory, AMD quad core, MySQL database on local machine.
Yum installed from Fedora 19 repository - digikam 3.5.0
Memory goes from a few GB linearly up to 11GB during the scan for changed/new thumbnails.

Reproducible: Always

Steps to Reproduce:
1.Select Maintainance->Rebuild thumbnails->scan for new
2. Watch the memory usage climb until you run out.
3.
Actual Results:  
Memory climbs linearly up to max available on the machine (12 GB)

Expected Results:  
Memory should be stable

I can't build thumbnails until this is fixed.
Comment 1 caulier.gilles 2013-10-24 05:30:56 UTC 
Not reproducible here. 16 Gb ram with more than 300 Gb of picture. I rebuild all thumbnails without any problem.

Sound like a dysfunction introduced by packaging rules. But which one ?

I suspect that digiKam have been compiled with an external libpgf shared lib. PGF is image format used to stored thumb in DB. Mine use internal implementation of PGF included in digiKam core.

To confirm, you need to run digiKam into valgrind and post here the memory leak trace reported by this tool. Follow instruction here :

http://www.digikam.org/contrib

Note : you need to have digiKam debug symbols package installed.

It will help too to copy and paste the liste of external components used by digiKam. Go to help/components info dialog for details.

Gilles Caulier
Comment 2 Jim Shipman 2013-10-24 18:24:09 UTC 
Output of components on my system.  Getting debug files now.  Will report more later.  Jim

digiKam version 3.5.0
Exiv2 can write to Jp2: Yes
Exiv2 can write to Jpeg: Yes
Exiv2 can write to Pgf: Yes
Exiv2 can write to Png: Yes
Exiv2 can write to Tiff: Yes
Exiv2 supports XMP metadata: Yes
LibCImg: 130
LibEigen: 3.1.4
LibExiv2: 0.23
LibJPEG: 62
LibJasper: 1.900.1
LibKDE: 4.11.2
LibKExiv2: 2.3.1
LibKGeoMap: 2.0.0
LibKdcraw: 2.3.1
LibLCMS: 2050
LibLensFun: 0.2.7-0
LibPGF: 6.12.24 - external shared library
LibPNG: 1.5.13
LibQt: 4.8.5
LibRaw: 0.15.3
LibTIFF: LIBTIFF, Version 4.0.3 Copyright (c) 1988-1996 Sam Leffler Copyright (c) 1991-1996 Silicon Graphics, Inc.
Marble Widget: 0.16.1 (stable version)
Parallelized PGF codec: No
Parallelized demosaicing: No
RawSpeed codec support: No
Database backend: QMYSQL
Database internal server: No
Kipi-Plugins: 3.5.0
LibGphoto2: 2.5.2
LibKface: 3.0.0
LibKipi: 2.1.0
LibOpenCV: 2.4.6.1
Comment 3 caulier.gilles 2013-10-24 18:25:19 UTC 
>LibPGF: 6.12.24 - external shared library

==> This is what i suspected...

Gilles Caulier
Comment 4 Rex Dieter 2013-10-25 01:49:27 UTC 
Interestingly, that's the same version as the bundled/embedded one.
Comment 5 caulier.gilles 2013-10-25 05:52:02 UTC 
Rex,

If i remember, there is some little difference in PGF embedded and PGF bundled.

I remember some Coverity entries in PGF code about memory leak that i reported to PGF team without any feedback...

PGF become a problem if code is no more maintained.  ì think that an alternative must be found as WEBp which provide a fast wavelets lossless compression as PGF :

http://en.wikipedia.org/wiki/WebP

It's maintained by Google.

Gilles Caulier
Comment 6 nucleo 2013-10-25 11:43:26 UTC 
Last released libpgf 6.12.24.
Trying 6.12.27 from svn.
Comment 7 Rex Dieter 2013-10-25 11:47:26 UTC 
Gilles, was your report to libpgf documented anywhere?  a bug or mailing list?
Comment 8 caulier.gilles 2013-10-25 12:33:24 UTC 
Hi Nueclo,

6 or 9 month ago, i work on PGF project to port library to Cmake and to improve binary compatibility. I suppose that svn trunk (6.12.27) must include my changes at least. I'm not sure because i don't check PGF code and i don't receive feedback from PGF team since a long time...

But what's i sure, it's 2 or 3 memory leak entries detected in PGF codec by Coverity tool.

Rex, do you have an account as KDE developer to Coverity ?

https://scan.coverity.com/

If yes, i can add you to digiKam project, you can take a look of details...

Another question is about future WebP QImageIO plugin for konqueror ? I remember to have see a thread about in reviewboard this summer. I don't follow anymore this entry since my serious car crash in september... If a WebP QImage plugin exist (readwrite  plus data stream), it's possible to switch thumb storage from PGF to WebP easily...

Gilles Caulier
Comment 9 caulier.gilles 2013-10-25 12:37:10 UTC 
Rex,

I remember to have contacted PGF maintainer directly by private mail. I pointed entries in Coverity about memory leak. No more. Problem : i cannot retrieve this email from Gmail archive (i a little bit old, at least 9 month, when i start to use Coverity to scan digiKam code)

Gilles
Comment 10 e.longuemare 2013-10-25 14:02:06 UTC 
Hello,

I have made a few test in rebuilding thumbnails with digikam 4.0.0 beta1 :  digikam crash here too.

Please view log file : digikam_thumbnail_rebuild.log


Components :
digiKam version 4.0.0-beta1
Codec PGF parallélisé: Non
Composant graphique Marble: 0.16.2 (stable version)
Demosaic GPL2 pack support: Non
Demosaic GPL3 pack support: Non
Dématriçage parallélisé: Non
Exiv2 peut écrire dans un fichier JP2: Oui
Exiv2 peut écrire dans un fichier JPEG: Oui
Exiv2 peut écrire dans un fichier PGF: Oui
Exiv2 peut écrire dans un fichier PNG: Oui
Exiv2 peut écrire dans un fichier TIFF: Oui
Exiv2 prend en charge les métadonnées XMP: Oui
LibCImg: 130
LibEigen: 3.0.5
LibExiv2: 0.22
LibJPEG: 80
LibJasper: 1.900.1
LibKDE: 4.11.2
LibKExiv2: 2.3.1
LibKGeoMap: 2.0.0
LibKdcraw: 2.4.0
LibLCMS: 2040
LibLensFun: 0.2.7-0
LibPGF: 6.12.24 - bibliothèque partagée externe
LibPNG: 1.2.46
LibQt: 4.8.2
LibRaw: 0.15.4
LibTIFF: LIBTIFF, Version 3.9.5 Copyright (c) 1988-1996 Sam Leffler Copyright (c) 1991-1996 Silicon Graphics, Inc.
Prise en charge du codec RawSpeed: Non
LibGphoto2: 2.4.14
LibKface: 3.0.0
LibKipi: 2.1.0
LibOpenCV: 2.4.5
Modules externes KIPI: 3.5.0
Moteur de base de données: QMYSQL
Serveur interne de base de données: Non

Date de compilation : Oct 24 2013 (cible : debugfull)
Rév. : c7dcf5224f6eb9c681707e6e0ca7578d40804152
Comment 11 e.longuemare 2013-10-25 14:03:33 UTC 
Created attachment 83110 [details]
digikam_thumbnail_rebuild.log
Comment 12 e.longuemare 2013-10-25 14:55:01 UTC 
Created attachment 83113 [details]
New crash information added by DrKonqi

digikam (4.0.0-beta1) on KDE Platform 4.11.2 using Qt 4.8.2

- What I was doing when the application crashed:

1 - go to maintenance tool
2 - launch thumbnails rebuild with analyse check

console log :

digikam(15929)/digikam (core) Digikam::MaintenanceMngr::setSettings:
wholeAlbums         : true
wholeTags           : true
Albums              : 387
Tags                : 495
useMutiCoreCPU      : false
newItems            : false
thumbnails          : true
scanThumbs          : true
fingerPrints        : false
scanFingerPrints    : false
duplicates          : false
similarity          : 90
faceManagement      : false
faceScannedHandling : 0
qualitySort         : false
quality             :
EnableSorter      : false
DetectBlur        : true
DetectNoise       : true
DetectCompression : true
LowQRejected      : true
MediumQPending    : true
HighQAccepted     : true
Speed             : 1
Rejected Threshold        : 10
Pending Threshold             : 40
Accepted Threshold             : 60
Blur Weight             : 100
Noise Weight             : 100
Compression Weight             : 100

qualityScanMode     : 0
metadataSync        : false
syncDirection       : 0

digikam(15929)/digikam (core) Digikam::MaintenanceMngr::stage1: stage1
digikam(15929)/digikam (core) Digikam::MaintenanceMngr::stage2: stage2
digikam(15929)/KDCRAW KDcrawIface::RActionThreadBase::RActionThreadBase: Starting Main Thread
digikam(15929)/KDCRAW KDcrawIface::RActionThreadBase::run: In action thread Run
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadStarted: RWeaverObserver: thread  1  started
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0019.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0020.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0021.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0022.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0023.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0024.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0025.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0026.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0027.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0028.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0029.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0030.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0031.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0032.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0033.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0034.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0035.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0036.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0037.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0038.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0039.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0040.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0041.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0042.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0043.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0044.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0045.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0046.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0047.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 1/0048.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 4/0000.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 4/0001.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 4/0002.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 4/0003.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 4/0004.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 4/0005.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 4/0006.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 4/0007.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 4/0008.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 4/0009.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 4/0010.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 4/0011.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 4/0012.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 4/0013.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 4/0014.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0000.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0001.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0002.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0003.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0004.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0005.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0006.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0007.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0008.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0009.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0010.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0011.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0012.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0013.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0014.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0015.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0016.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0017.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0018.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0019.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0020.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0021.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0022.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0023.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0024.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0025.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0026.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0027.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0028.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0030.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0031.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0032.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0033.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0034.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0035.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0036.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0037.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0038.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0039.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0040.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0041.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0042.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0043.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0044.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0045.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0046.jpg"  : JPEG file identified
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0047.jpg"  : JPEG file identified
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0048.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0049.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0050.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0051.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0052.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0053.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0054.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0055.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0056.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0057.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0058.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0059.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0060.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0061.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0062.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0063.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0064.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0065.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0066.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003 - 5/0067.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-2/0000.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-2/0001.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-2/0002_000.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-2/0003_001.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-2/0004_002.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-2/0005_003.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-2/0006_004.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-2/0007_000.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-2/0008_001.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-2/0009_002.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-2/0010_003.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-2/0011_004.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-2/0012_000.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-2/0013_001.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-2/0014_002.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-2/0015_003.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-2/0016_004.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0000_000.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0001_001.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0002_002.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0003_003.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0004_004.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0005.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0006.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0007.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0008.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0009.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0010.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0011.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0012.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0013.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0014.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0015.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0016.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0017.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0018.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
digikam(15929)/digikam (core) Digikam::DImg::load: "/media/DiskphotosTests/Saint Gilles 2003/Saint Gilles 2003-3/0019.jpg"  : JPEG file identified
digikam(15929)/KDCRAW KDcrawIface::RWeaverObserver::slotThreadBusy: RWeaverObserver: thread  1  busy
Qt Concurrent has caught an exception thrown from a worker thread.
This is not supported, exceptions thrown in worker threads must be
caught before control returns to Qt Concurrent.
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
KCrash: Application 'digikam' crashing...
KCrash: Attempting to start /usr/lib/kde4/libexec/drkonqi from kdeinit
QSocketNotifier: Invalid socket 13 and type 'Read', disabling...
QSocketNotifier: Invalid socket 16 and type 'Read', disabling...
QSocketNotifier: Invalid socket 22 and type 'Read', disabling...
QSocketNotifier: Invalid socket 66 and type 'Read', disabling...
QSocketNotifier: Invalid socket 30 and type 'Read', disabling...
digikam: Fatal IO error 9 (Mauvais descripteur de fichier) on X server :0.0.
QSocketNotifier: Invalid socket 17 and type 'Read', disabling...

-- Backtrace (Reduced):
#8  0x02edc1df in __GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#9  0x02edf825 in __GI_abort () at abort.c:91
[...]
#14 0x01eba5ee in QThreadPoolThread::run (this=0x9a870f0) at concurrent/qthreadpool.cpp:114
#15 0x01ec7eb0 in QThreadPrivate::start (arg=0x9a870f0) at thread/qthread_unix.cpp:307
#16 0x007d4d4c in start_thread (arg=0xa06d1b40) at pthread_create.c:308
Comment 13 caulier.gilles 2013-10-25 16:04:20 UTC 
Instead to use an external version of libPGF, can you force digiKam to recompile internal version of libpgf, where i DISABLED parallellization (using OpenMP) which is completely bugous.

If you look into digiKam component list, you will see PGF parallelization disabled, but i'm not sure if this state is right here using an external version of PGF...

Gilles Caulier
Comment 14 caulier.gilles 2013-10-25 16:13:52 UTC 
Rex,

Look this entry about WebP support in KDELibs :

https://git.reviewboard.kde.org/r/106300/

Sound like code have been submited. right ?

Gilles
Comment 15 caulier.gilles 2013-10-25 16:35:22 UTC 
Rex,

Look cppcheck reports into PGF code 

http://build.kde.org/view/extragear/job/digikam_master/800/cppcheckResult/

Just search "pgf" strings in this page.

Gilles
Comment 16 caulier.gilles 2013-10-25 16:54:24 UTC 
Entries list from Coverity Scan about libpgf :

981290	Uninitialized scalar field	Medium	Dismissed	1	02/13/13	Unassigned	Intentional	Unspecified	Ignore	Other	Uninitialized members	CDecoder::CMacroBlock::CMacroBlock(CDecoder*)	/mnt/devel/GIT/3.x/core/libs/3rdparty/libpgf/Decoder.h

981172	Out-of-bounds write	High	Dismissed	1	02/13/13	Unassigned	Intentional	Unspecified	Ignore	Other	Memory - corruptions	CEncoder::WriteValue(CSubband *, int)	/mnt/devel/GIT/3.x/core/libs/3rdparty/libpgf/Encoder.cpp

981291	Uninitialized pointer field	Medium	Dismissed	1	02/13/13	Unassigned	Intentional	Unspecified	Ignore	Other	Uninitialized members	CEncoder::CEncoder(CPGFStream *, PGFPreHeader, PGFHeader, const PGFPostHeader &, unsigned long long &, bool)	/mnt/devel/GIT/3.x/core/libs/3rdparty/libpgf/Encoder.cpp

981292	Uninitialized scalar field	Medium	Dismissed	1	02/13/13	Unassigned	Intentional	Unspecified	Ignore	Other	Uninitialized members	CEncoder::CMacroBlock::CMacroBlock(CEncoder*)	/mnt/devel/GIT/3.x/core/libs/3rdparty/libpgf/Encoder.h

981188	Resource leak	High	Dismissed	9	02/13/13	Unassigned	Intentional	Unspecified	Ignore	Other	Resource leaks	CPGFImage::GetBitmap(int, unsigned char *, unsigned char, int *, bool (*)(double, bool, void *), void *) const	/mnt/devel/GIT/3.x/core/libs/3rdparty/libpgf/PGFimage.cpp

981293	Uninitialized scalar field	Medium	Dismissed	1	02/13/13	Unassigned	Intentional	Unspecified	Ignore	Other	Uninitialized members	CPGFImage::CPGFImage()	/mnt/devel/GIT/3.x/core/libs/3rdparty/libpgf/PGFimage.cpp

981497	Read from pointer after free	High	Dismissed	1	02/13/13	Unassigned	False Positive	Unspecified	Undecided	Other	Memory - illegal accesses	CPGFImage::Write(int, bool (*)(double, bool, void *), void *)	/mnt/devel/GIT/3.x/core/libs/3rdparty/libpgf/PGFimage.cpp

981294	Uninitialized scalar field	Medium	Dismissed	1	02/13/13	Unassigned	Intentional	Unspecified	Ignore	Other	Uninitialized members	CSubband::CSubband()	/mnt/devel/GIT/3.x/core/libs/3rdparty/libpgf/Subband.cpp

Gilles
Comment 17 Jim Shipman 2013-10-25 17:20:49 UTC 
I saw the comments about parallelization and noted that I had checked the "work on all processor cores" checkbox in the tools->maintainance dialog.  I unchecked that and ran the thumbnailer again.  This time it ran to completion and only consumed 1.3GB of memory (that was not released when digikam finished), but at least it ran all the way through my  86000 pictures.
Jim Shipman
Comment 18 caulier.gilles 2013-10-25 17:39:28 UTC 
No. parallelization of libPGF is to decode/encode image through OpenMP.

In Maintenance tool, parallelization is performed to process more than one items at the same time. It use ThreadWaver API from KDELibs. It's completly different...

In all case PGF must run in on thread. parallelization do not work and crash digiKam. I'm sure i tested it in-deep, without to found where is the problem to PGF code. This is why libPGF is in digiKam with this option forced to disabled.

But, go back to this problem. Here digiKam do not crash imediatly, until memory is full due to memory leak. I'm sure that problem detected in libpgf by Coverity Scan tool is the source.

As i'm not PGF algorithm developer, i cannot fix it easily. This must be done by PGF team.
If PGF team do not respond, i need to found an alternative, as WebP codec for ex. Look like PGF project sound like in stand-by since a long moment. In opensource, it's really problematic...

Gilles Caulier
Comment 19 caulier.gilles 2013-10-25 17:41:29 UTC 
Important : I suspect that a lots of bug-reports are relevant of PGF codec bugs. For ex in face management, we have serious and stranges crashes when face thumbs data are set/get to database.

Gilles Caulier
Comment 20 Rex Dieter 2013-10-25 17:46:57 UTC 
Re: comment #14

I can confirm https://git.reviewboard.kde.org/r/110649/ landed in kde-runtime 4.11 branch
Comment 21 caulier.gilles 2013-10-25 17:51:22 UTC 
Thanks Rex,

Sadly, under my Linux, i run KDE 4.10.

But under my MacBook, it's last KDE 4.11. So i can add try to make a WebP wrapper for thumb database, using KImageIO plugin.

Gilles
Comment 22 caulier.gilles 2013-10-25 17:52:52 UTC 
Note : another advantage to use WebP with digiKam thumbs DB is to drop definitively LibPGF code (yes another one!), and to remove another external dependency...

Gilles Caulier
Comment 23 caulier.gilles 2013-10-26 05:09:14 UTC 
There is a simple test to confirm that memory leak is relevant of PGF codec implementation.

In digiKam thumbnails database interface, we have code to change image compression format to store in DB. Currently PGF is hardcoded in this line :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/threadimageio/thumbnailcreator.cpp#L687

Just change :

dbInfo.type             = DatabaseThumbnail::PGF;

by :

dbInfo.type             = DatabaseThumbnail::JPEG;

... and recompile and install digiKam. Re-run maintenance tool and look if memory leak still here. 

Note, with this change, JPEG will be used instead PGF to store thumbnails in DB.

Gilles Caulier
Comment 24 caulier.gilles 2013-10-26 08:13:23 UTC 
Marcel,

Running digiKam into valgrind give me these backtraces :

There is a lots of memory allocation into ThumbnailImageCatcher. This is not a memory leak. When digiKam is stopped, all memory is free properly. Look below :

==7668== 7,128,144 bytes in 15,102 blocks are possibly lost in loss record 25,758 of 25,761
==7668==    at 0x4C285F1: operator new(unsigned long) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==7668==    by 0x80B40EE: QList<Digikam::ThumbnailImageCatcher::Private::CatcherResult>::node_construct(QList<Digikam::ThumbnailImageCatcher::Private::CatcherResult>::Node*, Digikam::ThumbnailImageCatcher::Private::CatcherResult const&) (qlist.h:372)
==7668==    by 0x80B2D2D: QList<Digikam::ThumbnailImageCatcher::Private::CatcherResult>::append(Digikam::ThumbnailImageCatcher::Private::CatcherResult const&) (qlist.h:512)
==7668==    by 0x80B197E: QList<Digikam::ThumbnailImageCatcher::Private::CatcherResult>::operator<<(Digikam::ThumbnailImageCatcher::Private::CatcherResult const&) (qlist.h:334)
==7668==    by 0x80AFC76: Digikam::ThumbnailImageCatcher::slotThumbnailLoaded(Digikam::LoadingDescription const&, QImage const&) (thumbnailloadthread.cpp:1142)
==7668==    by 0x80ABB70: Digikam::ThumbnailImageCatcher::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) (thumbnailloadthread.moc:172)
==7668==    by 0xF9AE8CE: QMetaObject::activate(QObject*, QMetaObject const*, int, void**) (in /usr/lib64/libQtCore.so.4.8.5)
==7668==    by 0x808FCEA: Digikam::LoadSaveThread::signalThumbnailLoaded(Digikam::LoadingDescription const&, QImage const&) (loadsavethread.moc:174)
==7668==    by 0x8090538: Digikam::LoadSaveThread::thumbnailLoaded(Digikam::LoadingDescription const&, QImage const&) (loadsavethread.cpp:204)
==7668==    by 0x80ADFEC: Digikam::ThumbnailLoadThread::thumbnailLoaded(Digikam::LoadingDescription const&, QImage const&) (thumbnailloadthread.cpp:684)
==7668==    by 0x80B5565: Digikam::ThumbnailLoadingTask::execute() (thumbnailtask.cpp:229)
==7668==    by 0x809029C: Digikam::LoadSaveThread::run() (loadsavethread.cpp:136)
==7668== 
==7668== 335,044,008 bytes in 709,839 blocks are possibly lost in loss record 25,761 of 25,761
==7668==    at 0x4C285F1: operator new(unsigned long) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==7668==    by 0x80B40EE: QList<Digikam::ThumbnailImageCatcher::Private::CatcherResult>::node_construct(QList<Digikam::ThumbnailImageCatcher::Private::CatcherResult>::Node*, Digikam::ThumbnailImageCatcher::Private::CatcherResult const&) (qlist.h:372)
==7668==    by 0x80B2D59: QList<Digikam::ThumbnailImageCatcher::Private::CatcherResult>::append(Digikam::ThumbnailImageCatcher::Private::CatcherResult const&) (qlist.h:521)
==7668==    by 0x80B197E: QList<Digikam::ThumbnailImageCatcher::Private::CatcherResult>::operator<<(Digikam::ThumbnailImageCatcher::Private::CatcherResult const&) (qlist.h:334)
==7668==    by 0x80AFC76: Digikam::ThumbnailImageCatcher::slotThumbnailLoaded(Digikam::LoadingDescription const&, QImage const&) (thumbnailloadthread.cpp:1142)
==7668==    by 0x80ABB70: Digikam::ThumbnailImageCatcher::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) (thumbnailloadthread.moc:172)
==7668==    by 0xF9AE8CE: QMetaObject::activate(QObject*, QMetaObject const*, int, void**) (in /usr/lib64/libQtCore.so.4.8.5)
==7668==    by 0x808FCEA: Digikam::LoadSaveThread::signalThumbnailLoaded(Digikam::LoadingDescription const&, QImage const&) (loadsavethread.moc:174)
==7668==    by 0x8090538: Digikam::LoadSaveThread::thumbnailLoaded(Digikam::LoadingDescription const&, QImage const&) (loadsavethread.cpp:204)
==7668==    by 0x80ADFEC: Digikam::ThumbnailLoadThread::thumbnailLoaded(Digikam::LoadingDescription const&, QImage const&) (thumbnailloadthread.cpp:684)
==7668==    by 0x80B5565: Digikam::ThumbnailLoadingTask::execute() (thumbnailtask.cpp:229)
==7668==    by 0x809029C: Digikam::LoadSaveThread::run() (loadsavethread.cpp:136)

Typically, ThumbnailImageCatcher is called in ThumbsTask from maintenance tool :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/utilities/maintenance/thumbstask.cpp#L76

and ThumbnailImageCatcher store thumbs in a QList cache here, without limit :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/threadimageio/thumbnailloadthread.cpp#L1142

... until memory is full...

Question : there is something missing in ThumbsTask::run() to prevent this problem ?

Gilles
Comment 25 caulier.gilles 2013-10-26 08:22:19 UTC 
Raphael,

In same way that thumb cache memory allocation problem from digiKam core, we have also a memory problems with PGF codec :

memory leak :

==7668== 2,208 bytes in 6 blocks are possibly lost in loss record 25,044 of 25,761
==7668==    at 0x4C26DFF: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==7668==    by 0x4011008: _dl_allocate_tls (in /usr/lib64/ld-2.17.so)
==7668==    by 0xFD008B8: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.17.so)
==7668==    by 0x10AAACBD: ??? (in /usr/lib64/libgomp.so.1.0.0)
==7668==    by 0x80BC413: CPGFImage::WriteHeader(CPGFStream*) (PGFimage.cpp:924)
==7668==    by 0x80BCE86: CPGFImage::Write(CPGFStream*, unsigned int*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1145)
==7668==    by 0x80C876E: Digikam::PGFUtils::writePGFImageDataToStream(QImage const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:280)
==7668==    by 0x80C7BF0: Digikam::PGFUtils::writePGFImageData(QImage const&, QByteArray&, int, bool) (pgfutils.cpp:178)
==7668==    by 0x80A8E99: Digikam::ThumbnailCreator::storeInDatabase(Digikam::ThumbnailInfo const&, Digikam::ThumbnailImage const&) const (thumbnailcreator.cpp:695)
==7668==    by 0x80A6818: Digikam::ThumbnailCreator::load(QString const&, QRect const&, bool) const (thumbnailcreator.cpp:267)
==7668==    by 0x80A63C5: Digikam::ThumbnailCreator::load(QString const&) const (thumbnailcreator.cpp:199)
==7668==    by 0x80B51F9: Digikam::ThumbnailLoadingTask::execute() (thumbnailtask.cpp:172)

memory corruption : 

==7668== Thread 16:
==7668== Conditional jump or move depends on uninitialised value(s)
==7668==    at 0x80C47CF: CSubband::Quantize(int) (Subband.cpp:132)
==7668==    by 0x80C5772: CWaveletTransform::ForwardTransform(int, int) (WaveletTransform.cpp:164)
==7668==    by 0x80C3023: CPGFImage::WriteHeader(CPGFStream*) [clone ._omp_fn.0] (PGFimage.cpp:949)
==7668==    by 0x10AAA809: ??? (in /usr/lib64/libgomp.so.1.0.0)
==7668==    by 0xFCFFD17: start_thread (in /usr/lib64/libpthread-2.17.so)
==7668==    by 0x10FB67CC: clone (in /usr/lib64/libc-2.17.so)
==7668== 
==7668== Conditional jump or move depends on uninitialised value(s)
==7668==    at 0x80C4828: CSubband::Quantize(int) (Subband.cpp:134)
==7668==    by 0x80C5772: CWaveletTransform::ForwardTransform(int, int) (WaveletTransform.cpp:164)
==7668==    by 0x80C3023: CPGFImage::WriteHeader(CPGFStream*) [clone ._omp_fn.0] (PGFimage.cpp:949)
==7668==    by 0x10AAA809: ??? (in /usr/lib64/libgomp.so.1.0.0)
==7668==    by 0xFCFFD17: start_thread (in /usr/lib64/libpthread-2.17.so)
==7668==    by 0x10FB67CC: clone (in /usr/lib64/libc-2.17.so)
==7668== 
==7668== Thread 14:
==7668== Conditional jump or move depends on uninitialised value(s)
==7668==    at 0x80B8979: CEncoder::WriteValue(CSubband*, int) (Encoder.cpp:327)
==7668==    by 0x80B8640: CEncoder::Partition(CSubband*, int, int, int, int) (Encoder.cpp:258)
==7668==    by 0x80C49FB: CSubband::ExtractTile(CEncoder&, bool, unsigned int, unsigned int) (Subband.cpp:185)
==7668==    by 0x80BC9C4: CPGFImage::WriteLevel() (PGFimage.cpp:1029)
==7668==    by 0x80BCD55: CPGFImage::WriteImage(CPGFStream*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1103)
==7668==    by 0x80BCEA1: CPGFImage::Write(CPGFStream*, unsigned int*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1148)
==7668==    by 0x80C876E: Digikam::PGFUtils::writePGFImageDataToStream(QImage const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:280)
==7668==    by 0x80C7BF0: Digikam::PGFUtils::writePGFImageData(QImage const&, QByteArray&, int, bool) (pgfutils.cpp:178)
==7668==    by 0x80A8E99: Digikam::ThumbnailCreator::storeInDatabase(Digikam::ThumbnailInfo const&, Digikam::ThumbnailImage const&) const (thumbnailcreator.cpp:695)
==7668==    by 0x80A6818: Digikam::ThumbnailCreator::load(QString const&, QRect const&, bool) const (thumbnailcreator.cpp:267)
==7668==    by 0x80A63C5: Digikam::ThumbnailCreator::load(QString const&) const (thumbnailcreator.cpp:199)
==7668==    by 0x80B51F9: Digikam::ThumbnailLoadingTask::execute() (thumbnailtask.cpp:172)
==7668== 
==7668== Conditional jump or move depends on uninitialised value(s)
==7668==    at 0x80B8979: CEncoder::WriteValue(CSubband*, int) (Encoder.cpp:327)
==7668==    by 0x80B872D: CEncoder::Partition(CSubband*, int, int, int, int) (Encoder.cpp:283)
==7668==    by 0x80C49FB: CSubband::ExtractTile(CEncoder&, bool, unsigned int, unsigned int) (Subband.cpp:185)
==7668==    by 0x80BC9C4: CPGFImage::WriteLevel() (PGFimage.cpp:1029)
==7668==    by 0x80BCD55: CPGFImage::WriteImage(CPGFStream*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1103)
==7668==    by 0x80BCEA1: CPGFImage::Write(CPGFStream*, unsigned int*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1148)
==7668==    by 0x80C876E: Digikam::PGFUtils::writePGFImageDataToStream(QImage const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:280)
==7668==    by 0x80C7BF0: Digikam::PGFUtils::writePGFImageData(QImage const&, QByteArray&, int, bool) (pgfutils.cpp:178)
==7668==    by 0x80A8E99: Digikam::ThumbnailCreator::storeInDatabase(Digikam::ThumbnailInfo const&, Digikam::ThumbnailImage const&) const (thumbnailcreator.cpp:695)
==7668==    by 0x80A6818: Digikam::ThumbnailCreator::load(QString const&, QRect const&, bool) const (thumbnailcreator.cpp:267)
==7668==    by 0x80A63C5: Digikam::ThumbnailCreator::load(QString const&) const (thumbnailcreator.cpp:199)
==7668==    by 0x80B51F9: Digikam::ThumbnailLoadingTask::execute() (thumbnailtask.cpp:172)

Sure, memory leak with PGF codec is lesser than thumb cache from digiKam core, but it exists...

See my comments #16 about Coverity Scan source code reports (https://scan.coverity.com/) which are fully relevant of this memory problems...

Gilles Caulier
Comment 26 caulier.gilles 2013-10-26 08:23:17 UTC 
Rex,

libpgf maintainer is now in CC. Wait and see...

Gilles
Comment 27 e.longuemare 2013-10-26 08:57:58 UTC 
Thank you Gilles,

I will try this and the compilation digikam with internal PGF.

Give you the result today or monday.

Greatings,

Eric

Le samedi 26 octobre 2013 à 05:09 +0000, Gilles Caulier a écrit :
> https://bugs.kde.org/show_bug.cgi?id=326525
> 
> --- Comment #23 from Gilles Caulier <caulier.gilles@gmail.com> ---
> There is a simple test to confirm that memory leak is relevant of PGF codec
> implementation.
> 
> In digiKam thumbnails database interface, we have code to change image
> compression format to store in DB. Currently PGF is hardcoded in this line :
> 
> https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/threadimageio/thumbnailcreator.cpp#L687
> 
> Just change :
> 
> dbInfo.type             = DatabaseThumbnail::PGF;
> 
> by :
> 
> dbInfo.type             = DatabaseThumbnail::JPEG;
> 
> ... and recompile and install digiKam. Re-run maintenance tool and look if
> memory leak still here. 
> 
> Note, with this change, JPEG will be used instead PGF to store thumbnails in
> DB.
> 
> Gilles Caulier
>
Comment 28 Marcel Wiesweg 2013-10-26 18:37:20 UTC 
Re Memory leak:
Please use a dedicated thumbnail load thread (*) for the catcher. The logic depends on a enqueue being called after immediately after requesting a thumbnail, it breaks if the thread receives requests from many places concurrently so that the "last request" is undefined when calling enqueue. 

Re PGF:
Even if we switch thumbnail creation to another algorithm, we
a) need to continue providing PGF read support for those gigabytes of thumbnails on our users machines
b) we break backwards compatibility to use a new thumbnail db with an older version, so it should be done only with a major release

(*)
    ThumbnailLoadThread* thumbnailLoadThread = new ThumbnailLoadThread;
    thumbnailLoadThread->setPixmapRequested(false);
    thumbnailLoadThread->setThumbnailSize(ThumbnailLoadThread::maximumThumbnailSize());

Note that switching off pixmap generation for this thread also avoids flooding the cache for the main UI.
Comment 29 caulier.gilles 2013-10-26 21:20:56 UTC 
Git commit 21eb0ed5e895742228d789077d6590af9ff7ab36 by Gilles Caulier.
Committed on 26/10/2013 at 21:18.
Pushed by cgilles into branch 'master'.

Fix memory leak with ThumbnailLoadThread instance passed to ThumbnailImageCatcher.
Pass a dedicated instace of thread to cacher, following Marcel tips.
FIXED-IN: 4.0.0

M  +6    -4    utilities/maintenance/thumbstask.cpp

http://commits.kde.org/digikam/21eb0ed5e895742228d789077d6590af9ff7ab36
Comment 30 caulier.gilles 2013-10-26 21:28:27 UTC 
Marcel,

Thanks for your tips. Huge memory leak due to ThumbnailImageThread instance is now fixed.

It still PGF codec memory leak. I will wait feedback of Raphael Schweizer from LibPGF team before to take a decision about to continue to use or not PGF as Thumb DB image codec.

I know that a change like this require a backward compatibility until few release... If PGF implementation can be fixed,, it will be great for the future. I hope that PGF team is not dead... 

Gilles
Comment 31 Raphael Schweizer 2013-10-28 09:33:32 UTC 
At last, news from the libPGF team: we will immediately look into this memory leak problem.

@Gilles: please apologize for the long silence. Unlike the coverity reports your contributions towards CMake and better binary compatibility are not lost nor forgotten. We started renewed efforts to migrate to CMake last week and will (hopefully soon) have good news. I'll also check the indicated coverity reports.

Thanks, Raphael
Comment 32 caulier.gilles 2013-10-28 10:06:17 UTC 
Hi Raphael,

Great to hear you in this room...

I don't know if you have a Coverity Scan account to be able to check reports. I send you an invitation using your email, like this, you will be able to review all information from Coverity web interface.

Gilles Caulier
Comment 33 caulier.gilles 2013-10-28 10:11:44 UTC 
Raphael,

Coverity said "The following email addresses are invalid: rschweizer@schweizer-informatik.ch"

Probably it don't like minus in your email domain name...

Please use this page :

https://scan.coverity.com/projects

Search "digiKam" entry and use "Add me to project" link...

Gilles Caulier
Comment 34 caulier.gilles 2013-10-28 10:15:19 UTC 
Raphael,

About my previous libpgf contributions not yet release, i recommend to use Cmake port in priority. Forget BC branch for the moment. I will replay my fixes in this branch later, including these future memory fixes. Also i want to pass more test about BC code from libpgf.

Cmake branch do not touch in-deep libpgf implementation. Most of code are cmake scripts. It safe to release. BC branch can come later...

Gilles Caulier
Comment 35 caulier.gilles 2013-10-28 10:21:56 UTC 
Raphael,

As you can see in comment #13, OpenMp support from libpgf do not work  in digiKam.

PGF codec in digiKam will run into a separated thread (QThread class : http://qt-project.org/doc/qt-4.8/qthread.html) and when libpgf paralellize decoding to get PGF thumbnail from database, it crash violently.

I written simple a code outside wholde digiKam implementation to reproduce the problem :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/tests/qtpgftest.cpp

When libpgf do not use OpenMp, all work fine.

I can give you GDB backtrace after that you fix memory corruption/leak previously listed here. Perhaps openmp has rise up some side-effects.

Gilles Caulier
Comment 36 Raphael Schweizer 2013-10-28 17:02:34 UTC 
Gilles,

I synced our internal repository with SourceForce, now containing at least a bunch of ROI related out of bounds errors less. This will, however, not fix the issue reported here. I'll keep you posted on our progress.

Raphael Schweizer
Comment 37 Ananta Palani 2013-10-29 08:31:17 UTC 
(In reply to comment #28)
> Re PGF:
> Even if we switch thumbnail creation to another algorithm, we
> a) need to continue providing PGF read support for those gigabytes of
> thumbnails on our users machines
> b) we break backwards compatibility to use a new thumbnail db with an older
> version, so it should be done only with a major release

I agree with Marcel. Also, the WebP is inferior in some ways to PGF (vastly reduced color depth support), but superior in others (animation, color management). Might be nice to use WebP in certain cases, like to create animated thumbnails for animated gifs or videos, but I suppose most users won't want their thumbnail list to be animated. Perhaps it could be an option if the user would like it.
Comment 38 caulier.gilles 2013-10-29 09:18:14 UTC 
Hi Ananta,

I agree with you that PGF is superior than WebP. 

But the topic here is not to use WebP to store photo, but only thumbnails in database, compressed with Wavelets algorithm. In this case WebP can be suitable. We don't need 16 bits color depth.

As you can see, i recieve PGF maintainer feedback and it's good for the future. If PGF memory corruption/leak problems are fixed there is no reason to switch to WebP.

There is also this problem with OpenMP support in PGF which don't work. I just re-tested here and crash still here. I wait to see if PGF memory fixes improve OPenMP support before look in-deep in this problem.

Gilles
Comment 39 caulier.gilles 2013-10-29 09:27:31 UTC 
Ananta,

Just to talk more about wavelets image format, there is another challenger named JPEG-XR done by ... Microsoft, and valided as a standard bu JPEG group to include it in digital camera to replace usual JPEG !

This format is now available to developers though an "open-source like" library named "jxlib"

https://jxrlib.codeplex.com/

Gilles Caulier
Comment 40 caulier.gilles 2013-10-29 09:38:36 UTC 
Raphael,

Do you see my comment #33 ? 

I can add you to digiKam Coverity Scan project. Like this you can analyse in-deep all Coverity entries about libPGF. Reports are really detailled and instructive...

Gilles Caulier
Comment 41 Raphael Schweizer 2013-10-29 09:43:11 UTC 
(In reply to comment #40)

Yes, I applied for access using a different mail address (minus' not allowed at coverity).

Raphael
Comment 42 caulier.gilles 2013-10-29 09:58:25 UTC 
Raphael,

I send you invitation with your new email...

As i list in comment #16, Coverity Scan entries ID relevant to libPGF are :

- 981290
- 981172
- 981291
- 981292
- 981188
- 981293
- 981497
- 981294

Gilles Caulier
Comment 43 caulier.gilles 2013-11-02 13:15:11 UTC 
*** Bug 327033 has been marked as a duplicate of this bug. ***
Comment 44 caulier.gilles 2013-11-06 16:20:34 UTC 
Raphael,

Any progress about libpgf memory leak fixes ?

Gilles Caulier
Comment 45 Raphael Schweizer 2013-11-07 08:44:46 UTC 
Gilles,

at last:
- regarding memory leak @CPGFImage::WriteHeader(CPGFStream*) (PGFimage.cpp:924): we believe we only leaked memory if we eventually ran out of it (fixed in latest sourceforfe repo version) (though I'm not sure how OMP is related here)
- uninitialized values @CEncoder::WriteValue(CSubband*, int) (Encoder.cpp:327) are false positives
- uninitialized values @CSubband::Quantize(int) (Subband.cpp:134) are also believed to be false positives

We've just set up a new linux box for more testing (including your test from comment #35)

- Raphael
Comment 46 e.longuemare 2013-11-07 09:55:48 UTC 
Hello,

I know the question is not for me.

But what I can say is that use of "internal libpgf" reduce most of the
memory leak. Before the use of memory was growing fast to 1 gb and go
away to crash. Now, for scanning the all collection (same condition and
same pictures) it's almost use 400 mo for digikam to the end of
thumbnail scan.

Is libpgf has something to do with face scan and/or recognition :

- thumbnails for faces storage in digikam thumb db?

- and other stuff in recognition,db ?

Greatings,

Eric
Comment 47 caulier.gilles 2013-11-07 10:21:19 UTC 
Raphael,

I will backport last libpgf code into digiKam core and look valgrind report to see if any improvements have been done about memory leak.

Gilles Caulier
Comment 48 caulier.gilles 2013-11-07 10:24:14 UTC 
Eric,

libpgf has nothing releated to face recognition memory leak. All problem with face management are already reported in another report and problem is relevant of OpenCV bugs.

https://bugs.kde.org/show_bug.cgi?id=323888

Gilles Caulier
Comment 49 Raphael Schweizer 2013-11-07 10:29:29 UTC 
Gilles,

Does digiKam "internal libpgf" have other improvements (comment #5) besides disabling OpenMP regarding better "memory performance"? I'd then patch our sources before you do the work of backporting.
Also, I'd like to fix some "uninitialized values in constructor" issued reported by coverity (though I don't think they actually have any effect).

- Raphael
Comment 50 caulier.gilles 2013-11-07 10:36:07 UTC 
Raphael,

No, as i remember, i don't patch internal libpgf code. I just only include a cmake switch to enable/disable OpenMP support. That all...

Gilles Caulier
Comment 51 e.longuemare 2013-11-07 11:07:59 UTC 
Created attachment 83399 [details]
Build traces for digikam with external libpgf
Comment 52 e.longuemare 2013-11-07 11:10:17 UTC 
Created attachment 83400 [details]
Build traces for digikam with inrenal libpgf
Comment 53 caulier.gilles 2013-11-07 14:03:15 UTC 
Git commit 8c41f33ff8489e863130003a6f2bfea1bb6393e5 by Gilles Caulier.
Committed on 07/11/2013 at 14:02.
Pushed by cgilles into branch 'master'.

update internal libpgf source code to last revision #r121 from SF repository

M  +1    -1    libs/3rdparty/libpgf/Decoder.cpp
M  +1    -1    libs/3rdparty/libpgf/Decoder.h
M  +1    -1    libs/3rdparty/libpgf/Encoder.h
M  +26   -11   libs/3rdparty/libpgf/PGFimage.cpp
M  +4    -4    libs/3rdparty/libpgf/PGFplatform.h
M  +4    -4    libs/3rdparty/libpgf/PGFstream.h
M  +4    -4    libs/3rdparty/libpgf/PGFtypes.h
M  +2    -2    libs/3rdparty/libpgf/Subband.cpp
M  +56   -5    libs/3rdparty/libpgf/WaveletTransform.cpp

http://commits.kde.org/digikam/8c41f33ff8489e863130003a6f2bfea1bb6393e5
Comment 54 caulier.gilles 2013-11-07 14:29:28 UTC 
Raphael,

As you can see in my last comment, i commited last code from libpgf repository and recompiled digiKam. Running it through valgrind report again some issue about memory problems. Below, it's not a memory leak : 

=2438== Thread 21:
==2438== Conditional jump or move depends on uninitialised value(s)
==2438==    at 0x80BAA10: CEncoder::CMacroBlock::NumberOfBitplanes() (Encoder.cpp:740)
==2438==    by 0x80BA233: CEncoder::CMacroBlock::BitplaneEncode() (Encoder.cpp:495)
==2438==    by 0x80B9E7A: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:347)
==2438==    by 0x80B9D2B: CEncoder::Flush() (Encoder.cpp:313)
==2438==    by 0x80BE2CD: CPGFImage::WriteImage(CPGFStream*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1128)
==2438==    by 0x80BE381: CPGFImage::Write(CPGFStream*, unsigned int*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1163)
==2438==    by 0x80CA106: Digikam::PGFUtils::writePGFImageDataToStream(QImage const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:288)
==2438==    by 0x80C9562: Digikam::PGFUtils::writePGFImageData(QImage const&, QByteArray&, int, bool) (pgfutils.cpp:184)
==2438==    by 0x80AA279: Digikam::ThumbnailCreator::storeInDatabase(Digikam::ThumbnailInfo const&, Digikam::ThumbnailImage const&) const (thumbnailcreator.cpp:695)
==2438==    by 0x80A7BF8: Digikam::ThumbnailCreator::load(QString const&, QRect const&, bool) const (thumbnailcreator.cpp:267)
==2438==    by 0x80A77A5: Digikam::ThumbnailCreator::load(QString const&) const (thumbnailcreator.cpp:199)
==2438==    by 0x80B6685: Digikam::ThumbnailLoadingTask::execute() (thumbnailtask.cpp:172)
==2438== 
==2438== Conditional jump or move depends on uninitialised value(s)
==2438==    at 0x80BAA41: CEncoder::CMacroBlock::NumberOfBitplanes() (Encoder.cpp:741)
==2438==    by 0x80BA233: CEncoder::CMacroBlock::BitplaneEncode() (Encoder.cpp:495)
==2438==    by 0x80B9E7A: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:347)
==2438==    by 0x80B9D2B: CEncoder::Flush() (Encoder.cpp:313)
==2438==    by 0x80BE2CD: CPGFImage::WriteImage(CPGFStream*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1128)
==2438==    by 0x80BE381: CPGFImage::Write(CPGFStream*, unsigned int*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1163)
==2438==    by 0x80CA106: Digikam::PGFUtils::writePGFImageDataToStream(QImage const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:288)
==2438==    by 0x80C9562: Digikam::PGFUtils::writePGFImageData(QImage const&, QByteArray&, int, bool) (pgfutils.cpp:184)
==2438==    by 0x80AA279: Digikam::ThumbnailCreator::storeInDatabase(Digikam::ThumbnailInfo const&, Digikam::ThumbnailImage const&) const (thumbnailcreator.cpp:695)
==2438==    by 0x80A7BF8: Digikam::ThumbnailCreator::load(QString const&, QRect const&, bool) const (thumbnailcreator.cpp:267)
==2438==    by 0x80A77A5: Digikam::ThumbnailCreator::load(QString const&) const (thumbnailcreator.cpp:199)
==2438==    by 0x80B6685: Digikam::ThumbnailLoadingTask::execute() (thumbnailtask.cpp:172)
==2438== 

At end of application, valgrind report memory leak. It's not too huge, because i stopped thumbnail maintenance task 30 secondes after to start (1% completed)...

==2438== 9,200 bytes in 25 blocks are possibly lost in loss record 25,377 of 25,727
==2438==    at 0x4C26DFF: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==2438==    by 0x4011008: _dl_allocate_tls (in /usr/lib64/ld-2.17.so)
==2438==    by 0xFD028B8: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.17.so)
==2438==    by 0x10AACCBD: ??? (in /usr/lib64/libgomp.so.1.0.0)
==2438==    by 0x80BD89F: CPGFImage::WriteHeader(CPGFStream*) (PGFimage.cpp:924)
==2438==    by 0x80BE366: CPGFImage::Write(CPGFStream*, unsigned int*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1160)
==2438==    by 0x80CA106: Digikam::PGFUtils::writePGFImageDataToStream(QImage const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:288)
==2438==    by 0x80C9562: Digikam::PGFUtils::writePGFImageData(QImage const&, QByteArray&, int, bool) (pgfutils.cpp:184)
==2438==    by 0x80AA279: Digikam::ThumbnailCreator::storeInDatabase(Digikam::ThumbnailInfo const&, Digikam::ThumbnailImage const&) const (thumbnailcreator.cpp:695)
==2438==    by 0x80A7BF8: Digikam::ThumbnailCreator::load(QString const&, QRect const&, bool) const (thumbnailcreator.cpp:267)
==2438==    by 0x80A77A5: Digikam::ThumbnailCreator::load(QString const&) const (thumbnailcreator.cpp:199)
==2438==    by 0x80B6685: Digikam::ThumbnailLoadingTask::execute() (thumbnailtask.cpp:172)
 
==2438== 2,208 bytes in 6 blocks are possibly lost in loss record 24,976 of 25,727
==2438==    at 0x4C26DFF: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==2438==    by 0x4011008: _dl_allocate_tls (in /usr/lib64/ld-2.17.so)
==2438==    by 0xFD028B8: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.17.so)
==2438==    by 0x10AACCBD: ??? (in /usr/lib64/libgomp.so.1.0.0)
==2438==    by 0x80BC372: CPGFImage::Read(int, bool (*)(double, bool, void*), void*) (PGFimage.cpp:428)
==2438==    by 0x80C8C51: Digikam::PGFUtils::readPGFImageData(QByteArray const&, QImage&, bool) (pgfutils.cpp:93)
==2438==    by 0x80AAE7E: Digikam::ThumbnailCreator::loadFromDatabase(Digikam::ThumbnailInfo const&) const (thumbnailcreator.cpp:886)
==2438==    by 0x80A7ACC: Digikam::ThumbnailCreator::load(QString const&, QRect const&, bool) const (thumbnailcreator.cpp:248)
==2438==    by 0x80A77A5: Digikam::ThumbnailCreator::load(QString const&) const (thumbnailcreator.cpp:199)
==2438==    by 0x80B6685: Digikam::ThumbnailLoadingTask::execute() (thumbnailtask.cpp:172)
==2438==    by 0x8091550: Digikam::LoadSaveThread::run() (loadsavethread.cpp:136)
==2438==    by 0x80CF451: Digikam::DynamicThread::DynamicThreadPriv::run() (dynamicthread.cpp:186)

Gilles Caulier
Comment 55 caulier.gilles 2013-11-07 14:31:57 UTC 
Raphael,

My last comment is about digiKam compiled with internal libpgf with OpenMP support disabled.

Gilles Caulier
Comment 56 Raphael Schweizer 2013-11-07 15:08:46 UTC 
Created attachment 83403 [details]
Patch out OpenMP #pragma s

Gilles,

first two must be false positives. The value in question (m_maxAbsValue) is initialized in a method (CMacroBlock::Init) called from the CMacroBlock constructor.

Memory leaks are reported on lines "#pragma omp parallel for default(shared)". Seemingly OpenMP leaks some memory here, even though being disabled. Searching the internet leads to other people having similar problems. As I'm not an OpenMP guru my best guess is to patch these lines out for the time being. I'll see for someone more into OpenMP to investigate this issue.

- Raphael
Comment 57 Raphael Schweizer 2013-11-07 15:32:08 UTC 
Gilles,

Do you compile using -fopenmp and disable OpenMP just for libPGF (how, which flag?) or are you using gcc default (AFAIK no OpenMP)? I'm puzzled by the fact that libPGF is calling into libgomp. Is there anything we can do to better support disabling OpenMP?

- Raphael

(In reply to comment #55)
> Raphael,
> 
> My last comment is about digiKam compiled with internal libpgf with OpenMP
> support disabled.
> 
> Gilles Caulier
Comment 58 caulier.gilles 2013-11-07 16:06:38 UTC 
Raphael,

At cmake configuration time, this flag is turned off (default) :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/CMakeLists.txt#L17

Which is managed here to set compilation flags :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/CMakeLists.txt#L272

In libpgf code, this flag is used here :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/3rdparty/libpgf/PGFplatform.h#L608

I don't know why libpgf call libgomp here...

Gilles
Comment 59 Raphael Schweizer 2013-11-07 17:32:14 UTC 
Gilles,

nostra culpa, I'll upload a fixed version tonight.

- Raphael
Comment 60 Raphael Schweizer 2013-11-07 20:58:24 UTC 
Gilles,

r122 @sourceforge ready with guarded OpenMP #pragma s and some improvements regarding the coverity issues.

- Raphael
Comment 61 caulier.gilles 2013-11-08 07:07:59 UTC 
Git commit b494c2a021e3fd85eeb1a5eba5a9e449e4dbe114 by Gilles Caulier.
Committed on 08/11/2013 at 07:07.
Pushed by cgilles into branch 'master'.

update internal libpgf source code to last revision #r121 from SF repository

M  +2    -1    libs/3rdparty/libpgf/Decoder.cpp
M  +4    -0    libs/3rdparty/libpgf/Decoder.h
M  +8    -1    libs/3rdparty/libpgf/Encoder.cpp
M  +5    -1    libs/3rdparty/libpgf/Encoder.h
M  +8    -1    libs/3rdparty/libpgf/PGFimage.cpp
M  +16   -16   libs/3rdparty/libpgf/PGFplatform.h
M  +2    -2    libs/3rdparty/libpgf/PGFtypes.h
M  +6    -1    libs/3rdparty/libpgf/Subband.cpp

http://commits.kde.org/digikam/b494c2a021e3fd85eeb1a5eba5a9e449e4dbe114
Comment 62 caulier.gilles 2013-11-08 07:09:25 UTC 
Sorry, commit from comment #61 is libpgf revision #122 from SF repository (not #121)

Gilles Caulier
Comment 63 caulier.gilles 2013-11-08 07:54:32 UTC 
Raphael,

All memory leak from libpgf are disappear with revision #122, but multiple conditional jump or move depends on uninitialized values (more than revision #121). Look below. This can be false warnings, but this must be checked :

==8229== Thread 17:
==8229== Conditional jump or move depends on uninitialised value(s)
==8229==    at 0x80C5B55: CSubband::Quantize(int) (Subband.cpp:137)
==8229==    by 0x80C6B0E: CWaveletTransform::ForwardTransform(int, int) (WaveletTransform.cpp:164)
==8229==    by 0x80BDBA7: CPGFImage::WriteHeader(CPGFStream*) (PGFimage.cpp:961)
==8229==    by 0x80BE720: CPGFImage::Write(CPGFStream*, unsigned int*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1167)
==8229==    by 0x80C9F2E: Digikam::PGFUtils::writePGFImageDataToStream(QImage const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:288)
==8229==    by 0x80C938A: Digikam::PGFUtils::writePGFImageData(QImage const&, QByteArray&, int, bool) (pgfutils.cpp:184)
==8229==    by 0x80AA259: Digikam::ThumbnailCreator::storeInDatabase(Digikam::ThumbnailInfo const&, Digikam::ThumbnailImage const&) const (thumbnailcreator.cpp:695)
==8229==    by 0x80A7BD8: Digikam::ThumbnailCreator::load(QString const&, QRect const&, bool) const (thumbnailcreator.cpp:267)
==8229==    by 0x80A7785: Digikam::ThumbnailCreator::load(QString const&) const (thumbnailcreator.cpp:199)
==8229==    by 0x80B6665: Digikam::ThumbnailLoadingTask::execute() (thumbnailtask.cpp:172)
==8229==    by 0x8091530: Digikam::LoadSaveThread::run() (loadsavethread.cpp:136)
==8229==    by 0x80CF279: Digikam::DynamicThread::DynamicThreadPriv::run() (dynamicthread.cpp:186)

==8229== Conditional jump or move depends on uninitialised value(s)
==8229==    at 0x80B9DDD: CEncoder::WriteValue(CSubband*, int) (Encoder.cpp:328)
==8229==    by 0x80B9AA4: CEncoder::Partition(CSubband*, int, int, int, int) (Encoder.cpp:259)
==8229==    by 0x80C5D81: CSubband::ExtractTile(CEncoder&, bool, unsigned int, unsigned int) (Subband.cpp:190)
==8229==    by 0x80BE25E: CPGFImage::WriteLevel() (PGFimage.cpp:1051)
==8229==    by 0x80BE5EF: CPGFImage::WriteImage(CPGFStream*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1125)
==8229==    by 0x80BE73B: CPGFImage::Write(CPGFStream*, unsigned int*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1170)
==8229==    by 0x80C9F2E: Digikam::PGFUtils::writePGFImageDataToStream(QImage const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:288)
==8229==    by 0x80C938A: Digikam::PGFUtils::writePGFImageData(QImage const&, QByteArray&, int, bool) (pgfutils.cpp:184)
==8229==    by 0x80AA259: Digikam::ThumbnailCreator::storeInDatabase(Digikam::ThumbnailInfo const&, Digikam::ThumbnailImage const&) const (thumbnailcreator.cpp:695)
==8229==    by 0x80A7BD8: Digikam::ThumbnailCreator::load(QString const&, QRect const&, bool) const (thumbnailcreator.cpp:267)
==8229==    by 0x80A7785: Digikam::ThumbnailCreator::load(QString const&) const (thumbnailcreator.cpp:199)
==8229==    by 0x80B6665: Digikam::ThumbnailLoadingTask::execute() (thumbnailtask.cpp:172)

==8229== Conditional jump or move depends on uninitialised value(s)
==8229==    at 0x80BA7B8: CEncoder::CMacroBlock::DecomposeBitplane(unsigned int, unsigned int, unsigned int, unsigned int*, unsigned int*, unsigned int*, unsigned int&, unsigned int&) (Encoder.cpp:654)
==8229==    by 0x80BA307: CEncoder::CMacroBlock::BitplaneEncode() (Encoder.cpp:520)
==8229==    by 0x80B9E52: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:348)
==8229==    by 0x80B9D74: CEncoder::WriteValue(CSubband*, int) (Encoder.cpp:324)
==8229==    by 0x80B9AA4: CEncoder::Partition(CSubband*, int, int, int, int) (Encoder.cpp:259)
==8229==    by 0x80C5D81: CSubband::ExtractTile(CEncoder&, bool, unsigned int, unsigned int) (Subband.cpp:190)
==8229==    by 0x80BE25E: CPGFImage::WriteLevel() (PGFimage.cpp:1051)
==8229==    by 0x80BE5EF: CPGFImage::WriteImage(CPGFStream*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1125)
==8229==    by 0x80BE73B: CPGFImage::Write(CPGFStream*, unsigned int*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1170)
==8229==    by 0x80C9F2E: Digikam::PGFUtils::writePGFImageDataToStream(QImage const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:288)
==8229==    by 0x80C938A: Digikam::PGFUtils::writePGFImageData(QImage const&, QByteArray&, int, bool) (pgfutils.cpp:184)
==8229==    by 0x80AA259: Digikam::ThumbnailCreator::storeInDatabase(Digikam::ThumbnailInfo const&, Digikam::ThumbnailImage const&) const (thumbnailcreator.cpp:695)

==8229== Thread 16:
==8229== Conditional jump or move depends on uninitialised value(s)
==8229==    at 0x80BA9F6: CEncoder::CMacroBlock::NumberOfBitplanes() (Encoder.cpp:747)
==8229==    by 0x80BA219: CEncoder::CMacroBlock::BitplaneEncode() (Encoder.cpp:502)
==8229==    by 0x80B9E52: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:348)
==8229==    by 0x80B9D03: CEncoder::Flush() (Encoder.cpp:314)
==8229==    by 0x80BE687: CPGFImage::WriteImage(CPGFStream*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1135)
==8229==    by 0x80BE73B: CPGFImage::Write(CPGFStream*, unsigned int*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1170)
==8229==    by 0x80C9F2E: Digikam::PGFUtils::writePGFImageDataToStream(QImage const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:288)
==8229==    by 0x80C938A: Digikam::PGFUtils::writePGFImageData(QImage const&, QByteArray&, int, bool) (pgfutils.cpp:184)
==8229==    by 0x80AA259: Digikam::ThumbnailCreator::storeInDatabase(Digikam::ThumbnailInfo const&, Digikam::ThumbnailImage const&) const (thumbnailcreator.cpp:695)
==8229==    by 0x80A7BD8: Digikam::ThumbnailCreator::load(QString const&, QRect const&, bool) const (thumbnailcreator.cpp:267)
==8229==    by 0x80A7785: Digikam::ThumbnailCreator::load(QString const&) const (thumbnailcreator.cpp:199)
==8229==    by 0x80B6665: Digikam::ThumbnailLoadingTask::execute() (thumbnailtask.cpp:172)
==8229== 
==8229== Conditional jump or move depends on uninitialised value(s)
==8229==    at 0x80BAA27: CEncoder::CMacroBlock::NumberOfBitplanes() (Encoder.cpp:748)
==8229==    by 0x80BA219: CEncoder::CMacroBlock::BitplaneEncode() (Encoder.cpp:502)
==8229==    by 0x80B9E52: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:348)
==8229==    by 0x80B9D03: CEncoder::Flush() (Encoder.cpp:314)
==8229==    by 0x80BE687: CPGFImage::WriteImage(CPGFStream*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1135)
==8229==    by 0x80BE73B: CPGFImage::Write(CPGFStream*, unsigned int*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1170)
==8229==    by 0x80C9F2E: Digikam::PGFUtils::writePGFImageDataToStream(QImage const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:288)
==8229==    by 0x80C938A: Digikam::PGFUtils::writePGFImageData(QImage const&, QByteArray&, int, bool) (pgfutils.cpp:184)
==8229==    by 0x80AA259: Digikam::ThumbnailCreator::storeInDatabase(Digikam::ThumbnailInfo const&, Digikam::ThumbnailImage const&) const (thumbnailcreator.cpp:695)
==8229==    by 0x80A7BD8: Digikam::ThumbnailCreator::load(QString const&, QRect const&, bool) const (thumbnailcreator.cpp:267)
==8229==    by 0x80A7785: Digikam::ThumbnailCreator::load(QString const&) const (thumbnailcreator.cpp:199)
==8229==    by 0x80B6665: Digikam::ThumbnailLoadingTask::execute() (thumbnailtask.cpp:172)

Gilles Caulier
Comment 64 caulier.gilles 2013-11-08 07:59:21 UTC 
Raphael,

Next stages here will are :

- to turn on OpenMP support for libpgf and check if crash is reproducible as explained in comment #35.
- to run again Coverity Scan to see if all main reports are fixed now.

Gilles Caulier
Comment 65 caulier.gilles 2013-11-08 08:12:14 UTC 
Raphael,

There are few warnings reported by cppcheck. Nothing urgent, just informative and good to know :

[gilles@localhost libpgf]$ cppcheck --enable=all .
...
[Decoder.h:46]: (style) The class 'CDecoder' does not have a constructor.
[Decoder.cpp:499]: (style) Throwing a copy of the caught exception instead of rethrowing the original exception.
[Decoder.cpp:536]: (warning) Redundant assignment of 'wordLen' to itself.
[Decoder.cpp:595]: (warning) Redundant assignment of 'wordLen' to itself.
[BitStream.h:129]: (style) The scope of the variable 'hiCount' can be reduced.
[Decoder.h:46]: (style) 'class CDecoder' does not have a copy constructor which is recommended since the class contains a pointer to allocated memory.
...
[Encoder.h:46]: (style) The class 'CEncoder' does not have a constructor.
[Encoder.cpp:405]: (style) Variable 'h' is assigned a value that is never used.
[Encoder.cpp:744]: (style) The scope of the variable 'cnt' can be reduced.
[Encoder.cpp:570]: (style) Unsigned variable 'wordPos' can't be negative so it is unnecessary to test it.
[Encoder.cpp:583]: (style) Unsigned variable 'wordPos' can't be negative so it is unnecessary to test it.
[Encoder.cpp:593]: (style) Unsigned variable 'wordPos' can't be negative so it is unnecessary to test it.
[Encoder.cpp:605]: (style) Unsigned variable 'wordPos' can't be negative so it is unnecessary to test it.
[Encoder.cpp:614]: (style) Unsigned variable 'm_codePos' can't be negative so it is unnecessary to test it.
[Encoder.cpp:769]: (style) Unsigned variable 'codePos' can't be negative so it is unnecessary to test it.
[Encoder.h:46]: (style) 'class CEncoder' does not have a copy constructor which is recommended since the class contains a pointer to allocated memory.
...
[PGFimage.cpp:1294]: (error) syntax error
[PGFimage.cpp:637]: (style) Unsigned variable 'len' can't be negative so it is unnecessary to test it.
[PGFimage.cpp:680]: (style) Unsigned variable 'len' can't be negative so it is unnecessary to test it.
...
[gilles@localhost libpgf]$ 

Gilles Caulier
Comment 66 caulier.gilles 2013-11-08 08:19:50 UTC 
Rapahel,

I don't check your change about OpenMP support, but now, crash cannot be reproduced as explained in comment #35. I can run qtpgftest command line tool :

[gilles@localhost tests]$ ./qtpgftest
unnamed app(22268)/digikam (core) main: Using LibPGF version:  "6.13.45"
unnamed app(22268)/digikam (core) main: Using OpenMP with LibPGF :  yes
unnamed app(22268)/digikam (core) main: Generate PGF file using file stream
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream: RGB => ARGB
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream: PGF image settings:
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream:    width:  256
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream:    height:  170
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream:    nLevels:  0
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream:    quality:  0
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream:    bpp:  32
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream:    channels:  4
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream:    mode:  17
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream:    usedBitsPerChannel:  0
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageFile: file size written :  80806
unnamed app(22268)/digikam (core) main: Generate PGF file using data stream
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageData: PGF stream memory allocation in bytes:  174080
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream: RGB => ARGB
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream: PGF image settings:
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream:    width:  256
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream:    height:  170
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream:    nLevels:  0
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream:    quality:  0
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream:    bpp:  32
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream:    channels:  4
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream:    mode:  17
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageDataToStream:    usedBitsPerChannel:  0
unnamed app(22268)/digikam (core) Digikam::PGFUtils::writePGFImageData: data size written :  80806
unnamed app(22268)/digikam (core) main: PGF Encoding time:  0.08  s
unnamed app(22268)/digikam (core) main: Load PGF file generated by data stream
unnamed app(22268)/digikam (core) Digikam::PGFUtils::readPGFImageData: image data stream size is :  80806
unnamed app(22268)/digikam (core) Digikam::PGFUtils::readPGFImageData: PGF image is open
unnamed app(22268)/digikam (core) Digikam::PGFUtils::readPGFImageData: PGF image is read
unnamed app(22268)/digikam (core) Digikam::PGFUtils::readPGFImageData: PGF image is decoded
unnamed app(22268)/digikam (core) main: PGF Decoding time:  0.1  s
unnamed app(22268)/digikam (core) main: Generate JPG file to compare performances
unnamed app(22268)/digikam (core) main: JPG Encoding time:  0  s
unnamed app(22268)/digikam (core) main: JPG Decoding time:  0  s
unnamed app(22268)/digikam (core) main: Generate PNG file to compare performances
unnamed app(22268)/digikam (core) main: PNG Encoding time:  0.01  s
libpng warning: Unknown iTXt compression type or method
unnamed app(22268)/digikam (core) main: PNG Decoding time:  0  s
[gilles@localhost tests]$

But valgrind report few traces about memory leak when OpenMP support is enabled :

[gilles@localhost tests]$ valgrind --tool=memcheck --leak-check=full --error-limit=no ./qtpgftest
==22448== Memcheck, a memory error detector
==22448== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==22448== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==22448== Command: ./qtpgftest
==22448== 
...
==22448== 
==22448== HEAP SUMMARY:
==22448==     in use at exit: 2,219,171 bytes in 470 blocks
==22448==   total heap usage: 23,798 allocs, 23,328 frees, 39,735,306 bytes allocated
...
==22448== 2,464 bytes in 7 blocks are possibly lost in loss record 247 of 260
==22448==    at 0x4C26DFF: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==22448==    by 0x4011008: _dl_allocate_tls (in /usr/lib64/ld-2.17.so)
==22448==    by 0x59CB8B8: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.17.so)
==22448==    by 0x4E38CBD: ??? (in /usr/lib64/libgomp.so.1.0.0)
==22448==    by 0x40AD03: CPGFImage::WriteHeader(CPGFStream*) (PGFimage.cpp:930)
==22448==    by 0x40B7C0: CPGFImage::Write(CPGFStream*, unsigned int*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1167)
==22448==    by 0x417611: Digikam::PGFUtils::writePGFImageDataToStream(QImage const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:288)
==22448==    by 0x4167B3: Digikam::PGFUtils::writePGFImageFile(QImage const&, QString const&, int, bool) (pgfutils.cpp:152)
==22448==    by 0x402992: main (qtpgftest.cpp:63)
...
==22448== 1,032,472 bytes in 7 blocks are definitely lost in loss record 259 of 260
==22448==    at 0x4C285F1: operator new(unsigned long) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==22448==    by 0x40686A: CEncoder::CEncoder(CPGFStream*, PGFPreHeader, PGFHeader, PGFPostHeader const&, unsigned long long&, bool) (Encoder.cpp:101)
==22448==    by 0x40AE60: CPGFImage::WriteHeader(CPGFStream*) (PGFimage.cpp:982)
==22448==    by 0x40B7C0: CPGFImage::Write(CPGFStream*, unsigned int*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1167)
==22448==    by 0x417611: Digikam::PGFUtils::writePGFImageDataToStream(QImage const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:288)
==22448==    by 0x416ABD: Digikam::PGFUtils::writePGFImageData(QImage const&, QByteArray&, int, bool) (pgfutils.cpp:184)
==22448==    by 0x402AF3: main (qtpgftest.cpp:74)
==22448== 
==22448== 1,032,472 bytes in 7 blocks are definitely lost in loss record 260 of 260
==22448==    at 0x4C285F1: operator new(unsigned long) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==22448==    by 0x40686A: CEncoder::CEncoder(CPGFStream*, PGFPreHeader, PGFHeader, PGFPostHeader const&, unsigned long long&, bool) (Encoder.cpp:101)
==22448==    by 0x40AE60: CPGFImage::WriteHeader(CPGFStream*) (PGFimage.cpp:982)
==22448==    by 0x40B7C0: CPGFImage::Write(CPGFStream*, unsigned int*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1167)
==22448==    by 0x417611: Digikam::PGFUtils::writePGFImageDataToStream(QImage const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:288)
==22448==    by 0x4167B3: Digikam::PGFUtils::writePGFImageFile(QImage const&, QString const&, int, bool) (pgfutils.cpp:152)
==22448==    by 0x402992: main (qtpgftest.cpp:63)
==22448== 
==22448== LEAK SUMMARY:
==22448==    definitely lost: 2,066,960 bytes in 35 blocks
==22448==    indirectly lost: 7,540 bytes in 63 blocks
==22448==      possibly lost: 2,464 bytes in 7 blocks
==22448==    still reachable: 142,207 bytes in 365 blocks
==22448==         suppressed: 0 bytes in 0 blocks
==22448== Reachable blocks (those to which a pointer was found) are not shown.
==22448== To see them, rerun with: --leak-check=full --show-reachable=yes
==22448== 
==22448== For counts of detected and suppressed errors, rerun with: -v
==22448== ERROR SUMMARY: 6 errors from 6 contexts (suppressed: 2 from 2)
[gilles@localhost tests]$
Comment 67 caulier.gilles 2013-11-08 08:31:47 UTC 
Raphael,

running digiKam into valgrind with libpgf OpenMP support confirm memory leak :

==22590== 368 bytes in 1 blocks are possibly lost in loss record 22,738 of 25,730
==22590==    at 0x4C26DFF: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==22590==    by 0x4011008: _dl_allocate_tls (in /usr/lib64/ld-2.17.so)
==22590==    by 0xFD028B8: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.17.so)
==22590==    by 0x10AACCBD: ??? (in /usr/lib64/libgomp.so.1.0.0)
==22590==    by 0x80BDA49: CPGFImage::WriteHeader(CPGFStream*) (PGFimage.cpp:930)
==22590==    by 0x80BE510: CPGFImage::Write(CPGFStream*, unsigned int*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1167)
==22590==    by 0x80CA2EE: Digikam::PGFUtils::writePGFImageDataToStream(QImage const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:288)
==22590==    by 0x80C974A: Digikam::PGFUtils::writePGFImageData(QImage const&, QByteArray&, int, bool) (pgfutils.cpp:184)
==22590==    by 0x80AA2F9: Digikam::ThumbnailCreator::storeInDatabase(Digikam::ThumbnailInfo const&, Digikam::ThumbnailImage const&) const (thumbnailcreator.cpp:695)
==22590==    by 0x80A7C78: Digikam::ThumbnailCreator::load(QString const&, QRect const&, bool) const (thumbnailcreator.cpp:267)
==22590==    by 0x80A7825: Digikam::ThumbnailCreator::load(QString const&) const (thumbnailcreator.cpp:199)
==22590==    by 0x80B6705: Digikam::ThumbnailLoadingTask::execute() (thumbnailtask.cpp:172)

==22590== 5,309,856 bytes in 36 blocks are possibly lost in loss record 25,727 of 25,730
==22590==    at 0x4C285F1: operator new(unsigned long) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==22590==    by 0x80B9531: CEncoder::CEncoder(CPGFStream*, PGFPreHeader, PGFHeader, PGFPostHeader const&, unsigned long long&, bool) (Encoder.cpp:101)
==22590==    by 0x80BDBAC: CPGFImage::WriteHeader(CPGFStream*) (PGFimage.cpp:982)
==22590==    by 0x80BE510: CPGFImage::Write(CPGFStream*, unsigned int*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1167)
==22590==    by 0x80CA2EE: Digikam::PGFUtils::writePGFImageDataToStream(QImage const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:288)
==22590==    by 0x80C974A: Digikam::PGFUtils::writePGFImageData(QImage const&, QByteArray&, int, bool) (pgfutils.cpp:184)
==22590==    by 0x80AA2F9: Digikam::ThumbnailCreator::storeInDatabase(Digikam::ThumbnailInfo const&, Digikam::ThumbnailImage const&) const (thumbnailcreator.cpp:695)
==22590==    by 0x80A7C78: Digikam::ThumbnailCreator::load(QString const&, QRect const&, bool) const (thumbnailcreator.cpp:267)
==22590==    by 0x80A7825: Digikam::ThumbnailCreator::load(QString const&) const (thumbnailcreator.cpp:199)
==22590==    by 0x80B6705: Digikam::ThumbnailLoadingTask::execute() (thumbnailtask.cpp:172)
==22590==    by 0x80915D0: Digikam::LoadSaveThread::run() (loadsavethread.cpp:136)
==22590==    by 0x80CF639: Digikam::DynamicThread::DynamicThreadPriv::run() (dynamicthread.cpp:186)

==22590== 59,735,880 bytes in 405 blocks are definitely lost in loss record 25,730 of 25,730
==22590==    at 0x4C285F1: operator new(unsigned long) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==22590==    by 0x80B9531: CEncoder::CEncoder(CPGFStream*, PGFPreHeader, PGFHeader, PGFPostHeader const&, unsigned long long&, bool) (Encoder.cpp:101)
==22590==    by 0x80BDBAC: CPGFImage::WriteHeader(CPGFStream*) (PGFimage.cpp:982)
==22590==    by 0x80BE510: CPGFImage::Write(CPGFStream*, unsigned int*, bool (*)(double, bool, void*), void*) (PGFimage.cpp:1167)
==22590==    by 0x80CA2EE: Digikam::PGFUtils::writePGFImageDataToStream(QImage const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:288)
==22590==    by 0x80C974A: Digikam::PGFUtils::writePGFImageData(QImage const&, QByteArray&, int, bool) (pgfutils.cpp:184)
==22590==    by 0x80AA2F9: Digikam::ThumbnailCreator::storeInDatabase(Digikam::ThumbnailInfo const&, Digikam::ThumbnailImage const&) const (thumbnailcreator.cpp:695)
==22590==    by 0x80A7C78: Digikam::ThumbnailCreator::load(QString const&, QRect const&, bool) const (thumbnailcreator.cpp:267)
==22590==    by 0x80A7825: Digikam::ThumbnailCreator::load(QString const&) const (thumbnailcreator.cpp:199)
==22590==    by 0x80B6705: Digikam::ThumbnailLoadingTask::execute() (thumbnailtask.cpp:172)
==22590==    by 0x80915D0: Digikam::LoadSaveThread::run() (loadsavethread.cpp:136)
==22590==    by 0x80CF639: Digikam::DynamicThread::DynamicThreadPriv::run() (dynamicthread.cpp:186)

Ask you can see, memory leaks are not negligible if OPenMP support is enabled. I always run digiKam in same condition : I stop thumbnails generator at 1% (else it too long under valgrind).

Nota : digiKam do not crash imediatly when openMp support is enabled, as i have seen in the pass.

To resume : 

- libpgf without OpenMP support : no memory leak.
- libpgf with OpenMP support : no crash but important memory leak.

Gilles Caulier
Comment 68 Raphael Schweizer 2013-11-08 10:30:33 UTC 
Gilles,

- UNINIT/JUMP warnings are false positives (I'll check and mark accordingly on coverity when new analysis is available).
- Memory leak @CEncoder::CEncoder(Encoder.cpp:101) FIXED (soon on SF); other OpenMP memory leak must remain low-priority for the time being (we do not have sufficient resources to check with valgrind/gcc/libgomp, if time allows we'll at least check with VC++ [and Intel])
- cppcheck: fixed two issued (others are related to little/big endian conditionals), soon on SF

Next steps: Dario will intermittently work on CMake toolchain and eventually OpenMP issues

- Raphael
Comment 69 Raphael Schweizer 2013-11-08 10:44:49 UTC 
r123 with said FIXes now available @SF
Comment 70 caulier.gilles 2013-11-08 11:00:15 UTC 
Git commit 2a5aadd60a23bd61b74722125dc25677556422a4 by Gilles Caulier.
Committed on 08/11/2013 at 10:59.
Pushed by cgilles into branch 'master'.

update internal libpgf source code to last revision #r123 from SF repository

M  +1    -1    libs/3rdparty/libpgf/Decoder.cpp
M  +8    -3    libs/3rdparty/libpgf/Encoder.cpp

http://commits.kde.org/digikam/2a5aadd60a23bd61b74722125dc25677556422a4
Comment 71 caulier.gilles 2013-11-08 11:02:27 UTC 
Raphael,

About CMake toolchain, you just need to sync svn libpgf cmakeport branch with trunk, and sync back this branch in trunk to complete job.

Gilles Caulier
Comment 72 Raphael Schweizer 2013-11-08 11:14:39 UTC 
Gilles,

unfortunately it's not that trivial (for us) as we have > 10 different build configurations in our Visual Studio project. We'd like to change everything to CMake but need to understand its configuration first (i.e. how to generate our configurations from CMake). Your CMake branch obviously is an excellent starting point. Thanks!

- Raphael
Comment 73 caulier.gilles 2013-11-08 12:49:19 UTC 
Raphael,

To configure cmake to build VisualC++ project, just run this script :

http://sourceforge.net/p/libpgf/code/HEAD/tree/cmakeport/libpgf/bootstrap.msvc.bat

this will make nmake rules to compile under Windows console. You can change cmake -G option to make VisualC++ project in the same way. For ex, to generate Visual C++ 2010 SLN files :

cmake -G "Visual Studio 10"

... as explained here :

http://stackoverflow.com/questions/395169/using-cmake-to-generate-visual-studio-c-project-files

Gilles Caulier
Comment 74 Raphael Schweizer 2013-11-08 15:14:47 UTC 
Gilles,

thanks for the SO link. I found the following there:
"One more important thing to know about is the lack of support (afaik) for 'Solution Configurations' in CMake. As it stands, you have to generate two directories with projects/solutions - one for each build type (debug, release, etc.). There is no direct support for more sophisticated features - in other words: switching between configurations won't give you what you might expect."
which is exactly our problem. As stated in my last post we have > 10 such "configurations" for release, debug, omp, 16bit, ROI/nonROI and combinations thereof. In Visual Studio we can easily switch between them by just selecting from a drop down menu. We'd like to keep this as convenient as possible... In principle we "just" need to translate our configurations to CMake, but there seem to be related issues with the VS generator (see CMake bugtracker) and we need to distil workarounds from their mailing list as the documentation is allegedly out of sync with the implementation... which in practice makes this a tedious task.

- Raphael
Comment 75 caulier.gilles 2013-11-08 15:24:00 UTC 
Raphael,

Well about cmakeport branch, i recommend to merge it in trunk, including multiple configuration framework support for a transition time. This will make cmake support official for the next release and permit to use other way to configure and compile the library. 

A new release must be done in a near future to include your precious fixes... I want also to set digiKam libpgf dependency restricted to this new release to be able to remove later all libpgf source code from digiKam core.

Gilles Caulier
Comment 76 caulier.gilles 2013-11-14 17:30:29 UTC 
As with commit of libpgf revision 123 fix memory leak when OpenMP is not used, i can close this file now...

Gilles Caulier
Comment 77 caulier.gilles 2013-11-19 15:22:29 UTC 
*** Bug 327815 has been marked as a duplicate of this bug. ***
Comment 78 matteo.fortini 2013-11-19 18:10:56 UTC 
(In reply to comment #76)
> As with commit of libpgf revision 123 fix memory leak when OpenMP is not
> used, i can close this file now...
> 
> Gilles Caulier

I just downloaded libpgf r123 from svn and installed it. I checked with ldd that the library was actually used. It was.

Problem is still there, memory usage skyrockets. Any clue?
Comment 79 Raphael Schweizer 2013-11-19 19:47:20 UTC 
(In reply to comment #78)
> I just downloaded libpgf r123 from svn and installed it. I checked with ldd
> that the library was actually used. It was.
> 
> Problem is still there, memory usage skyrockets. Any clue?

Without further information, hard to say. I assume you did use the code from trunk and not by chance from bincomp or cmakeport? Did you compile with disabled OpenMP support? We still have confirmed memory leaks when using libgomp (though these shouldn't be "skyrockets").
- Raphael
Comment 80 caulier.gilles 2013-12-12 21:37:30 UTC 
*** Bug 328732 has been marked as a duplicate of this bug. ***
Comment 81 caulier.gilles 2014-03-12 11:40:17 UTC 
To Raphael,

I continue to talk in this thread about future official release of libpgf including all fixes recently added.

What's the plan for you. I would to see libpgf released before digiKam 4.0.0 to be sure that all distro will use clean code. Also, did you take a look to all ccpcheck and Coverty reports about your code ? I can see some entries from Jenkins/ccpcheck in KDE server :

http://build.kde.org/view/extragear/job/digikam_master/1195/cppcheckResult/

The digiKam 4.0.0 plan is listed here :

http://www.digikam.org/about/releaseplan

Also, in the future, i would to drop libpgf code from digiKam code to only use external shared library instead.

Thanks to your feedback

Gilles Caulier