Bug 331224

Summary: Sync metadata eventually slow-down computer to unusable state
Product: [Applications] digikam Reporter: Jeff Robinson <jeffnik>
Component: Maintenance-MetadataAssignee: Digikam Developers <digikam-bugs-null>
Status: RESOLVED FIXED    
Severity: crash CC: caulier.gilles
Priority: NOR    
Version: 4.0.0   
Target Milestone: ---   
Platform: Compiled Sources   
OS: Linux   
Latest Commit: Version Fixed In: 4.6.0

Description Jeff Robinson 2014-02-17 01:31:06 UTC 
I have a collection of RAW files that were tagged using Digikam on another machine, the metadata written to the XMP files.  Accessing these files via a second installation of Digikam on a laptop the second install does not read in the XMP data.
Trying to use the Maintenance feature to sync the metadata to the database over the network creates a moving process indicator (with no percentage/activity text) but not activity seems to be taking place.  The hard drive on the laptop shows activity, but eventually the system slows so much that it doesn't respond to mouse-clicks.
I tried plugging a USB removable drive of the same images into the laptop with the same result.
As this was a rather large collection, I pointed the maintenance tools at different directories (including one containing only 22 images) and the same issue occurred.

Reproducible: Always

Steps to Reproduce:
1. Select Tools -> Maintenance...
2. Select a single album.
3. Select "Sync Metadata and Database" with sync direction "From image metadata to database"
Actual Results:  
Digikam runs but does does not complete.  The system eventually slows down to the point where the computer has to be shut off to stop the process.

Expected Results:  
Metadata would be read in from XMP files and populate the database.
Comment 1 caulier.gilles 2014-02-17 05:53:48 UTC 
Do you use multicore option from maintenance tool ?

Gilles Caulier
Comment 2 Jeff Robinson 2014-02-17 18:33:47 UTC 
(In reply to comment #1)
> Do you use multicore option from maintenance tool ?
> 
> Gilles Caulier

I did not have multicore enabled (I had missed that option), but I have the same results with that option turned on.  Performance gradually degrades until I have trouble even stopping the running programs.

To see if it was a network issue, I also copied a directory of 22 RAW Pentax files (.PEF) to my laptop and tried the maintenance feature again locally.  I appear to be getting the same results.

Is there a way I can check to make certain my Digikam database hasn't been corrupted and that's causing this problem?

I have a sneaking suspicion that I'm running out of memory (8G physical memory and swap).
Comment 3 caulier.gilles 2014-02-17 18:47:52 UTC 
 a memory leak can be very easily checkable using an external tool as htop for ex. Just check if memory allocated by digiKam raise progressively...

The second stage is to understand where memory is lost. this can be into Exiv2 shared library which process whole metadata management with file (outside DB of course).

If you have a suspected problem with DB, you can try to run a fresh digiKam from a test account on your computer. Import new image a run maintenance tool in same conditions.

Gilles Caulier
Comment 4 Jeff Robinson 2014-02-19 02:45:08 UTC 
(In reply to comment #3)
>  a memory leak can be very easily checkable using an external tool as htop
> for ex. Just check if memory allocated by digiKam raise progressively...
> 
> The second stage is to understand where memory is lost. this can be into
> Exiv2 shared library which process whole metadata management with file
> (outside DB of course).
> 
> If you have a suspected problem with DB, you can try to run a fresh digiKam
> from a test account on your computer. Import new image a run maintenance
> tool in same conditions.
> 
> Gilles Caulier

I created a new account and was able to have the same 22 files recognised properly, bringing the information from the XMP files in as expected.

Going back to the original account, I took the step of renaming the existing Digikam database and thumbnail data and restarting.

I added a local directory which was catalogued properly, including information from the XMP sidecar for the RAW files.

I then added my networked portfolio drive (which took about 3 hours to catalogue), and this also seems to be working properly.

I did try running the maintenance tool on this new install and the same issue seems to occur.  Using htop I can watch the Digikam process add about 2MB of memory usage every 3 seconds or so, with one of the cores always around 90%.

I may have been seeing two different issues with one caused by a damaged database.  The maintenance tool problem still seems to persist for me, though.

The Exiv2 library I'm using is 0.23 from Slackware-Current, but I wouldn't know how to go about finding out if it is the cause of a memory leak.
Comment 5 caulier.gilles 2014-02-20 20:20:07 UTC 
To identify where memory is leak, run digiKam in a console through valgrind, and reproduce all operation in GUI. Application will slow down with valgrind. Report console trace here for future investigations.

Look details here :

http://www.digikam.org/contrib

Gilles Caulier
Comment 6 Jeff Robinson 2014-02-21 04:18:10 UTC 
bash-4.2$ valgrind --tool=memcheck --leak-check=full --error-limit=no digikam
==19952== Memcheck, a memory error detector
==19952== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==19952== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==19952== Command: digikam
==19952== 
QSqlDatabasePrivate::removeDatabase: connection 'ConnectionTest' is still in use, all queries will cease to work.
==19952== Conditional jump or move depends on uninitialised value(s)
==19952==    at 0x1171ECD5: wcsnlen (in /lib64/libc-2.17.so)
==19952==    by 0x1171E44F: wcsrtombs (in /lib64/libc-2.17.so)
==19952==    by 0x116B4DA0: wcstombs (in /lib64/libc-2.17.so)
==19952==    by 0x7B978BB: ??? (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7B99465: cmsReadICCTextEx (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7B9A1E5: cmsTakeProductDesc (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7FCA82F: Digikam::IccProfile::description() (iccprofile.cpp:377)
==19952==    by 0x7FE1239: Digikam::IccSettings::loadAllProfilesProperties() (iccsettings.cpp:585)
==19952==    by 0x5AB43E: Digikam::DigikamApp::DigikamApp() (digikamapp.cpp:227)
==19952==    by 0x49A9A9: main (main.cpp:189)
==19952== 
==19952== Conditional jump or move depends on uninitialised value(s)
==19952==    at 0x1171ECEB: wcsnlen (in /lib64/libc-2.17.so)
==19952==    by 0x1171E44F: wcsrtombs (in /lib64/libc-2.17.so)
==19952==    by 0x116B4DA0: wcstombs (in /lib64/libc-2.17.so)
==19952==    by 0x7B978BB: ??? (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7B99465: cmsReadICCTextEx (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7B9A1E5: cmsTakeProductDesc (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7FCA82F: Digikam::IccProfile::description() (iccprofile.cpp:377)
==19952==    by 0x7FE1239: Digikam::IccSettings::loadAllProfilesProperties() (iccsettings.cpp:585)
==19952==    by 0x5AB43E: Digikam::DigikamApp::DigikamApp() (digikamapp.cpp:227)
==19952==    by 0x49A9A9: main (main.cpp:189)
==19952== 
==19952== Conditional jump or move depends on uninitialised value(s)
==19952==    at 0x1171ECB0: wcsnlen (in /lib64/libc-2.17.so)
==19952==    by 0x1171E44F: wcsrtombs (in /lib64/libc-2.17.so)
==19952==    by 0x116B4DA0: wcstombs (in /lib64/libc-2.17.so)
==19952==    by 0x7B978BB: ??? (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7B99465: cmsReadICCTextEx (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7B9A1E5: cmsTakeProductDesc (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7FCA82F: Digikam::IccProfile::description() (iccprofile.cpp:377)
==19952==    by 0x7FE1239: Digikam::IccSettings::loadAllProfilesProperties() (iccsettings.cpp:585)
==19952==    by 0x5AB43E: Digikam::DigikamApp::DigikamApp() (digikamapp.cpp:227)
==19952==    by 0x49A9A9: main (main.cpp:189)
==19952== 
==19952== Conditional jump or move depends on uninitialised value(s)
==19952==    at 0x1171ECC1: wcsnlen (in /lib64/libc-2.17.so)
==19952==    by 0x1171E44F: wcsrtombs (in /lib64/libc-2.17.so)
==19952==    by 0x116B4DA0: wcstombs (in /lib64/libc-2.17.so)
==19952==    by 0x7B978BB: ??? (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7B99465: cmsReadICCTextEx (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7B9A1E5: cmsTakeProductDesc (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7FCA82F: Digikam::IccProfile::description() (iccprofile.cpp:377)
==19952==    by 0x7FE1239: Digikam::IccSettings::loadAllProfilesProperties() (iccsettings.cpp:585)
==19952==    by 0x5AB43E: Digikam::DigikamApp::DigikamApp() (digikamapp.cpp:227)
==19952==    by 0x49A9A9: main (main.cpp:189)
==19952== 
==19952== Syscall param ioctl(generic) points to uninitialised byte(s)
==19952==    at 0x1176E249: syscall (in /lib64/libc-2.17.so)
==19952==    by 0x2431E3EF: ??? (in /usr/lib64/libv4lconvert.so.0.0.0)
==19952==    by 0x24333CFD: ??? (in /usr/lib64/libv4lconvert.so.0.0.0)
==19952==    by 0x2431EB9D: v4lconvert_create_with_dev_ops (in /usr/lib64/libv4lconvert.so.0.0.0)
==19952==    by 0x2257033D: v4l2_fd_open (in /usr/lib64/libv4l2.so.0.0.0)
==19952==    by 0x30755507: ??? (in /usr/lib64/gstreamer-0.10/libgstvideo4linux2.so)
==19952==    by 0x3074FC64: ??? (in /usr/lib64/gstreamer-0.10/libgstvideo4linux2.so)
==19952==    by 0x307536B0: ??? (in /usr/lib64/gstreamer-0.10/libgstvideo4linux2.so)
==19952==    by 0x1741C064: g_object_get_valist (in /usr/lib64/libgobject-2.0.so.0.3600.4)
==19952==    by 0x1741C4F6: g_object_get (in /usr/lib64/libgobject-2.0.so.0.3600.4)
==19952==    by 0x2FFF615B: ??? (in /usr/lib64/kde4/plugins/phonon_backend/phonon_gstreamer.so)
==19952==    by 0x2FFF71BA: ??? (in /usr/lib64/kde4/plugins/phonon_backend/phonon_gstreamer.so)
==19952==  Address 0xffeffda64 is on thread 1's stack
==19952== 
==19952== Syscall param ioctl(generic) points to uninitialised byte(s)
==19952==    at 0x1176E249: syscall (in /lib64/libc-2.17.so)
==19952==    by 0x2431E3EF: ??? (in /usr/lib64/libv4lconvert.so.0.0.0)
==19952==    by 0x24333532: ??? (in /usr/lib64/libv4lconvert.so.0.0.0)
==19952==    by 0x2431EB9D: v4lconvert_create_with_dev_ops (in /usr/lib64/libv4lconvert.so.0.0.0)
==19952==    by 0x2257033D: v4l2_fd_open (in /usr/lib64/libv4l2.so.0.0.0)
==19952==    by 0x30755507: ??? (in /usr/lib64/gstreamer-0.10/libgstvideo4linux2.so)
==19952==    by 0x3074FC64: ??? (in /usr/lib64/gstreamer-0.10/libgstvideo4linux2.so)
==19952==    by 0x307536B0: ??? (in /usr/lib64/gstreamer-0.10/libgstvideo4linux2.so)
==19952==    by 0x1741C064: g_object_get_valist (in /usr/lib64/libgobject-2.0.so.0.3600.4)
==19952==    by 0x1741C4F6: g_object_get (in /usr/lib64/libgobject-2.0.so.0.3600.4)
==19952==    by 0x2FFF615B: ??? (in /usr/lib64/kde4/plugins/phonon_backend/phonon_gstreamer.so)
==19952==    by 0x2FFF71BA: ??? (in /usr/lib64/kde4/plugins/phonon_backend/phonon_gstreamer.so)
==19952==  Address 0xffeffdecb is on thread 1's stack
==19952== 
digikam(19952) Phonon::KdePlatformPlugin::createBackend: using backend:  "GStreamer"

There is no additional output after the "GStreamer" line, even when running the maintenance tool.  I'll try the next instructions on the webpage for hangs and other dysfunctions next.
Comment 7 caulier.gilles 2014-05-16 07:29:05 UTC 
digiKam 4.0.0 is out :

http://www.digikam.org/node/713

Please check if this entry still valid with this new version.

Thanks in advance

Gilles Caulier
Comment 8 Jeff Robinson 2014-05-31 13:50:53 UTC 
(In reply to comment #7)
> digiKam 4.0.0 is out :
> 
> http://www.digikam.org/node/713
> 
> Please check if this entry still valid with this new version.
> 
> Thanks in advance
> 
> Gilles Caulier

I can confirm that this issue continues in my install of digikam 4.0.0 (x64).
Comment 9 caulier.gilles 2014-08-29 22:35:12 UTC 
What's about this file using last digiKam 4.2.0 ?

Gilles Caulier
Comment 10 caulier.gilles 2014-12-10 17:53:26 UTC 
digiKam 4.5.0 have been released.

Crash still reproducible with this release ?

Gilles Caulier
Comment 11 Jeff Robinson 2014-12-11 00:39:44 UTC 
(In reply to Gilles Caulier from comment #10)
> digiKam 4.5.0 have been released.
> 
> Crash still reproducible with this release ?
> 
> Gilles Caulier

I can confirm that with Digikam 4.5.0 I have successfully synchronized first a directory of 40 images and then two directories with more than 400 images, and everything seems to have worked correctly.

The system took approximately 3 minutes to do 400+ images.  I believe this bug should be closed.