Bug 210259

Can you give us a backtrace to see where is the problem ?

Can you try under linux to see if problem exist too ?

Can you provide a tiff file to test ?

Gilles Caulier

Can we assume that this happens due to the size of this image (doesn't happen with smaller tiffs?)
Backtrace would be the most interesting piece of information here.

Thanks for the interest. Being a noob you'll have to be a bit patient ;)

First I tested the size hypothesis, which appears valid. Using only "normal" mode for the layers, no mask (but some transparency), digiKam reads a 215MB file, but chokes on 236MB. Currently uploading files to file sharing site, but that's going to take a bit of time...

Looking into producing a backtrace under Windows XP.

Here are the file:
http://dl.free.fr/qZ7HT26Rk

Can't say the backtrace research has been particularly fruitful :(

Andreas, we have a memory allocation problem with exiv2 here.
I will attach output from massif (heap profiler). When scanning the image in question, libexiv2 will allocate a total of 1209MB of memory (in seven parts of around 172MB each)

Note: from the link posted by Francois above, you can download a ZIP file containing two sample TIFFs.

Created attachment 37634 [details]
Relevant output from Massif

(read with automatic line breaking disabled)

Is this related to bug #183171?

Marcel,

Exiv2 uses more memory than necessary when writing to TIFFs, that's a known issue. See here: http://dev.exiv2.org/issues/show/617

However, a normal "intrusive" write operation should not allocate more than about 2x the size of the image. Try simple commands like those in the issue to verify. (I can't download such a large file right now, slow internet connections seem to follow me around...)

Additional memory allocated is likely from suboptimal use of the library.

Andreas

So with a 236MB image, 2*image size is already about 500MB ;-)

So this is indeed an incarnation of bug 183171 (when reading, to store the read data, we do something supposed to be done only when writing).

Francois: Bug 183171 will be fixed soon, but it requires API changes in libkexiv2, so the fix will be in KDE4.4 only.

> So with a 236MB image, 2*image size is already about 500MB ;-)

That's why there is http://dev.exiv2.org/issues/show/617 

In the unstable branch in SVN it is partly fixed, the image is only loaded once, eventually it will not be necessary to load the image anymore at all. But that also requires API changes.

Andreas

The Exiv2 part is now complete and the results are quite drastic. See http://dev.exiv2.org/issues/show/617

This is still in the unstable branch in the Exiv2 SVN for the moment, but will go into the trunk and then become 0.19 (hopefully) soon. I'd be grateful if someone took the trouble to compile digiKam/libkexiv2 against exiv2-unstable and could report if there are issues with the API changes.

Andreas

digiKam 1.0.0 is out since few days...

http://www.digikam.org/drupal/node/491

Please try with this version coming with more than 400 bug-fixes.

Thanks in advance

Gilles Caulier

digiKam 1.1. release will be done in few days. Please check if this entry still valid.

Thanks in advance

Gilles Caulier

Thanks a lot Gilles, Marel and Andreas. Tried 1.0.0 but it crashes every time at startup so cannot test how well it scans large files. Hopefully I'll have more luck with 1.1.

Not 100% sure this is the same bug, but I did have a crash on scanning that was fixed by removing the .TIFFs from my photos folder. The largest one was very close to exactly 200MB.

Using linux version 1.2.0 on Ubuntu (downloaded from SVN and compiled on 2nd March 2010), not sure what the library versions are - I just used apt-get build-dep digikam.

Happy to provide any more info if needed, but I can't quite work out how to get the debug info properly - I have installed various '-dbg' packages, but the crash reporter still says that the crash information is not useful.

What version of exiv2 are you using? Run the following command to check (assuming you have the exiv2 utility installed, not only the library)

$ exiv2 -V

If it's 0.19, can you post the output of this:

$ ls -la *.TIFF
$ time exiv2 *.TIFF >/dev/null

If it's < 0.19, pls upgrade to 0.19 and re-compile digikam + libkexiv2 against that

Thanks,
Andreas

Okay, so now I'm stuck.

I ran 'exiv2 -V', and it said that I have version 0.18 (can't remember if there was a minor number) installed.

So I checked the latest version of exiv2 out of svn (using the instructions on the website), compiled that via cmake, make, sudo make install. Then I re-compiled digikam, via cmake, make, sudo make install.

Now if I hit Help->Component information, it says:
(abridged)

digiKam version 1.2.0 (rev.: 1098216)
LibExiv2: 0.18.2

I can't understand why the new version of libexiv2 isn't overwriting the old version. Any thoughts? I also notice that the command line program 'exiv2' doesn't get installed - I have to manually run it from the compiled src directory. Perhaps the install is just silently failing?

I suspect that fixing this would fix the problem, but the solution might end up being that I have to wait for the next release of Ubuntu - I notice that version .19 of exiv2 is in the list of packages for Lucid.

Then again, maybe its time to try a different distro...

> So I checked the latest version of exiv2 out of svn (using the instructions on
> the website), compiled that via cmake, make, sudo make install. 

Good. The current trunk is better than 0.19. But instead of cmake, do

$ make config; ./configure; make; sudo make install

(cmake is highly experimental - see README-cmake)

> Then I
> re-compiled digikam, via cmake, make, sudo make install.
> Now if I hit Help->Component information, it says:
> (abridged)
> 
> digiKam version 1.2.0 (rev.: 1098216)
> LibExiv2: 0.18.2
> 
> I can't understand why the new version of libexiv2 isn't overwriting the old
> version. Any thoughts? 

Did you re-compile libkexiv2? - note the 'k' If not, that's probably the issue. 

> I also notice that the command line program 'exiv2'
> doesn't get installed - I have to manually run it from the compiled src
> directory. Perhaps the install is just silently failing?

Compile & install exiv2 as suggested above and it will get installed. But this won't fix your other problem.

Okay, so I got exiv2 -V to show version 19.1

Then I tried to download and recompile the svn versions of both kdegraphics and digikam. The compile seemed to work, but now digikam won't even start up. It crashes on the 'loading kipi-plugins' part of the splash screen. I used the automatic bug tracker to add it as a separate bug - https://bugs.kde.org/show_bug.cgi?id=229344

Here is the output of those commands:

frith@sally:~/trouble_images/test$ ls -la
total 162468
drwxr-xr-x 2 frith frith      4096 2010-03-04 13:29 .
drwxr-xr-x 3 frith frith      4096 2010-03-04 13:27 ..
-rw-r--r-- 1 frith frith 164909860 2010-03-04 13:28 CardinalNewman_dinner.tif
-rwxr-xr-x 1 frith frith     75934 2009-06-17 14:02 import_010518.tif
-rwxr-xr-x 1 frith frith     76568 2009-06-17 14:02 import_010519.tif
-rwxr-xr-x 1 frith frith    171370 2009-06-17 14:04 import_011102.tif
-rwxr-xr-x 1 frith frith    440064 2009-06-16 12:12 pic_008434.tif
-rwxr-xr-x 1 frith frith    422358 2009-06-16 12:12 pic_009022.tif
-rwxr-xr-x 1 frith frith     18900 2009-06-16 21:53 pic_009335.tif
-rwxr-xr-x 1 frith frith    223594 2009-06-16 12:12 pic_009816.tif
frith@sally:~/trouble_images/test$ time exiv2 *.tif > /dev/null 
exiv2: tiffvisitor.cpp:1188: virtual void Exiv2::Internal::TiffReader::visitDirectory(Exiv2::Internal::TiffDirectory*): Assertion `tc.get()' failed.
Aborted

real	0m0.013s
user	0m0.004s
sys	0m0.008s
frith@sally:~/trouble_images/test$ 

What is interesting is that the exiv2 command works fine on any of the images individually, but something seems to happen when doing the whole lot.

The failed assertion is an exiv2 bug. And it's very weird if that happens only when the utility operates on all of the images, but not an individual one.

Can you run exiv2 -pa *.tif and post the output here? If possible with the last image processed before the crash (unless it's the big one...)

Thanks,
Andreas

Created attachment 41308 [details]
TIFF Image that makes digikam 1.2.0 crash

This is an image that seems to trigger a bug in exiv2.

Thanks for the image.

I've created an Exiv2 bug for this: http://dev.exiv2.org/issues/show/684
and checked-in a quickfix with r2034. This change is binary compatible, all you need to do is re-compile and re-install Exiv2, digiKam & Co don't need to be re-built.

The quickfix removes the assertion failure but Exiv2 still only handles the first for sub-IFDs. Further sub-IFDs are not decoded and, more important, if you write to the sample TIFF image using this revision of Exiv2, it may remove sub-IFDs beyond the first four.

Next I'll work on a better solution, but that will not be binary compatible anymore.

Andreas

> Next I'll work on a better solution, but that will not be binary compatible
anymore.

Checked-in Exiv2 r2035 with extended sub-IFD support. Make sure you re-compile digiKam & Co if you use this.

-ahu.

Fixed.

I updated from the SVN the digikam/kdegraphics/exiv2 libraries and now it compiles, installs, loads and correctly displays all the TIF files in my collection.

Thanks for all your help and efforts, this is a great piece of software.

-Frith.