Bug 307602

Created attachment 74240 [details]
large tiff mkes digiKam crash?

Some thoughts:
1.) the pic remained on my PC when HUGIN maks new panorama-pics. 
These panorama pics can reach up to 1 GB when finshed. 
Don't know, if these intermediate pics follow tiff definitions!?
Don't know, why these püic rmains sometimes on the PC

2.) I wonder, if a size limit would resolve the problems I face(d?)

Question / Suggestion:

Maybe it is a good idea to check pics (intermediatly and) continously in background.

GIMP does _not_ crash when opening such huge pics

Maybe there is no such effective error handling in digiKam?

Ok,

Now as you have isolated the tiff file which crash digiKam please share this TIFF with us to test in local. Please use a share free space on the web...

Gilles Caulier

sorry, can't find working free space- space limits, wheras file has 1.7GB

"minus-com"/"sendspace.com" won't work

Idea?


Axel
---
Am 30.09.2012 12:51, schrieb Gilles Caulier:
> https://bugs.kde.org/show_bug.cgi?id=307602
> 
> Gilles Caulier <caulier.gilles@gmail.com> changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |caulier.gilles@gmail.com
> 
> --- Comment #3 from Gilles Caulier <caulier.gilles@gmail.com> ---
> Ok,
> 
> Now as you have isolated the tiff file which crash digiKam please share this
> TIFF with us to test in local. Please use a share free space on the web...
> 
> Gilles Caulier
>

Cut file with rar archover and use a decent files sharing service. "uptobox" for ex is very fast

Gilles Caulier

Am 30.09.2012 14:07, schrieb Gilles Caulier:
> Gilles Caulier
I didn't get it... if you don't mind, let me send you a DVD to a
_postal_ address.


Axel

(> Axel.Krebs@gmx.net)

Hum, it a tol complicated way.

How do to generate this huge TIFF file ? With hugin ? If yes, if original files are JPEG, it's more easy to share it, and let's me to regenerate TIFF with hugin. Right ?

Gilles Caulier

Am 01.10.2012 23:13, schrieb Gilles Caulier:
> https://bugs.kde.org/show_bug.cgi?id=307602
> 
> --- Comment #7 from Gilles Caulier <caulier.gilles@gmail.com> ---
> Hum, it a tol complicated way.
> 
> How do to generate this huge TIFF file ? With hugin ? If yes, if original files
> are JPEG, it's more easy to share it, and let's me to regenerate TIFF with
> hugin. Right ?
> 
> Gilles Caulier
> 
thanks for answer!

You want to reproduce the huge files with HUGIN directly?

As much as I see, sometimes I get those file sizes... sometimes, HUGIN
does no erase intermediate files, and the huge files seem to be such
remainings...

(P.S.: HUGIN: 2011.4.0.cf9be9344356)

Original pics are DSC_4052 .. DSC_4063: 12 pics, minimum size 3,7 MB,
Max 5.5 MB, together 53,2MB. with my sleepery connection, this would
take forever...

Shall we try reproduce HUGINs behavior or resolve digiKams crashes?

Why not sending a DVD with two huge TIF-files _and_ the 12 original JPGs?

The DVD is burned already...



Axel

We must to understand if hugin tiff generator make something wrong, and found the problem in digiKam TIFF loader.

If you want to send me a DVD why not. My mail adress is :

Gilles Caulier, 16 Impasse de la Cascade, 13770 Venelles, France.

Gilles

Marcel, 

I just recieve DVD from Alex with Large tiff files. Problem is fully reproducible with digiKam 3.0.0 :

digikam(22522)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation =>  8
digikam(22522)/digikam (core) Digikam::DMetadata::getImageHistory: Loading image history  ""
digikam(22522)/digikam (core) Digikam::ImageScanner::commit: Scanning took 10 ms
digikam(22522)/digikam (core) Digikam::ImageScanner::~ImageScanner: Finishing took 3 ms
digikam(22522)/digikam (core) Digikam::CollectionScanner::completeHistoryScanning: items to tag ()
digikam(22522)/digikam (core) Digikam::DImg::load: "/mnt/data/photos/TESTS/Alex Krebs/DSC_4052-DSC_40630004.tif"  : TIFF file identified
KCrash: Application 'digikam' crashing...

Thread 5 (Thread 0x7f8d56605700 (LWP 22562)):
[KCrash Handler]
#6  0x00007f8d8e5e5b0a in Digikam::TIFFLoader::load (this=0x7f8d56603f70, filePath=..., observer=0x306d580) at /mnt/devel/GIT/3.x/core/libs/dimg/loaders/tiffloader.cpp:548
#7  0x00007f8d8e5ae7a0 in Digikam::DImg::load (this=0x7f8d566043f0, filePath=..., loadFlagsInt=13, observer=0x306d580, rawDecodingSettings=...) at /mnt/devel/GIT/3.x/core/libs/dimg/dimg.cpp:449
#8  0x00007f8d8e5ae21e in Digikam::DImg::load (this=0x7f8d566043f0, filePath=..., loadMetadata=false, loadICCData=true, loadUniqueHash=false, loadImageHistory=false, observer=0x306d580, rawDecodingSettings=...) at /mnt/devel/GIT/3.x/core/libs/dimg/dimg.cpp:403
#9  0x00007f8d8e78eacb in Digikam::ThumbnailCreator::loadWithDImg (this=0x255de30, path=..., profile=0x7f8d566044c0) at /mnt/devel/GIT/3.x/core/libs/threadimageio/thumbnailcreator.cpp:561
#10 0x00007f8d8e78e2d9 in Digikam::ThumbnailCreator::createThumbnail (this=0x255de30, info=..., detailRect=..., isFace=false) at /mnt/devel/GIT/3.x/core/libs/threadimageio/thumbnailcreator.cpp:490
#11 0x00007f8d8e78ce99 in Digikam::ThumbnailCreator::load (this=0x255de30, path=..., rect=..., pregenerate=false) at /mnt/devel/GIT/3.x/core/libs/threadimageio/thumbnailcreator.cpp:260
#12 0x00007f8d8e78cac8 in Digikam::ThumbnailCreator::load (this=0x255de30, path=...) at /mnt/devel/GIT/3.x/core/libs/threadimageio/thumbnailcreator.cpp:199
#13 0x00007f8d8e79b7ac in Digikam::ThumbnailLoadingTask::execute (this=0x306d570) at /mnt/devel/GIT/3.x/core/libs/threadimageio/thumbnailtask.cpp:170
#14 0x00007f8d8e776ae1 in Digikam::LoadSaveThread::run (this=0x255db70) at /mnt/devel/GIT/3.x/core/libs/threadimageio/loadsavethread.cpp:136
#15 0x00007f8d8e7a2960 in Digikam::DynamicThread::DynamicThreadPriv::run (this=0x255dbe0) at /mnt/devel/GIT/3.x/core/libs/threads/dynamicthread.cpp:186
#16 0x00007f8d89aa74a2 in ?? () from /usr/lib64/libQtCore.so.4
#17 0x00007f8d89ab3c3b in ?? () from /usr/lib64/libQtCore.so.4
#18 0x00007f8d89823b99 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f8d888550cd in clone () from /lib64/libc.so.6
#20 0x0000000000000000 in ?? ()

Exiv2 can write to Jp2: Yes
Exiv2 can write to Jpeg: Yes
Exiv2 can write to Pgf: Yes
Exiv2 can write to Png: Yes
Exiv2 can write to Tiff: Yes
Exiv2 supports XMP metadata: Yes
LibCImg: 130
LibClapack: internal library
LibExiv2: 0.23
LibJPEG: 80
LibJasper: 1.900.1
LibKDE: 4.8.5 (4.8.5)
LibKExiv2: 2.3.0
LibKGeoMap: 2.0.0
LibKdcraw: 2.2.0
LibLCMS: 119
LibLensFun: external shared library
LibLqr: internal library
LibPGF: 6.11.42 - external shared library
LibPNG: 1.5.10
LibQt: 4.8.2
LibRaw: 0.15.0-Alpha4
LibTIFF: LIBTIFF, Version 4.0.1 Copyright (c) 1988-1996 Sam Leffler Copyright (c) 1991-1996 Silicon Graphics, Inc.
Marble Widget: 0.13.4 (stable release)
Parallelized PGF codec: No
Parallelized demosaicing: Yes
RawSpeed codec support: Yes

[gilles@localhost core (master)]$ ls -al "/mnt/data/photos/TESTS/Alex Krebs/"
total 5128152
drwxrwxr-x  3 gilles gilles       4096 oct.  11 13:55 ./
drwxr-xr-x 46 gilles gilles       4096 oct.  11 13:46 ../
-r--r--r--  1 gilles gilles   16614150 sept. 18 07:06 DSC_4052-DSC_40630000.tif
-r--r--r--  1 gilles gilles   32852286 sept. 18 07:06 DSC_4052-DSC_40630001.tif
-r--r--r--  1 gilles gilles 1477235108 sept. 18 07:18 DSC_4052-DSC_40630002.tif
-r--r--r--  1 gilles gilles 1698042226 sept. 18 07:27 DSC_4052-DSC_40630003.tif
-r--r--r--  1 gilles gilles 1823832214 sept. 18 07:37 DSC_4052-DSC_40630004.tif
-r--r--r--  1 gilles gilles  100241338 sept. 18 07:38 DSC_4052-DSC_40630005.tif
-r--r--r--  1 gilles gilles   26953084 sept. 18 07:38 DSC_4052-DSC_40630006.tif
-r--r--r--  1 gilles gilles   22224704 sept. 18 07:38 DSC_4052-DSC_40630007.tif
-r--r--r--  1 gilles gilles    5217105 sept. 16 13:04 DSC_4052.JPG
-r--r--r--  1 gilles gilles    5172771 sept. 16 13:04 DSC_4053.JPG
-r--r--r--  1 gilles gilles    4843119 sept. 16 13:04 DSC_4054.JPG
-r--r--r--  1 gilles gilles    4670839 sept. 16 13:04 DSC_4055.JPG
-r--r--r--  1 gilles gilles    4480745 sept. 16 13:04 DSC_4056.JPG
-r--r--r--  1 gilles gilles    4489894 sept. 16 13:04 DSC_4057.JPG
-r--r--r--  1 gilles gilles    4231733 sept. 16 13:04 DSC_4058.JPG
-r--r--r--  1 gilles gilles    4041280 sept. 16 13:04 DSC_4059.JPG
-r--r--r--  1 gilles gilles    4157543 sept. 16 13:04 DSC_4060.JPG
-r--r--r--  1 gilles gilles    4261651 sept. 16 13:04 DSC_4061.JPG
-r--r--r--  1 gilles gilles    3877228 sept. 16 13:04 DSC_4062.JPG
-r--r--r--  1 gilles gilles    3722625 sept. 16 13:04 DSC_4063.JPG
drwxrwxr-x  2 gilles gilles       4096 oct.   3 11:01 screen/
[gilles@localhost core (master)]$

digiKam crash when image DSC_4052-DSC_40630004.tif is parsed (1,7Gb)

This is not an allocation problem on my computer to load image. My i5 Box has 16 Gb or RAM...

identification of image with ImageMagick :

[gilles@localhost core (master)]$ identify -verbose "/mnt/data/photos/TESTS/Alex Krebs/DSC_4052-DSC_40630004.tif" -version
Image: /mnt/data/photos/TESTS/Alex Krebs/DSC_4052-DSC_40630004.tif
  Format: TIFF (Tagged Image File Format)
  Class: DirectClass
  Geometry: 11939x108859+0+0
  Resolution: 150x150
  Print size: 79.5933x725.727
  Units: PixelsPerInch
  Type: TrueColorMatte
  Base type: TrueColor
  Endianess: MSB
  Colorspace: sRGB
  Depth: 8-bit
  Channel depth:
    red: 8-bit
    green: 8-bit
    blue: 8-bit
    alpha: 1-bit
  Channel statistics:
    Red:
      min: 0 (0)
      max: 255 (1)
      mean: 132.861 (0.521023)
      standard deviation: 94.7031 (0.371385)
      kurtosis: -1.90999
      skewness: -0.0217065
    Green:
      min: 0 (0)
      max: 255 (1)
      mean: 150.436 (0.589946)
      standard deviation: 99.5435 (0.390367)
      kurtosis: -1.90405
      skewness: -0.055626
    Blue:
      min: 0 (0)
      max: 255 (1)
      mean: 154.395 (0.60547)
      standard deviation: 101.835 (0.399353)
      kurtosis: -1.89304
      skewness: -0.0871236
    Alpha:
      min: 0 (0)
      max: 255 (1)
      mean: 247.312 (0.969851)
      standard deviation: 43.6041 (0.170997)
      kurtosis: 28.1999
      skewness: 5.49544
  Image statistics:
    Overall:
      min: 0 (0)
      max: 255 (1)
      mean: 111.345 (0.436647)
      standard deviation: 88.2458 (0.346062)
      kurtosis: -0.166605
      skewness: 0.660303
  Alpha: none   #00000000
  Rendering intent: Perceptual
  Gamma: 0.45455
  Chromaticity:
    red primary: (0.64,0.33)
    green primary: (0.3,0.6)
    blue primary: (0.15,0.06)
    white point: (0.3127,0.329)
  Interlace: None
  Background color: white
  Border color: srgba(223,223,223,1)
  Matte color: grey74
  Transparent color: none
  Compose: Over
  Page geometry: 11939x108859+0+108860
  Origin geometry: +0+108860
  Dispose: Undefined
  Iterations: 0
  Compression: LZW
  Orientation: TopLeft
  Properties:
    date:create: 2012-10-11T13:54:58+02:00
    date:modify: 2012-09-18T07:37:35+02:00
    signature: b1f1e065fc3bd5dd454084b54d283d9e909e73c974f6c29f035f21afb17444d6
    tiff:alpha: unassociated
    tiff:endian: lsb
    tiff:photometric: RGB
    tiff:rows-per-strip: 21
  Artifacts:
    filename: /mnt/data/photos/TESTS/Alex Krebs/DSC_4052-DSC_40630004.tif
    verbose: true
  Tainted: False
  Filesize: 1.8238GB
  Number pixels: 1.2997G
  Pixels per second: 48.23MB
  User time: 25.840u
  Elapsed time: 0:27.950
  Version: ImageMagick 6.7.5-10 2012-08-22 Q16 http://www.imagemagick.org

Marcel,

The crash appears when data are read from tiff and Red and Blue channels are switched :

                uchar* stripPtr = (uchar*)(strip.data());
                uchar* dataPtr  = (uchar*)(data.data() + offset);
                uchar* p;

                // Reverse red and blue

                for (int i = 0; i < pixelsRead; ++i)
                {
                    p = dataPtr;

                    p[2] = *stripPtr++;              // <========== CRASh here !
                    p[1] = *stripPtr++;
                    p[0] = *stripPtr++;
                    p[3] = *stripPtr++;

                    dataPtr += 4;
                }

Why ?

Note : If i want to display image with ImageMagick, it crash too !!!


[gilles@localhost core (master)]$ gdb --args display "/mnt/data/photos/TESTS/Alex Krebs/DSC_4052-DSC_40630004.tif"
GNU gdb (GDB) 7.3.50.20110722-4.mga2 (Mageia release 2)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-mageia-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/display...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/bin/display /mnt/data/photos/TESTS/Alex\ Krebs/DSC_4052-DSC_40630004.tif
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Program received signal SIGBUS, Bus error.
0x00007ffff7a533e0 in SetImageBackgroundColor () from /usr/lib64/libMagickCore.so.5
(gdb) bt
#0  0x00007ffff7a533e0 in SetImageBackgroundColor () from /usr/lib64/libMagickCore.so.5
#1  0x00007ffc5168cff6 in ?? () from /usr/lib64/ImageMagick-6.7.5/modules-Q16/coders/pattern.so
#2  0x00007ffff79d3590 in ReadImage () from /usr/lib64/libMagickCore.so.5
#3  0x00007ffff7b04dd2 in ?? () from /usr/lib64/libMagickCore.so.5
#4  0x00007ffff7b0e70c in XMakeImage () from /usr/lib64/libMagickCore.so.5
#5  0x00007ffff79f9847 in XDisplayImage () from /usr/lib64/libMagickCore.so.5
#6  0x00007ffff76a5810 in DisplayImageCommand () from /usr/lib64/libMagickWand.so.5
#7  0x00007ffff76f06a1 in MagickCommandGenesis () from /usr/lib64/libMagickWand.so.5
#8  0x00000000004007e9 in ?? ()
#9  0x00007ffff70be32d in __libc_start_main () from /lib64/libc.so.6
#10 0x0000000000400839 in ?? ()
#11 0x00007fffffffda68 in ?? ()
#12 0x000000000000001c in ?? ()
#13 0x0000000000000002 in ?? ()
#14 0x00007fffffffdedf in ?? ()
#15 0x00007fffffffdef0 in ?? ()
#16 0x0000000000000000 in ?? ()

The image can be loaded in Gimp, but it's really strange... Image is vertically oriented and completely deformed...

Gilles

Gwenview cannot load image, but do not crash. It report quickly an error in GUI (nothing in the console). Sound like a simple "no support error", not an image decoding error....

Important : i see a lots of fee space fully used here :


[gilles@localhost manager (master)]$ df
Sys. fich.               Taille Util. Dispo Uti% Monté sur
rootfs                      30G   29G     0 100% /
devtmpfs                   7,8G     0  7,8G   0% /dev
tmpfs                      7,8G   17M  7,8G   1% /dev/shm
tmpfs                      7,8G  2,9M  7,8G   1% /run
/dev/sda1                   30G   29G     0 100% /
tmpfs                      7,8G     0  7,8G   0% /sys/fs/cgroup
/dev/sda6                  146G   15G  124G  11% /mnt/devel
/dev/sda3                   20G   19G  4,0K 100% /tmp
/dev/sda5                   20G   15G  5,5G  72% /home
/dev/sdd1                  1,9T  508G  1,3T  29% /mnt/data
/dev/mapper/lvm--video-1   5,5T  2,1T  3,2T  40% /mnt/videos
[gilles@localhost manager (master)]$

In fact / and /tmp are full where normally i have a full of free space to work...

Gilles

In /tmp, IM has make some dirty think :


[root@localhost manager (master)]# du -hms /tmp/* | sort -nr | head
9916    /tmp/magick-l66Ht8cw
8572    /tmp/magick-zlaovCfC
16      /tmp/magick-EkjCOp7E
6       /tmp/kde-gilles
1       /tmp/systemd-namespace-zlJBOl
1       /tmp/systemd-namespace-XYqfQi
1       /tmp/systemd-namespace-X36Tsv
1       /tmp/systemd-namespace-uMQ6Tj
1       /tmp/systemd-namespace-UHtbR2
1       /tmp/systemd-namespace-Q8z6Bu

2 first files are giant (:=))

Gilles Caulier

on /, i can see this :

[root@localhost /]# find . -ls | awk '{printf(" %12d %s \n ",$7 ,$11)}' | sort -nr | head
  140737486266368 ./proc/kcore

... probably due to memory lost with IM crash...

Gilles Caulier

All buffers are allocated in lines 461/2:
            data.reset(new_failureTolerant(w * h * 4));
            QScopedArrayPointer<uchar> strip(new_failureTolerant(w * rows_per_strip * 4));
and checked later.

Does it crash already on the first iteration, or sometime late during the decoding?
Is it possible to get a valgrind trace?

Am 11.10.2012 20:54, schrieb Marcel Wiesweg:
> https://bugs.kde.org/show_bug.cgi?id=307602
> 
> --- Comment #18 from Marcel Wiesweg <marcel.wiesweg@gmx.de> ---
> All buffers are allocated in lines 461/2:
>             data.reset(new_failureTolerant(w * h * 4));
>             QScopedArrayPointer<uchar> strip(new_failureTolerant(w *
> rows_per_strip * 4));
> and checked later.
> 
> Does it crash already on the first iteration, or sometime late during the
> decoding?
> Is it possible to get a valgrind trace?
> 
Marcel,

you address to me. Unfortunatly, I am not able to analyse software. What
do you mean with "iteration"? I observed crashes, when digiKam was
idling som tim...


Axel

Marcel,

I will take a valgrind trace.

I will trying to put file on the web for you to download it. It's take time...

Gilles

Marcel,

I shared with you in private tiff file through Google Drive web service. Do you recieve the url ?

Gilles

Axel: all this was for Gilles.
Gilles: Downloading the file

I've got a new camera D800 with 36 mio pixels.
It looks like large pixels-numbers can improve doing panoramas.

First observation: all takes huge space and _much_ more time.

Storing times in digiKam are _much_ longer, even when doing (only) small
modifictions in picture editor.
Maybe a new storage concept could overcome this potential handicap? I think about
working with virtual pics-copies and when finishing work on pics, the
output will be written on disk- not before.

When working with tags (geotagging, e.g.), it feels _much_ slowlier than with smaller pics (14, 3 as before). I do not understand why, as metadata should scale constant (that is independent form pic size).

Crashes occour, when HUGIN as an external program finishes, as described
previously (bug #...?).

When finishing digiKam, I still get a crash event rather than simply closing (bug #...?).

?? Do you need specific information, some minimal statistics, maybe?
Loadings times, storing times for small (7 mio)  medium size (14 mio)  and large pics (30 mio)??

Have a nice Sunday-



Axel

My statistics right now:
digiKam version 2.8.0
Bilder:
BMP: 3
GIF: 99
JP2: 14
JPEG: 273
JPG: 118942
NIKON-NEF: 640
PGF: 2
PNG: 1608
RAW-CR2: 632
RAW-CRW: 11258
RAW-DNG: 7
RAW-NEF: 80755
TIFF: 1780
XCF: 1
Gesamt: 216014
:
:
Videos:
AVI: 60
MOV: 23
MPEG: 3
Gesamt: 86
:
:
Gesamtzahl der Einträge: 216100
Alben: 2222
Stichwörter: 109
Datenbanktreiber: QSQLITE

*** Bug 308572 has been marked as a duplicate of this bug. ***

==20316== Invalid write of size 1
==20316==    at 0x7C7DB9B: Digikam::TIFFLoader::load(QString const&, Digikam::DImgLoaderObserver*) (tiffloader.cpp:548)
==20316==    by 0x7C5AA53: Digikam::DImg::load(QString const&, int, Digikam::DImgLoaderObserver*, Digikam::DRawDecoding const&) (dimg.cpp:449)
==20316==    by 0x7E2AD9D: Digikam::ThumbnailCreator::loadWithDImg(QString const&, Digikam::IccProfile*) const (thumbnailcreator.cpp:561)
==20316==    by 0x7E2BED7: Digikam::ThumbnailCreator::createThumbnail(Digikam::ThumbnailInfo const&, QRect const&, bool) const (thumbnailcreator.cpp:490)
==20316==    by 0x7E2F692: Digikam::ThumbnailCreator::load(QString const&, QRect const&, bool) const (thumbnailcreator.cpp:260)
==20316==    by 0x7E304F1: Digikam::ThumbnailCreator::load(QString const&) const (thumbnailcreator.cpp:199)
==20316==    by 0x7E3DEC6: Digikam::ThumbnailLoadingTask::execute() (thumbnailtask.cpp:170)
==20316==    by 0x7E1243D: Digikam::LoadSaveThread::run() (loadsavethread.cpp:136)
==20316==    by 0x7E4FACD: Digikam::DynamicThread::DynamicThreadPriv::run() (dynamicthread.cpp:186)
==20316==    by 0xB605A9C: QThreadPoolThread::run() (qthreadpool.cpp:107)
==20316==    by 0xB611E8A: QThreadPrivate::start(void*) (qthread_unix.cpp:307)
==20316==    by 0xBA6BE0D: start_thread (in /lib64/libpthread-2.15.so)
==20316==  Address 0xeb502a86 is 2 bytes after a block of size 903,703,108 alloc'd
==20316==    at 0x4C2A147: operator new[](unsigned long) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==20316==    by 0x7C76E21: Digikam::DImgLoader::new_failureTolerant(unsigned long) (dimgloader.h:147)
==20316==    by 0x7C7D1A7: Digikam::TIFFLoader::load(QString const&, Digikam::DImgLoaderObserver*) (tiffloader.cpp:461)
==20316==    by 0x7C5AA53: Digikam::DImg::load(QString const&, int, Digikam::DImgLoaderObserver*, Digikam::DRawDecoding const&) (dimg.cpp:449)
==20316==    by 0x7E2AD9D: Digikam::ThumbnailCreator::loadWithDImg(QString const&, Digikam::IccProfile*) const (thumbnailcreator.cpp:561)
==20316==    by 0x7E2BED7: Digikam::ThumbnailCreator::createThumbnail(Digikam::ThumbnailInfo const&, QRect const&, bool) const (thumbnailcreator.cpp:490)
==20316==    by 0x7E2F692: Digikam::ThumbnailCreator::load(QString const&, QRect const&, bool) const (thumbnailcreator.cpp:260)
==20316==    by 0x7E304F1: Digikam::ThumbnailCreator::load(QString const&) const (thumbnailcreator.cpp:199)
==20316==    by 0x7E3DEC6: Digikam::ThumbnailLoadingTask::execute() (thumbnailtask.cpp:170)
==20316==    by 0x7E1243D: Digikam::LoadSaveThread::run() (loadsavethread.cpp:136)
==20316==    by 0x7E4FACD: Digikam::DynamicThread::DynamicThreadPriv::run() (dynamicthread.cpp:186)
==20316==    by 0xB605A9C: QThreadPoolThread::run() (qthreadpool.cpp:107)

Am 17.10.2012 21:00, schrieb Gilles Caulier:
> https://bugs.kde.org/show_bug.cgi?id=307602
> 
> --- Comment #24 from Gilles Caulier <caulier.gilles@gmail.com> ---
> *** Bug 308572 has been marked as a duplicate of this bug. ***
> 
How can bug #308572 be "duplicate of bug #307602", if the 1 GB pics have
been taken away from my pics collection?


Axel

Perhaps another image has same dysfunction. At least, it's always the same libtiff error...

Gilles Caulier

Am 18.10.2012 10:28, schrieb Gilles Caulier:
> https://bugs.kde.org/show_bug.cgi?id=307602
> 
> --- Comment #27 from Gilles Caulier <caulier.gilles@gmail.com> ---
> Perhaps another image has same dysfunction. At least, it's always the same
> libtiff error...
> 
> Gilles Caulier
> 
If problem is understood, it should be possible to find suspicious files
which cause this type of errors inversely?

And, if, digiKam could (and should!!) avoid "touching those files". Right?


Axel

For me the crash appear when files are scanned. So files are not touched by digiKam. Scan is read only.

If we found the problem in Tiff loader (used to scan file properties, load thumbnail, and load image, we can avoid (corrupted ?) tiff items to prevent future problem.

Gilles Caulier

This was a tricky bug. The information from valgrind gives the clue: "a block of size 903,703,108". In fact, the image needs about 5,9 GB of memory. This memory is computed: w*h*4. And the TIFF loader uses a 32 bit type for storing width and height!

Git commit 767a14c626384d646276972c531e9a617e36face by Marcel Wiesweg.
Committed on 18/10/2012 at 22:18.
Pushed by mwiesweg into branch 'master'.

Prevent integer overflows when calculating the memory necessary for loading an image.

This can happen particularly for images requiring chunks of >4GB and when using a 32 bit
variable to store width and height. Worst case, a smaller chunk of memory was allocated,
resulting in a crash.
FIXED-IN: 3.0.0

M  +5    -1    NEWS
M  +10   -0    libs/dimg/loaders/dimgloader.cpp
M  +21   -0    libs/dimg/loaders/dimgloader.h
M  +2    -2    libs/dimg/loaders/jp2kloader.cpp
M  +1    -1    libs/dimg/loaders/jpegloader.cpp
M  +2    -2    libs/dimg/loaders/pgfloader.cpp
M  +2    -2    libs/dimg/loaders/pngloader.cpp
M  +1    -1    libs/dimg/loaders/qimageloader.cpp
M  +2    -2    libs/dimg/loaders/rawloader.cpp
M  +3    -3    libs/dimg/loaders/tiffloader.cpp

http://commits.kde.org/digikam/767a14c626384d646276972c531e9a617e36face

A sidenote: It's clear that digikam should not crash here, but it is not designed with such large files in mind. It will probably work with a 64bit Linux and sufficient physical memory, but we still allocate this as one contiguous chunk, which can be a problem even with enough free memory. For proper support of such large files, tiled memory allocation would be necessary which we dont support.

Marcel,

Qt API do not provide any storage container to :

1/ compress data to optimize memory allocation of RAW data ?
2/ tile access of data from a file stream ?

For 1/, if we use QByteArray into DImg to store image data instead a uchar pointer, we can compress it :

http://qt-project.org/doc/qt-4.8/qbytearray.html#qCompress

QByteArray will also prevent memory allocation problem...

For 2/, i have no idea...

Gilles

If we change the data layout of DImg, we need to adapt everything access the image data, most notably the image filters. It's a bit of work, but possible.
I believe the path to choose would be to encapsulate the added complexity in an iterator class:
The underlying data storage in memory is intransparent through the DImg class. The method returning an uchar* are removed, instead they return an Iterator class. This iterator offers a few accessors:
uchar* row(int r), uchar* operator[](uint byte). This must be very carefully designed as it has enormous performance impact. I believe Krita has adopted a similar scheme, dont know how complex it is implemented.
Regarding in-memory compression, which also requires an extra access layer like described above, there are well-suited algorithms available, most notably LZO: 
http://www.oberhumer.com/opensource/lzo/
We would need to verify that this algorithm is also well-suited for our special case of uncompressed, natural image data.

For both tiling and in-memory compression, we'd need to support inside the implementation to use it or not use it for a DImg, depending on the use case (small image vs. very large image, larger image but loaded to image editor, smaller image but stored in cache etc.)

Interesting project, and needs benchmarking all along the way.

LZ does not compress well with natural images; you can expect 90% of original file size. They are fast to decompress, but only because they do "memcpy" 90% of the time. For benchmarks of photo compression, see http://www.imagecompression.info/gralic/LPCB.html

Thanks Christoph for this benchmark.

From my side, i have experimented some format. To compress image the best ratio will be given by Wavelets algorithm in lossless :

- PGF : need to be improved around OpenMP support. Good candidate (GPL)
- Jasper : JPEG 2000 : good but very time consuming due to floating computation. Not optimized and now, not maintained (GPL). Don't use...
- OpenJPEG : another JPEG 2000 implementation. Very well maintained, but not yet optimized. In progress. (GPL).
- WebP : from Google, to replace JPEG on web. fast and Open-Source like compatible, but patents still active ?
- JPEG_XR : from Microsoft. fast and optimized, but not open source compatible. Re-usable only. 20 active patents... Note that it's have been selected by "JPEG organization" to replace current JPEG in digital still cameras. The war is open (:=)))... In all case, to ignore here...

Gilles Caulier

We are talking about in-memory compression, which means the uncompressed image data is recompressed to save memory, but needs to be uncompressed each time the image data is used (viewing, editing). That means the performance must be suitable. Any sophisticated compression will be so slow that it cannot play a role; LZO would be acceptable in speed, but obviously does not give any usable benefit with our use case. Here comes ImageZero, which will give a benefit regarding compression. 
The performance number given is 180MB/s for compression or decompression.
A 25 MPx image in 16 bit needs roughly these 180 MB of memory.
There are real-life situations where applying this compression to images in our in-memory cache could give us a benefit, but there are as many situations to be expected where we win nothing.
Regarding the problem of extremely large images: Axel's example here would need 30 seconds for compression, and another 30 seconds for decompression.

The "180 MB/s" figure is missleading; it should probably say "50 cycles per pixel". If your pixels are 48 bit rather than 24 bit, you would get 300 MB/s per core on a 2.5 gigacycles/s CPU.

On the other hand, 48 bit pixels are not as compressible as 24 bit pixels. You probably would end up compressing only the higher 8 bits per component, and store the lower 8 bits uncompressed. But I never did any tests with 48 bit, due to lack of representative data.

It seems I reopend this bug myself. I didnt mean to.

*** Bug 312063 has been marked as a duplicate of this bug. ***