>there is some way to know from where the data is feed into libpgf?

I/O PGF image data are wrapped to QImage in this source code :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/threadimageio/pgfutils.cpp

And managed in DB with this class :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/threadimageio/thumbnailcreator.cpp#L886

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/threadimageio/thumbnailcreator.cpp#L886

>BTW the crash happen even with external libpgf


Are you sure, because in your backtrace, you use libpgf from 3rdparty digiKam core...

Gilles Caulier

Are steps 1) to 3) really necessary to reproduce? Could it be there is "simply" a PGF file produced by digikam which crashes digikam when opened?

Created attachment 60243 [details]
getThumbs.py extract pgf thumbnail from database

(In reply to comment #1)
[snip]
> >BTW the crash happen even with external libpgf
> 
> Are you sure, because in your backtrace, you use libpgf from 3rdparty digiKam
> core...

yes, I've recompiled digiKam with internal libpgf only later and the backtrace is the same
I'm digging the sources right now at least to understund how it crashes, thanks for the pointers

(In reply to comment #2)
> Are steps 1) to 3) really necessary to reproduce? Could it be there is "simply"
> a PGF file produced by digikam which crashes digikam when opened?

No, that's how the problem generated initially, there is _only_ a pgf broken:
http://xn--wtf-xs0a.ws/Bug273765/_dsc4243.pgf

I've also extracted all thumbnails from the database to pgf files and they are all readable

I'm looking into showfoto, since it's reproducible there and there are less threads involved ;)

Self-compiled without external libpgf crashes when I press F4 to edit the image. Preview etc. works ok.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xa2b2ab70 (LWP 32539)]
0xb672036b in ClearBitBlock (stream=0xa2b28724, pos=73170, len=16384)
    at /home/rabauke/devel/src/digikam-software-compilation/core/libs/3rdparty/libpgf/BitStream.h:159
159                             stream[i] = 0;
(gdb) bt
#0  0xb672036b in ClearBitBlock (stream=0xa2b28724, pos=73170, len=16384)
    at /home/rabauke/devel/src/digikam-software-compilation/core/libs/3rdparty/libpgf/BitStream.h:159
#1  0xb671ff7b in CDecoder::RLDSigsAndSigns (this=0xa1ac3738, bufferSize=16384, codeLen=522, sigBits=0xa2b28724, signBits=0xa2b27f24)
    at /home/rabauke/devel/src/digikam-software-compilation/core/libs/3rdparty/libpgf/Decoder.cpp:726
#2  0xb671f9e6 in CDecoder::BitplaneDecode (this=0x0, bufferSize=0)
    at /home/rabauke/devel/src/digikam-software-compilation/core/libs/3rdparty/libpgf/Decoder.cpp:536
#3  0x00000000 in ?? ()
(gdb) 292056 FLAGS ($IGNORED)

#0  0xb672036b in ClearBitBlock (stream=0xa2b28724, pos=73170, len=16384)
    at /home/rabauke/devel/src/digikam-software-compilation/core/libs/3rdparty/libpgf/BitStream.h:159
#1  0xb671ff7b in CDecoder::RLDSigsAndSigns (this=0xa1ac3738, bufferSize=16384, codeLen=522, sigBits=0xa2b28724, signBits=0xa2b27f24)
    at /home/rabauke/devel/src/digikam-software-compilation/core/libs/3rdparty/libpgf/Decoder.cpp:726
#2  0xb671f9e6 in CDecoder::BitplaneDecode (this=0x0, bufferSize=0)
    at /home/rabauke/devel/src/digikam-software-compilation/core/libs/3rdparty/libpgf/Decoder.cpp:536
#3  0x00000000 in ?? ()


konsole output:

QSqlDatabasePrivate::removeDatabase: connection 'ConnectionTest' is still in use, all queries will cease to work.
digikam(32499)/KIPI (loading) KIPI::PluginLoader::Info::plugin: KIPI::PluginLoader:: createInstance returned 0 for  "Kalender"  ( "kipiplugin_calendar" )  with error:  "Could not find plugin 'Kalender' for application 'digikam'" 
digikam(32499)/KIPI (loading) KIPI::PluginLoader::Info::plugin: KIPI::PluginLoader:: createInstance returned 0 for  "DNGkonverter"  ( "kipiplugin_dngconverter" )  with error:  "Could not find plugin 'DNGkonverter' for application 'digikam'" 
Exiv2::PgfImage::readMetadata: Reading PGF file /home/rabauke/Pictures/_dsc4243.pgf
Exiv2::PgfImage: PGF header size : 79242 bytes
Exiv2::PgfImage::readMetadata: Found Image data (79226 bytes)
Exiv2::PgfImage::readMetadata: Reading PGF file /home/rabauke/Pictures/_dsc4243.pgf
Exiv2::PgfImage: PGF header size : 79242 bytes
Exiv2::PgfImage::readMetadata: Found Image data (79226 bytes)
Exiv2::PgfImage::readMetadata: Reading PGF file /home/rabauke/Pictures/_dsc4243.pgf
Exiv2::PgfImage: PGF header size : 79242 bytes
Exiv2::PgfImage::readMetadata: Found Image data (79226 bytes)
Exiv2::PgfImage::readMetadata: Reading PGF file /home/rabauke/Pictures/_dsc4243.pgf
Exiv2::PgfImage: PGF header size : 79242 bytes
Exiv2::PgfImage::readMetadata: Found Image data (79226 bytes)
Exiv2::PgfImage::readMetadata: Reading PGF file /home/rabauke/Pictures/_dsc4243.pgf
Exiv2::PgfImage: PGF header size : 79242 bytes
Exiv2::PgfImage::readMetadata: Found Image data (79226 bytes)
Exiv2::PgfImage::readMetadata: Reading PGF file /home/rabauke/Pictures/_dsc4243.pgf
Exiv2::PgfImage: PGF header size : 79242 bytes
Exiv2::PgfImage::readMetadata: Found Image data (79226 bytes)
Exiv2::PgfImage::readMetadata: Reading PGF file /home/rabauke/Pictures/_dsc4243.pgf
Exiv2::PgfImage: PGF header size : 79242 bytes
Exiv2::PgfImage::readMetadata: Found Image data (79226 bytes)
Speicherzugriffsfehler

(memory corruption)

digiKam version 2.0.0-beta6
Exiv2 can write to Jp2: Yes
Exiv2 can write to Jpeg: Yes
Exiv2 can write to Pgf: Yes
Exiv2 can write to Png: Yes
Exiv2 can write to Tiff: Yes
Exiv2 supports XMP metadata: Yes
LibCImg: 130
LibClapack: internal library
LibExiv2: 0.20
LibJPEG: 62
LibJasper: 1.900.1
LibKDE: 4.6.3 (4.6.3)
LibKExiv2: 2.0.0
LibKMap: 2.0.0
LibKdcraw: 2.0.0
LibLCMS: 119
LibPGF: 6.09.44 - internal library
LibPNG: 1.4.4
LibQt: 4.7.3
LibRaw: 0.13.5
LibTIFF: LIBTIFF, Version 3.9.4 Copyright (c) 1988-1996 Sam Leffler Copyright (c) 1991-1996 Silicon Graphics, Inc.
Marble Widget: 0.11.2 (Stable Release)
Parallelized demosaicing: Yes
Database backend: QSQLITE
LibGphoto2: 2.4.10
LibKface: 2.0.0
LibKipi: 1.2.0
LibOpenCV: 2.2.0
Libface: 0.2

ok, after a lot of learning of gdb, and some reading on signals and exceptions and also pgf specs:

There are two parts of this bug:
The first is a bad pgf file generated from the batch queque manager, I would consider this a non issue, since there was intervention from the shell while digikam was still running

The second is a unpleasant crash when a corrupted file is loaded:

The file is corrupted (redoing it give a correct one #1), _dsc4243.pgf headers are correct #2 so it pass all initial checks and then if asserts are enabled it trigger
ASSERT(sigPos + runlen <= BufferSize);
in Decoder.cpp:725 CDecoder::RLDSigsAndSigns(...)

For this example it's impossible to catch the problem early as it's done for other things in pgfloader.cpp

Seem to me that the only solutions available are:
- rewrite libpgf to handle exceptions (slowing it down) and to fit better in a qt env (no no no)
- fork an external process to read te image (like plasma does with plasmoids) which can be inefficient at best
- register a SIGINT catcher
- close as wont fix

any ideas?

Note to self: converting .nef in batch resize+pgf does not give byte exact files if run twice there are usually at least 3 or 4 bytes that differ

#1 http://xn--wtf-xs0a.ws/Bug273765/_dsc4243_ok.pgf
#2 I've created a structure for okteta available here:
http://kde-files.org/content/show.php/PGF+file+header+definition?content=142016

(In reply to comment #5)
> - rewrite libpgf to handle exceptions (slowing it down) and to fit better in a

PGFplatform.h already contain the code below:

#ifndef ASSERT
	#ifdef _DEBUG
		#define ASSERT(x)	assert(x)
	#else
		#if defined(__GNUC__)
			#define ASSERT(ignore)((void) 0) 
		#elif _MSC_VER >= 1300 
			#define ASSERT		__noop
		#else
			#define ASSERT ((void)0)
		#endif
	#endif //_DEBUG
#endif //ASSERT

Maybe something easy can be added here and the Decoder.ccp line //ASSERT(sigPos + runlen <= BufferSize); uncommented

now something what?

Francesco,

I CC Raphael Schweizer, who is LibPGF author and maintainer. We will see if he can help us to hack this indeep problem in PGF library.

PS: Can you check if valgrind give more info about libpgf problem ?

Gilles Caulier

Created attachment 60435 [details]
valgrind-broken-pgf.zip

valgrind-broken-pgf.zip include the output from:

valgrind --tool=memcheck --leak-check=full \
  --error-limit=no --suppressions=project/digikam.supp \
  showfoto --nocrashhandler _dsc4243_ok.pgf \
  > valgrind-pgf-ok-1.txt \
  2> valgrind-pgf-ok-2.txt

and similar for the broken pdf

lines 273-335 of valgrind-pgf-bad-2.txt show the program starting to crash there is a "Invalid write of size 4" followed by "Invalid read of size 4"

"Conditional jump or move depends on uninitialised value" it's an error that in the opening of a "good" pgf file happen only once, opening the bad pgf it's ubiquitus

libpgf should not crash, but crashes happen (unexpected by the programmer) and are fixed when seen. 

But it must never never never abort because of an error that it has seen and has checked for (expected by the programmer).
It must fail loading the invalid file, but not crash the whole process!!

Francesco,

thanks for reporting this issue and gathering all the helpful details.
We will investigate further and provide a patch shortly.

Raphael

Thanks Raphael. I may also once again bring this image to your attention:
http://digikam3rdparty.free.fr/TEST_IMAGES/PGF/DSC03274_v1.pgf
which is also created by digikam and crashes at the same place

Valgrind:

==14132== Conditional jump or move depends on uninitialised value(s)
==14132==    at 0x7FBC016: CDecoder::ComposeBitplane(unsigned int, unsigned 
int, unsigned int*, unsigned int*, unsigned int*) (BitStream.h:210)
==14132==    by 0x7FBC86B: CDecoder::BitplaneDecode(unsigned int) 
(Decoder.cpp:603)
==14132==    by 0x7FBCAA9: CDecoder::DecodeBuffer() (Decoder.cpp:451)
==14132==    by 0x7FBCB41: CDecoder::DequantizeValue(CSubband*, unsigned int, 
int) (Decoder.cpp:394)
==14132==    by 0x7FBD036: CDecoder::Partition(CSubband*, int, int, int, int, 
int) (Decoder.cpp:213)
==14132==    by 0x7FC4D75: CSubband::PlaceTile(CDecoder&, int, bool, unsigned 
int, unsigned int) (Subband.cpp:227)
==14132==    by 0x7FBF69D: CPGFImage::Read(int, bool (*)(double, bool, void*), 
void*) (PGFimage.cpp:286)

==14132== 
==14132== Conditional jump or move depends on uninitialised value(s)
==14132==    at 0x7FBC71A: CDecoder::BitplaneDecode(unsigned int) 
(Decoder.cpp:526)
==14132==    by 0x7FBCAA9: CDecoder::DecodeBuffer() (Decoder.cpp:451)
==14132==    by 0x7FBCB41: CDecoder::DequantizeValue(CSubband*, unsigned int, 
int) (Decoder.cpp:394)
==14132==    by 0x7FBD036: CDecoder::Partition(CSubband*, int, int, int, int, 
int) (Decoder.cpp:213)
==14132==    by 0x7FC4D75: CSubband::PlaceTile(CDecoder&, int, bool, unsigned 
int, unsigned int) (Subband.cpp:227)
==14132==    by 0x7FBF69D: CPGFImage::Read(int, bool (*)(double, bool, void*), 
void*) (PGFimage.cpp:286)
==14132==    by 0x7E079B3: Digikam::PGFLoader::load(QString const&, 
Digikam::DImgLoaderObserver*) (pgfloader.cpp:290)

==14132== 
==14132== Conditional jump or move depends on uninitialised value(s)
==14132==    at 0x7FBC162: CDecoder::RLDSigsAndSigns(unsigned int, unsigned 
int, unsigned int*, unsigned int*) (Decoder.cpp:690)
==14132==    by 0x7FBC911: CDecoder::BitplaneDecode(unsigned int) 
(Decoder.cpp:536)
==14132==    by 0x7FBCAA9: CDecoder::DecodeBuffer() (Decoder.cpp:451)
==14132==    by 0x7FBCB41: CDecoder::DequantizeValue(CSubband*, unsigned int, 
int) (Decoder.cpp:394)
==14132==    by 0x7FBD036: CDecoder::Partition(CSubband*, int, int, int, int, 
int) (Decoder.cpp:213)
==14132==    by 0x7FC4D75: CSubband::PlaceTile(CDecoder&, int, bool, unsigned 
int, unsigned int) (Subband.cpp:227)
==14132==    by 0x7FBF69D: CPGFImage::Read(int, bool (*)(double, bool, void*), 
void*) (PGFimage.cpp:286)

==14132== 
==14132== Conditional jump or move depends on uninitialised value(s)
==14132==    at 0x7FBC183: CDecoder::RLDSigsAndSigns(unsigned int, unsigned 
int, unsigned int*, unsigned int*) (Decoder.cpp:691)
==14132==    by 0x7FBC911: CDecoder::BitplaneDecode(unsigned int) 
(Decoder.cpp:536)
==14132==    by 0x7FBCAA9: CDecoder::DecodeBuffer() (Decoder.cpp:451)
==14132==    by 0x7FBCB41: CDecoder::DequantizeValue(CSubband*, unsigned int, 
int) (Decoder.cpp:394)
==14132==    by 0x7FBD036: CDecoder::Partition(CSubband*, int, int, int, int, 
int) (Decoder.cpp:213)
==14132==    by 0x7FC4D75: CSubband::PlaceTile(CDecoder&, int, bool, unsigned 
int, unsigned int) (Subband.cpp:227)
==14132==    by 0x7FBF69D: CPGFImage::Read(int, bool (*)(double, bool, void*), 
void*) (PGFimage.cpp:286)

==14132== 
==14132== Conditional jump or move depends on uninitialised value(s)
==14132==    at 0x7FBC21E: CDecoder::RLDSigsAndSigns(unsigned int, unsigned 
int, unsigned int*, unsigned int*) (Decoder.cpp:699)
==14132==    by 0x7FBC911: CDecoder::BitplaneDecode(unsigned int) 
(Decoder.cpp:536)
==14132==    by 0x7FBCAA9: CDecoder::DecodeBuffer() (Decoder.cpp:451)
==14132==    by 0x7FBCB41: CDecoder::DequantizeValue(CSubband*, unsigned int, 
int) (Decoder.cpp:394)
==14132==    by 0x7FBD036: CDecoder::Partition(CSubband*, int, int, int, int, 
int) (Decoder.cpp:213)
==14132==    by 0x7FC4D75: CSubband::PlaceTile(CDecoder&, int, bool, unsigned 
int, unsigned int) (Subband.cpp:227)
==14132==    by 0x7FBF69D: CPGFImage::Read(int, bool (*)(double, bool, void*), 
void*) (PGFimage.cpp:286)

And when the reader bug is fixed (and the image is indeed invalid), the next bug to hunt is in the writer, when these images are created ;-)

As a hint for the writer bug: valgrind gives the following eight errors when saving a PGF.
Uninitialized value problems tend to work in 99% of cases but break for the rest (could explain this bug is rarely seen)

==28252== Conditional jump or move depends on uninitialised value(s)
==28252==    at 0x80A6813: CEncoder::RLESigsAndSigns(unsigned int*, unsigned int, unsigned int*, unsigned int) (BitStream.h:203)
==28252==    by 0x80A7054: CEncoder::BitplaneEncode(unsigned int) (Encoder.cpp:380)
==28252==    by 0x80A72F7: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:274)
==28252==    by 0x80A7461: CEncoder::WriteValue(CSubband*, int) (Encoder.cpp:253)
==28252==    by 0x80A7544: CEncoder::Partition(CSubband*, int, int, int, int) (Encoder.cpp:152)
==28252==    by 0x80AE01D: CSubband::ExtractTile(CEncoder&, int, bool, unsigned int, unsigned int) (Subband.cpp:188)
==28252==    by 0x80A956C: CPGFImage::Write(CPGFStream*, int, bool (*)(double, bool, void*), unsigned int*, void*) (PGFimage.cpp:922)

==28252== 
==28252== Conditional jump or move depends on uninitialised value(s)
==28252==    at 0x80A686A: CEncoder::RLESigsAndSigns(unsigned int*, unsigned int, unsigned int*, unsigned int) (Encoder.cpp:628)
==28252==    by 0x80A7054: CEncoder::BitplaneEncode(unsigned int) (Encoder.cpp:380)
==28252==    by 0x80A72F7: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:274)
==28252==    by 0x80A7461: CEncoder::WriteValue(CSubband*, int) (Encoder.cpp:253)
==28252==    by 0x80A7544: CEncoder::Partition(CSubband*, int, int, int, int) (Encoder.cpp:152)
==28252==    by 0x80AE01D: CSubband::ExtractTile(CEncoder&, int, bool, unsigned int, unsigned int) (Subband.cpp:188)
==28252==    by 0x80A956C: CPGFImage::Write(CPGFStream*, int, bool (*)(double, bool, void*), unsigned int*, void*) (PGFimage.cpp:922)

==28252== 
==28252== Syscall param write(buf) points to uninitialised byte(s)
==28252==    at 0xBA6131D: ??? (in /lib64/libpthread-2.11.3.so)
==28252==    by 0x80AD921: CPGFFileStream::Write(int*, void*) (PGFplatform.h:510)
==28252==    by 0x80A7352: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:311)
==28252==    by 0x80A7461: CEncoder::WriteValue(CSubband*, int) (Encoder.cpp:253)
==28252==    by 0x80A7544: CEncoder::Partition(CSubband*, int, int, int, int) (Encoder.cpp:152)
==28252==    by 0x80AE01D: CSubband::ExtractTile(CEncoder&, int, bool, unsigned int, unsigned int) (Subband.cpp:188)
==28252==    by 0x80A956C: CPGFImage::Write(CPGFStream*, int, bool (*)(double, bool, void*), unsigned int*, void*) (PGFimage.cpp:922)
==28252==  Address 0x359a3b35 is on thread 5's stack


==28252== Conditional jump or move depends on uninitialised value(s)
==28252==    at 0x80A6AB6: CEncoder::RLESigns(unsigned int*, unsigned int) (BitStream.h:233)
==28252==    by 0x80A7133: CEncoder::BitplaneEncode(unsigned int) (Encoder.cpp:418)
==28252==    by 0x80A72F7: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:274)
==28252==    by 0x80A7461: CEncoder::WriteValue(CSubband*, int) (Encoder.cpp:253)
==28252==    by 0x80A7544: CEncoder::Partition(CSubband*, int, int, int, int) (Encoder.cpp:152)
==28252==    by 0x80AE01D: CSubband::ExtractTile(CEncoder&, int, bool, unsigned int, unsigned int) (Subband.cpp:188)
==28252==    by 0x80A963B: CPGFImage::Write(CPGFStream*, int, bool (*)(double, bool, void*), unsigned int*, void*) (PGFimage.cpp:932)

==28252== 
==28252== Conditional jump or move depends on uninitialised value(s)
==28252==    at 0x80A6A9C: CEncoder::RLESigns(unsigned int*, unsigned int) (BitStream.h:233)
==28252==    by 0x80A7133: CEncoder::BitplaneEncode(unsigned int) (Encoder.cpp:418)
==28252==    by 0x80A72F7: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:274)
==28252==    by 0x80A7461: CEncoder::WriteValue(CSubband*, int) (Encoder.cpp:253)
==28252==    by 0x80A7544: CEncoder::Partition(CSubband*, int, int, int, int) (Encoder.cpp:152)
==28252==    by 0x80AE01D: CSubband::ExtractTile(CEncoder&, int, bool, unsigned int, unsigned int) (Subband.cpp:188)
==28252==    by 0x80A966F: CPGFImage::Write(CPGFStream*, int, bool (*)(double, bool, void*), unsigned int*, void*) (PGFimage.cpp:933)

==28252== 
==28252== Conditional jump or move depends on uninitialised value(s)
==28252==    at 0x80A6813: CEncoder::RLESigsAndSigns(unsigned int*, unsigned int, unsigned int*, unsigned int) (BitStream.h:203)
==28252==    by 0x80A7054: CEncoder::BitplaneEncode(unsigned int) (Encoder.cpp:380)
==28252==    by 0x80A72F7: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:274)
==28252==    by 0x80A76BC: CEncoder::Flush() (Encoder.cpp:216)
==28252==    by 0x80A96D1: CPGFImage::Write(CPGFStream*, int, bool (*)(double, bool, void*), unsigned int*, void*) (PGFimage.cpp:953)

==28252== Conditional jump or move depends on uninitialised value(s)
==28252==    at 0x80A686A: CEncoder::RLESigsAndSigns(unsigned int*, unsigned int, unsigned int*, unsigned int) (Encoder.cpp:628)
==28252==    by 0x80A7054: CEncoder::BitplaneEncode(unsigned int) (Encoder.cpp:380)
==28252==    by 0x80A72F7: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:274)
==28252==    by 0x80A76BC: CEncoder::Flush() (Encoder.cpp:216)
==28252==    by 0x80A96D1: CPGFImage::Write(CPGFStream*, int, bool (*)(double, bool, void*), unsigned int*, void*) (PGFimage.cpp:953)

==28252== Syscall param write(buf) points to uninitialised byte(s)
==28252==    at 0xBA6131D: ??? (in /lib64/libpthread-2.11.3.so)
==28252==    by 0x80AD921: CPGFFileStream::Write(int*, void*) (PGFplatform.h:510)
==28252==    by 0x80A7352: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:311)
==28252==    by 0x80A76BC: CEncoder::Flush() (Encoder.cpp:216)
==28252==    by 0x80A96D1: CPGFImage::Write(CPGFStream*, int, bool (*)(double, bool, void*), unsigned int*, void*) (PGFimage.cpp:953)
==28252==    by 0x7EE1544: Digikam::PGFLoader::save(QString const&, Digikam::DImgLoaderObserver*) (pgfloader.cpp:438)

Currently there is ongoing work on a new, slightly 'OMP'ified, version of the codec which does not contain anymore the notorious CEncoder::RLESigsAndSigns method.
So, contradictory to "we [...] provide a patch shortly", there will be no prompt fix, unless you have objections.

Marcel, could you also provide the source image for http://digikam3rdparty.free.fr/TEST_IMAGES/PGF/DSC03274_v1.pgf ? Was this image converted using digikam without any obvious errors? We never had any signs of libPGF encoding sane images into corrupted PGF images.
And also "converting .nef in batch resize+pgf does not give byte exact
files if run twice there are usually at least 3 or 4 bytes that differ" sounds strange, as PGF really should be deterministic.
Francesco, could you check whether this is PGF or the batch resize? I.e. whether the resized pictures always are byte-equal?

- Raphael

(In reply to comment #13)

> [SNIP]
> And also "converting .nef in batch resize+pgf does not give byte exact
> files if run twice there are usually at least 3 or 4 bytes that differ" sounds
> strange, as PGF really should be deterministic.
> Francesco, could you check whether this is PGF or the batch resize? I.e.
> whether the resized pictures always are byte-equal?
> 
> - Raphael

I've tried batch resize+PNG and it give two different images if run twice, probably somewhere between raw converter and resizer there is some random value added nothing too worrysome and
PGF encoder has nothing to do with that

> - Raphael

also, just out of curiosity, have you considered orc, successors of liboil (http://code.entropywave.com/projects/orc/)
it claim to make easy to optimize operations on contiguous arrays, only because you mentioned "omp"ification

Francesco,

thanks for checking (glad we don't have to deal with unintended randomness in PGF).
Orc looks like an interesting approach (having dealt with SSE intrinsics, making them more abstract is tempting). However, I don't think we will see Orc code within libPGF anytime soon. Windows users are not as used to complicated build processes as others are, and Orc's instructions (respectively the lack of WIN related ones) look frightening. Also, my test results / benchmarks using SSE are not very encouraging. Extrapolating them could make AVX interesting, though. But this would require a major remake of most array intensive code in libPGF, while still maintaining compatibility with older CPUs.

- Raphael

Yes I have the source, but I could not reproduce an image that crashes. At this one day, I produced three images that crashed; since then, none has. The source are normal JPG images, some filters were applied. Nothing special in that regard.

What I can reproduce are the uninitialized-value errors. (As I said, it would be well possible that an uninitialized memory area is almost always as expected. I once found the status of uninitialized memory very much reproducible)
They may just as well be completely misleading though.
I would guess that refBits and signBits are not set to 0; I cannot judge if this is relevant.

Thank you very much. There's another new report of indeterminism (http://sourceforge.net/projects/libpgf/forums/forum/530086/topic/4566988), possibly due to padding w/ uninitialized buffers. I'll definitely look into this once more and make sure this is solved for the next release.

- Raphael

And you will surely also fix the crash when reading the invalid file (instead, failing to load it)? ;-)

New version out now: including
- fixed crashes
- deterministic encoder
- OpenMP support

We are looking forward to your test result.

- Raphael

Raphael,

I patched digiKam with your last libpgf tarball...

There is a problem with OpenMP dependency : it's not optional. It must be.

Un MACOSX, GCC do not support OpenMP very well, especially when parallelized code run into separated thread. We have already see very weird side-effect with libraw under Mac.

I think it can be easily patched. I see in your code that you include hard omp.h. It must be wrapped in prepocessor like this :

https://projects.kde.org/projects/kde/kdegraphics/libs/libkdcraw/repository/revisions/master/entry/libraw/libraw/libraw_types.h#L32

Gilles Caulier

Git commit 46c8af586c53127ea48eebad23ff5968bcd9aa5c by Gilles Caulier.
Committed on 17/06/2011 at 08:55.
Pushed by cgilles into branch 'master'.

update libpgf from digiKam core to last 6.11.24 including OpenMP support
CCBUGS: 273765

M  +22   -1    CMakeLists.txt     
M  +1    -0    NEWS     
M  +3    -0    digikam/CMakeLists.txt     
M  +56   -53   libs/3rdparty/libpgf/BitStream.h     
M  +458  -272  libs/3rdparty/libpgf/Decoder.cpp     
M  +62   -35   libs/3rdparty/libpgf/Decoder.h     
M  +358  -307  libs/3rdparty/libpgf/Encoder.cpp     
M  +96   -36   libs/3rdparty/libpgf/Encoder.h     
M  +703  -483  libs/3rdparty/libpgf/PGFimage.cpp     
M  +114  -58   libs/3rdparty/libpgf/PGFimage.h     
M  +70   -61   libs/3rdparty/libpgf/PGFplatform.h     
A  +272  -0    libs/3rdparty/libpgf/PGFstream.cpp         [License: BSD]  *
A  +185  -0    libs/3rdparty/libpgf/PGFstream.h         [License: BSD]  *
M  +35   -21   libs/3rdparty/libpgf/PGFtypes.h     
D  +0    -254  libs/3rdparty/libpgf/Stream.cpp     
D  +0    -169  libs/3rdparty/libpgf/Stream.h     
M  +16   -13   libs/3rdparty/libpgf/Subband.cpp     
M  +14   -9    libs/3rdparty/libpgf/Subband.h     
M  +55   -48   libs/3rdparty/libpgf/WaveletTransform.cpp     
M  +17   -10   libs/3rdparty/libpgf/WaveletTransform.h     

The files marked with a * at the end have a non valid license. Please read: http://techbase.kde.org/Policies/Licensing_Policy and use the headers which are listed at that page.


http://commits.kde.org/digikam/46c8af586c53127ea48eebad23ff5968bcd9aa5c

Git commit 781571dd5b72d50d3e26e70bd4023938f20df94a by Gilles Caulier.
Committed on 17/06/2011 at 09:14.
Pushed by cgilles into branch 'master'.

patch against original libpgf 6.11.24 to :
- Handle properly OpenMP depending of compiler/OS (inspired from libraw.org).
- Include only one omp.h instance in PGFplatform.h
- Support no OpenMP availability (else libpgf do not compile without OpenMP).
This patch must be included in next stable libpgf.

Raphael, i recommend too to change EOL as Unix like and to replace tab by 4 spaces.

CCBUGS: 273765
CCMAIL: rschweizer@schweizer-informatik.ch

M  +3    -1    libs/3rdparty/libpgf/Decoder.cpp     
M  +3    -1    libs/3rdparty/libpgf/Encoder.cpp     
M  +25   -0    libs/3rdparty/libpgf/PGFplatform.h     

http://commits.kde.org/digikam/781571dd5b72d50d3e26e70bd4023938f20df94a

Raphael,

digiKam with new libpgf 6.11.24 compile fine under Linux with or without OpenMP with my personnal patch from #23. Please include it to your official repository.

digiKam run and do not crash under my Linux Mandriva  (thumbnails generation and file save as/load PGF in image editor)

I will check if all compile and run fine under MACosX and Windows (using MSVC compiler)

Gilles Caulier

Raphael,

digiKam with new libpgf 6.11.24 compile fine under MacOSX. Note that OpenMP support is disabled, as for libraw, to prevent crash when parallelized code run in a separated thread (current limitation of OpenMp support under MacOSX).

Gilles Caulier

Git commit dbb1f15dbe0998b9cc806ef32f7c15afbde1614c by Gilles Caulier.
Committed on 18/06/2011 at 09:50.
Pushed by cgilles into branch 'master'.

Another patch about OpenMp support for libpgf :

- M$ Visual C++ 2008 support well OpenMP. Do not limit OpenMP support to Visual C++ 2010.
- use dedicated OpenMP detection define everywhere if OpenMP will be used with libpgf implementation.

Raphael, now, digiKam+libpgf compile fine with M$ Visual C++. Please include this patch into official and current libpgf repository. Thanks in advance.

CCBUGS: 273765
CCMAIL: rschweizer@schweizer-informatik.ch

M  +2    -2    libs/3rdparty/libpgf/Decoder.cpp     
M  +2    -2    libs/3rdparty/libpgf/Encoder.cpp     
M  +2    -2    libs/3rdparty/libpgf/PGFplatform.h     

http://commits.kde.org/digikam/dbb1f15dbe0998b9cc806ef32f7c15afbde1614c

To all in this room.

digiKam from git master include now the new libpgf library + few patches for OpenMP support. Please checkout last digiKam code and try again to see if crash is reproducible.

Thanks in advance

Gilles Caulier

(In reply to comment #27)
> To all in this room.
> 
> digiKam from git master include now the new libpgf library + few patches for
> OpenMP support. Please checkout last digiKam code and try again to see if crash
> is reproducible.
> 
> Thanks in advance
> 
> Gilles Caulier

_dsc4243.pgf does not crash showfoto any more, thanks to everybody involved

The two pgf examples (in tests/) do not link anymore. Any change in symbol visibility?

Marcel,

I don't change visibility. I don't check test source code compilation too. I will take a look.

Gilles Caulier

Marcel,

Since libpgf have been updated into digiKam, under windows, thumbnails database do not sound to work as espected. There is no thumbnails cache effects visible between 2 digiKam sessions.

Can you confirm this problem under Linux ?

Gilles Caulier

Thanks for all the testing and the patches. I'm pleasantly surprised of your fast pace. Our source server is currently down, I'll apply patches and update sourceforge as soon as I get access again.

Thanks, Raphael

Raphael,

use test compilation mode from KDE sdk, more GCC warnings options are turned on. I can see this :

[ 55%] Building CXX object core/digikam/CMakeFiles/digikamdatabase.dir/__/libs/3rdparty/libpgf/Decoder.cpp.o                                         
In file included from /mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.cpp:29:                                                              
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h: In constructor ‘CDecoder::CMacroBlock::CMacroBlock(CDecoder*)’:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:74: attention : ‘CDecoder::CMacroBlock::m_decoder’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:62: attention :   ‘ROIBlockHeader CDecoder::CMacroBlock::m_header’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:52: attention :   when initialized here
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h: In constructor ‘CDecoder::CDecoder(CPGFStream*, PGFPreHeader&, PGFHeader&, PGFPostHeader&, UINT32*&, bool)’:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:180: attention : ‘CDecoder::m_encodedHeaderLength’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:179: attention :   ‘UINT64 CDecoder::m_streamSizeEstimation’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.cpp:65: attention :   when initialized here
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:185: attention : ‘CDecoder::m_macroBlocksAvailable’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:183: attention :   ‘int CDecoder::m_currentBlockIndex’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.cpp:65: attention :   when initialized here
[ 55%] Building CXX object core/digikam/CMakeFiles/digikamdatabase.dir/__/libs/3rdparty/libpgf/Encoder.cpp.o
In file included from /mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.cpp:29:                                                              
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h: In constructor ‘CEncoder::CMacroBlock::CMacroBlock(CEncoder*)’:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:82: attention : ‘CEncoder::CMacroBlock::m_encoder’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:63: attention :   ‘ROIBlockHeader CEncoder::CMacroBlock::m_header’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:52: attention :   when initialized here
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h: In constructor ‘CEncoder::CEncoder(CPGFStream*, PGFPreHeader, PGFHeader, const PGFPostHeader&, UINT32*&, bool)’:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:188: attention : ‘CEncoder::m_forceWriting’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:186: attention :   ‘UINT8 CEncoder::m_nLevels’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.cpp:67: attention :   when initialized here
[ 55%] Building CXX object core/digikam/CMakeFiles/digikamdatabase.dir/__/libs/3rdparty/libpgf/PGFimage.cpp.o
In file included from /mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/PGFimage.cpp:30:                                                             
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h: In constructor ‘CDecoder::CMacroBlock::CMacroBlock(CDecoder*)’:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:74: attention : ‘CDecoder::CMacroBlock::m_decoder’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:62: attention :   ‘ROIBlockHeader CDecoder::CMacroBlock::m_header’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:52: attention :   when initialized here
In file included from /mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/PGFimage.cpp:31:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h: In constructor ‘CEncoder::CMacroBlock::CMacroBlock(CEncoder*)’:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:82: attention : ‘CEncoder::CMacroBlock::m_encoder’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:63: attention :   ‘ROIBlockHeader CEncoder::CMacroBlock::m_header’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:52: attention :   when initialized here
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/PGFimage.h: In constructor ‘CPGFImage::CPGFImage()’:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/PGFimage.h:509: attention : ‘CPGFImage::m_cbArg’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/PGFimage.h:502: attention :   ‘bool CPGFImage::m_levelwise’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/PGFimage.cpp:56: attention :   when initialized here
[ 55%] Building CXX object core/digikam/CMakeFiles/digikamdatabase.dir/__/libs/3rdparty/libpgf/PGFstream.cpp.o
[ 55%] Building CXX object core/digikam/CMakeFiles/digikamdatabase.dir/__/libs/3rdparty/libpgf/Subband.cpp.o                                         
In file included from /mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Subband.cpp:30:                                                              
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h: In constructor ‘CEncoder::CMacroBlock::CMacroBlock(CEncoder*)’:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:82: attention : ‘CEncoder::CMacroBlock::m_encoder’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:63: attention :   ‘ROIBlockHeader CEncoder::CMacroBlock::m_header’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:52: attention :   when initialized here
In file included from /mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Subband.cpp:31:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h: In constructor ‘CDecoder::CMacroBlock::CMacroBlock(CDecoder*)’:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:74: attention : ‘CDecoder::CMacroBlock::m_decoder’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:62: attention :   ‘ROIBlockHeader CDecoder::CMacroBlock::m_header’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:52: attention :   when initialized here
[ 55%] Building CXX object core/digikam/CMakeFiles/digikamdatabase.dir/__/libs/3rdparty/libpgf/WaveletTransform.cpp.o
[ 55%] Building CXX object core/digikam/CMakeFiles/digikamdatabase.dir/__/libs/threadimageio/pgfutils.cpp.o                                          
Linking CXX shared library ../../lib/libdigikamdatabase.so                                                                                           

I will fix it tomorow...

Gilles Caulier

To Marcel, #29

All digiKam tests compile fine here :

[ 98%] Built target advancedrenametest_automoc
Linking CXX executable advancedrenametest
[ 98%] Built target advancedrenametest
[ 98%] Built target cameranamehelpertest_automoc
Linking CXX executable cameranamehelpertest
[ 98%] Built target cameranamehelpertest
[ 98%] Built target dimagefilteractiontest_automoc
Linking CXX executable dimagefilteractiontest
[ 98%] Built target dimagefilteractiontest
[ 98%] Built target dimagehistorygraphtest_automoc
Scanning dependencies of target dimagehistorygraphtest
[ 98%] Building CXX object core/tests/CMakeFiles/dimagehistorygraphtest.dir/dimagehistorygraphtest_automoc.cpp.o
Linking CXX executable dimagehistorygraphtest
[ 98%] Built target dimagehistorygraphtest
[ 98%] Built target dimagehistorytest_automoc
Linking CXX executable dimagehistorytest
[ 98%] Built target dimagehistorytest
[ 98%] Built target filesaveoptionsboxtest_automoc
Linking CXX executable filesaveoptionsboxtest
[ 98%] Built target filesaveoptionsboxtest
[ 98%] Built target freerotationtest_automoc
Linking CXX executable freerotationtest
[ 98%] Built target freerotationtest
[ 98%] Built target pgfscaled_automoc
Scanning dependencies of target pgfscaled
[ 98%] Building CXX object core/tests/CMakeFiles/pgfscaled.dir/__/libs/3rdparty/libpgf/Decoder.cpp.o
Linking CXX executable pgfscaled
[100%] Built target pgfscaled
[100%] Built target qtpgftest_automoc
Scanning dependencies of target qtpgftest
[100%] Building CXX object core/tests/CMakeFiles/qtpgftest.dir/__/libs/3rdparty/libpgf/Decoder.cpp.o
Linking CXX executable qtpgftest
[100%] Built target qtpgftest
[100%] Built target searchtextbartest_automoc
Linking CXX executable searchtextbartest
[100%] Built target searchtextbartest
[100%] Built target statesavingobjecttest_automoc
Linking CXX executable statesavingobjecttest
[100%] Built target statesavingobjecttest
Scanning dependencies of target uifilevalidatortest_automoc
Generating uifilevalidatortest.moc
[100%] Built target uifilevalidatortest_automoc
Scanning dependencies of target uifilevalidatortest
[100%] Building CXX object core/tests/CMakeFiles/uifilevalidatortest.dir/uifilevalidatortest_automoc.cpp.o
[100%] Building CXX object core/tests/CMakeFiles/uifilevalidatortest.dir/uifilevalidatortest.cpp.o
Linking CXX executable uifilevalidatortest
[100%] Built target uifilevalidatortest

Gilles Caulier

Git commit cf8cdc2cd50a63f6aa6d81c327005ed4faf37592 by Gilles Caulier.
Committed on 19/06/2011 at 01:20.
Pushed by cgilles into branch 'master'.

fix gcc warnings about order of classes member initialization
Raphael, please include this patch to official libpgf repository. Thanks in advance
CCBUGS: 273765
CCMAIL: rschweizer@schweizer-informatik.ch

M  +2    -2    libs/3rdparty/libpgf/Decoder.cpp     
M  +3    -3    libs/3rdparty/libpgf/Decoder.h     
M  +1    -1    libs/3rdparty/libpgf/Encoder.cpp     
M  +2    -2    libs/3rdparty/libpgf/Encoder.h     
M  +2    -2    libs/3rdparty/libpgf/PGFimage.cpp     

http://commits.kde.org/digikam/cf8cdc2cd50a63f6aa6d81c327005ed4faf37592

Marcel,

As in #31, i can reproduce the thumbnail cache problem under Linux, as under Windows.

there is something to patch in thumbnail management Database code about PGF image data handling. Thumbs are displayed very slowly. Sound like all thumb from an album are recompute between each digiKam session. 

Of course, if you change current album in same digiKam session, thumbs cache from memory work fine.

Can you reproduce the problem ?

Gilles Caulier

Marcel,

Under Linux, I removed all DB file, restarted digiKam, and rebuild all thumbnails. Problem still here. Icon view is very slow in all case to display thumbs.

Can you reproduce ?

Gilles Caulier

I see it as well, not only with PGFs.

Git commit bf9645f52d22ba43b7b5bc58b73f915ecd713df4 by Marcel Wiesweg.
Committed on 19/06/2011 at 15:46.
Pushed by mwiesweg into branch 'master'.

Fix linking of libpgf tests
CCBUG: 273765

M  +2    -2    tests/CMakeLists.txt     

http://commits.kde.org/digikam/bf9645f52d22ba43b7b5bc58b73f915ecd713df4

Yes, I can fully reproduce. See these error messages on the console:
digikam(5323)/digikam (core) Digikam::readPGFImageData: Error running libpgf ( 2 )!
digikam(5323)/digikam (core) Digikam::ThumbnailCreator::loadFromDatabase: Cannot load PGF thumb from DB 

This means the data is loaded from the db, but libpgf fails to load the data with error code 2. Apparently this happens only with every second or third image.
Backtrace of the thrown exception:

#0  0x00007fffef9d7d90 in __cxa_throw () from /usr/lib64/libstdc++.so.6
#1  0x00007ffff4aa9912 in CPGFMemoryStream::Read (this=0x7fffcbe98460, count=0x7fffcbe97cfc, buffPtr=<value optimized out>)
    at /home/marcel/freshmeat/multimedia/kde4/src/extragear/graphics/digikam/core/libs/3rdparty/libpgf/PGFstream.cpp:157
#2  0x00007ffff4aa06f3 in CDecoder::ReadMacroBlock (this=0x7fffd021d430, block=0x7fffd0450370)
    at /home/marcel/freshmeat/multimedia/kde4/src/extragear/graphics/digikam/core/libs/3rdparty/libpgf/Decoder.cpp:461
#3  0x00007ffff4aa14d0 in CDecoder::DecodeBuffer (this=0x7fffd021d430)
    at /home/marcel/freshmeat/multimedia/kde4/src/extragear/graphics/digikam/core/libs/3rdparty/libpgf/Decoder.cpp:571
#4  0x00007ffff4aa169a in CDecoder::DequantizeValue (this=0x7fffd021d430, band=<value optimized out>, 
    bandPos=<value optimized out>, quantParam=<value optimized out>)
    at /home/marcel/freshmeat/multimedia/kde4/src/extragear/graphics/digikam/core/libs/3rdparty/libpgf/Decoder.cpp:429
#5  0x00007ffff4aa1b87 in CDecoder::Partition (this=0x7fffd021d430, band=0x7fffd056a7b8, quantParam=3, width=<value optimized out>, 
    height=<value optimized out>, startPos=0, pitch=42)
    at /home/marcel/freshmeat/multimedia/kde4/src/extragear/graphics/digikam/core/libs/3rdparty/libpgf/Decoder.cpp:249
#6  0x00007ffff4aaa1f6 in CSubband::PlaceTile (this=0x7fffd056a7b8, decoder=..., quantParam=3, tile=false, tileX=0, tileY=0)
    at /home/marcel/freshmeat/multimedia/kde4/src/extragear/graphics/digikam/core/libs/3rdparty/libpgf/Subband.cpp:230
#7  0x00007ffff4aa92b9 in CPGFImage::Read (this=0x7fffcbe97f30, level=0, cb=0, data=0x0)
    at /home/marcel/freshmeat/multimedia/kde4/src/extragear/graphics/digikam/core/libs/3rdparty/libpgf/PGFimage.cpp:408
#8  0x00007ffff4aab93f in Digikam::readPGFImageData (data=<value optimized out>, img=...)
    at /home/marcel/freshmeat/multimedia/kde4/src/extragear/graphics/digikam/core/libs/threadimageio/pgfutils.cpp:71

Created attachment 61150 [details]
Thumbnail PGF data which throws exception

Sample thumbnail data (written to disk the first time an exception is caught at startup from libpgf)

Thanks Marcel, 

Sound like something must be adapted in PGFUtils interface with libpgf API to load RAW PGF image data from a bytearray take from thumbs database:

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/threadimageio/pgfutils.cpp

Raphael, any sugestions ?

Gilles Caulier

Git commit e7e53a9c288b3b2aa83a3b7dd09c7a136f6253b1 by Gilles Caulier.
Committed on 20/06/2011 at 08:58.
Pushed by cgilles into branch 'master'.

Note : MSVC 2008 patched with SP1 work fine. MSVC 2008 official has a problem with OpenMP.
CCBUGS: 273765

M  +1    -1    libs/3rdparty/libpgf/PGFplatform.h     

http://commits.kde.org/digikam/e7e53a9c288b3b2aa83a3b7dd09c7a136f6253b1

Git commit cb9435fae37f32553e8e354705e9abfe77783dad by Gilles Caulier.
Committed on 20/06/2011 at 12:10.
Pushed by cgilles into branch 'master'.

fix argment call for PGFImage::write method, which have changed with libpgf 6.11.24
CCBUGS: 273765

M  +1    -1    libs/threadimageio/pgfutils.cpp     

http://commits.kde.org/digikam/cb9435fae37f32553e8e354705e9abfe77783dad

Marcel, Raphael

I hacked a lot digiKam PGF Utils interface and found a bug in lipgf API call arguments (in fact, an evolution of API to update in digiKam core).

I tested with pgf test program from digiKam, and encoding and decoding PGF work fine.

BUT, the problem still here about thumb database. I set a debug print to see if PGF data taken from thumb db are not null. It's not the case :

digikam(11244)/digikam (core) Digikam::readPGFImageData: image data stream size is :  8722
digikam(11244)/digikam (core) Digikam::readPGFImageData: Error running libpgf ( 2 )!
digikam(11244)/digikam (core) Digikam::ThumbnailCreator::loadFromDatabase: Cannot load PGF thumb from DB 
digikam(11244)/KDCRAW KDcrawIface::KDcraw::loadEmbeddedPreview: Preview data size: 345669
digikam(11244)/KDCRAW KDcrawIface::KDcraw::loadEmbeddedPreview: Using embedded RAW preview extraction
digikam(11244)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
digikam(11244)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation =>  1
digikam(11244)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation =>  1
digikam(11244)/digikam (core) Digikam::readPGFImageData: image data stream size is :  8002
digikam(11244)/digikam (core) Digikam::readPGFImageData: Error running libpgf ( 2 )!
digikam(11244)/digikam (core) Digikam::ThumbnailCreator::loadFromDatabase: Cannot load PGF thumb from DB 
digikam(11244)/KDCRAW KDcrawIface::KDcraw::loadEmbeddedPreview: Preview data size: 101661
digikam(11244)/KDCRAW KDcrawIface::KDcraw::loadEmbeddedPreview: Using embedded RAW preview extraction
digikam(11244)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation =>  1
digikam(11244)/digikam (core) Digikam::readPGFImageData: image data stream size is :  8814
digikam(11244)/digikam (core) Digikam::readPGFImageData: Error running libpgf ( 2 )!
digikam(11244)/digikam (core) Digikam::ThumbnailCreator::loadFromDatabase: Cannot load PGF thumb from DB 
digikam(11244)/KDCRAW KDcrawIface::KDcraw::loadEmbeddedPreview: Preview data size: 331522
digikam(11244)/KDCRAW KDcrawIface::KDcraw::loadEmbeddedPreview: Using embedded RAW preview extraction
digikam(11244)/kio (KIOConnection) KIO::ConnectionServer::listenForRemote: digikam(11244)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
digikam(11244)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation =>  1
digikam(11244)/digikam (core) Digikam::readPGFImageData: image data stream size is :  10966
digikam(11244)/digikam (core) Digikam::readPGFImageData: Error running libpgf ( 2 )!
digikam(11244)/digikam (core) Digikam::ThumbnailCreator::loadFromDatabase: Cannot load PGF thumb from DB

Thumbs size are around 8/10Kb (256x256 pixels). So it sound fine.

The exception generated by libPGF appear when image data taken from DB is passed to CPGFImage::Open() as a memory stream :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/threadimageio/pgfutils.cpp#L67

What's error #2 given by libPGF ?

Note : in my pgf test code, i use this method, and all work fine ! PGF image data are taken from a file, not DB of course.

Gilles Caulier

Sorry, 

The exception is not generated by libpgf to CPGFImage::Open(), by to CPGFImage::Read() :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/threadimageio/pgfutils.cpp#L76

Gilles

Marcel,

Your PGF thumb from #41 work fine here. I wan load it into showfoto and pgfscaled test program do not report an error.

Gilles Caulier

Git commit 3382b0cb2901b599b8e593c9cc07353cea045ee6 by Gilles Caulier.
Committed on 20/06/2011 at 14:30.
Pushed by cgilles into branch 'master'.

new test program to load PGF data from image file and convert it to QImg using PGFUtils::readPGFImageData()
CCBUGS: 273765

M  +3    -0    tests/CMakeLists.txt     
A  +76   -0    tests/loadpgfdata.cpp         [License: GPL (v2+)]

http://commits.kde.org/digikam/3382b0cb2901b599b8e593c9cc07353cea045ee6

Marcel, with your PGF thumb image data, i can also pass loadpgfdata test program... How do you explain that ?

Gilles Caulier

Ok, i find the problem, but not the real solution.

I disable OpenMp support to libpgf, and now, it work fine. Thumbnails can be loaded from database and decoded through libpgf without an error.

As thumbnail process run into a separated thread, using QThread, i suspect a problem here.

The fact that loadpgdfdata work as well is simple : i do not run in a separate thead, but in main thread, as it's a single a small test application.

Raphael,

Do you have tested into libpgf test collection tools a case where libpgf is used through a separated thread, as Posix thread ?

Note that Windows thread as the same problem. It's not Unix thread in this case...


Marcel,

Another question : image editor use a separated thread to load PGF image. Why it work fine in this case, if libpgf is compiled with OpenMP support ?

Gilles Caulier

AFAIK there were no such tests. Unfortunately I still have no access to our source server (it went down shortly after some migration, no words from IT yet). I'm currently at home - preparing for next week's exams - and ha

AFAIK there were no explicit tests (a background thread is used by the ImageViewer [http://xeraina.ch/pgf/pgfdownload.html] to load the images, no problems there). Unfortunately I still have no access to our source server (it went down shortly after some migration, no words from IT yet). I'm currently at home - preparing for next two week's exams - and have no access to our test system nor the latest source.
I really appreciate the work you have done so far. Is it feasible to disable OpenMP for the time being and suspend development till start of July?

- Raphael

Raphael,

Trying to find a solution to OpenMP support in libpgf, is tried to turn off OpenMP with PGF encoder/decoder in digiKam PGFUtils methods relevant of thumbs database.

libPGF still compiled with OpenMP support. digiKam crash into libpgf :

Thread 2 (Thread 0xa8c3ab70 (LWP 7763)):
[KCrash Handler]
#6  0xb67ede11 in CDecoder::DecodeBuffer (this=0xa746628) at /mnt/data/Devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.cpp:571
#7  0xb67ed900 in CDecoder::DequantizeValue (this=0xa746628, band=0xa732f2c, bandPos=0, quantParam=0) at /mnt/data/Devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.cpp:429
#8  0xb67ed187 in CDecoder::Partition (this=0xa746628, band=0xa732f2c, quantParam=0, width=64, height=43, startPos=0, pitch=64) at /mnt/data/Devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.cpp:249
#9  0xb67f9fe7 in CSubband::PlaceTile (this=0xa732f2c, decoder=..., quantParam=0, tile=false, tileX=0, tileY=0) at /mnt/data/Devel/GIT/2.x/core/libs/3rdparty/libpgf/Subband.cpp:230
#10 0xb67f19b9 in CPGFImage::Read (this=0xa8c398d0, level=0, cb=0, data=0x0) at /mnt/data/Devel/GIT/2.x/core/libs/3rdparty/libpgf/PGFimage.cpp:398
#11 0xb67fc342 in Digikam::readPGFImageData (data=..., img=...) at /mnt/data/Devel/GIT/2.x/core/libs/threadimageio/pgfutils.cpp:78
#12 0xb67ddbfd in Digikam::ThumbnailCreator::loadFromDatabase (this=0xa3a8f00, info=...) at /mnt/data/Devel/GIT/2.x/core/libs/threadimageio/thumbnailcreator.cpp:884
#13 0xb67da8fc in Digikam::ThumbnailCreator::load (this=0xa3a8f00, path=..., rect=..., pregenerate=false) at /mnt/data/Devel/GIT/2.x/core/libs/threadimageio/thumbnailcreator.cpp:245
#14 0xb67da599 in Digikam::ThumbnailCreator::load (this=0xa3a8f00, path=...) at /mnt/data/Devel/GIT/2.x/core/libs/threadimageio/thumbnailcreator.cpp:196
#15 0xb67e8181 in Digikam::ThumbnailLoadingTask::execute (this=0xa733768) at /mnt/data/Devel/GIT/2.x/core/libs/threadimageio/thumbnailtask.cpp:169
#16 0xb67c574a in Digikam::LoadSaveThread::run (this=0xa3841f8) at /mnt/data/Devel/GIT/2.x/core/libs/threadimageio/loadsavethread.cpp:118
#17 0xb68024ca in Digikam::DynamicThread::DynamicThreadPriv::run (this=0xa3a71a8) at /mnt/data/Devel/GIT/2.x/core/libs/threads/dynamicthread.cpp:328
#18 0xb48fa9d6 in ?? () from /usr/lib/libQtCore.so.4
#19 0xb490555f in ?? () from /usr/lib/libQtCore.so.4
#20 0xb4892ae5 in start_thread () from /lib/i686/libpthread.so.0
#21 0xb466203e in clone () from /lib/i686/libc.so.6

Why ???

So, my only solution is to temporally disable OpenMP support with libpgf to be able to run properly digiKam.

I recommend to improve better OpenMP support in libpgf. I will write some code to be able to force compilation of libpgf without OpenMP, in case of OpenMP is detected on target computer. I recommend to backport my patch to current implementation of libpgf

Gilles Caulier

Git commit 4ca8c22c4368306457aed9d62bbf48f7436553da by Gilles Caulier.
Committed on 20/06/2011 at 15:32.
Pushed by cgilles into branch 'master'.

add a flag to compile libpgf without openmp support at compilation time.
Raphael, please add this patch to current libpgf implementation
CCMAIL: rschweizer@schweizer-informatik.ch
CCBUGS: 273765

M  +10   -4    libs/3rdparty/libpgf/PGFplatform.h     

http://commits.kde.org/digikam/4ca8c22c4368306457aed9d62bbf48f7436553da

Git commit 6349f79b8a059034bfe218f51ef26e46581a64d4 by Gilles Caulier.
Committed on 20/06/2011 at 15:51.
Pushed by cgilles into branch 'master'.

disable openmp support to libpgf
CCBUGS: 273765

M  +4    -0    CMakeLists.txt     

http://commits.kde.org/digikam/6349f79b8a059034bfe218f51ef26e46581a64d4

Francesco,

Lets' me hear if this file can be closed now...

Gilles Caulier

yes it is, liked very much the collaborative work on this one, congratz

Git commit 22018b3ec500ee76178d486456598f511eb74455 by Gilles Caulier.
Committed on 21/06/2011 at 09:53.
Pushed by cgilles into branch 'master'.

Add a PGF codec version ID to be able to make rules with clien application accodingly with libpgf API changes.
Raphael, it's not possible to do it with PGFCodecVersion string and preprocessor. Of course both must be updated at each libpgf version.
We use already this way in libraw and libexiv2.
CCMAIL: rschweizer@schweizer-informatik.ch
CCBUGS: 273765

M  +2    -0    libs/3rdparty/libpgf/PGFtypes.h     

http://commits.kde.org/digikam/22018b3ec500ee76178d486456598f511eb74455

Git commit 67029dd34cc4314885db49258b948de4c77d7d58 by Gilles Caulier.
Committed on 22/06/2011 at 08:22.
Pushed by cgilles into branch 'master'.

fix MSVC warnings
CCBUGS: 273765
CCMAIL: rschweizer@schweizer-informatik.ch

M  +2    -2    libs/3rdparty/libpgf/PGFplatform.h     

http://commits.kde.org/digikam/67029dd34cc4314885db49258b948de4c77d7d58

Not reproducible with digiKam 7.3.0 and Exiv2 0.27.4