Bug 273765

Summary: replacing pgf files with an open digikam lead to reproducible crash
Product: [Applications] digikam Reporter: Francesco Riosa <vivo75+kde>
Component: Portability-RuntimeAssignee: Digikam Developers <digikam-bugs-null>
Status: RESOLVED FIXED    
Severity: crash CC: caulier.gilles, nucleo, rdieter, rschweizer, sven.burmeister
Priority: NOR    
Version: 2.0.0   
Target Milestone: ---   
Platform: Compiled Sources   
OS: Linux   
Latest Commit: Version Fixed In: 7.3.0
Sentry Crash Report:
Attachments: getThumbs.py extract pgf thumbnail from database
valgrind-broken-pgf.zip
Thumbnail PGF data which throws exception

Description Francesco Riosa 2011-05-21 01:24:55 UTC
Version:           2.0.0 (using KDE 4.6.3) 
OS:                Linux

ok, I'm a bit stuck in debugging this one, while it's a corner case it would be interesting to discover why the crash happen.

The point I'm stuck at is how to correlate the backtrace of the signalled thread, working in pgf library to a call in digikam

there is some way to know from where the data is feed into libpgf?

BTW the crash happen even with external libpgf

Reproducible: Always

Steps to Reproduce:
1) download a bunch of .nef files in raw/
2) convert it to lossy pgf (resized) to be put in tmp/
3) change your mind, w/o close digikam open a shell and remove all files in tmp/
4) convert raw/ and put it again in tmp/ this time with lossless pgf compression
5) close and reopen digikam, then try to navigate into tmp/
6) crash




tried to use gdb for the live program but found that working on a core file has more flexibility so:

digikam.sh --nocrashhandler
# steps to reproduce
gdb /home/vivo/usr/bin/digikam  core

(gdb) info threads
  7 Thread 18247  0x00007fdf6625b54e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  6 Thread 16802  0x00007fdf652927e9 in ?? () from /lib64/libc.so.6
  5 Thread 18245  0x00007fdf6625b54e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  4 Thread 16807  0x00007fdf6625b1e4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  3 Thread 16808  0x00007fdf6527db23 in poll () from /lib64/libc.so.6
  2 Thread 16895  0x00007fdf6625b1e4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
* 1 Thread 18246  ClearBitBlock (stream=0x7fdf26ba94c0, pos=82071, len=<value optimized out>)
    at /home/vivo/digikam-devel/digikam-sc/core/libs/3rdparty/libpgf/BitStream.h:159

(gdb) info source
Current source file is /home/vivo/digikam-devel/digikam-sc/core/libs/3rdparty/libpgf/BitStream.h
Compilation directory is /home/vivo/digikam-devel/digikam-sc/build2/core/digikam
Located in /srv/git/digikam-sc/core/libs/3rdparty/libpgf/BitStream.h
Contains 269 lines.
Source language is c++.
Compiled with unknown debugging format.
Includes preprocessor macro info.

(gdb) info line
Line 159 of "/home/vivo/digikam-devel/digikam-sc/core/libs/3rdparty/libpgf/BitStream.h"
   starts at address 0x7fdf6b2ab8e0 <ClearBitBlock(UINT32*, UINT32, UINT32)+48> and ends at 0x7fdf6b2ab8e3 <ClearBitBlock(UINT32*, UINT32, UINT32)+51>.
(gdb) 

(gdb) frame 1
#1  0x00007fdf6b2aaaab in CDecoder::RLDSigsAndSigns (this=0x7fdf44095d90, bufferSize=16384, codeLen=109243, sigBits=0x7fdf26ba94c0, signBits=0x7fdf26ba8cc0)
    at /home/vivo/digikam-devel/digikam-sc/core/libs/3rdparty/libpgf/Decoder.cpp:726
726                             ClearBitBlock(sigBits, sigPos, runlen); 
(gdb) line 
Undefined command: "line".  Try "help".
(gdb) info line
Line 726 of "/home/vivo/digikam-devel/digikam-sc/core/libs/3rdparty/libpgf/Decoder.cpp"
   starts at address 0x7fdf6b2aaa93 <CDecoder::RLDSigsAndSigns(UINT32, UINT32, UINT32*, UINT32*)+291>
   and ends at 0x7fdf6b2aaa96 <CDecoder::RLDSigsAndSigns(UINT32, UINT32, UINT32*, UINT32*)+294>.

(gdb) frame 2
#2  0x00007fdf6b2aaea0 in CDecoder::BitplaneDecode (this=0x7fdf44095d90, bufferSize=16384)
    at /home/vivo/digikam-devel/digikam-sc/core/libs/3rdparty/libpgf/Decoder.cpp:536
536                             sigLen = RLDSigsAndSigns(bufferSize, codeLen, sigBits, signBits); ASSERT(sigLen <= bufferSize);
(gdb) info line
Line 536 of "/home/vivo/digikam-devel/digikam-sc/core/libs/3rdparty/libpgf/Decoder.cpp"
   starts at address 0x7fdf6b2aad46 <CDecoder::BitplaneDecode(UINT32)+278> and ends at 0x7fdf6b2aad4e <CDecoder::BitplaneDecode(UINT32)+286>.


(gdb) thread apply all bt

Thread 7 (Thread 18247):
#0  0x00007fdf6625b54e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fdf664ec1b8 in wait (this=<value optimized out>, mutex=0x18fc5a0, time=<value optimized out>) at thread/qwaitcondition_unix.cpp:86
#2  QWaitCondition::wait (this=<value optimized out>, mutex=0x18fc5a0, time=<value optimized out>) at thread/qwaitcondition_unix.cpp:160
#3  0x00007fdf664df19f in QThreadPoolThread::run (this=0x54496b0) at concurrent/qthreadpool.cpp:140
#4  0x00007fdf664eba8a in QThreadPrivate::start (arg=0x54496b0) at thread/qthread_unix.cpp:320
#5  0x00007fdf66256c00 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fdf65285b0d in clone () from /lib64/libc.so.6


Thread 6 (Thread 16802):
#0  0x00007fdf652927e9 in ?? () from /lib64/libc.so.6
#1  0x00007fdf65229ea6 in ?? () from /lib64/libc.so.6
#2  0x00007fdf65227eca in malloc () from /lib64/libc.so.6
#3  0x00007fdf65a8f1ad in operator new(unsigned long) () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.5.2/libstdc++.so.6
#4  0x00007fdf67395ddc in QPainter::save (this=<value optimized out>) at painting/qpainter.cpp:1604
#5  0x00007fdf4e5973b3 in Oxygen::Style::drawControl (this=0x172e0b0, element=QStyle::CE_ShapedFrame, option=0x7ffffa529200, painter=0x7ffffa529290, widget=
    0x17f5170) at /usr/src/debug/kde-base/kstyles-4.6.3/kstyles-4.6.3/kstyles/oxygen/oxygenstyle.cpp:1026
#6  0x00007fdf676702ed in QFrame::drawFrame (this=0x17f5170, p=0x7ffffa529290) at widgets/qframe.cpp:534
#7  0x00007fdf676703a8 in QFrame::paintEvent (this=0x17f5170) at widgets/qframe.cpp:496
#8  0x00007fdf67292590 in QWidget::event (this=0x17f5170, event=0x7ffffa529b00) at kernel/qwidget.cpp:8405
#9  0x00007fdf67670446 in QFrame::event (this=0x17f5170, e=0x7ffffa529b00) at widgets/qframe.cpp:557
#10 0x00007fdf67238ea4 in QApplicationPrivate::notify_helper (this=0x16fbc80, receiver=0x17f5170, e=0x7ffffa529b00) at kernel/qapplication.cpp:4462
#11 0x00007fdf6723e351 in QApplication::notify (this=<value optimized out>, receiver=0x17f5170, e=0x7ffffa529b00) at kernel/qapplication.cpp:4341
#12 0x00007fdf67ff4002 in KApplication::notify (this=0x7ffffa52b5b0, receiver=0x17f5170, event=0x7ffffa529b00)
    at /usr/src/debug/kde-base/kdelibs-4.6.3-r1/kdelibs-4.6.3/kdeui/kernel/kapplication.cpp:311
#13 0x00007fdf665e4a9b in QCoreApplication::notifyInternal (this=0x7ffffa52b5b0, receiver=0x17f5170, event=0x7ffffa529b00) at kernel/qcoreapplication.cpp:731
#14 0x00007fdf6728f17e in sendSpontaneousEvent (this=0x18e3ae0, pdev=0x18e3998, rgn=..., offset=..., flags=4, sharedPainter=0x0, backingStore=0x19347e0)
    at ../../src/corelib/kernel/qcoreapplication.h:218
#15 QWidgetPrivate::drawWidget (this=0x18e3ae0, pdev=0x18e3998, rgn=..., offset=..., flags=4, sharedPainter=0x0, backingStore=0x19347e0)
    at kernel/qwidget.cpp:5492
#16 0x00007fdf6728fddd in QWidgetPrivate::paintSiblingsRecursive (this=0x18dfed0, pdev=0x18e3998, siblings=..., index=<value optimized out>, rgn=..., 
    offset=..., flags=4, sharedPainter=0x0, backingStore=0x19347e0) at kernel/qwidget.cpp:5699
#17 0x00007fdf6728fc93 in QWidgetPrivate::paintSiblingsRecursive (this=0x18dfed0, pdev=0x18e3998, siblings=..., index=132, rgn=..., offset=..., flags=4, 
    sharedPainter=0x0, backingStore=0x19347e0) at kernel/qwidget.cpp:5686
#18 0x00007fdf6728fc93 in QWidgetPrivate::paintSiblingsRecursive (this=0x18dfed0, pdev=0x18e3998, siblings=..., index=151, rgn=..., offset=..., flags=4, 
    sharedPainter=0x0, backingStore=0x19347e0) at kernel/qwidget.cpp:5686
#19 0x00007fdf6728ee40 in QWidgetPrivate::drawWidget (this=0x18dfed0, pdev=0x18e3998, rgn=..., offset=..., flags=<value optimized out>, sharedPainter=0x0, 
    backingStore=0x19347e0) at kernel/qwidget.cpp:5545
#20 0x00007fdf6747fdd5 in QWidgetBackingStore::sync (this=0x19347e0) at painting/qbackingstore.cpp:1333
#21 0x00007fdf67283bd0 in QWidgetPrivate::syncBackingStore (this=0x18dfed0) at kernel/qwidget.cpp:1842
#22 0x00007fdf67292d02 in QWidget::event (this=0x1879de0, event=0x7fdf4400bc80) at kernel/qwidget.cpp:8552
#23 0x00007fdf6768badb in QMainWindow::event (this=0x1879de0, event=0x7fdf4400bc80) at widgets/qmainwindow.cpp:1480
#24 0x00007fdf680e6990 in KXmlGuiWindow::event (this=0x1879de0, ev=0x7fdf4400bc80)
    at /usr/src/debug/kde-base/kdelibs-4.6.3-r1/kdelibs-4.6.3/kdeui/xmlgui/kxmlguiwindow.cpp:126
#25 0x00007fdf67238ea4 in QApplicationPrivate::notify_helper (this=0x16fbc80, receiver=0x1879de0, e=0x7fdf4400bc80) at kernel/qapplication.cpp:4462
#26 0x00007fdf6723e351 in QApplication::notify (this=<value optimized out>, receiver=0x1879de0, e=0x7fdf4400bc80) at kernel/qapplication.cpp:4341
#27 0x00007fdf67ff4002 in KApplication::notify (this=0x7ffffa52b5b0, receiver=0x1879de0, event=0x7fdf4400bc80)
    at /usr/src/debug/kde-base/kdelibs-4.6.3-r1/kdelibs-4.6.3/kdeui/kernel/kapplication.cpp:311
#28 0x00007fdf665e4a9b in QCoreApplication::notifyInternal (this=0x7ffffa52b5b0, receiver=0x1879de0, event=0x7fdf4400bc80) at kernel/qcoreapplication.cpp:731
#29 0x00007fdf665e89be in sendEvent (receiver=0x0, event_type=0, data=0x16b2a00) at kernel/qcoreapplication.h:215
#30 QCoreApplicationPrivate::sendPostedEvents (receiver=0x0, event_type=0, data=0x16b2a00) at kernel/qcoreapplication.cpp:1372
#31 0x00007fdf66612c53 in sendPostedEvents (s=<value optimized out>) at kernel/qcoreapplication.h:220
#32 postEventSourceDispatch (s=<value optimized out>) at kernel/qeventdispatcher_glib.cpp:277
#33 0x00007fdf64acdb3e in g_main_context_dispatch () from /usr/lib64/libglib-2.0.so.0
#34 0x00007fdf64ace328 in ?? () from /usr/lib64/libglib-2.0.so.0
#35 0x00007fdf64ace5bd in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0
#36 0x00007fdf66612def in QEventDispatcherGlib::processEvents (this=0x16faff0, flags=<value optimized out>) at kernel/qeventdispatcher_glib.cpp:422
#37 0x00007fdf672ea03e in QGuiEventDispatcherGlib::processEvents (this=<value optimized out>, flags=<value optimized out>)
    at kernel/qguieventdispatcher_glib.cpp:204
#38 0x00007fdf665e3562 in QEventLoop::processEvents (this=<value optimized out>, flags=...) at kernel/qeventloop.cpp:149
#39 0x00007fdf665e37a5 in QEventLoop::exec (this=0x7ffffa52b3f0, flags=...) at kernel/qeventloop.cpp:201
#40 0x00007fdf665e8cb9 in QCoreApplication::exec () at kernel/qcoreapplication.cpp:1008
#41 0x000000000063635f in main (argc=7818496, argv=0x7ffffa52bc58) at /home/vivo/digikam-devel/digikam-sc/core/digikam/main/main.cpp:232

Thread 5 (Thread 18245):
#0  0x00007fdf6625b54e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fdf664ec1b8 in wait (this=<value optimized out>, mutex=0x18fc5a0, time=<value optimized out>) at thread/qwaitcondition_unix.cpp:86
#2  QWaitCondition::wait (this=<value optimized out>, mutex=0x18fc5a0, time=<value optimized out>) at thread/qwaitcondition_unix.cpp:160
#3  0x00007fdf664df19f in QThreadPoolThread::run (this=0x5f1c430) at concurrent/qthreadpool.cpp:140
#4  0x00007fdf664eba8a in QThreadPrivate::start (arg=0x5f1c430) at thread/qthread_unix.cpp:320
#5  0x00007fdf66256c00 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fdf65285b0d in clone () from /lib64/libc.so.6

Thread 4 (Thread 16807):
#0  0x00007fdf6625b1e4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fdf664ec29f in wait (this=<value optimized out>, mutex=0x17fc0d8, time=<value optimized out>) at thread/qwaitcondition_unix.cpp:88
#2  QWaitCondition::wait (this=<value optimized out>, mutex=0x17fc0d8, time=<value optimized out>) at thread/qwaitcondition_unix.cpp:160
#3  0x000000000058b2a1 in Digikam::ScanController::run (this=0x17fbd60) at /home/vivo/digikam-devel/digikam-sc/core/digikam/database/scancontroller.cpp:618
#4  0x00007fdf664eba8a in QThreadPrivate::start (arg=0x17fbd60) at thread/qthread_unix.cpp:320
#5  0x00007fdf66256c00 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fdf65285b0d in clone () from /lib64/libc.so.6

Thread 3 (Thread 16808):
#0  0x00007fdf6527db23 in poll () from /lib64/libc.so.6
#1  0x00007fdf64ace08d in ?? () from /usr/lib64/libglib-2.0.so.0
#2  0x00007fdf64ace5bd in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0
#3  0x00007fdf66612def in QEventDispatcherGlib::processEvents (this=0x182d6d0, flags=<value optimized out>) at kernel/qeventdispatcher_glib.cpp:422
#4  0x00007fdf665e3562 in QEventLoop::processEvents (this=<value optimized out>, flags=...) at kernel/qeventloop.cpp:149
#5  0x00007fdf665e37a5 in QEventLoop::exec (this=0x7fdf49672d20, flags=...) at kernel/qeventloop.cpp:201
#6  0x00007fdf664e8db8 in QThread::exec (this=<value optimized out>) at thread/qthread.cpp:492
#7  0x00007fdf665c3480 in QInotifyFileSystemWatcherEngine::run (this=0x183a020) at io/qfilesystemwatcher_inotify.cpp:248
#8  0x00007fdf664eba8a in QThreadPrivate::start (arg=0x183a020) at thread/qthread_unix.cpp:320
#9  0x00007fdf66256c00 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fdf65285b0d in clone () from /lib64/libc.so.6

Thread 2 (Thread 16895):
#0  0x00007fdf6625b1e4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fdf664ec29f in wait (this=<value optimized out>, mutex=0x18cf348, time=<value optimized out>) at thread/qwaitcondition_unix.cpp:88
#2  QWaitCondition::wait (this=<value optimized out>, mutex=0x18cf348, time=<value optimized out>) at thread/qwaitcondition_unix.cpp:160
#3  0x00007fdf6b2b89c6 in Digikam::ParkingThread::run (this=0x18cf330) at /home/vivo/digikam-devel/digikam-sc/core/libs/threads/threadmanager.cpp:119
#4  0x00007fdf664eba8a in QThreadPrivate::start (arg=0x18cf330) at thread/qthread_unix.cpp:320
#5  0x00007fdf66256c00 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fdf65285b0d in clone () from /lib64/libc.so.6

Thread 1 (Thread 18246):
#0  ClearBitBlock (stream=0x7fdf26ba94c0, pos=82071, len=<value optimized out>)
    at /home/vivo/digikam-devel/digikam-sc/core/libs/3rdparty/libpgf/BitStream.h:159
#1  0x00007fdf6b2aaaab in CDecoder::RLDSigsAndSigns (this=0x7fdf44095d90, bufferSize=16384, codeLen=109243, sigBits=0x7fdf26ba94c0, signBits=0x7fdf26ba8cc0)
    at /home/vivo/digikam-devel/digikam-sc/core/libs/3rdparty/libpgf/Decoder.cpp:726
#2  0x00007fdf6b2aaea0 in CDecoder::BitplaneDecode (this=0x7fdf44095d90, bufferSize=16384)
    at /home/vivo/digikam-devel/digikam-sc/core/libs/3rdparty/libpgf/Decoder.cpp:536
#3  0x0000000000000000 in ?? ()
Comment 1 caulier.gilles 2011-05-21 11:11:22 UTC
>there is some way to know from where the data is feed into libpgf?

I/O PGF image data are wrapped to QImage in this source code :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/threadimageio/pgfutils.cpp

And managed in DB with this class :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/threadimageio/thumbnailcreator.cpp#L886

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/threadimageio/thumbnailcreator.cpp#L886

>BTW the crash happen even with external libpgf


Are you sure, because in your backtrace, you use libpgf from 3rdparty digiKam core...

Gilles Caulier
Comment 2 Marcel Wiesweg 2011-05-21 12:40:27 UTC
Are steps 1) to 3) really necessary to reproduce? Could it be there is "simply" a PGF file produced by digikam which crashes digikam when opened?
Comment 3 Francesco Riosa 2011-05-23 16:21:05 UTC
Created attachment 60243 [details]
getThumbs.py extract pgf thumbnail from database

(In reply to comment #1)
[snip]
> >BTW the crash happen even with external libpgf
> 
> Are you sure, because in your backtrace, you use libpgf from 3rdparty digiKam
> core...

yes, I've recompiled digiKam with internal libpgf only later and the backtrace is the same
I'm digging the sources right now at least to understund how it crashes, thanks for the pointers

(In reply to comment #2)
> Are steps 1) to 3) really necessary to reproduce? Could it be there is "simply"
> a PGF file produced by digikam which crashes digikam when opened?

No, that's how the problem generated initially, there is _only_ a pgf broken:
http://xn--wtf-xs0a.ws/Bug273765/_dsc4243.pgf

I've also extracted all thumbnails from the database to pgf files and they are all readable

I'm looking into showfoto, since it's reproducible there and there are less threads involved ;)
Comment 4 S. Burmeister 2011-05-23 22:43:49 UTC
Self-compiled without external libpgf crashes when I press F4 to edit the image. Preview etc. works ok.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xa2b2ab70 (LWP 32539)]
0xb672036b in ClearBitBlock (stream=0xa2b28724, pos=73170, len=16384)
    at /home/rabauke/devel/src/digikam-software-compilation/core/libs/3rdparty/libpgf/BitStream.h:159
159                             stream[i] = 0;
(gdb) bt
#0  0xb672036b in ClearBitBlock (stream=0xa2b28724, pos=73170, len=16384)
    at /home/rabauke/devel/src/digikam-software-compilation/core/libs/3rdparty/libpgf/BitStream.h:159
#1  0xb671ff7b in CDecoder::RLDSigsAndSigns (this=0xa1ac3738, bufferSize=16384, codeLen=522, sigBits=0xa2b28724, signBits=0xa2b27f24)
    at /home/rabauke/devel/src/digikam-software-compilation/core/libs/3rdparty/libpgf/Decoder.cpp:726
#2  0xb671f9e6 in CDecoder::BitplaneDecode (this=0x0, bufferSize=0)
    at /home/rabauke/devel/src/digikam-software-compilation/core/libs/3rdparty/libpgf/Decoder.cpp:536
#3  0x00000000 in ?? ()
(gdb) 292056 FLAGS ($IGNORED)

#0  0xb672036b in ClearBitBlock (stream=0xa2b28724, pos=73170, len=16384)
    at /home/rabauke/devel/src/digikam-software-compilation/core/libs/3rdparty/libpgf/BitStream.h:159
#1  0xb671ff7b in CDecoder::RLDSigsAndSigns (this=0xa1ac3738, bufferSize=16384, codeLen=522, sigBits=0xa2b28724, signBits=0xa2b27f24)
    at /home/rabauke/devel/src/digikam-software-compilation/core/libs/3rdparty/libpgf/Decoder.cpp:726
#2  0xb671f9e6 in CDecoder::BitplaneDecode (this=0x0, bufferSize=0)
    at /home/rabauke/devel/src/digikam-software-compilation/core/libs/3rdparty/libpgf/Decoder.cpp:536
#3  0x00000000 in ?? ()


konsole output:

QSqlDatabasePrivate::removeDatabase: connection 'ConnectionTest' is still in use, all queries will cease to work.
digikam(32499)/KIPI (loading) KIPI::PluginLoader::Info::plugin: KIPI::PluginLoader:: createInstance returned 0 for  "Kalender"  ( "kipiplugin_calendar" )  with error:  "Could not find plugin 'Kalender' for application 'digikam'" 
digikam(32499)/KIPI (loading) KIPI::PluginLoader::Info::plugin: KIPI::PluginLoader:: createInstance returned 0 for  "DNGkonverter"  ( "kipiplugin_dngconverter" )  with error:  "Could not find plugin 'DNGkonverter' for application 'digikam'" 
Exiv2::PgfImage::readMetadata: Reading PGF file /home/rabauke/Pictures/_dsc4243.pgf
Exiv2::PgfImage: PGF header size : 79242 bytes
Exiv2::PgfImage::readMetadata: Found Image data (79226 bytes)
Exiv2::PgfImage::readMetadata: Reading PGF file /home/rabauke/Pictures/_dsc4243.pgf
Exiv2::PgfImage: PGF header size : 79242 bytes
Exiv2::PgfImage::readMetadata: Found Image data (79226 bytes)
Exiv2::PgfImage::readMetadata: Reading PGF file /home/rabauke/Pictures/_dsc4243.pgf
Exiv2::PgfImage: PGF header size : 79242 bytes
Exiv2::PgfImage::readMetadata: Found Image data (79226 bytes)
Exiv2::PgfImage::readMetadata: Reading PGF file /home/rabauke/Pictures/_dsc4243.pgf
Exiv2::PgfImage: PGF header size : 79242 bytes
Exiv2::PgfImage::readMetadata: Found Image data (79226 bytes)
Exiv2::PgfImage::readMetadata: Reading PGF file /home/rabauke/Pictures/_dsc4243.pgf
Exiv2::PgfImage: PGF header size : 79242 bytes
Exiv2::PgfImage::readMetadata: Found Image data (79226 bytes)
Exiv2::PgfImage::readMetadata: Reading PGF file /home/rabauke/Pictures/_dsc4243.pgf
Exiv2::PgfImage: PGF header size : 79242 bytes
Exiv2::PgfImage::readMetadata: Found Image data (79226 bytes)
Exiv2::PgfImage::readMetadata: Reading PGF file /home/rabauke/Pictures/_dsc4243.pgf
Exiv2::PgfImage: PGF header size : 79242 bytes
Exiv2::PgfImage::readMetadata: Found Image data (79226 bytes)
Speicherzugriffsfehler

(memory corruption)

digiKam version 2.0.0-beta6
Exiv2 can write to Jp2: Yes
Exiv2 can write to Jpeg: Yes
Exiv2 can write to Pgf: Yes
Exiv2 can write to Png: Yes
Exiv2 can write to Tiff: Yes
Exiv2 supports XMP metadata: Yes
LibCImg: 130
LibClapack: internal library
LibExiv2: 0.20
LibJPEG: 62
LibJasper: 1.900.1
LibKDE: 4.6.3 (4.6.3)
LibKExiv2: 2.0.0
LibKMap: 2.0.0
LibKdcraw: 2.0.0
LibLCMS: 119
LibPGF: 6.09.44 - internal library
LibPNG: 1.4.4
LibQt: 4.7.3
LibRaw: 0.13.5
LibTIFF: LIBTIFF, Version 3.9.4 Copyright (c) 1988-1996 Sam Leffler Copyright (c) 1991-1996 Silicon Graphics, Inc.
Marble Widget: 0.11.2 (Stable Release)
Parallelized demosaicing: Yes
Database backend: QSQLITE
LibGphoto2: 2.4.10
LibKface: 2.0.0
LibKipi: 1.2.0
LibOpenCV: 2.2.0
Libface: 0.2
Comment 5 Francesco Riosa 2011-05-24 21:22:48 UTC
ok, after a lot of learning of gdb, and some reading on signals and exceptions and also pgf specs:

There are two parts of this bug:
The first is a bad pgf file generated from the batch queque manager, I would consider this a non issue, since there was intervention from the shell while digikam was still running

The second is a unpleasant crash when a corrupted file is loaded:

The file is corrupted (redoing it give a correct one #1), _dsc4243.pgf headers are correct #2 so it pass all initial checks and then if asserts are enabled it trigger
ASSERT(sigPos + runlen <= BufferSize);
in Decoder.cpp:725 CDecoder::RLDSigsAndSigns(...)

For this example it's impossible to catch the problem early as it's done for other things in pgfloader.cpp

Seem to me that the only solutions available are:
- rewrite libpgf to handle exceptions (slowing it down) and to fit better in a qt env (no no no)
- fork an external process to read te image (like plasma does with plasmoids) which can be inefficient at best
- register a SIGINT catcher
- close as wont fix

any ideas?

Note to self: converting .nef in batch resize+pgf does not give byte exact files if run twice there are usually at least 3 or 4 bytes that differ

#1 http://xn--wtf-xs0a.ws/Bug273765/_dsc4243_ok.pgf
#2 I've created a structure for okteta available here:
http://kde-files.org/content/show.php/PGF+file+header+definition?content=142016
Comment 6 Francesco Riosa 2011-05-24 21:27:28 UTC
(In reply to comment #5)
> - rewrite libpgf to handle exceptions (slowing it down) and to fit better in a

PGFplatform.h already contain the code below:

#ifndef ASSERT
	#ifdef _DEBUG
		#define ASSERT(x)	assert(x)
	#else
		#if defined(__GNUC__)
			#define ASSERT(ignore)((void) 0) 
		#elif _MSC_VER >= 1300 
			#define ASSERT		__noop
		#else
			#define ASSERT ((void)0)
		#endif
	#endif //_DEBUG
#endif //ASSERT

Maybe something easy can be added here and the Decoder.ccp line //ASSERT(sigPos + runlen <= BufferSize); uncommented

now something what?
Comment 7 caulier.gilles 2011-05-26 19:54:16 UTC
Francesco,

I CC Raphael Schweizer, who is LibPGF author and maintainer. We will see if he can help us to hack this indeep problem in PGF library.

PS: Can you check if valgrind give more info about libpgf problem ?

Gilles Caulier
Comment 8 Francesco Riosa 2011-05-29 15:11:15 UTC
Created attachment 60435 [details]
valgrind-broken-pgf.zip

valgrind-broken-pgf.zip include the output from:

valgrind --tool=memcheck --leak-check=full \
  --error-limit=no --suppressions=project/digikam.supp \
  showfoto --nocrashhandler _dsc4243_ok.pgf \
  > valgrind-pgf-ok-1.txt \
  2> valgrind-pgf-ok-2.txt

and similar for the broken pdf

lines 273-335 of valgrind-pgf-bad-2.txt show the program starting to crash there is a "Invalid write of size 4" followed by "Invalid read of size 4"

"Conditional jump or move depends on uninitialised value" it's an error that in the opening of a "good" pgf file happen only once, opening the bad pgf it's ubiquitus
Comment 9 Marcel Wiesweg 2011-05-29 17:15:48 UTC
libpgf should not crash, but crashes happen (unexpected by the programmer) and are fixed when seen. 

But it must never never never abort because of an error that it has seen and has checked for (expected by the programmer).
It must fail loading the invalid file, but not crash the whole process!!
Comment 10 Raphael Schweizer 2011-05-30 13:06:27 UTC
Francesco,

thanks for reporting this issue and gathering all the helpful details.
We will investigate further and provide a patch shortly.

Raphael
Comment 11 Marcel Wiesweg 2011-05-30 21:45:40 UTC
Thanks Raphael. I may also once again bring this image to your attention:
http://digikam3rdparty.free.fr/TEST_IMAGES/PGF/DSC03274_v1.pgf
which is also created by digikam and crashes at the same place

Valgrind:

==14132== Conditional jump or move depends on uninitialised value(s)
==14132==    at 0x7FBC016: CDecoder::ComposeBitplane(unsigned int, unsigned 
int, unsigned int*, unsigned int*, unsigned int*) (BitStream.h:210)
==14132==    by 0x7FBC86B: CDecoder::BitplaneDecode(unsigned int) 
(Decoder.cpp:603)
==14132==    by 0x7FBCAA9: CDecoder::DecodeBuffer() (Decoder.cpp:451)
==14132==    by 0x7FBCB41: CDecoder::DequantizeValue(CSubband*, unsigned int, 
int) (Decoder.cpp:394)
==14132==    by 0x7FBD036: CDecoder::Partition(CSubband*, int, int, int, int, 
int) (Decoder.cpp:213)
==14132==    by 0x7FC4D75: CSubband::PlaceTile(CDecoder&, int, bool, unsigned 
int, unsigned int) (Subband.cpp:227)
==14132==    by 0x7FBF69D: CPGFImage::Read(int, bool (*)(double, bool, void*), 
void*) (PGFimage.cpp:286)

==14132== 
==14132== Conditional jump or move depends on uninitialised value(s)
==14132==    at 0x7FBC71A: CDecoder::BitplaneDecode(unsigned int) 
(Decoder.cpp:526)
==14132==    by 0x7FBCAA9: CDecoder::DecodeBuffer() (Decoder.cpp:451)
==14132==    by 0x7FBCB41: CDecoder::DequantizeValue(CSubband*, unsigned int, 
int) (Decoder.cpp:394)
==14132==    by 0x7FBD036: CDecoder::Partition(CSubband*, int, int, int, int, 
int) (Decoder.cpp:213)
==14132==    by 0x7FC4D75: CSubband::PlaceTile(CDecoder&, int, bool, unsigned 
int, unsigned int) (Subband.cpp:227)
==14132==    by 0x7FBF69D: CPGFImage::Read(int, bool (*)(double, bool, void*), 
void*) (PGFimage.cpp:286)
==14132==    by 0x7E079B3: Digikam::PGFLoader::load(QString const&, 
Digikam::DImgLoaderObserver*) (pgfloader.cpp:290)

==14132== 
==14132== Conditional jump or move depends on uninitialised value(s)
==14132==    at 0x7FBC162: CDecoder::RLDSigsAndSigns(unsigned int, unsigned 
int, unsigned int*, unsigned int*) (Decoder.cpp:690)
==14132==    by 0x7FBC911: CDecoder::BitplaneDecode(unsigned int) 
(Decoder.cpp:536)
==14132==    by 0x7FBCAA9: CDecoder::DecodeBuffer() (Decoder.cpp:451)
==14132==    by 0x7FBCB41: CDecoder::DequantizeValue(CSubband*, unsigned int, 
int) (Decoder.cpp:394)
==14132==    by 0x7FBD036: CDecoder::Partition(CSubband*, int, int, int, int, 
int) (Decoder.cpp:213)
==14132==    by 0x7FC4D75: CSubband::PlaceTile(CDecoder&, int, bool, unsigned 
int, unsigned int) (Subband.cpp:227)
==14132==    by 0x7FBF69D: CPGFImage::Read(int, bool (*)(double, bool, void*), 
void*) (PGFimage.cpp:286)

==14132== 
==14132== Conditional jump or move depends on uninitialised value(s)
==14132==    at 0x7FBC183: CDecoder::RLDSigsAndSigns(unsigned int, unsigned 
int, unsigned int*, unsigned int*) (Decoder.cpp:691)
==14132==    by 0x7FBC911: CDecoder::BitplaneDecode(unsigned int) 
(Decoder.cpp:536)
==14132==    by 0x7FBCAA9: CDecoder::DecodeBuffer() (Decoder.cpp:451)
==14132==    by 0x7FBCB41: CDecoder::DequantizeValue(CSubband*, unsigned int, 
int) (Decoder.cpp:394)
==14132==    by 0x7FBD036: CDecoder::Partition(CSubband*, int, int, int, int, 
int) (Decoder.cpp:213)
==14132==    by 0x7FC4D75: CSubband::PlaceTile(CDecoder&, int, bool, unsigned 
int, unsigned int) (Subband.cpp:227)
==14132==    by 0x7FBF69D: CPGFImage::Read(int, bool (*)(double, bool, void*), 
void*) (PGFimage.cpp:286)

==14132== 
==14132== Conditional jump or move depends on uninitialised value(s)
==14132==    at 0x7FBC21E: CDecoder::RLDSigsAndSigns(unsigned int, unsigned 
int, unsigned int*, unsigned int*) (Decoder.cpp:699)
==14132==    by 0x7FBC911: CDecoder::BitplaneDecode(unsigned int) 
(Decoder.cpp:536)
==14132==    by 0x7FBCAA9: CDecoder::DecodeBuffer() (Decoder.cpp:451)
==14132==    by 0x7FBCB41: CDecoder::DequantizeValue(CSubband*, unsigned int, 
int) (Decoder.cpp:394)
==14132==    by 0x7FBD036: CDecoder::Partition(CSubband*, int, int, int, int, 
int) (Decoder.cpp:213)
==14132==    by 0x7FC4D75: CSubband::PlaceTile(CDecoder&, int, bool, unsigned 
int, unsigned int) (Subband.cpp:227)
==14132==    by 0x7FBF69D: CPGFImage::Read(int, bool (*)(double, bool, void*), 
void*) (PGFimage.cpp:286)

And when the reader bug is fixed (and the image is indeed invalid), the next bug to hunt is in the writer, when these images are created ;-)
Comment 12 Marcel Wiesweg 2011-05-30 21:53:33 UTC
As a hint for the writer bug: valgrind gives the following eight errors when saving a PGF.
Uninitialized value problems tend to work in 99% of cases but break for the rest (could explain this bug is rarely seen)

==28252== Conditional jump or move depends on uninitialised value(s)
==28252==    at 0x80A6813: CEncoder::RLESigsAndSigns(unsigned int*, unsigned int, unsigned int*, unsigned int) (BitStream.h:203)
==28252==    by 0x80A7054: CEncoder::BitplaneEncode(unsigned int) (Encoder.cpp:380)
==28252==    by 0x80A72F7: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:274)
==28252==    by 0x80A7461: CEncoder::WriteValue(CSubband*, int) (Encoder.cpp:253)
==28252==    by 0x80A7544: CEncoder::Partition(CSubband*, int, int, int, int) (Encoder.cpp:152)
==28252==    by 0x80AE01D: CSubband::ExtractTile(CEncoder&, int, bool, unsigned int, unsigned int) (Subband.cpp:188)
==28252==    by 0x80A956C: CPGFImage::Write(CPGFStream*, int, bool (*)(double, bool, void*), unsigned int*, void*) (PGFimage.cpp:922)

==28252== 
==28252== Conditional jump or move depends on uninitialised value(s)
==28252==    at 0x80A686A: CEncoder::RLESigsAndSigns(unsigned int*, unsigned int, unsigned int*, unsigned int) (Encoder.cpp:628)
==28252==    by 0x80A7054: CEncoder::BitplaneEncode(unsigned int) (Encoder.cpp:380)
==28252==    by 0x80A72F7: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:274)
==28252==    by 0x80A7461: CEncoder::WriteValue(CSubband*, int) (Encoder.cpp:253)
==28252==    by 0x80A7544: CEncoder::Partition(CSubband*, int, int, int, int) (Encoder.cpp:152)
==28252==    by 0x80AE01D: CSubband::ExtractTile(CEncoder&, int, bool, unsigned int, unsigned int) (Subband.cpp:188)
==28252==    by 0x80A956C: CPGFImage::Write(CPGFStream*, int, bool (*)(double, bool, void*), unsigned int*, void*) (PGFimage.cpp:922)

==28252== 
==28252== Syscall param write(buf) points to uninitialised byte(s)
==28252==    at 0xBA6131D: ??? (in /lib64/libpthread-2.11.3.so)
==28252==    by 0x80AD921: CPGFFileStream::Write(int*, void*) (PGFplatform.h:510)
==28252==    by 0x80A7352: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:311)
==28252==    by 0x80A7461: CEncoder::WriteValue(CSubband*, int) (Encoder.cpp:253)
==28252==    by 0x80A7544: CEncoder::Partition(CSubband*, int, int, int, int) (Encoder.cpp:152)
==28252==    by 0x80AE01D: CSubband::ExtractTile(CEncoder&, int, bool, unsigned int, unsigned int) (Subband.cpp:188)
==28252==    by 0x80A956C: CPGFImage::Write(CPGFStream*, int, bool (*)(double, bool, void*), unsigned int*, void*) (PGFimage.cpp:922)
==28252==  Address 0x359a3b35 is on thread 5's stack


==28252== Conditional jump or move depends on uninitialised value(s)
==28252==    at 0x80A6AB6: CEncoder::RLESigns(unsigned int*, unsigned int) (BitStream.h:233)
==28252==    by 0x80A7133: CEncoder::BitplaneEncode(unsigned int) (Encoder.cpp:418)
==28252==    by 0x80A72F7: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:274)
==28252==    by 0x80A7461: CEncoder::WriteValue(CSubband*, int) (Encoder.cpp:253)
==28252==    by 0x80A7544: CEncoder::Partition(CSubband*, int, int, int, int) (Encoder.cpp:152)
==28252==    by 0x80AE01D: CSubband::ExtractTile(CEncoder&, int, bool, unsigned int, unsigned int) (Subband.cpp:188)
==28252==    by 0x80A963B: CPGFImage::Write(CPGFStream*, int, bool (*)(double, bool, void*), unsigned int*, void*) (PGFimage.cpp:932)

==28252== 
==28252== Conditional jump or move depends on uninitialised value(s)
==28252==    at 0x80A6A9C: CEncoder::RLESigns(unsigned int*, unsigned int) (BitStream.h:233)
==28252==    by 0x80A7133: CEncoder::BitplaneEncode(unsigned int) (Encoder.cpp:418)
==28252==    by 0x80A72F7: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:274)
==28252==    by 0x80A7461: CEncoder::WriteValue(CSubband*, int) (Encoder.cpp:253)
==28252==    by 0x80A7544: CEncoder::Partition(CSubband*, int, int, int, int) (Encoder.cpp:152)
==28252==    by 0x80AE01D: CSubband::ExtractTile(CEncoder&, int, bool, unsigned int, unsigned int) (Subband.cpp:188)
==28252==    by 0x80A966F: CPGFImage::Write(CPGFStream*, int, bool (*)(double, bool, void*), unsigned int*, void*) (PGFimage.cpp:933)

==28252== 
==28252== Conditional jump or move depends on uninitialised value(s)
==28252==    at 0x80A6813: CEncoder::RLESigsAndSigns(unsigned int*, unsigned int, unsigned int*, unsigned int) (BitStream.h:203)
==28252==    by 0x80A7054: CEncoder::BitplaneEncode(unsigned int) (Encoder.cpp:380)
==28252==    by 0x80A72F7: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:274)
==28252==    by 0x80A76BC: CEncoder::Flush() (Encoder.cpp:216)
==28252==    by 0x80A96D1: CPGFImage::Write(CPGFStream*, int, bool (*)(double, bool, void*), unsigned int*, void*) (PGFimage.cpp:953)

==28252== Conditional jump or move depends on uninitialised value(s)
==28252==    at 0x80A686A: CEncoder::RLESigsAndSigns(unsigned int*, unsigned int, unsigned int*, unsigned int) (Encoder.cpp:628)
==28252==    by 0x80A7054: CEncoder::BitplaneEncode(unsigned int) (Encoder.cpp:380)
==28252==    by 0x80A72F7: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:274)
==28252==    by 0x80A76BC: CEncoder::Flush() (Encoder.cpp:216)
==28252==    by 0x80A96D1: CPGFImage::Write(CPGFStream*, int, bool (*)(double, bool, void*), unsigned int*, void*) (PGFimage.cpp:953)

==28252== Syscall param write(buf) points to uninitialised byte(s)
==28252==    at 0xBA6131D: ??? (in /lib64/libpthread-2.11.3.so)
==28252==    by 0x80AD921: CPGFFileStream::Write(int*, void*) (PGFplatform.h:510)
==28252==    by 0x80A7352: CEncoder::EncodeBuffer(ROIBlockHeader) (Encoder.cpp:311)
==28252==    by 0x80A76BC: CEncoder::Flush() (Encoder.cpp:216)
==28252==    by 0x80A96D1: CPGFImage::Write(CPGFStream*, int, bool (*)(double, bool, void*), unsigned int*, void*) (PGFimage.cpp:953)
==28252==    by 0x7EE1544: Digikam::PGFLoader::save(QString const&, Digikam::DImgLoaderObserver*) (pgfloader.cpp:438)
Comment 13 Raphael Schweizer 2011-06-06 11:35:15 UTC
Currently there is ongoing work on a new, slightly 'OMP'ified, version of the codec which does not contain anymore the notorious CEncoder::RLESigsAndSigns method.
So, contradictory to "we [...] provide a patch shortly", there will be no prompt fix, unless you have objections.

Marcel, could you also provide the source image for http://digikam3rdparty.free.fr/TEST_IMAGES/PGF/DSC03274_v1.pgf ? Was this image converted using digikam without any obvious errors? We never had any signs of libPGF encoding sane images into corrupted PGF images.
And also "converting .nef in batch resize+pgf does not give byte exact
files if run twice there are usually at least 3 or 4 bytes that differ" sounds strange, as PGF really should be deterministic.
Francesco, could you check whether this is PGF or the batch resize? I.e. whether the resized pictures always are byte-equal?

- Raphael
Comment 14 Francesco Riosa 2011-06-06 16:53:12 UTC
(In reply to comment #13)

> [SNIP]
> And also "converting .nef in batch resize+pgf does not give byte exact
> files if run twice there are usually at least 3 or 4 bytes that differ" sounds
> strange, as PGF really should be deterministic.
> Francesco, could you check whether this is PGF or the batch resize? I.e.
> whether the resized pictures always are byte-equal?
> 
> - Raphael

I've tried batch resize+PNG and it give two different images if run twice, probably somewhere between raw converter and resizer there is some random value added nothing too worrysome and
PGF encoder has nothing to do with that
Comment 15 Francesco Riosa 2011-06-06 16:57:21 UTC
> - Raphael

also, just out of curiosity, have you considered orc, successors of liboil (http://code.entropywave.com/projects/orc/)
it claim to make easy to optimize operations on contiguous arrays, only because you mentioned "omp"ification
Comment 16 Raphael Schweizer 2011-06-06 17:48:35 UTC
Francesco,

thanks for checking (glad we don't have to deal with unintended randomness in PGF).
Orc looks like an interesting approach (having dealt with SSE intrinsics, making them more abstract is tempting). However, I don't think we will see Orc code within libPGF anytime soon. Windows users are not as used to complicated build processes as others are, and Orc's instructions (respectively the lack of WIN related ones) look frightening. Also, my test results / benchmarks using SSE are not very encouraging. Extrapolating them could make AVX interesting, though. But this would require a major remake of most array intensive code in libPGF, while still maintaining compatibility with older CPUs.

- Raphael
Comment 17 Marcel Wiesweg 2011-06-07 21:16:51 UTC
Yes I have the source, but I could not reproduce an image that crashes. At this one day, I produced three images that crashed; since then, none has. The source are normal JPG images, some filters were applied. Nothing special in that regard.

What I can reproduce are the uninitialized-value errors. (As I said, it would be well possible that an uninitialized memory area is almost always as expected. I once found the status of uninitialized memory very much reproducible)
They may just as well be completely misleading though.
I would guess that refBits and signBits are not set to 0; I cannot judge if this is relevant.
Comment 18 Raphael Schweizer 2011-06-09 19:24:10 UTC
Thank you very much. There's another new report of indeterminism (http://sourceforge.net/projects/libpgf/forums/forum/530086/topic/4566988), possibly due to padding w/ uninitialized buffers. I'll definitely look into this once more and make sure this is solved for the next release.

- Raphael
Comment 19 Marcel Wiesweg 2011-06-11 16:40:44 UTC
And you will surely also fix the crash when reading the invalid file (instead, failing to load it)? ;-)
Comment 20 Raphael Schweizer 2011-06-16 17:02:28 UTC
New version out now: including
- fixed crashes
- deterministic encoder
- OpenMP support

We are looking forward to your test result.

- Raphael
Comment 21 caulier.gilles 2011-06-16 19:53:21 UTC
Raphael,

I patched digiKam with your last libpgf tarball...

There is a problem with OpenMP dependency : it's not optional. It must be.

Un MACOSX, GCC do not support OpenMP very well, especially when parallelized code run into separated thread. We have already see very weird side-effect with libraw under Mac.

I think it can be easily patched. I see in your code that you include hard omp.h. It must be wrapped in prepocessor like this :

https://projects.kde.org/projects/kde/kdegraphics/libs/libkdcraw/repository/revisions/master/entry/libraw/libraw/libraw_types.h#L32

Gilles Caulier
Comment 22 caulier.gilles 2011-06-17 08:57:26 UTC
Git commit 46c8af586c53127ea48eebad23ff5968bcd9aa5c by Gilles Caulier.
Committed on 17/06/2011 at 08:55.
Pushed by cgilles into branch 'master'.

update libpgf from digiKam core to last 6.11.24 including OpenMP support
CCBUGS: 273765

M  +22   -1    CMakeLists.txt     
M  +1    -0    NEWS     
M  +3    -0    digikam/CMakeLists.txt     
M  +56   -53   libs/3rdparty/libpgf/BitStream.h     
M  +458  -272  libs/3rdparty/libpgf/Decoder.cpp     
M  +62   -35   libs/3rdparty/libpgf/Decoder.h     
M  +358  -307  libs/3rdparty/libpgf/Encoder.cpp     
M  +96   -36   libs/3rdparty/libpgf/Encoder.h     
M  +703  -483  libs/3rdparty/libpgf/PGFimage.cpp     
M  +114  -58   libs/3rdparty/libpgf/PGFimage.h     
M  +70   -61   libs/3rdparty/libpgf/PGFplatform.h     
A  +272  -0    libs/3rdparty/libpgf/PGFstream.cpp         [License: BSD]  *
A  +185  -0    libs/3rdparty/libpgf/PGFstream.h         [License: BSD]  *
M  +35   -21   libs/3rdparty/libpgf/PGFtypes.h     
D  +0    -254  libs/3rdparty/libpgf/Stream.cpp     
D  +0    -169  libs/3rdparty/libpgf/Stream.h     
M  +16   -13   libs/3rdparty/libpgf/Subband.cpp     
M  +14   -9    libs/3rdparty/libpgf/Subband.h     
M  +55   -48   libs/3rdparty/libpgf/WaveletTransform.cpp     
M  +17   -10   libs/3rdparty/libpgf/WaveletTransform.h     

The files marked with a * at the end have a non valid license. Please read: http://techbase.kde.org/Policies/Licensing_Policy and use the headers which are listed at that page.


http://commits.kde.org/digikam/46c8af586c53127ea48eebad23ff5968bcd9aa5c
Comment 23 caulier.gilles 2011-06-17 09:16:55 UTC
Git commit 781571dd5b72d50d3e26e70bd4023938f20df94a by Gilles Caulier.
Committed on 17/06/2011 at 09:14.
Pushed by cgilles into branch 'master'.

patch against original libpgf 6.11.24 to :
- Handle properly OpenMP depending of compiler/OS (inspired from libraw.org).
- Include only one omp.h instance in PGFplatform.h
- Support no OpenMP availability (else libpgf do not compile without OpenMP).
This patch must be included in next stable libpgf.

Raphael, i recommend too to change EOL as Unix like and to replace tab by 4 spaces.

CCBUGS: 273765
CCMAIL: rschweizer@schweizer-informatik.ch

M  +3    -1    libs/3rdparty/libpgf/Decoder.cpp     
M  +3    -1    libs/3rdparty/libpgf/Encoder.cpp     
M  +25   -0    libs/3rdparty/libpgf/PGFplatform.h     

http://commits.kde.org/digikam/781571dd5b72d50d3e26e70bd4023938f20df94a
Comment 24 caulier.gilles 2011-06-17 09:21:40 UTC
Raphael,

digiKam with new libpgf 6.11.24 compile fine under Linux with or without OpenMP with my personnal patch from #23. Please include it to your official repository.

digiKam run and do not crash under my Linux Mandriva  (thumbnails generation and file save as/load PGF in image editor)

I will check if all compile and run fine under MACosX and Windows (using MSVC compiler)

Gilles Caulier
Comment 25 caulier.gilles 2011-06-17 17:38:03 UTC
Raphael,

digiKam with new libpgf 6.11.24 compile fine under MacOSX. Note that OpenMP support is disabled, as for libraw, to prevent crash when parallelized code run in a separated thread (current limitation of OpenMp support under MacOSX).

Gilles Caulier
Comment 26 caulier.gilles 2011-06-18 09:54:56 UTC
Git commit dbb1f15dbe0998b9cc806ef32f7c15afbde1614c by Gilles Caulier.
Committed on 18/06/2011 at 09:50.
Pushed by cgilles into branch 'master'.

Another patch about OpenMp support for libpgf :

- M$ Visual C++ 2008 support well OpenMP. Do not limit OpenMP support to Visual C++ 2010.
- use dedicated OpenMP detection define everywhere if OpenMP will be used with libpgf implementation.

Raphael, now, digiKam+libpgf compile fine with M$ Visual C++. Please include this patch into official and current libpgf repository. Thanks in advance.

CCBUGS: 273765
CCMAIL: rschweizer@schweizer-informatik.ch

M  +2    -2    libs/3rdparty/libpgf/Decoder.cpp     
M  +2    -2    libs/3rdparty/libpgf/Encoder.cpp     
M  +2    -2    libs/3rdparty/libpgf/PGFplatform.h     

http://commits.kde.org/digikam/dbb1f15dbe0998b9cc806ef32f7c15afbde1614c
Comment 27 caulier.gilles 2011-06-18 10:39:21 UTC
To all in this room.

digiKam from git master include now the new libpgf library + few patches for OpenMP support. Please checkout last digiKam code and try again to see if crash is reproducible.

Thanks in advance

Gilles Caulier
Comment 28 Francesco Riosa 2011-06-18 16:48:05 UTC
(In reply to comment #27)
> To all in this room.
> 
> digiKam from git master include now the new libpgf library + few patches for
> OpenMP support. Please checkout last digiKam code and try again to see if crash
> is reproducible.
> 
> Thanks in advance
> 
> Gilles Caulier

_dsc4243.pgf does not crash showfoto any more, thanks to everybody involved
Comment 29 Marcel Wiesweg 2011-06-18 16:52:05 UTC
The two pgf examples (in tests/) do not link anymore. Any change in symbol visibility?
Comment 30 caulier.gilles 2011-06-18 17:28:04 UTC
Marcel,

I don't change visibility. I don't check test source code compilation too. I will take a look.

Gilles Caulier
Comment 31 caulier.gilles 2011-06-18 17:37:36 UTC
Marcel,

Since libpgf have been updated into digiKam, under windows, thumbnails database do not sound to work as espected. There is no thumbnails cache effects visible between 2 digiKam sessions.

Can you confirm this problem under Linux ?

Gilles Caulier
Comment 32 Raphael Schweizer 2011-06-18 18:41:32 UTC
Thanks for all the testing and the patches. I'm pleasantly surprised of your fast pace. Our source server is currently down, I'll apply patches and update sourceforge as soon as I get access again.

Thanks, Raphael
Comment 33 caulier.gilles 2011-06-18 20:02:02 UTC
Raphael,

use test compilation mode from KDE sdk, more GCC warnings options are turned on. I can see this :

[ 55%] Building CXX object core/digikam/CMakeFiles/digikamdatabase.dir/__/libs/3rdparty/libpgf/Decoder.cpp.o                                         
In file included from /mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.cpp:29:                                                              
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h: In constructor ‘CDecoder::CMacroBlock::CMacroBlock(CDecoder*)’:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:74: attention : ‘CDecoder::CMacroBlock::m_decoder’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:62: attention :   ‘ROIBlockHeader CDecoder::CMacroBlock::m_header’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:52: attention :   when initialized here
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h: In constructor ‘CDecoder::CDecoder(CPGFStream*, PGFPreHeader&, PGFHeader&, PGFPostHeader&, UINT32*&, bool)’:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:180: attention : ‘CDecoder::m_encodedHeaderLength’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:179: attention :   ‘UINT64 CDecoder::m_streamSizeEstimation’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.cpp:65: attention :   when initialized here
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:185: attention : ‘CDecoder::m_macroBlocksAvailable’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:183: attention :   ‘int CDecoder::m_currentBlockIndex’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.cpp:65: attention :   when initialized here
[ 55%] Building CXX object core/digikam/CMakeFiles/digikamdatabase.dir/__/libs/3rdparty/libpgf/Encoder.cpp.o
In file included from /mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.cpp:29:                                                              
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h: In constructor ‘CEncoder::CMacroBlock::CMacroBlock(CEncoder*)’:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:82: attention : ‘CEncoder::CMacroBlock::m_encoder’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:63: attention :   ‘ROIBlockHeader CEncoder::CMacroBlock::m_header’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:52: attention :   when initialized here
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h: In constructor ‘CEncoder::CEncoder(CPGFStream*, PGFPreHeader, PGFHeader, const PGFPostHeader&, UINT32*&, bool)’:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:188: attention : ‘CEncoder::m_forceWriting’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:186: attention :   ‘UINT8 CEncoder::m_nLevels’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.cpp:67: attention :   when initialized here
[ 55%] Building CXX object core/digikam/CMakeFiles/digikamdatabase.dir/__/libs/3rdparty/libpgf/PGFimage.cpp.o
In file included from /mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/PGFimage.cpp:30:                                                             
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h: In constructor ‘CDecoder::CMacroBlock::CMacroBlock(CDecoder*)’:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:74: attention : ‘CDecoder::CMacroBlock::m_decoder’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:62: attention :   ‘ROIBlockHeader CDecoder::CMacroBlock::m_header’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:52: attention :   when initialized here
In file included from /mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/PGFimage.cpp:31:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h: In constructor ‘CEncoder::CMacroBlock::CMacroBlock(CEncoder*)’:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:82: attention : ‘CEncoder::CMacroBlock::m_encoder’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:63: attention :   ‘ROIBlockHeader CEncoder::CMacroBlock::m_header’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:52: attention :   when initialized here
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/PGFimage.h: In constructor ‘CPGFImage::CPGFImage()’:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/PGFimage.h:509: attention : ‘CPGFImage::m_cbArg’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/PGFimage.h:502: attention :   ‘bool CPGFImage::m_levelwise’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/PGFimage.cpp:56: attention :   when initialized here
[ 55%] Building CXX object core/digikam/CMakeFiles/digikamdatabase.dir/__/libs/3rdparty/libpgf/PGFstream.cpp.o
[ 55%] Building CXX object core/digikam/CMakeFiles/digikamdatabase.dir/__/libs/3rdparty/libpgf/Subband.cpp.o                                         
In file included from /mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Subband.cpp:30:                                                              
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h: In constructor ‘CEncoder::CMacroBlock::CMacroBlock(CEncoder*)’:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:82: attention : ‘CEncoder::CMacroBlock::m_encoder’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:63: attention :   ‘ROIBlockHeader CEncoder::CMacroBlock::m_header’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Encoder.h:52: attention :   when initialized here
In file included from /mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Subband.cpp:31:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h: In constructor ‘CDecoder::CMacroBlock::CMacroBlock(CDecoder*)’:
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:74: attention : ‘CDecoder::CMacroBlock::m_decoder’ will be initialized after
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:62: attention :   ‘ROIBlockHeader CDecoder::CMacroBlock::m_header’
/mnt/data/devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.h:52: attention :   when initialized here
[ 55%] Building CXX object core/digikam/CMakeFiles/digikamdatabase.dir/__/libs/3rdparty/libpgf/WaveletTransform.cpp.o
[ 55%] Building CXX object core/digikam/CMakeFiles/digikamdatabase.dir/__/libs/threadimageio/pgfutils.cpp.o                                          
Linking CXX shared library ../../lib/libdigikamdatabase.so                                                                                           

I will fix it tomorow...

Gilles Caulier
Comment 34 caulier.gilles 2011-06-19 01:05:17 UTC
To Marcel, #29

All digiKam tests compile fine here :

[ 98%] Built target advancedrenametest_automoc
Linking CXX executable advancedrenametest
[ 98%] Built target advancedrenametest
[ 98%] Built target cameranamehelpertest_automoc
Linking CXX executable cameranamehelpertest
[ 98%] Built target cameranamehelpertest
[ 98%] Built target dimagefilteractiontest_automoc
Linking CXX executable dimagefilteractiontest
[ 98%] Built target dimagefilteractiontest
[ 98%] Built target dimagehistorygraphtest_automoc
Scanning dependencies of target dimagehistorygraphtest
[ 98%] Building CXX object core/tests/CMakeFiles/dimagehistorygraphtest.dir/dimagehistorygraphtest_automoc.cpp.o
Linking CXX executable dimagehistorygraphtest
[ 98%] Built target dimagehistorygraphtest
[ 98%] Built target dimagehistorytest_automoc
Linking CXX executable dimagehistorytest
[ 98%] Built target dimagehistorytest
[ 98%] Built target filesaveoptionsboxtest_automoc
Linking CXX executable filesaveoptionsboxtest
[ 98%] Built target filesaveoptionsboxtest
[ 98%] Built target freerotationtest_automoc
Linking CXX executable freerotationtest
[ 98%] Built target freerotationtest
[ 98%] Built target pgfscaled_automoc
Scanning dependencies of target pgfscaled
[ 98%] Building CXX object core/tests/CMakeFiles/pgfscaled.dir/__/libs/3rdparty/libpgf/Decoder.cpp.o
Linking CXX executable pgfscaled
[100%] Built target pgfscaled
[100%] Built target qtpgftest_automoc
Scanning dependencies of target qtpgftest
[100%] Building CXX object core/tests/CMakeFiles/qtpgftest.dir/__/libs/3rdparty/libpgf/Decoder.cpp.o
Linking CXX executable qtpgftest
[100%] Built target qtpgftest
[100%] Built target searchtextbartest_automoc
Linking CXX executable searchtextbartest
[100%] Built target searchtextbartest
[100%] Built target statesavingobjecttest_automoc
Linking CXX executable statesavingobjecttest
[100%] Built target statesavingobjecttest
Scanning dependencies of target uifilevalidatortest_automoc
Generating uifilevalidatortest.moc
[100%] Built target uifilevalidatortest_automoc
Scanning dependencies of target uifilevalidatortest
[100%] Building CXX object core/tests/CMakeFiles/uifilevalidatortest.dir/uifilevalidatortest_automoc.cpp.o
[100%] Building CXX object core/tests/CMakeFiles/uifilevalidatortest.dir/uifilevalidatortest.cpp.o
Linking CXX executable uifilevalidatortest
[100%] Built target uifilevalidatortest

Gilles Caulier
Comment 35 caulier.gilles 2011-06-19 01:22:58 UTC
Git commit cf8cdc2cd50a63f6aa6d81c327005ed4faf37592 by Gilles Caulier.
Committed on 19/06/2011 at 01:20.
Pushed by cgilles into branch 'master'.

fix gcc warnings about order of classes member initialization
Raphael, please include this patch to official libpgf repository. Thanks in advance
CCBUGS: 273765
CCMAIL: rschweizer@schweizer-informatik.ch

M  +2    -2    libs/3rdparty/libpgf/Decoder.cpp     
M  +3    -3    libs/3rdparty/libpgf/Decoder.h     
M  +1    -1    libs/3rdparty/libpgf/Encoder.cpp     
M  +2    -2    libs/3rdparty/libpgf/Encoder.h     
M  +2    -2    libs/3rdparty/libpgf/PGFimage.cpp     

http://commits.kde.org/digikam/cf8cdc2cd50a63f6aa6d81c327005ed4faf37592
Comment 36 caulier.gilles 2011-06-19 01:30:43 UTC
Marcel,

As in #31, i can reproduce the thumbnail cache problem under Linux, as under Windows.

there is something to patch in thumbnail management Database code about PGF image data handling. Thumbs are displayed very slowly. Sound like all thumb from an album are recompute between each digiKam session. 

Of course, if you change current album in same digiKam session, thumbs cache from memory work fine.

Can you reproduce the problem ?

Gilles Caulier
Comment 37 caulier.gilles 2011-06-19 10:33:44 UTC
Marcel,

Under Linux, I removed all DB file, restarted digiKam, and rebuild all thumbnails. Problem still here. Icon view is very slow in all case to display thumbs.

Can you reproduce ?

Gilles Caulier
Comment 38 S. Burmeister 2011-06-19 12:13:52 UTC
I see it as well, not only with PGFs.
Comment 39 Marcel Wiesweg 2011-06-19 16:33:11 UTC
Git commit bf9645f52d22ba43b7b5bc58b73f915ecd713df4 by Marcel Wiesweg.
Committed on 19/06/2011 at 15:46.
Pushed by mwiesweg into branch 'master'.

Fix linking of libpgf tests
CCBUG: 273765

M  +2    -2    tests/CMakeLists.txt     

http://commits.kde.org/digikam/bf9645f52d22ba43b7b5bc58b73f915ecd713df4
Comment 40 Marcel Wiesweg 2011-06-19 16:43:34 UTC
Yes, I can fully reproduce. See these error messages on the console:
digikam(5323)/digikam (core) Digikam::readPGFImageData: Error running libpgf ( 2 )!
digikam(5323)/digikam (core) Digikam::ThumbnailCreator::loadFromDatabase: Cannot load PGF thumb from DB 

This means the data is loaded from the db, but libpgf fails to load the data with error code 2. Apparently this happens only with every second or third image.
Backtrace of the thrown exception:

#0  0x00007fffef9d7d90 in __cxa_throw () from /usr/lib64/libstdc++.so.6
#1  0x00007ffff4aa9912 in CPGFMemoryStream::Read (this=0x7fffcbe98460, count=0x7fffcbe97cfc, buffPtr=<value optimized out>)
    at /home/marcel/freshmeat/multimedia/kde4/src/extragear/graphics/digikam/core/libs/3rdparty/libpgf/PGFstream.cpp:157
#2  0x00007ffff4aa06f3 in CDecoder::ReadMacroBlock (this=0x7fffd021d430, block=0x7fffd0450370)
    at /home/marcel/freshmeat/multimedia/kde4/src/extragear/graphics/digikam/core/libs/3rdparty/libpgf/Decoder.cpp:461
#3  0x00007ffff4aa14d0 in CDecoder::DecodeBuffer (this=0x7fffd021d430)
    at /home/marcel/freshmeat/multimedia/kde4/src/extragear/graphics/digikam/core/libs/3rdparty/libpgf/Decoder.cpp:571
#4  0x00007ffff4aa169a in CDecoder::DequantizeValue (this=0x7fffd021d430, band=<value optimized out>, 
    bandPos=<value optimized out>, quantParam=<value optimized out>)
    at /home/marcel/freshmeat/multimedia/kde4/src/extragear/graphics/digikam/core/libs/3rdparty/libpgf/Decoder.cpp:429
#5  0x00007ffff4aa1b87 in CDecoder::Partition (this=0x7fffd021d430, band=0x7fffd056a7b8, quantParam=3, width=<value optimized out>, 
    height=<value optimized out>, startPos=0, pitch=42)
    at /home/marcel/freshmeat/multimedia/kde4/src/extragear/graphics/digikam/core/libs/3rdparty/libpgf/Decoder.cpp:249
#6  0x00007ffff4aaa1f6 in CSubband::PlaceTile (this=0x7fffd056a7b8, decoder=..., quantParam=3, tile=false, tileX=0, tileY=0)
    at /home/marcel/freshmeat/multimedia/kde4/src/extragear/graphics/digikam/core/libs/3rdparty/libpgf/Subband.cpp:230
#7  0x00007ffff4aa92b9 in CPGFImage::Read (this=0x7fffcbe97f30, level=0, cb=0, data=0x0)
    at /home/marcel/freshmeat/multimedia/kde4/src/extragear/graphics/digikam/core/libs/3rdparty/libpgf/PGFimage.cpp:408
#8  0x00007ffff4aab93f in Digikam::readPGFImageData (data=<value optimized out>, img=...)
    at /home/marcel/freshmeat/multimedia/kde4/src/extragear/graphics/digikam/core/libs/threadimageio/pgfutils.cpp:71
Comment 41 Marcel Wiesweg 2011-06-19 16:50:20 UTC
Created attachment 61150 [details]
Thumbnail PGF data which throws exception

Sample thumbnail data (written to disk the first time an exception is caught at startup from libpgf)
Comment 42 caulier.gilles 2011-06-19 19:19:03 UTC
Thanks Marcel, 

Sound like something must be adapted in PGFUtils interface with libpgf API to load RAW PGF image data from a bytearray take from thumbs database:

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/threadimageio/pgfutils.cpp

Raphael, any sugestions ?

Gilles Caulier
Comment 43 caulier.gilles 2011-06-20 09:00:30 UTC
Git commit e7e53a9c288b3b2aa83a3b7dd09c7a136f6253b1 by Gilles Caulier.
Committed on 20/06/2011 at 08:58.
Pushed by cgilles into branch 'master'.

Note : MSVC 2008 patched with SP1 work fine. MSVC 2008 official has a problem with OpenMP.
CCBUGS: 273765

M  +1    -1    libs/3rdparty/libpgf/PGFplatform.h     

http://commits.kde.org/digikam/e7e53a9c288b3b2aa83a3b7dd09c7a136f6253b1
Comment 44 caulier.gilles 2011-06-20 12:12:00 UTC
Git commit cb9435fae37f32553e8e354705e9abfe77783dad by Gilles Caulier.
Committed on 20/06/2011 at 12:10.
Pushed by cgilles into branch 'master'.

fix argment call for PGFImage::write method, which have changed with libpgf 6.11.24
CCBUGS: 273765

M  +1    -1    libs/threadimageio/pgfutils.cpp     

http://commits.kde.org/digikam/cb9435fae37f32553e8e354705e9abfe77783dad
Comment 45 caulier.gilles 2011-06-20 13:06:12 UTC
Marcel, Raphael

I hacked a lot digiKam PGF Utils interface and found a bug in lipgf API call arguments (in fact, an evolution of API to update in digiKam core).

I tested with pgf test program from digiKam, and encoding and decoding PGF work fine.

BUT, the problem still here about thumb database. I set a debug print to see if PGF data taken from thumb db are not null. It's not the case :

digikam(11244)/digikam (core) Digikam::readPGFImageData: image data stream size is :  8722
digikam(11244)/digikam (core) Digikam::readPGFImageData: Error running libpgf ( 2 )!
digikam(11244)/digikam (core) Digikam::ThumbnailCreator::loadFromDatabase: Cannot load PGF thumb from DB 
digikam(11244)/KDCRAW KDcrawIface::KDcraw::loadEmbeddedPreview: Preview data size: 345669
digikam(11244)/KDCRAW KDcrawIface::KDcraw::loadEmbeddedPreview: Using embedded RAW preview extraction
digikam(11244)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
digikam(11244)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation =>  1
digikam(11244)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation =>  1
digikam(11244)/digikam (core) Digikam::readPGFImageData: image data stream size is :  8002
digikam(11244)/digikam (core) Digikam::readPGFImageData: Error running libpgf ( 2 )!
digikam(11244)/digikam (core) Digikam::ThumbnailCreator::loadFromDatabase: Cannot load PGF thumb from DB 
digikam(11244)/KDCRAW KDcrawIface::KDcraw::loadEmbeddedPreview: Preview data size: 101661
digikam(11244)/KDCRAW KDcrawIface::KDcraw::loadEmbeddedPreview: Using embedded RAW preview extraction
digikam(11244)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation =>  1
digikam(11244)/digikam (core) Digikam::readPGFImageData: image data stream size is :  8814
digikam(11244)/digikam (core) Digikam::readPGFImageData: Error running libpgf ( 2 )!
digikam(11244)/digikam (core) Digikam::ThumbnailCreator::loadFromDatabase: Cannot load PGF thumb from DB 
digikam(11244)/KDCRAW KDcrawIface::KDcraw::loadEmbeddedPreview: Preview data size: 331522
digikam(11244)/KDCRAW KDcrawIface::KDcraw::loadEmbeddedPreview: Using embedded RAW preview extraction
digikam(11244)/kio (KIOConnection) KIO::ConnectionServer::listenForRemote: digikam(11244)/digikam (core) Digikam::DMetadata::getIccProfile: Exif color-space tag is sRGB. Using default sRGB ICC profile.
digikam(11244)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation =>  1
digikam(11244)/digikam (core) Digikam::readPGFImageData: image data stream size is :  10966
digikam(11244)/digikam (core) Digikam::readPGFImageData: Error running libpgf ( 2 )!
digikam(11244)/digikam (core) Digikam::ThumbnailCreator::loadFromDatabase: Cannot load PGF thumb from DB

Thumbs size are around 8/10Kb (256x256 pixels). So it sound fine.

The exception generated by libPGF appear when image data taken from DB is passed to CPGFImage::Open() as a memory stream :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/threadimageio/pgfutils.cpp#L67

What's error #2 given by libPGF ?

Note : in my pgf test code, i use this method, and all work fine ! PGF image data are taken from a file, not DB of course.

Gilles Caulier
Comment 46 caulier.gilles 2011-06-20 13:45:49 UTC
Sorry, 

The exception is not generated by libpgf to CPGFImage::Open(), by to CPGFImage::Read() :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/threadimageio/pgfutils.cpp#L76

Gilles
Comment 47 caulier.gilles 2011-06-20 14:08:17 UTC
Marcel,

Your PGF thumb from #41 work fine here. I wan load it into showfoto and pgfscaled test program do not report an error.

Gilles Caulier
Comment 48 caulier.gilles 2011-06-20 14:32:14 UTC
Git commit 3382b0cb2901b599b8e593c9cc07353cea045ee6 by Gilles Caulier.
Committed on 20/06/2011 at 14:30.
Pushed by cgilles into branch 'master'.

new test program to load PGF data from image file and convert it to QImg using PGFUtils::readPGFImageData()
CCBUGS: 273765

M  +3    -0    tests/CMakeLists.txt     
A  +76   -0    tests/loadpgfdata.cpp         [License: GPL (v2+)]

http://commits.kde.org/digikam/3382b0cb2901b599b8e593c9cc07353cea045ee6
Comment 49 caulier.gilles 2011-06-20 14:35:13 UTC
Marcel, with your PGF thumb image data, i can also pass loadpgfdata test program... How do you explain that ?

Gilles Caulier
Comment 50 caulier.gilles 2011-06-20 14:49:40 UTC
Ok, i find the problem, but not the real solution.

I disable OpenMp support to libpgf, and now, it work fine. Thumbnails can be loaded from database and decoded through libpgf without an error.

As thumbnail process run into a separated thread, using QThread, i suspect a problem here.

The fact that loadpgdfdata work as well is simple : i do not run in a separate thead, but in main thread, as it's a single a small test application.

Raphael,

Do you have tested into libpgf test collection tools a case where libpgf is used through a separated thread, as Posix thread ?

Note that Windows thread as the same problem. It's not Unix thread in this case...


Marcel,

Another question : image editor use a separated thread to load PGF image. Why it work fine in this case, if libpgf is compiled with OpenMP support ?

Gilles Caulier
Comment 51 Raphael Schweizer 2011-06-20 15:11:49 UTC
AFAIK there were no such tests. Unfortunately I still have no access to our source server (it went down shortly after some migration, no words from IT yet). I'm currently at home - preparing for next week's exams - and ha
Comment 52 Raphael Schweizer 2011-06-20 15:20:47 UTC
AFAIK there were no explicit tests (a background thread is used by the ImageViewer [http://xeraina.ch/pgf/pgfdownload.html] to load the images, no problems there). Unfortunately I still have no access to our source server (it went down shortly after some migration, no words from IT yet). I'm currently at home - preparing for next two week's exams - and have no access to our test system nor the latest source.
I really appreciate the work you have done so far. Is it feasible to disable OpenMP for the time being and suspend development till start of July?

- Raphael
Comment 53 caulier.gilles 2011-06-20 15:25:55 UTC
Raphael,

Trying to find a solution to OpenMP support in libpgf, is tried to turn off OpenMP with PGF encoder/decoder in digiKam PGFUtils methods relevant of thumbs database.

libPGF still compiled with OpenMP support. digiKam crash into libpgf :

Thread 2 (Thread 0xa8c3ab70 (LWP 7763)):
[KCrash Handler]
#6  0xb67ede11 in CDecoder::DecodeBuffer (this=0xa746628) at /mnt/data/Devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.cpp:571
#7  0xb67ed900 in CDecoder::DequantizeValue (this=0xa746628, band=0xa732f2c, bandPos=0, quantParam=0) at /mnt/data/Devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.cpp:429
#8  0xb67ed187 in CDecoder::Partition (this=0xa746628, band=0xa732f2c, quantParam=0, width=64, height=43, startPos=0, pitch=64) at /mnt/data/Devel/GIT/2.x/core/libs/3rdparty/libpgf/Decoder.cpp:249
#9  0xb67f9fe7 in CSubband::PlaceTile (this=0xa732f2c, decoder=..., quantParam=0, tile=false, tileX=0, tileY=0) at /mnt/data/Devel/GIT/2.x/core/libs/3rdparty/libpgf/Subband.cpp:230
#10 0xb67f19b9 in CPGFImage::Read (this=0xa8c398d0, level=0, cb=0, data=0x0) at /mnt/data/Devel/GIT/2.x/core/libs/3rdparty/libpgf/PGFimage.cpp:398
#11 0xb67fc342 in Digikam::readPGFImageData (data=..., img=...) at /mnt/data/Devel/GIT/2.x/core/libs/threadimageio/pgfutils.cpp:78
#12 0xb67ddbfd in Digikam::ThumbnailCreator::loadFromDatabase (this=0xa3a8f00, info=...) at /mnt/data/Devel/GIT/2.x/core/libs/threadimageio/thumbnailcreator.cpp:884
#13 0xb67da8fc in Digikam::ThumbnailCreator::load (this=0xa3a8f00, path=..., rect=..., pregenerate=false) at /mnt/data/Devel/GIT/2.x/core/libs/threadimageio/thumbnailcreator.cpp:245
#14 0xb67da599 in Digikam::ThumbnailCreator::load (this=0xa3a8f00, path=...) at /mnt/data/Devel/GIT/2.x/core/libs/threadimageio/thumbnailcreator.cpp:196
#15 0xb67e8181 in Digikam::ThumbnailLoadingTask::execute (this=0xa733768) at /mnt/data/Devel/GIT/2.x/core/libs/threadimageio/thumbnailtask.cpp:169
#16 0xb67c574a in Digikam::LoadSaveThread::run (this=0xa3841f8) at /mnt/data/Devel/GIT/2.x/core/libs/threadimageio/loadsavethread.cpp:118
#17 0xb68024ca in Digikam::DynamicThread::DynamicThreadPriv::run (this=0xa3a71a8) at /mnt/data/Devel/GIT/2.x/core/libs/threads/dynamicthread.cpp:328
#18 0xb48fa9d6 in ?? () from /usr/lib/libQtCore.so.4
#19 0xb490555f in ?? () from /usr/lib/libQtCore.so.4
#20 0xb4892ae5 in start_thread () from /lib/i686/libpthread.so.0
#21 0xb466203e in clone () from /lib/i686/libc.so.6

Why ???

So, my only solution is to temporally disable OpenMP support with libpgf to be able to run properly digiKam.

I recommend to improve better OpenMP support in libpgf. I will write some code to be able to force compilation of libpgf without OpenMP, in case of OpenMP is detected on target computer. I recommend to backport my patch to current implementation of libpgf

Gilles Caulier
Comment 54 caulier.gilles 2011-06-20 15:33:53 UTC
Git commit 4ca8c22c4368306457aed9d62bbf48f7436553da by Gilles Caulier.
Committed on 20/06/2011 at 15:32.
Pushed by cgilles into branch 'master'.

add a flag to compile libpgf without openmp support at compilation time.
Raphael, please add this patch to current libpgf implementation
CCMAIL: rschweizer@schweizer-informatik.ch
CCBUGS: 273765

M  +10   -4    libs/3rdparty/libpgf/PGFplatform.h     

http://commits.kde.org/digikam/4ca8c22c4368306457aed9d62bbf48f7436553da
Comment 55 caulier.gilles 2011-06-20 15:52:26 UTC
Git commit 6349f79b8a059034bfe218f51ef26e46581a64d4 by Gilles Caulier.
Committed on 20/06/2011 at 15:51.
Pushed by cgilles into branch 'master'.

disable openmp support to libpgf
CCBUGS: 273765

M  +4    -0    CMakeLists.txt     

http://commits.kde.org/digikam/6349f79b8a059034bfe218f51ef26e46581a64d4
Comment 56 caulier.gilles 2011-06-20 17:50:37 UTC
Francesco,

Lets' me hear if this file can be closed now...

Gilles Caulier
Comment 57 Francesco Riosa 2011-06-20 20:58:35 UTC
yes it is, liked very much the collaborative work on this one, congratz
Comment 58 caulier.gilles 2011-06-21 09:57:39 UTC
Git commit 22018b3ec500ee76178d486456598f511eb74455 by Gilles Caulier.
Committed on 21/06/2011 at 09:53.
Pushed by cgilles into branch 'master'.

Add a PGF codec version ID to be able to make rules with clien application accodingly with libpgf API changes.
Raphael, it's not possible to do it with PGFCodecVersion string and preprocessor. Of course both must be updated at each libpgf version.
We use already this way in libraw and libexiv2.
CCMAIL: rschweizer@schweizer-informatik.ch
CCBUGS: 273765

M  +2    -0    libs/3rdparty/libpgf/PGFtypes.h     

http://commits.kde.org/digikam/22018b3ec500ee76178d486456598f511eb74455
Comment 59 caulier.gilles 2011-06-22 08:23:55 UTC
Git commit 67029dd34cc4314885db49258b948de4c77d7d58 by Gilles Caulier.
Committed on 22/06/2011 at 08:22.
Pushed by cgilles into branch 'master'.

fix MSVC warnings
CCBUGS: 273765
CCMAIL: rschweizer@schweizer-informatik.ch

M  +2    -2    libs/3rdparty/libpgf/PGFplatform.h     

http://commits.kde.org/digikam/67029dd34cc4314885db49258b948de4c77d7d58
Comment 60 caulier.gilles 2021-05-09 14:52:17 UTC
Not reproducible with digiKam 7.3.0 and Exiv2 0.27.4