Bug 317943

Summary: (Auto) rotate/flip using EXIF Information destroys pictures
Product: [Applications] digikam Reporter: Nicofo <nicofo>
Component: Plugin-DImg-JPEGAssignee: Digikam Developers <digikam-bugs-null>
Status: RESOLVED FIXED    
Severity: critical CC: alexanders83, bbc.quincy, caulier.gilles, nous, px79, r.biegel, vh59foto
Priority: NOR    
Version: 3.3.0   
Target Milestone: ---   
Platform: Other   
OS: Linux   
Latest Commit: Version Fixed In: 4.0.0
Attachments: Destroyed picture 1 (bottom part of the picture lost !!)
Destroyed picture 2 (left part of the picture lost !!)
Temporary file
Temporary file2
digikam output
digikam console log (rotation of 56 images)
digikam console log (rotation of 4 images)
Black bordered thumbnail in Picasa

Description Nicofo 2013-04-06 18:09:41 UTC
In an album, I have used the tool:
Image > Auto rotate/flip using EXIF Information

As a result, 95% of the pictures were rotated as expected, but a few pictures were not or worse were destroyed (i.e. some part of the picture is lost -> I have put this bug 'critical') !

Actually there are differents problems:
- two pictures were not rotated (orientation remains 'right-top', even after runing again the autorotate tool)
- some pictures were destroyed: the bottom or the left side of the picture is lost: see attachments
- some temporary files (JpegRotator-xxxx.digikamtempfile.jpgxxxx) remained in the pictures' directory (see attachments)

Reproducible: Sometimes




Version of digikam: 3.0 and 3.1
No problem with previous versions (2.*)
Comment 1 Nicofo 2013-04-06 18:11:20 UTC
Created attachment 78687 [details]
Destroyed picture 1 (bottom part of the picture lost !!)
Comment 2 Nicofo 2013-04-06 18:11:45 UTC
Created attachment 78688 [details]
Destroyed picture 2 (left part of the picture lost !!)
Comment 3 Nicofo 2013-04-06 18:13:57 UTC
Created attachment 78689 [details]
Temporary file

This file should have been deleted. As it was not, that probably indicates that something went wrong...
Comment 4 Nicofo 2013-04-06 18:14:40 UTC
Created attachment 78690 [details]
Temporary file2

Rem: there was also an empty file:
JpegRotator-aq8526.digikamtempfile.jpg8526
Comment 5 caulier.gilles 2013-04-06 18:15:14 UTC
Which image format do you try to rotate : JPEG ?

Which libJPEG you use ? Go to Help/components Info for details.

Check also in your system if jpegturbo lib is installed...

Gilles Caulier
Comment 6 Nicofo 2013-04-06 18:28:13 UTC
(In reply to comment #5)
> Which image format do you try to rotate : JPEG ?
Yes. ~170 jpg files
> Which libJPEG you use ? Go to Help/components Info for details.
LibJPEG: 62
> Check also in your system if jpegturbo lib is installed...
Yes, I guess so:
$ rpm -qa|grep jpeg|sort
libjpeg-turbo-1.2.90-1.fc18.i686
libjpeg-turbo-utils-1.2.90-1.fc18.i686
mjpegtools-libs-2.0.0-5.fc18.i686
openjpeg-libs-1.5.1-4.fc18.i686

> Gilles Caulier
Thanks for your rapid answer !
Comment 7 caulier.gilles 2013-04-06 18:31:28 UTC
JPEGTurbo can be the problem. We have never tested it.

Can you run: 

- kdebugdialog and trun on digiKam and KExiv2 debug spaces.
- digiKam from a console and process JPEG rotation.
- report console trace here.

Gilles Caulier
Comment 8 Nicofo 2013-04-06 18:50:17 UTC
Created attachment 78691 [details]
digikam output

Log file attached, as demanded.
This time, I have still problems with:
- IMG_4072.JPG (left side of the picture lost + other distortions)
- IMG_4086.JPG => empty file => picture entirely lost !
- JpegRotator-w11195.digikamtempfile.jpg temporay file (which seems to be the bottom part of the previous picture IMG_4086)
- JpegRotator-M11195.digikamtempfile.jpg temporay file
Comment 9 Nicofo 2013-04-06 18:53:50 UTC
(In reply to comment #7)
> JPEGTurbo can be the problem. We have never tested it.
Additional info: I have now Fedora 18 and digikam 3.1,
but with my previous Fedora installation (Fedora 16, digikam 2.5), I had also libjpeg-turbo.i686 (1.2.1-1.fc16) and no bug.
Comment 10 caulier.gilles 2013-04-06 19:18:01 UTC
Strange no ?

digikam(11195)/digikam (core) Digikam::JPEGUtils::jpegutils_jpeg_error_exit: Jpegutils error, aborting operation: Output file write error --- out of disk space?

Gilles Caulier
Comment 11 Nicofo 2013-04-06 19:39:37 UTC
(In reply to comment #10)
> Strange no ?
> 
> digikam(11195)/digikam (core) Digikam::JPEGUtils::jpegutils_jpeg_error_exit:
> Jpegutils error, aborting operation: Output file write error --- out of disk
> space?
My pictures are on a NTFS windows partition, with 12Go free space.
Comment 12 caulier.gilles 2013-04-08 10:27:17 UTC
On my computer, i cannot reproduce the problem.

Sure i don't use NTFS, only Ext4. Perhaps it's the problem.

On my computer libjpeg-turbo is installed, including the compatibilty component to emulate old libjpeg (version 8) with MMX/SSE2 accelerated compresion/decompression support.

libjpeg-devel:
        The libjpeg-devel package includes the header files necessary for
        developing programs which will manipulate JPEG files using
        the libjpeg library.

        If you are going to develop programs which will manipulate JPEG images,
        you should install libjpeg-devel.  You'll also need to have the libjpeg
        package installed.
:http://sourceforge.net/projects/libjpeg-turbo

libjpeg8:
        This package contains the library needed to run programs dynamically
        linked with libjpeg-turbo.
:http://sourceforge.net/projects/libjpeg-turbo

Can you try to use a non NTFS file system to host JPEG file and perform rotation to see if problem is reproducible or not ?

Gilles Caulier
Comment 13 Reinhard Biegel 2013-04-09 08:48:38 UTC
I can confirm the problem here on my PC. Symptoms exacly as described by user Nicofo. Noticed it when batch-rotating some images (selecting multiple files and execute rotation with keyboard shortcut).

I'm using digikam 3.0.0 on a gentoo x86_64 system with libjpeg-turbo 1.2.90, images located on ext4.
Comment 14 caulier.gilles 2013-04-09 09:04:54 UTC
Reinhard,

I also use ext4 and libjpeg-turbo 1.2.90 under Linux Mageia2, and i cannot reproduce the problem.

Perhaps do you have a better debug trace in the console following my comment #7 ?

Gilles Caulier
Comment 15 Nicofo 2013-04-14 10:28:03 UTC
I have tried on an EXT4 fs -> as Reinhard Biegel, the bug is still there -> it's not related to the files system.

I have the impression that the pictures destroyed are more numerous if computer is busy by other programs. Or also if for instance dolphin (in preview mode) updates the pictures at the same time that digikam rotates them.
Comment 16 Reinhard Biegel 2013-04-15 06:54:30 UTC
Created attachment 78914 [details]
digikam console log (rotation of 56 images)
Comment 17 Reinhard Biegel 2013-04-15 07:02:26 UTC
Created attachment 78916 [details]
digikam console log (rotation of 4 images)

Sorry for the delay. Updated to 3.1.0, KDE 4.10.2, Qt is 4.8.4 btw.

First log: rotation of 56 images at once (single run). Leaves 9 temporary Files with filesizes of 0 B, 2 B, a few KiB and 2 files with a few MiB. The bigger ones seem to contain jpeg data, one of them can partially be read, the other one cannot be read at all. 2 B files contain 0xFF 0xD8.

Second log: rotation of 4 images at once (multiple runs until images are damaged). Left 2 Files with 0 B and 2 B respectively. I think first damages occured at second last run.
Comment 18 Nicofo 2013-05-01 15:52:08 UTC
Anything I can do (other log...) to help you to resolve the bug ?
Comment 19 Nicofo 2013-05-02 21:09:30 UTC
Hi,
I have made some tests with other software:
- jhead -autorot *.JPG     =>  no problem
- ImageMagick : convert -auto-orient IMG.JPG IMG_NEW.JPG  (done for 50 images in batch)  => no problem
- gwenview : gwenview doesn't have an autorotate function, so I've made a lot of rotations with several images at the same time instead => never had any problem

=> I can only reproduce this bug in digikam.
Comment 20 Alexander Stein 2013-05-05 10:07:05 UTC
I also have this problem that after auto-rotate from within an album some pictures (jpegs) got destroyed completely (0 Bytes) or only 1/3 of the right side is still there (the left 2/3 are pure gray) as with destroyed picture attachments from Nicofo.
There seem to be a race condition during rotation. I store both raw and jpeg from my camera and noticed today that the companion jpeg to the raw file (both file name prefixes are identical) doesn't match. I took photos of cocktails and now DSC00954.ARW is an orange one and DSC00954.JPG is a light blue one which I shot 9 days later.
Comment 21 Nicofo 2013-05-19 16:28:12 UTC
I have made additional tests:

1) I have downgraded libjpeg-turbo(-utils) to the version 1.2.1-3.fc18.i686 (instead of 1.2.90-1.fc18.i686) => bug was already present with that version

2) actually, the bug is not only present with the 'autorotate' function: the bug also happened with the normal rotate function. As already said, I think the chance is bigger for the bug to happen if you do multiple operations on the pictures: in my case, I have selected several pictures in digikam and clicked a lot of time (> 10 in a row) on the rotate left button => all my pictures where destroyed !!
Comment 22 Quincy 2013-05-30 02:35:50 UTC
Not really news, but just to push the really critical bug (backup of raw images is essential, otherwise data loss occurs) to a "confirmed" status:

Using a similar configuration as Reinhard Biegel I'm also on Gentoo x86_64 with digikam-3.1.0, images on ext4 FS, but libjpeg-turbo-1.2.1 and experiencing similar problems. I'm rotating groups of selected pictures via shortcut and images get destroyed in various ways as described above (size 0, reduced or even increased). Rotating larger amounts of images at the same time seems to make the problem worse, as until now I could not reproduce the error while rotating single files.

Interestingly not the pictures being reported as "JPEG transform failed" but others are destroyed. For example I rotated 6 pictures at the same time (58, 60, 61, 62, 63, 64), 60 and 61 were reported as failed, but 58, 60, 61 and 62 have been rotated, 63 and 64 have been destroyed. 63 has zero size, 64 is now about 75% larger than before (concatenated data of 63?). Additionally there is a temporary file of 61 with digikams PID as suffix leftover.
Comment 23 Peter Albrecht 2013-06-24 18:27:00 UTC
And here is another gentoo linux user confirming this problem:
  - arch: x86_64
  - digiKam 3.1.0
  - libjpeg-turbo  1.2.1
  - KDE 4.10.2
  - pictures on an ext4 filesystem on SSD

I did not encounter this problem with 
  - digiKam 2.9.0 
  - libjpeg-turbo 1.2.1 (same as above)
  - KDE 4.9.5

Components Information:
---------------------------------- 8< ----------------------------------
digiKam version 3.1.0
Exiv2 can write to Jp2: Yes
Exiv2 can write to Jpeg: Yes
Exiv2 can write to Pgf: Yes
Exiv2 can write to Png: Yes
Exiv2 can write to Tiff: Yes
Exiv2 supports XMP metadata: Yes
LibCImg: 130
LibClapack: external shared library
LibExiv2: 0.23
LibJPEG: 80
LibJasper: 1.900.1
LibKDE: 4.10.2
LibKExiv2: 2.3.0
LibKGeoMap: 2.0.0
LibKdcraw: 2.2.0
LibLCMS: 119
LibPGF: 6.12.27 - external shared library
LibPNG: 1.5.15
LibQt: 4.8.4
LibRaw: 0.15.0-Beta1
LibTIFF: LIBTIFF, Version 4.0.2 Copyright (c) 1988-1996 Sam Leffler Copyright (c) 1991-1996 Silicon Graphics, Inc.
Marble Widget: 0.15.1 (stable version)
Parallelized PGF codec: No
Parallelized demosaicing: No
RawSpeed codec support: No
Database backend: QSQLITE
Kipi-Plugins: 3.0.0
LibKface: 2.0.0
LibKipi: 2.0.0
LibOpenCV: 2.4.3
Libface: 0.2
---------------------------------- >8 ----------------------------------

It is strange, but since I know of this bug and having turned on debug output, no further image got destroyed. But a few days ago, I lost about five pictures (out of 300) on the same system. I did not update any packages during these days.
Today I selected 160 images of mixed rotation and started "Image -> Auto Rotate/Flip using EXIF information".
Comment 24 caulier.gilles 2013-06-24 20:36:29 UTC
If i understand you, when in debug, no problem, without debug, dysfunctions. Right ?

This can be a race condition in code.

Gilles Caulier
Comment 25 Peter Albrecht 2013-06-26 19:00:52 UTC
Today I found time to run another test:
I copied an album of 415 images unrotated images and started "Image -> Auto Rotate/Flip using EXIF information". Once with debug-output enabled and once with debug-output disabled:

 - debug-output enabled: 2 images died (half gray)
 - debug-output disabled: 1 image died (reduced to 12 Byte in size)

So the problem seems not to be affected by debug-output beeing enabled or not.

In both directories some "JpegRotator-rx2207.digikamtempfile.jpg" files keep existing after the rotation operation is finished. It is strange that one of these "JpegRotator..." files is 12,2 MB in size. But my largest JPEG is only 9,7 MB in size!?!?

So there seems to be some mixing up of rotation operations as Alexander writes in comment #20 and  Quincy writes in comment #22.
Comment 26 caulier.gilles 2013-06-26 19:07:43 UTC
Peter, 

I thinking not about debug trace in fact, printed on the console.

I suspect more debug symbols included during compilation time.

Here with full debug symbol, i never reproduce this dysfunction.

In release compilation, all debug symbols are dropped and code in more optimized (in speed in general). This can have any side effects...

Gilles Caulier
Comment 27 Peter Albrecht 2013-06-26 19:14:07 UTC
(In reply to comment #26)
> I thinking not about debug trace in fact, printed on the console.
> I suspect more debug symbols included during compilation time.
> Here with full debug symbol, i never reproduce this dysfunction.
> In release compilation, all debug symbols are dropped and code in more
> optimized (in speed in general). This can have any side effects...

What I did was enabling debug output with "kdebugdialog". Rebuilding digikam with debug symbols would be a lot more effort. ;)
Comment 28 Peter Albrecht 2013-06-26 19:29:45 UTC
(In reply to comment #21)
> 2) actually, the bug is not only present with the 'autorotate' function: the
> bug also happened with the normal rotate function. As already said, I think
> the chance is bigger for the bug to happen if you do multiple operations on
> the pictures: in my case, I have selected several pictures in digikam and
> clicked a lot of time (> 10 in a row) on the rotate left button => all my
> pictures where destroyed !!

I can reproduce this error on my machine: 
 - take 6 images
 - select them all
 - hit 10 times CTRL + SHIFT + "Cursor Right"  (to perform loss-less rotation)
=> 2 images of 6 are destroyed

  - doing another 20 rotations
=> no further images are destroyed

  - doing another 10 rotations
=> one half gray images gets reduced to a 0 Byte file

----------------------------------------------
Doing one CTRL + SHIFT + "Cursor Right" on my album of 415 images, I end up with 10 images destroyed. So there is a 2,4% chance to destroy an image using rotation.
Comment 29 Peter Albrecht 2013-06-26 20:20:36 UTC
(In reply to comment #27)
> (In reply to comment #26)
> > I thinking not about debug trace in fact, printed on the console.
> > I suspect more debug symbols included during compilation time.
> > Here with full debug symbol, i never reproduce this dysfunction.
> > In release compilation, all debug symbols are dropped and code in more
> > optimized (in speed in general). This can have any side effects...
> 
> What I did was enabling debug output with "kdebugdialog". Rebuilding digikam
> with debug symbols would be a lot more effort. ;)

Ok, I tried to build digikam with debug symbols, to help, finding out why you can't reproduce the error.
On my gentoo system, I ...
   - added "-ggdb" to the CFLAGS in "make.conf"
   - added 'FEATURE="nostrip"' in "make.conf"
   - enabled the useflag "debug" for package "media-gfx/digikam"
  (according to http://www.gentoo.org/proj/en/qa/backtraces.xml)
Than I rebuild digikam. Now "/usr/bin/digikam" is 82 MByte large, so I guess, there are a lot of debug symbols included. ;)

Additionally, I enabled every "*digikam*" setting in kdebugdialog.

Now I rerun my test of comment #28: "Doing one CTRL + SHIFT + "Cursor Right" on my album of 415 images". This time I ended up with:
   - a total of 30 images destroyed
   - 16 "JpegRotator-XXXXXXX.digikamtempfile.jpg" files left after rotation batch finished
   - 7 file of 0 Byte
   - 2 of those 0 Byte files with digikams thread-id attached to the filename
   - 2 files are larger than the larges original file

Another interesting fact: The destroyed images are not spread equally. Among the 30 destroyed images, I have two blocks of 6 images with subsequent image numbers:
   - IMG_7864.jpg to IMG_7869.jpg
   - IMG_8074.jpg to IMG_8079.jpg

=> To sum up: I still can reproduce this error with debug symbols enabled.

Do you have any idea, what I could test / debug / log next to help, find this bug?
Comment 30 Volker 2013-07-07 14:02:25 UTC
*** This bug has been confirmed by popular vote. ***
Comment 31 Volker 2013-07-07 14:10:39 UTC
(In reply to comment #30)
> *** This bug has been confirmed by popular vote. ***

I confirm this problem with a lot of DK versions, as I remember from 2.6 on , all on OpenSuseLinux.
Only possibility for me: Do not select more then 20 pictures at the same time, otherwise lot of pictures are destroyed and must be reloaded from the camera
Comment 32 nous 2013-07-21 13:28:47 UTC
I'm confirming a similar problem with EXIF auto-rotation: DK won't rotate at all pictures with pertinent EXIF rotation information and I have to rotate them manually. Picasa shows them correctly. If I rotate manually with DK sometimes the thumbnail in Picasa retains the original orientation and gets surrounded by a larger black frame of with the requested orientation and some garbage (see attachment).

Arch linux with DK 3.2.0, libjpeg-turbo 1.3.0 and libpeg7.
Comment 33 nous 2013-07-21 13:30:29 UTC
Created attachment 81240 [details]
Black bordered thumbnail in Picasa
Comment 34 Nicofo 2013-07-29 19:45:09 UTC
Still present with Digikam 3.2 and Fedora 19
Comment 35 nous 2013-08-16 18:00:52 UTC
Still present in Digikam 3.3.0 on Archlinux.
Comment 36 Nicofo 2013-08-22 20:59:33 UTC
I have not been able to reproduce this bug in digikam 3.3.0.
Maybe some other modification in this version has solved this issue ?

Rem: not sure the bug reported by 'nous' is the same in comment #32 because the 'destruction' of the picture is not similar to mine ? For him: the original picture is still there, but with strange strips around ; for me: an entire part of the picture was removed.
Comment 37 caulier.gilles 2013-10-31 08:00:48 UTC
digiKam 3.5.0 is out.

Can you give a fresh feedback about your report ? Problem still reproducible ?

Thanks in advance

Gilles Caulier
Comment 38 Nicofo 2013-11-06 19:07:02 UTC
(In reply to comment #37)
> digiKam 3.5.0 is out.
> Can you give a fresh feedback about your report ? Problem still reproducible ?

Hello,
I have tested again with digikam 3.5 and have not been able to reproduce this problem (and there are no temporary files left in the pictures folder).

Do you have an idea what could have 'solved' this bug between DK 3.2 (last version were I observed the bug) and DK 3.3 ?? As it seemed there could be some race condition in the code, if the buggy code has not been clearly identified and corrected, I fear the bug could reappear in a future version ...
Comment 39 caulier.gilles 2013-11-06 21:47:06 UTC
Nicofo,

To be honest, i'm cannot to be sure how this bug can be fixed. I remember some commit from Marcel around rotate/flip code in digiKam core. He can probably said more about than me.

This can be also, a side effect from a shared libs used by digiKam (as libjpeg for ex)...

I will close this file now. If problem re-appears, don't hesitate to re-open this file.

Gilles Caulier