| Summary: | digikam image download from memorycard problems | ||
|---|---|---|---|
| Product: | [Applications] digikam | Reporter: | Simon Oosthoek <kdebugs> |
| Component: | Import-UMS | Assignee: | Digikam Developers <digikam-bugs-null> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | m.spring, marcel.wiesweg |
| Priority: | NOR | ||
| Version: | 0.9.5 | ||
| Target Milestone: | --- | ||
| Platform: | Compiled Sources | ||
| OS: | Linux | ||
| Latest Commit: | Version Fixed In: | ||
| Sentry Crash Report: | |||
| Attachments: | corrupted sony a100 .arw raw image file | ||
|
Description
Simon Oosthoek
2009-01-08 21:41:12 UTC
Oh and BTW, I now see that all of my .arw files are smaller and corrupted when downloaded! I'm using the KDE3 svn version of 0.9.5 (with my ratiocrop patch, but that shouldn't matter) I have this same error using 0.10.x from svn on kde 4.2rc1 (kubuntu intrepid packages from ppa) The card I'm using is working fine and the actual image files are not corrupted on the card, only after downloading by digikam... (This would seem like a blocker bug if confirmed...) Cheers Simon Please let me know if you need more info! Choose a file. Access the mountpoint with dolphin and copy the file to your harddisk. Confirm that it is not corrupted. Then access the card with digikam, download the same one file, and confirm that it is corrupted. Watch for any error messages on the console. I've done just this, and via dolphin there's no problem, the file is 100% ok, it's around 11 MB large and loads fine in ufraw (or digikam/editor for that matter)
digikam gives some output on the console, but none very spectacular I think:
QGridLayout: Multi-cell fromRow greater than toRow
QGridLayout: Multi-cell fromRow greater than toRow
QGridLayout: Multi-cell fromRow greater than toRow
QGridLayout: Multi-cell fromRow greater than toRow
Starting to load Plugins.
Files: "CompassFloatItem.so"
Files: "MapScaleFloatItem.so"
Files: "MarbleCrosshairsPlugin.so"
Files: "MarbleGeoDataPlugin.so"
Files: "MarbleOverviewMap.so"
Files: "MarbleStarsPlugin.so"
Files: "NavigationFloatItem.so"
=== MarbleDirs: ===
Local Path: "/home/simon/.marble/data"
Plugin Local Path: "/home/simon/.marble/plugins"
Marble Data Path (Run Time) : ""
Marble Data Path (Compile Time): "/usr/share/kde4/apps/marble/data"
Marble Plugin Path (Run Time) : ""
Marble Plugin Path (Compile Time): "/usr/lib/kde4/plugins/marble"
System Path: "/usr/share/kde4/apps/marble/data"
Plugin System Path: "/usr/lib/kde4/plugins/marble"
===================
"CompassFloatItem.so" - "/usr/lib/kde4/plugins/marble/CompassFloatItem.so"
"MapScaleFloatItem.so" - "/usr/lib/kde4/plugins/marble/MapScaleFloatItem.so"
"MarbleCrosshairsPlugin.so" - "/usr/lib/kde4/plugins/marble/MarbleCrosshairsPlugin.so"
"MarbleGeoDataPlugin.so" - "/usr/lib/kde4/plugins/marble/MarbleGeoDataPlugin.so"
"MarbleOverviewMap.so" - "/usr/lib/kde4/plugins/marble/MarbleOverviewMap.so"
"MarbleStarsPlugin.so" - "/usr/lib/kde4/plugins/marble/MarbleStarsPlugin.so"
"NavigationFloatItem.so" - "/usr/lib/kde4/plugins/marble/NavigationFloatItem.so"
Time elapsed: 5 ms
Use workaround: 0
loadMapTheme "earth/citylights/citylights.dgml"
loadMapTheme "earth/bluemarble/bluemarble.dgml"
MapThemeId "earth/srtm/srtm.dgml"
loadMapTheme "earth/srtm/srtm.dgml"
DGML2 Name : "Atlas"
adding container: "cityplacemarks.kml" false
starting parser for "cityplacemarks"
"Loading Default Placemark Cache File:/usr/share/kde4/apps/marble/data/placemarks/cityplacemarks.cache"
adding container: "baseplacemarks.kml" false
starting parser for "baseplacemarks"
"Loading Default Placemark Cache File:/usr/share/kde4/apps/marble/data/placemarks/baseplacemarks.cache"
Loading ended true
adding container: "elevplacemarks.kml" false
adding container: "otherplacemarks.kml" false
starting parser for "elevplacemarks"
starting parser for "otherplacemarks"
adding container: "boundaryplacemarks" true
"Loading Default Placemark Cache File:/usr/share/kde4/apps/marble/data/placemarks/otherplacemarks.cache"
starting parser for "boundaryplacemarks"
Loading ended true
Loading ended true
"Loading Default Placemark Cache File:/usr/share/kde4/apps/marble/data/placemarks/elevplacemarks.cache"
"Loading Default Placemark Cache File:/usr/share/kde4/apps/marble/data/placemarks/boundaryplacemarks.cache"
Loading ended true
Loading ended true
THEME CHANGED: *** "earth/srtm/srtm.dgml"
SunLocator::setBody( QString )
DatasetProvider "1:3:0"
DatasetProvider "1:0:0"
DatasetProvider "1:2:0"
DatasetProvider "1:1:0"
DatasetProvider "1:1:1"
DatasetProvider "1:2:1"
Style reset requested.
Style reset requested.
Style reset requested.
Style reset requested.
start generate indexes
generated indexes
MarblePlacemarkModel (generateIndex): Time elapsed: 270 ms
Style reset requested.
Style reset requested.
Error: Directory SubImage1 with 33639 entries considered invalid; not read.
Error: Directory SubImage1 with 33639 entries considered invalid; not read.
Error: Directory SubImage1 with 33639 entries considered invalid; not read.
RESET started
RESET stopped
Detecting maxLabelHeight ...
Error: Directory SubImage1 with 33639 entries considered invalid; not read.
timeChanged
void SunLocator::update()
Error: Directory SubImage1 with 33639 entries considered invalid; not read.
Error: Directory SubImage1 with 33639 entries considered invalid; not read.
Error: Directory SubImage1 with 33639 entries considered invalid; not read.
Error: Directory SubImage1 with 33639 entries considered invalid; not read.
Error: Directory SubImage1 with 33639 entries considered invalid; not read.
Error: Directory SubImage1 with 33639 entries considered invalid; not read.
Error: Directory SubImage1 with 33639 entries considered invalid; not read.
The file downloaded by digikam is 34.3 KiB, the original is 11.8 MB
I'll attach the corrupted file.
Cheers
Simon
Created attachment 30648 [details]
corrupted sony a100 .arw raw image file
this was downloaded using digikam 0.10 svn r916749 from a cardreader mounted CF card of a sony a100. It's supposed to be a sony raw file, the original size is file is 11.8MB
Which digiKam version you use ? for KDE3 or KDE4 ? Which options you use in camera interface to download file ? Which camera driver you use (look Camera informations dialog) Gilles Caulier Start digiKam froma console. Download some ARW files and look all messages printed. Gilles Caulier Strange ther eis no messages relevant of camera interface in #4 The string : "Error: Directory SubImage1 with 33639 entries considered invalid; not read." ... come from Exiv2 libary used to manage metadata. But with ARW files (all raw files in fact), we don't touch anything. The file is copied as well. No more. Here, i have a Minolta Dynax 5D. I have no problem to download MRW raw files. Which Exiv2 library you use on your computer ? Go to Help Components Info dialog for details. Also, try to recompile digiKam using full debug option and try again. Look in README file for details. Gilles Caulier AFAIK the wiki instructions for compiling have fulldebug in all the options of the cmake line, so I assume that it's already compiled with full debugging. I didn't change anything in the lines from the instructions. This is the link where I got the instructions and I followed them to the letter. http://wiki.kde.org/tiki-index.php?page=Digikam+Compilation+on+Kubuntu+Intrepid The exiv2 version I use is the released exiv2-0.18 (file dates around 16-12-2008) from a tarball. FYI: from the help -> component info digiKam version 0.10.0-rc2 (rev.: 916823) Exiv2 can write to Jp2: Yes Exiv2 can write to Jpeg: Yes Exiv2 can write to Png: Yes Exiv2 can write to Tiff: Yes Exiv2 support XMP metadata: Yes LibCImg: 129 LibExiv2: 0.18 LibJPEG: 62 LibJasper: 1.900.1 LibKDE: 4.1.96 (KDE 4.1.96 (KDE 4.2 RC1)) LibKExiv2: 0.6.0 LibKdcraw: 0.5.0 LibLCMS: 116 LibPNG: 1.2.27 LibQt: 4.4.3 LibRaw: 0.7.0-Alpha4 LibTIFF: LIBTIFF, Version 3.8.2 Copyright (c) 1988-1996 Sam Leffler Copyright (c) 1991-1996 Silicon Graphics, Inc. Marble widget: pre 0.7 SVN LibGphoto2: 2.4.2 LibKipi: 0.4.0 Do you have any of the options on the camera gui right sidebar activated? I see the string "digikam-0.10.0-rc2" when I open the attached file in a binary editor, and this should certainly not happen when the file is untouched straight from the camera. Switch off all "File Renaming" and "On-the-fly" operations. All these operations that you mention should not apply to RAW files, so whether I've set these or not, it shouldn't matter when I download .ARW files from my memory card. I do believe I've set options like case setting (all uppercase files from my camera) and set author, but they do not apply to raw files and if digikam tries to apply them to raw files, that is a bug, IMO. Would it be useful to upload my configuration file of digikam? Which files should I upload? Cheers Simon Simon, Perhaps your problem is relevant of a stupid bug in libkexiv2. Look here : http://bugs.kde.org/show_bug.cgi?id=182412 Please, checkout libkexiv2 code from svn trunk and recompile, install it, and try again Gilles Caulier Simon, just try to switch off all options. If this fixes the bug, then we found the bug. This is the idea behind this. Of course it is a bug if digikam applies options to RAW files to which it cannot write, but we are hunting for a bug here. Gilles, Marcel, I'll try both suggestions, first the update of libkexiv2 and then turn off "all" options, however could you be specific about what is in "all" options? Cheers Simon I think I found it, in the jpeg options of the download dialogue, the setting "set default photographer identity" seems to attempt modification of the RAW image file and break it in the process... It still valid after to have updated libkexiv2 ? Gilles Caulier Markus, HAs you have a Sony camera and you shot ARW photo, can you reproduce the problem ? Gilles Caulier Yes Gilles, it is still valid after the upgrade, unless I did something wrong ;-) I ran svn update of kdegraphics and rebuilt libkexiv2, installed them and then started digikam. Or should I have updated digikam as well? Cheers Simon Just tried with 0.10.0rc2, SVN 883547 under Kubuntu Intrepid Ibex, KDE 4.1 Importing via digikam Sony .arw and .jpg files via a USB CF cardreader (vivanco brand) imports all files without and corruption. jpg files display without failures in digikam editor or gimp, arw files can be opened and converted in bibble. Not a single hint of a problem here on my side. In theory, yes, because binary compatibility is respected... But to be sure, try to recompile and install digiKam. Gilles Markus, Are you set the same downloading options than Simon ? Gilles OK, my first test was with no changing options in the sidebar dialogue of the download window active, i.e. no renumbering, no writing of photographers identity etc. there were no corruptions visible. A second test with active writing of a photographers identity during download, set from the download window options dialog, indeed resulted in writing of the raw and jpeg files. Now neither the jpg nor the raw files create thumbnails in digikam, they are displayable outside of digikam in gimp (jpg) resp. bibble (arw), but again the metadata of the arw files are corrupted and create problems in bibble. A cross-check showed that with this options "off", the files are imported again without problems. I guess its a problem in exiv2 (I am using 0.18pre2) and digikam - at least the raw files should not be modified. I updated and recompiled/installed digikam svn, same problem and I see Markus also confirmed this bug. Corrupted file when setting the option "Set default photographer identity", no corruption when un-setting this option. /Simon NB: the setting that affects my raw files is under "On the fly operations (JPG only!)", which makes it harder to track down when it affects RAW files... Marcel, Look in code: http://lxr.kde.org/source/extragear/graphics/digikam/utilities/cameragui/cameracontroller.cpp#523 This is camera controller code which play with image metadata. we call DMetadata::applyChanges() which call Kexiv2:save(). In this method we test if Raw files is supported : http://lxr.kde.org/source/KDE/kdegraphics/libs/libkexiv2/libkexiv2/kexiv2.cpp#308 So in case of ARW, nothing must be changed in metadata. Note : we can test if it's really a JPEG file in camera controller before to play with DMetadata. Marcus, Can you give the console trace when an ARW file is downloaded (and touched of course) Gilles Caulier First: "JPEG-only" is obviously wrong, at least the exiv2 related options are done for all supported files. And now this is the problem here: digikam(7684)/KEXIV2 KExiv2Iface::KExiv2::save: File Extension: "digikam-camera-tmp1-7684" is supported for writing mode We operate on a temp file which does not have any file suffix. Yes, Marcel, i just discovered this problem. A fix is on my computer... I will extend "JPEG only" operation to PNG/TIFF/JPEG2000 later 0.10.0. Gilles SVN commit 919053 by cgilles: digiKam from trunk: Camera GUI : append orginal file name to temp url used to download file from camera (not prepend). Like this, original file type mime is not lost and JPEG operation during download are not applied to RAW files. CCBUGS: 180065 M +1 -1 cameracontroller.cpp WebSVN link: http://websvn.kde.org/?view=rev&revision=919053 Great that you found the fix! however I must note that mime types should not be inferred from filenames or their extensions, this may be ok for 99% of all situations, but it's safer to test a file based on the internal file structure (like the file command does). Anyway, I'm already quite happy to know that the problem should now be fixed... Cheers Simon Well, try to detect type mime use file content is impossible to do... with RAW file. Why??? because camera maker have the good idea to use a derivated TIFF/EP format to make RAW files... For example KDELib typemime detection method find TIFF for ARW. Do you feel the problem ? This is why in digiKAm we only use file extension... (else it will be easy of course) I would to thanks camera maker to increase complexity of photo workflow !!! This is why i'm a support DNG format (it's not perfect of course, but it's a first start) Gilles Caulier SVN commit 919064 by cgilles: digiKam from trunk : to prevent any problem, only apply JPEG fix to JPEG files BUG: 180065 M +42 -39 cameracontroller.cpp WebSVN link: http://websvn.kde.org/?view=rev&revision=919064 |