Bug 316948

Summary:	Add a workflow check when RAW files are processed without an export tool
Product:	[Applications] digikam	Reporter:	cfab <fabien.crepin>
Component:	BatchQueueManager-RAWProcessing	Assignee:	Digikam Developers <digikam-bugs-null>
Status:	REPORTED ---
Severity:	wishlist	CC:	caulier.gilles
Priority:	NOR
Version:	3.1.0
Target Milestone:	---
Platform:	Ubuntu
OS:	Linux
Latest Commit:		Version Fixed In:
Sentry Crash Report:
Attachments:	Settings to reproduce the problem

Description cfab 2013-03-18 05:55:53 UTC

When processing multiple files, it leads to 'unable to process element'.

When processing a single file, it is ok.

Operations invoked were resizing and/or jpg conversion.

Result files are saved, but with 0 size.

Reproducible: Always

Steps to Reproduce:
1. Choose 2 files to process
2. Add an operation
3. Process
Actual Results:  
'unable to process element' (sorry, I've the French message, so something like that I guess) for every elements
Files saved with 0 size

Expected Results:  
Files correctly processed

Comment 1 caulier.gilles 2013-03-18 06:08:53 UTC

Not reproducible here. Can you take a shot of BQM in action with all workflow settings.

Also, run kdebugdialog, turn on digiKam, KExiv2, and KDcraw debug space, and run digiKam from a console. Post here all debug traces.

Which libjpeg version you use ? Do you use turbo-jpeg ?

Gilles Caulier

Comment 2 cfab 2013-03-18 06:23:53 UTC

Created attachment 78157 [details]
Settings to reproduce the problem

I don't have KExiv2 or KDcraw installed.

How can I tell you what JPG lib I'm using ?

Here is the console output (I've not understood what KDebugDialog is intended to do) :

fabien@pc-bureau:~$ digikam
Fontconfig warning: "/etc/fonts/conf.d/50-user.conf", line 9: reading configurations from ~/.fonts.conf is deprecated.
digikam(5293)/digikam (core) Digikam::KInotify::Private::open: Successfully opened connection to inotify: 12
digikam(5293)/digikam (core) Digikam::AlbumManager::setDatabase: DatabaseParameters: [ Type "QSQLITE", Name "/media/HDD_DATA1/Photos/digikam4.db" (Thumbnails Name "/media/HDD_DATA1/Photos/thumbnails-digikam.db"); ]
QSqlDatabasePrivate::removeDatabase: connection 'ConnectionTest' is still in use, all queries will cease to work.
digikam(5293)/digikam (core) Digikam::DatabaseConfigElementLoader::readConfig: Loading SQL code from config file "/usr/share/kde4/apps/digikam/database/dbconfig.xml"
digikam(5293)/digikam (core) Digikam::DatabaseConfigElementLoader::readConfig: false "1" 1 1
digikam(5293)/digikam (core) Digikam::SchemaUpdater::update: SchemaUpdater update
digikam(5293)/digikam (core) Digikam::SchemaUpdater::startUpdates: Have a database structure version  7
digikam(5293)/digikam (core) Digikam::SchemaUpdater::makeUpdates: makeUpdates  7  to  7
digikam(5293)/digikam (core) Digikam::AlbumRootLocation::AlbumRootLocation: Creating new Location  "/Photos"  uuid  "volumeid:?uuid=52afc0b6-d5f9-4c7d-8eb0-47f5739dd00b"
digikam(5293)/digikam (core) Digikam::CollectionManager::updateLocations: location for  "/media/HDD_DATA1/Photos"  is available  true
digikam(5293)/digikam (core) Digikam::KMemoryInfo::update: Platform identified :  "LINUX"
digikam(5293)/digikam (core) Digikam::KMemoryInfo::bytes: TotalRam:  8469602304
digikam(5293)/digikam (core) Digikam::LoadingCache::setCacheSize: Allowing a cache size of 200 MB
digikam(5293)/digikam (core) Digikam::ThumbnailSchemaUpdater::startUpdates: Have a thumbnail database structure version  "2"
digikam(5293)/digikam (core) Digikam::ThumbnailLoadThread::initializeThumbnailDatabase: Thumbnail db ready for use
digikam(5293)/digikam (core) Digikam::CollectionScanner::completeScan: Complete scan (file scanning deferred) took: 210 msecs.
digikam(5293)/digikam (core) Digikam::IccSettings::Private::scanDirectories: ("/usr/share/color/icc", "/home/fabien/.local/share/icc")
digikam(5293)/digikam (core) Digikam::LensFunCameraSelector::populateLensCombo: variant:  QVariant(Digikam::LensFunIface::DevicePtr, )
digikam(5293)/digikam (core) Digikam::LensFunCameraSelector::populateLensCombo: dev:  Asahi Optical Co.,Ltd  ::  PENTAX Optio 430  ::  4.85
digikam(5293)/digikam (core) Digikam::LensFunIface::findCamera: Search for camera  "" - ""  ==> false
digikam(5293)/digikam (core) Digikam::LensFunIface::findLens: Search for lens  ""  ==> false
digikam(5293)/digikam (core) Digikam::CurvesWidget::updateData: updating data
digikam(5293)/digikam (core) Digikam::ImageCurves::fillFromOtherCurves: Filling this curve from other curve  0xb0c4640
digikam(5293)/digikam (core) Digikam::ImageCurves::fillFromOtherCurves: Both curves have same type: isSixteenBits =  true
digikam(5293)/digikam (core) Digikam::ImageCurves::fillFromOtherCurves: Other is CURVE_SMOOTH
digikam(5293)/digikam (core) Digikam::ImageCurves::fillFromOtherCurves: Other is CURVE_SMOOTH
digikam(5293)/digikam (core) Digikam::ImageCurves::fillFromOtherCurves: Other is CURVE_SMOOTH
digikam(5293)/digikam (core) Digikam::ImageCurves::fillFromOtherCurves: Other is CURVE_SMOOTH
digikam(5293)/digikam (core) Digikam::ImageCurves::fillFromOtherCurves: Other is CURVE_SMOOTH
digikam(5293)/digikam (core) Digikam::BlurFilter::cimgBlurImage: Radius:  0.1
digikam(5293)/digikam (core) Digikam::BlurFilter::cimgBlurImage: Radius:  0.2
digikam(5293)/digikam (core) Digikam::BlurFilter::cimgBlurImage: Radius:  0.5
digikam(5293)/digikam (core) Digikam::BlurFilter::cimgBlurImage: BlurFilter::Process Computation...
digikam(5293)/digikam (core) Digikam::BlurFilter::cimgBlurImage: BlurFilter::Process Computation...
digikam(5293)/digikam (core) Digikam::BlurFilter::cimgBlurImage: BlurFilter::Finalization...
digikam(5293)/digikam (core) Digikam::BlurFilter::cimgBlurImage: Radius:  0.3
digikam(5293)/digikam (core) Digikam::BlurFilter::cimgBlurImage: BlurFilter::Process Computation...
digikam(5293)/digikam (core) Digikam::BlurFilter::cimgBlurImage: BlurFilter::Finalization...
digikam(5293)/digikam (core) Digikam::BlurFilter::cimgBlurImage: BlurFilter::Process Computation...
digikam(5293)/digikam (core) Digikam::BlurFilter::cimgBlurImage: BlurFilter::Finalization...
digikam(5293)/digikam (core) Digikam::CurvesWidget::updateData: updating data
digikam(5293)/digikam (core) Digikam::BlurFilter::cimgBlurImage: BlurFilter::Finalization...
digikam(5293)/digikam (core) Digikam::ImageCurves::fillFromOtherCurves: Filling this curve from other curve  0xaebb7b8
digikam(5293)/digikam (core) Digikam::ImageCurves::fillFromOtherCurves: Both curves have same type: isSixteenBits =  true
digikam(5293)/digikam (core) Digikam::ImageCurves::fillFromOtherCurves: Other is CURVE_SMOOTH
digikam(5293)/digikam (core) Digikam::ImageCurves::fillFromOtherCurves: Other is CURVE_SMOOTH
digikam(5293)/digikam (core) Digikam::ImageCurves::fillFromOtherCurves: Other is CURVE_SMOOTH
digikam(5293)/digikam (core) Digikam::ImageCurves::fillFromOtherCurves: Other is CURVE_SMOOTH
digikam(5293)/digikam (core) Digikam::ImageCurves::fillFromOtherCurves: Other is CURVE_SMOOTH
digikam(5293)/digikam (core) Digikam::AlbumFilterModel::setSearchTextSettings: new search text settings:  "" : hasResult =  false , validRows =  0
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::ThemeManager::slotChangePalette: ""
digikam(5293)/digikam (core) Digikam::ThemeManager::slotChangePalette: ""
digikam(5293)/digikam (core) Digikam::ThemeManager::slotChangePalette: ""
digikam(5293)/digikam (core) Digikam::ThemeManager::slotChangePalette: ""
digikam(5293)/digikam (core) Digikam::KInotify::Private::open: Successfully opened connection to inotify: 63
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (editor plugins) DigikamColorImagePlugin::ImagePlugin_Color::ImagePlugin_Color: ImagePlugin_Color plugin loaded
digikam(5293)/digikam (core) Digikam::ImagePluginLoader::loadPluginsFromList: ImagePluginLoader: Loaded plugin  "ImagePlugin_Color"
digikam(5293)/digikam (editor plugins) DigikamDecorateImagePlugin::ImagePlugin_Decorate::ImagePlugin_Decorate: ImagePlugin_Decorate plugin loaded
digikam(5293)/digikam (core) Digikam::ImagePluginLoader::loadPluginsFromList: ImagePluginLoader: Loaded plugin  "ImagePlugin_Decorate"
digikam(5293)/digikam (editor plugins) DigikamEnhanceImagePlugin::ImagePlugin_Enhance::ImagePlugin_Enhance: ImagePlugin_Enhance plugin loaded
digikam(5293)/digikam (core) Digikam::ImagePluginLoader::loadPluginsFromList: ImagePluginLoader: Loaded plugin  "ImagePlugin_Enhance"
digikam(5293)/digikam (editor plugins) DigikamFxFiltersImagePlugin::ImagePlugin_FxFilters::ImagePlugin_FxFilters: ImagePlugin_FxFilters plugin loaded
digikam(5293)/digikam (core) Digikam::ImagePluginLoader::loadPluginsFromList: ImagePluginLoader: Loaded plugin  "ImagePlugin_FxFilters"
digikam(5293)/digikam (editor plugins) DigikamTransformImagePlugin::ImagePlugin_Transform::ImagePlugin_Transform: ImagePlugin_Transform plugin loaded
digikam(5293)/digikam (core) Digikam::ImagePluginLoader::loadPluginsFromList: ImagePluginLoader: Loaded plugin  "ImagePlugin_Transform"
digikam(5293)/digikam (core) Digikam::NewItemsFinder::slotStart: scan mode: ScanDeferredFiles
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: ""
digikam(5293)/digikam (core) Digikam::QueueMgrWindow::checkTargetAlbum: Target album for queue  "#1"  is:  "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG/"
digikam(5293)/digikam (core) Digikam::Task::run: Tool : index=  1  :: name=  "Resize"  :: group=  5  :: wurl=  KUrl("file:///media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG/")
digikam(5293)/digikam (core) Digikam::Task::run: Tool : index=  1  :: name=  "Resize"  :: group=  5  :: wurl=  KUrl("file:///media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG/")
digikam(5293)/digikam (core) Digikam::BatchTool::setOutputUrlFromInputUrl: path:  "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG/BatchTool-MT5293.digikamtempfile.ARW"
digikam(5293)/digikam (core) Digikam::BatchTool::setOutputUrlFromInputUrl: path:  "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG/BatchTool-TJ5293.digikamtempfile.ARW"
digikam(5293)/digikam (core) Digikam::BatchTool::apply: Tool:        "Redimensionnement"
digikam(5293)/digikam (core) Digikam::BatchTool::apply: Tool:        "Redimensionnement"
digikam(5293)/digikam (core) Digikam::BatchTool::apply: Input url:   KUrl("file:///media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/DSC04924.ARW")
digikam(5293)/digikam (core) Digikam::BatchTool::apply: Input url:   KUrl("file:///media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/DSC04925.ARW")
digikam(5293)/digikam (core) Digikam::BatchTool::apply: Output url:  KUrl("file:///media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG/BatchTool-TJ5293.digikamtempfile.ARW")
digikam(5293)/digikam (core) Digikam::BatchTool::apply: Output url:  KUrl("file:///media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG/BatchTool-MT5293.digikamtempfile.ARW")
digikam(5293)/digikam (core) Digikam::DImg::load: "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/DSC04925.ARW"  : RAW file identified
digikam(5293)/digikam (core) Digikam::DImg::load: "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/DSC04924.ARW"  : RAW file identified
/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/DSC04924.ARW: data corrupted at 65537
No itemShortInfo could be retrieved from the database for image "DSC04924_1.ARW" 
digikam(5293)/digikam (core) Digikam::DImg::load: "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG/DSC04924_1.ARW"  : RAW file identified
digikam(5293)/digikam (core) Digikam::ImageScanner::prepareAddImage: Adding new item "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG/DSC04924_1.ARW"
digikam(5293)/digikam (core) Digikam::ImageScanner::commit: Scanning took 1 ms
digikam(5293)/digikam (core) Digikam::ImageScanner::~ImageScanner: Finishing took 126 ms
digikam(5293)/digikam (core) Digikam::CollectionScanner::copyFileProperties: Copying properties from 21726 to 26280
/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/DSC04925.ARW: data corrupted at 4823332
No itemShortInfo could be retrieved from the database for image "DSC04925.ARW" 
digikam(5293)/digikam (core) Digikam::DImg::load: "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG/DSC04925.ARW"  : RAW file identified
digikam(5293)/digikam (core) Digikam::ImageScanner::prepareAddImage: Adding new item "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG/DSC04925.ARW"
digikam(5293)/digikam (core) Digikam::ImageScanner::commit: Scanning took 1 ms
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: "1000"
digikam(5293)/digikam (core) Digikam::ImageScanner::~ImageScanner: Finishing took 106 ms
digikam(5293)/digikam (core) Digikam::CollectionScanner::copyFileProperties: Copying properties from 21727 to 26281
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: "1000"
digikam(5293)/digikam (core) Digikam::AlbumWatch::rescanDirectory: Detected change, triggering rescan of directory "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG"
digikam(5293)/digikam (core) Digikam::AlbumWatch::rescanDirectory: Detected change, triggering rescan of directory "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG"
digikam(5293)/digikam (core) Digikam::AlbumWatch::rescanDirectory: Detected change, triggering rescan of directory "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG"
digikam(5293)/digikam (core) Digikam::AlbumWatch::rescanDirectory: Detected change, triggering rescan of directory "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG"
digikam(5293)/digikam (core) Digikam::AlbumWatch::rescanDirectory: Detected change, triggering rescan of directory "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG"
digikam(5293)/digikam (core) Digikam::AlbumWatch::rescanDirectory: Detected change, triggering rescan of directory "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG"
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: "1000"
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: "1000"
digikam(5293)/digikam (core) Digikam::ActionThread::slotThreadFinished: List of Pending Jobs is empty
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: "1000"
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: "1000"
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: "1000"
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: "1000"
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: "1000"
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: "1000"
digikam(5293)/digikam (core) Digikam::StatusProgressBar::currentProgressItem: "1000"
digikam(5293)/digikam (core) Digikam::DImg::load: "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG/DSC04924_1.ARW"  : RAW file identified
digikam(5293)/digikam (core) Digikam::JPEGUtils::isJpegImage: mimetype =  ""  ext =  "ARW"
digikam(5293)/digikam (core) Digikam::ThumbnailCreator::createThumbnail: Cannot create thumbnail for  "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG/DSC04924_1.ARW" 
digikam(5293)/digikam (core) Digikam::ThumbnailCreator::load: Thumbnail is null for  "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG/DSC04924_1.ARW" 
digikam(5293)/digikam (core) Digikam::DImg::load: "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG/DSC04925.ARW"  : RAW file identified
digikam(5293)/digikam (core) Digikam::JPEGUtils::isJpegImage: mimetype =  ""  ext =  "ARW"
digikam(5293)/digikam (core) Digikam::ThumbnailCreator::createThumbnail: Cannot create thumbnail for  "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG/DSC04925.ARW" 
digikam(5293)/digikam (core) Digikam::ThumbnailCreator::load: Thumbnail is null for  "/media/HDD_DATA1/Photos/Randos/2012-03-17 Collège Hitiia/JPEG/DSC04925.ARW"

Comment 3 cfab 2013-03-18 06:34:52 UTC

I've seen some "data corrupted" on the log. The point is for ONE single file, there's no problem and the process achieves successfully.

Comment 4 caulier.gilles 2013-03-18 06:45:43 UTC

Go to Help/Components Info for details.

Gilles Caulier

Comment 5 cfab 2013-03-18 06:50:58 UTC

LibJPG : 80

Comment 6 cfab 2013-03-20 04:52:23 UTC

Is there anything else that I can do to get a more precise log ?

Thanks for your time!

Comment 7 caulier.gilles 2013-03-20 08:02:02 UTC

Ah, yes, now, i understand your problem.

You process ARW files which are read only images. With RAW files, you must export to another format all operations (JPEG, tiff, PNG, etc...)

There is not yet a check in BQM to see if you have assigned an "export to" tool in your workflow when you process RAW files.

Gilles Caulier

Comment 8 cfab 2013-03-20 21:30:34 UTC

I don't think this is the problem because of these two points : 

1°) If I had an export operation, the problem remains the same.

2°) If I process only ONE file in the BQM, there's no problem.

Comment 9 cfab 2013-03-25 02:56:17 UTC

Well, if you can't reproduce the problem, maybe there's something to do with external libraries. What is the libraries to check version ?

This problem is very annoying since processing pictures individually is very time comsupting.
Is there any other Linux tools that can process Raw files in batch mode ?

Comment 10 caulier.gilles 2013-03-25 05:53:11 UTC

I think the lead problem is fixed now following this report :

https://bugs.kde.org/show_bug.cgi?id=313938

Marie Noel has exactly the same problem than you with 0 byte JPEG file size generated.

About RAW, processing is slow, because demosaicing require high computing. Image data to process are more important than JPEG.

But with digiKam 3.x, BQM use multiprocessor (if you have a multi CPU). As example, an i7 will be able to process 8 items in parallel

Gilles Caulier

Comment 11 caulier.gilles 2015-08-13 08:01:37 UTC

digiKam 4.12.0 is out.

https://www.digikam.org/node/741

Problem still reproducible ?

Gilles Caulier

Comment 12 caulier.gilles 2018-12-31 11:49:31 UTC

Can you reproduce the dysfunction using the last digiKam 6.0.0-beta3 just
released ?

https://www.digikam.org/news/2018-12-30-6.0.0-beta3_release_announcement/

Comment 13 caulier.gilles 2020-07-14 09:41:26 UTC

Hi,

Can you check if this problem still exist with last weekly bundle build of digiKam 7.0.0 available here:

https://files.kde.org/digikam/

Thanks in advance

Gilles Caulier