Bug 318577

Summary:	Batch queue fails to process multiple files simultaneously (in parallel)
Product:	[Applications] digikam	Reporter:	Eduard Huguet <eduardhc>
Component:	BatchQueueManager-Workflow	Assignee:	Digikam Developers <digikam-bugs-null>
Status:	RESOLVED FIXED
Severity:	major	CC:	bugs, caulier.gilles, mailto.mikes, scramer
Priority:	NOR
Version:	3.1.0
Target Milestone:	---
Platform:	Fedora RPMs
OS:	Linux
Latest Commit:		Version Fixed In:	4.3.0
Sentry Crash Report:

Description Eduard Huguet 2013-04-19 06:58:26 UTC

I'm trying to process a batch of RAW files in Digikam in order to convert them to uncompressed PNG format. To do so, I add the RAW files to the queue, then set up RAW open options and the "Convert to PNG".

When files are processed, only a minor part of them are opened succesfully, while most of them fail. Logging traces in .xsession-errors show error messages regarding corruption found while trying to open the RAW files.

However, the RAW files (from a Sony Alpha-1, which is a long-time supported model) are not corrupted, and can be imported with no issues if I open them individually.

The interesting part, however, is that if I boot the system with "maxcpus=1" kernel setting, then the files are batch-processed sucessfully, which suggest some kind of problem related to multiple processing of RAW files simultaneosly (my computer has 4 cores, so the BQM attempts to process 4 files at a time without that option).

I suppose the problem is related to libdcraw or similar, not Digikam itself, but maybe Digikam should allow for some workaround (like an option to limit MT in BQM, or either disabling it completely if RAW files are processed until the error gets fixed).

Reproducible: Always

Steps to Reproduce:
1. select multiple RAW files from main Digikam screen, add them to a new queue
2. set up RAW opening options, then a conversion tool (in my case "Convert to PNG")
3. start processing files
Actual Results:
Most files are processed with errors, thus leading to empty result PNG files. The error appears to happen when RAW import tool tries to load the file.

Expected Results:
All RAW files successfully converted to PNG.

Comment 1 caulier.gilles 2013-04-19 08:29:50 UTC

With 3.2.0-beta2, i can process Alpha A67 RAW in PNG without any problem here (i7 computer, 6 cores)...

http://www.flickr.com/photos/digikam/8661749113/sizes/o/in/photostream/

What's your RAW workflow in BQM ?

Gilles Caulier

Comment 2 Eduard Huguet 2013-04-19 08:38:13 UTC

(In reply to comment #1)
> With 3.2.0-beta2, i can process Alpha A67 RAW in PNG without any problem
> here (i7 computer, 6 cores)...
> 
> http://www.flickr.com/photos/digikam/8661749113/sizes/o/in/photostream/
> 
> What's your RAW workflow in BQM ?
> 
> Gilles Caulier

Hi, and thank you for the quick reaction :-)

In this  case, the one described above: I simply select the RAW files I want to convert (which I downloaded from the camera), and I add them to a new queue. Once in BQM, I adjust RAW opening options (nothing special: AMAZe filtering and leave most other options as default - have checked using Bilinear with the same results...), then add a single process step consisting on the "Convert to PNG" tool. 

I do this because I prefer to batch-convert all RAW files to PNG first, then edit every PNG one by one. I used to do that using old "Batch RAW converter" tool, but now this one has gone, as it has been replaced by BQM (which totally makes sense to me, anyway). However, I'm finding this unexpected problem. IIRC, the old tool was not processing files in parallel, so that might be the reason this problem arises now.

It might be something related to specific versions of Digikam, libdcraw, etc... packed in Fedora 18.

Comment 3 caulier.gilles 2013-04-19 08:53:45 UTC

Since 3.0.0, kipi RAW converter processed files in parallel. You can try it from Gwenview  for ex. Tool is just disabled in digiKam. It use the same background process than BQM to paralelize operations, but implementation is more complex of course due of capability to chain tools in a workflow.

You don't need to turn your computer to single cpu through linux kernel option. In BQM, it's possible to turn off paralelization by patch source code like this :

//-----------------------------------------------------------------------------------------------------------------
diff --git a/utilities/queuemanager/manager/actionthread.cpp b/utilities/queuemanager/manager/actionthread.cpp
index 7a0a004..5f33727 100644
--- a/utilities/queuemanager/manager/actionthread.cpp
+++ b/utilities/queuemanager/manager/actionthread.cpp
@@ -55,6 +55,8 @@ public:
 ActionThread::ActionThread(QObject* const parent)
     : RActionThreadBase(parent), d(new Private)
 {
+    setMaximumNumberOfThreads(1);
+
     qRegisterMetaType<ActionData>();

     connect(this, SIGNAL(finished()),
//-----------------------------------------------------------------------------------------------------------------

Please try it just to confirm the problem with paralelization.

Q : why i cannot reproduce this problem on my computer. I use Sony Alpha camera in RAW and as you, i play with processed images in PNG...

Gilles Caulier

Comment 4 Eduard Huguet 2013-04-19 09:01:57 UTC

(In reply to comment #3)
> Since 3.0.0, kipi RAW converter processed files in parallel. You can try it
> from Gwenview  for ex. Tool is just disabled in digiKam. It use the same
> background process than BQM to paralelize operations, but implementation is
> more complex of course due of capability to chain tools in a workflow.
> 
> You don't need to turn your computer to single cpu through linux kernel
> option. In BQM, it's possible to turn off paralelization by patch source
> code like this :
> 
> //---------------------------------------------------------------------------
> --------------------------------------
> diff --git a/utilities/queuemanager/manager/actionthread.cpp
> b/utilities/queuemanager/manager/actionthread.cpp
> index 7a0a004..5f33727 100644
> --- a/utilities/queuemanager/manager/actionthread.cpp
> +++ b/utilities/queuemanager/manager/actionthread.cpp
> @@ -55,6 +55,8 @@ public:
>  ActionThread::ActionThread(QObject* const parent)
>      : RActionThreadBase(parent), d(new Private)
>  {
> +    setMaximumNumberOfThreads(1);
> +
>      qRegisterMetaType<ActionData>();
> 
>      connect(this, SIGNAL(finished()),
> //---------------------------------------------------------------------------
> --------------------------------------
> 
> Please try it just to confirm the problem with paralelization.
> 
> Q : why i cannot reproduce this problem on my computer. I use Sony Alpha
> camera in RAW and as you, i play with processed images in PNG...
> 
> Gilles Caulier

Interesting. I'll try later at home with the old batch convert tool in Gwenview, hadn't thought about it. 

Regarding about disabling parallelization by patching source code, I might give it a try later. I have a long experience regarding C++ & Qt, I just need to find the time to set up the compilation environment for Digikam, which I assume I will be complex (lots of dependencies needed, I guess...)

And regarding final question: I don't know, frankly :-( . I'll retry later at home just to be sure that the problem is related to parallelization. Maybe it has to be with the version of the RAW decoding library (libdcraw, libopenraw...?) used in F18 (as I suspect that the bug is really not in Digikam itself, but there instead...).

Comment 5 caulier.gilles 2013-07-27 09:19:26 UTC

Git commit dd3dc1cdd2292f2c12b2927250619ae4781f3a61 by Gilles Caulier.
Committed on 27/07/2013 at 09:10.
Pushed by cgilles into branch 'master'.

Batch Queue Manager : add new option to turn on/off multi-core support. This option is now turn off by default due to some dysfunctions under Windows, until it will be hack and fixed.
Note that problem are not reproducible under OSX and Linux here. Options is implemnted for digiKam 3.3.0 release.
Related: bug 315025, bug 318198, bug 320358

M  +13   -1    utilities/queuemanager/manager/actionthread.cpp
M  +1    -1    utilities/queuemanager/manager/actionthread.h
M  +4    -1    utilities/queuemanager/manager/queuesettings.h
M  +9    -1    utilities/queuemanager/manager/workflowmanager.cpp
M  +14   -2    utilities/queuemanager/views/queuesettingsview.cpp

http://commits.kde.org/digikam/dd3dc1cdd2292f2c12b2927250619ae4781f3a61

Comment 6 caulier.gilles 2014-09-01 11:41:17 UTC

*** Bug 320358 has been marked as a duplicate of this bug. ***

Comment 7 caulier.gilles 2014-09-01 11:42:13 UTC

*** Bug 315025 has been marked as a duplicate of this bug. ***

Comment 8 caulier.gilles 2014-09-01 11:43:27 UTC

Eduard,

This file still valid using last digiKam 4.2.0 ?

Gilles Caulier

Comment 9 Eduard Huguet 2014-09-01 21:02:20 UTC

Hi, Gilles
Just testing it now using 4.2.0 from Fedora 20, and it seems to work fine. Thank you very much for your hard work!