Version: 1.5.0 (using KDE 4.5.2) OS: Linux The digikam batch queue manager seems to run in the same thread as digikam itself, and is only using one CPU core. If I am correct, and I hope I didn't overlook an existing enhancement request, then it would be great if the batch queue manager would use all or a selectable maximum of the available CPU cores on the system. I think 3-4 cores is the average of today's CPUs. Optionally using a lower thread priority could help using the system while large batches are running. Of course many of the image tools are already partially multithreaded, but that's not helping a lot. Reproducible: Didn't try Expected Results: Almost 400% CPU usage on a system with 4 cores. ;)
I have the wish as you do. Just a remark: I think there is a missunderstanding. In the changelog of previous versions of DigiKam, the fact that image tools are multithreaded, only meant that they can be run in a separate thread from the gui, it does not mean they are parallelized to use several cores. Julien
BQM use multithreading, through QThread : http://lxr.kde.org/source/extragear/graphics/digikam/utilities/queuemanager/actionthread.h#44 using multi core processor is planed using OpenMP API Gilles Caulier
*** Bug 195538 has been marked as a duplicate of this bug. ***
BQM use threading, but every queue resists in it's own thread. Every item of this queue is executed in a linear manner, it is currently not possible to execute the batch tool stack to N images at the same time of the same queue (if you have N cores). That is the footpoint I think.
A new Thread manager must be written to : - Process each queue in a dedicated thread - Process item in parallel. No need to wait than one item is processed to work on next one. We will use OpenMp at this place. - Separate GUI of thread processing. Currently, GUi is used to manage the item and queue to process in a separate thread. This must be done in a low level. GUi must show only the result of thread processing. These changes will break a lots of code in BQM. i'm not sure if we can do it in 1.x. Jens, are you interrested to work with me on this subject ? Note : before to work on this file, this one must be solved : https://bugs.kde.org/show_bug.cgi?id=195784 All is ready in settings to pass dedicated RAW decoding settings to queue processing. Only GUi must be factored with Raw import tools (settings) and shared with BQM. Gilles Caulier
(In reply to comment #5) > A new Thread manager must be written to : > > - Process each queue in a dedicated thread Yes. > - Process item in parallel. No need to wait than one item is processed to work > on next one. We will use OpenMp at this place. Speaking from bird eyes view, I do not think so. All filters are build on top of dynamicthread class and that one is build on qrunnable, right? So we have to chain the filter stack for each image into another qrunnable and execute all of them in parallel. But I have get a deep lock at the code first. > - Separate GUI of thread processing. Currently, GUi is used to manage the item > and queue to process in a separate thread. This must be done in a low level. Yes. Another problem is the chaining of output messages that refer to one image. > GUi must show only the result of thread processing. Yes. > > These changes will break a lots of code in BQM. i'm not sure if we can do it in > 1.x. I do not think that this is a simple task, so not for 1.6. > > Jens, are you interrested to work with me on this subject ? Yes, but I lack in time. > > Note : before to work on this file, this one must be solved : > > https://bugs.kde.org/show_bug.cgi?id=195784 > > All is ready in settings to pass dedicated RAW decoding settings to queue > processing. Only GUi must be factored with Raw import tools (settings) and > shared with BQM. > > Gilles Caulier
There are two approaches which could be called linear and parallel: 1) Linear: Each processing step ("worker object") lives in its own thread. An image is sent to the first worker, processed, then sent to the next worker. 2) Parallel: A processing step is provided by n threads, processing n different images at a time. The maximum version of this would be to provide the full pipeline parallely in n threads. You can combine the approaches. If you look at the FacePipeline for face detection, it is mainly based on the linear approach, but for one segment (detection, the most expensive step), it parallelizes using more than one detector. The situation is a bit different though, because some tasks cannot be parallelized because they are mutex-protected (database access, recognition database), so the linear approach is natural. With BQM, I think, all tasks can be parallelized without restriction. We must only care for memory demands - if you've got a hundred images to process, you dont want to load them into memory at once. There's QThreadPool::idealThreadCount as well. Have a look in libs/threads/. Probably neither DynamicThread nor WorkerObject fit perfectly into the task, but could be helpful. We can also write a third taste of threading class.
Is there something new about this bug?
not yet. We need to find student to work on it... Gilles Caulier
Git commit 987d78c787849ba8f8dfab408afd83a7172f0162 by Gilles Caulier. Committed on 05/12/2012 at 14:24. Pushed by cgilles into branch 'master'. Add support multicore CPU in Batch Queue Manager to process items from queues in parallel. This will speed up items processing depending of cores available in your computer. Detection of core suitable is done through KDE::Solid API. Registration of items properties is not paralellized yet and still in main thread. It can be a bottleneck in the workflow. This point must be analyzed later with core Database interface. FIXED-IN: 3.0.0 M +1 -0 utilities/queuemanager/CMakeLists.txt M +13 -8 utilities/queuemanager/basetools/color/autocorrection.cpp M +4 -0 utilities/queuemanager/basetools/color/autocorrection.h M +12 -8 utilities/queuemanager/basetools/color/bcgcorrection.cpp M +4 -0 utilities/queuemanager/basetools/color/bcgcorrection.h M +4 -5 utilities/queuemanager/basetools/color/bwconvert.cpp M +4 -1 utilities/queuemanager/basetools/color/bwconvert.h M +12 -7 utilities/queuemanager/basetools/color/channelmixer.cpp M +4 -0 utilities/queuemanager/basetools/color/channelmixer.h M +10 -6 utilities/queuemanager/basetools/color/colorbalance.cpp M +4 -0 utilities/queuemanager/basetools/color/colorbalance.h M +0 -1 utilities/queuemanager/basetools/color/convert16to8.cpp M +2 -0 utilities/queuemanager/basetools/color/convert16to8.h M +0 -1 utilities/queuemanager/basetools/color/convert8to16.cpp M +2 -0 utilities/queuemanager/basetools/color/convert8to16.h M +4 -2 utilities/queuemanager/basetools/color/curvesadjust.cpp M +4 -1 utilities/queuemanager/basetools/color/curvesadjust.h M +10 -6 utilities/queuemanager/basetools/color/hslcorrection.cpp M +4 -0 utilities/queuemanager/basetools/color/hslcorrection.h M +5 -6 utilities/queuemanager/basetools/color/iccconvert.cpp M +4 -1 utilities/queuemanager/basetools/color/iccconvert.h M +0 -1 utilities/queuemanager/basetools/color/invert.cpp M +2 -0 utilities/queuemanager/basetools/color/invert.h M +22 -18 utilities/queuemanager/basetools/color/whitebalance.cpp M +4 -0 utilities/queuemanager/basetools/color/whitebalance.h M +10 -5 utilities/queuemanager/basetools/convert/convert2jp2.cpp M +4 -0 utilities/queuemanager/basetools/convert/convert2jp2.h M +10 -5 utilities/queuemanager/basetools/convert/convert2jpeg.cpp M +4 -0 utilities/queuemanager/basetools/convert/convert2jpeg.h M +10 -5 utilities/queuemanager/basetools/convert/convert2pgf.cpp M +4 -0 utilities/queuemanager/basetools/convert/convert2pgf.h M +10 -5 utilities/queuemanager/basetools/convert/convert2png.cpp M +4 -0 utilities/queuemanager/basetools/convert/convert2png.h M +10 -5 utilities/queuemanager/basetools/convert/convert2tiff.cpp M +4 -0 utilities/queuemanager/basetools/convert/convert2tiff.h M +10 -6 utilities/queuemanager/basetools/decorate/border.cpp M +4 -0 utilities/queuemanager/basetools/decorate/border.h M +11 -5 utilities/queuemanager/basetools/decorate/watermark.cpp M +4 -0 utilities/queuemanager/basetools/decorate/watermark.h M +4 -4 utilities/queuemanager/basetools/enhance/antivignetting.cpp M +4 -1 utilities/queuemanager/basetools/enhance/antivignetting.h M +6 -6 utilities/queuemanager/basetools/enhance/blur.cpp M +4 -1 utilities/queuemanager/basetools/enhance/blur.h M +4 -4 utilities/queuemanager/basetools/enhance/lensautofix.cpp M +4 -1 utilities/queuemanager/basetools/enhance/lensautofix.h M +4 -4 utilities/queuemanager/basetools/enhance/localcontrast.cpp M +4 -1 utilities/queuemanager/basetools/enhance/localcontrast.h M +8 -5 utilities/queuemanager/basetools/enhance/noisereduction.cpp M +4 -1 utilities/queuemanager/basetools/enhance/noisereduction.h M +4 -2 utilities/queuemanager/basetools/enhance/restoration.cpp M +4 -1 utilities/queuemanager/basetools/enhance/restoration.h M +4 -4 utilities/queuemanager/basetools/enhance/sharpen.cpp M +4 -1 utilities/queuemanager/basetools/enhance/sharpen.h M +10 -6 utilities/queuemanager/basetools/filters/colorfx.cpp M +4 -0 utilities/queuemanager/basetools/filters/colorfx.h M +42 -38 utilities/queuemanager/basetools/filters/filmgrain.cpp M +4 -0 utilities/queuemanager/basetools/filters/filmgrain.h M +9 -4 utilities/queuemanager/basetools/metadata/assigntemplate.cpp M +4 -0 utilities/queuemanager/basetools/metadata/assigntemplate.h M +13 -9 utilities/queuemanager/basetools/metadata/removemetadata.cpp M +4 -0 utilities/queuemanager/basetools/metadata/removemetadata.h M +10 -5 utilities/queuemanager/basetools/transform/flip.cpp M +4 -0 utilities/queuemanager/basetools/transform/flip.h M +16 -10 utilities/queuemanager/basetools/transform/resize.cpp M +4 -0 utilities/queuemanager/basetools/transform/resize.h M +11 -5 utilities/queuemanager/basetools/transform/rotate.cpp M +4 -0 utilities/queuemanager/basetools/transform/rotate.h M +0 -3 utilities/queuemanager/main/queuemgrwindow.cpp M +15 -4 utilities/queuemanager/manager/actionthread.cpp M +6 -7 utilities/queuemanager/manager/actionthread.h M +31 -49 utilities/queuemanager/manager/batchtool.cpp M +35 -104 utilities/queuemanager/manager/batchtool.h M +45 -88 utilities/queuemanager/manager/batchtoolsmanager.cpp M +2 -3 utilities/queuemanager/manager/batchtoolsmanager.h C +34 -21 utilities/queuemanager/manager/batchtoolutils.cpp [from: utilities/queuemanager/manager/batchtoolsmanager.h - 056% similarity] A +98 -0 utilities/queuemanager/manager/batchtoolutils.h [License: GPL (v2+)] M +0 -2 utilities/queuemanager/views/assignedlist.cpp http://commits.kde.org/digikam/987d78c787849ba8f8dfab408afd83a7172f0162