Bug 426309

Nghia,

Look the topic of this entry, this is what we talk together last winter about the capability to download data model on demand.

Supporting Yolo model is a good candidate to be hosted somewhere in digikam.org (for ex), and users can use this model instead SSDMobilenet for a better detection.

digiKam will upload data file from Internet, store the file somewhere in user space, and use it.

Gilles

I also experience poor face detection when the faces in the picture are smaller (typical of group pictures with many people in them).  

Is there currently any way to manually download and use other models?  The source repository contains the large yolo model files.  Could those just be downloaded by the user or are code changes required to utilize them?

A code change is currently necessary.

Maik

(In reply to caulier.gilles from comment #1)
> Nghia,
> 
> Look the topic of this entry, this is what we talk together last winter
> about the capability to download data model on demand.
> 
> Supporting Yolo model is a good candidate to be hosted somewhere in
> digikam.org (for ex), and users can use this model instead SSDMobilenet for
> a better detection.
> 
> digiKam will upload data file from Internet, store the file somewhere in
> user space, and use it.
> 
> Gilles

I will look into it. 

Nghia

Hi all,

Say that we can support customized face detection backend. How do you intend to use it after importing the data files?

For example, how would you like to configure it and the DNN would be another pre-trained model, or do you want to train the model yourself.

Could you describe to me more about the use-case scenario of the configuration to make to install a customized model?

Nghia

(In reply to Minh Nghia Duong from comment #5)
> Hi all,
> 
> Say that we can support customized face detection backend. How do you intend
> to use it after importing the data files?
> 
> For example, how would you like to configure it and the DNN would be another
> pre-trained model, or do you want to train the model yourself.
> 
> Could you describe to me more about the use-case scenario of the
> configuration to make to install a customized model?
> 
> Nghia

I think I would use pre-trained models.  As time goes on, the ability for the digikam devs to add additional detection methods could be handy.  YOLOv3 works quite well.  Maybe in a few months, someone comes out with YOLOv4 and it's a better detector.  It would be nice to open up prefs, and select the new face detection model, and have the required training data downloaded and installed if it doesn't already exist.

(In reply to Rob D from comment #6)

Hi Rob,

I agree with your concern. Normally, for the current CNN models of object detection, the pre-processing and post-processing are nearly the same between different neural network. The only significant differences are the backend type for network importation and the input size of the network. 

The input size seems easy to configure in user interface but the backend type supported by OpenCV is limited and therefore needs better specialization to avoid errors.

About YOLOv4 and v5, it's already out and I tried it on digikam. It requires the latest version of OpenCV and doesn't seem to make a significant change in performance on CPU. Might be it would work better on GPU but the only GPU driver supported by OpenCV is CUDA.

Can you describe to me more about the workflow of loading a customized neural network that you have in mind?

Nghia

(In reply to Minh Nghia Duong from comment #7)
> (In reply to Rob D from comment #6)

I was thinking of maybe having an additional selector for choose an available detection model, with a button that would bring up a dialogue do download additional models (or to re-download a existing model in case of issues)

I would expect it to display current download status and verify a checksum to ensure file integrity.

My concert would be the existing face detection settings box is too small to fit the additional items, and moving these settings back into the main digikam settings window would negate all recent interface changes.

Rob

Just wondering if you'd had any further thoughts on this topic.  Given that the limitations of the current face detection model are basically preventing me from using the software, I'd be happy to test a branch build (Windows) when one is ready.

(In reply to princeanthony.tiago from comment #9)
> Just wondering if you'd had any further thoughts on this topic.  Given that
> the limitations of the current face detection model are basically preventing
> me from using the software, I'd be happy to test a branch build (Windows)
> when one is ready.

Yes, this feature is definately on my TODO list. However, my schedule now is a little stacked but I will works on this feature in the following weaks. I hope to release it with version 7.3. 

If you have any suggestion about the workflow, how it should look or function, I'm all ear.

Nghia

Hi all,

About different detection models, could you give me the link to some models that you're having in mind? I would like to dig into the different models to see their implementation with opencv in order to prepare for this new module.

Thank you in advance.

Nghia

At the moment the "yolov3-wider_16000.weights" is of interest to us. This is already included in the digiKam source code. It is not installed because the file is commented out in the CMakeList.txt. In the source code you only have to switch to YOLO (change one line). Other models available on the web recognize objects or do not currently work with digiKam. The YOLO-V3 is ideal for detection faces, but is also very slow. There is a smaller version of the YOLO-V3 on the web, but I couldn't get good results with it, so I'm looking for the download again.

Maik

(In reply to Maik Qualmann from comment #12)
> At the moment the "yolov3-wider_16000.weights" is of interest to us. This is
> already included in the digiKam source code. It is not installed because the
> file is commented out in the CMakeList.txt. In the source code you only have
> to switch to YOLO (change one line). Other models available on the web
> recognize objects or do not currently work with digiKam. The YOLO-V3 is
> ideal for detection faces, but is also very slow. There is a smaller version
> of the YOLO-V3 on the web, but I couldn't get good results with it, so I'm
> looking for the download again.
> 
> Maik

I will agree with this model.  

I think putting the infastructure in place to add this one model will allow additional models to be added more easily as they become available in the future.

It might be good to have a little status indicator to indicate if a GPU is detected and available for OpenCV to make use of.

Hi Nghia, Hi Maik,

thank you for the great work and enhancements of v7.2.0-beta1.

Are there any news on this topic? In case you decided to implement this feature, is there a schedule available?

Really looking forward to an improvement of the face detection (#420128)

Git commit fc52e19cc3b50b981da7a7d80e62380cf09a4582 by Maik Qualmann.
Committed on 14/11/2020 at 23:29.
Pushed by mqualmann into branch 'master'.

first step to add a face model downloader
Class is still not used yet.

M  +2    -0    core/utilities/facemanagement/CMakeLists.txt
A  +268  -0    core/utilities/facemanagement/downloader/facemodeldownloader.cpp     [License: GPL (v2+)]
A  +76   -0    core/utilities/facemanagement/downloader/facemodeldownloader.h     [License: GPL (v2+)]

https://invent.kde.org/graphics/digikam/commit/fc52e19cc3b50b981da7a7d80e62380cf09a4582

Git commit 1519c40809cd3c6f20c63f73efea874cd39895cd by Maik Qualmann.
Committed on 16/11/2020 at 07:04.
Pushed by mqualmann into branch 'master'.

activate the file downloader for testing
The large binary files contained in the
source code are no longer installed.
Later, if there are no problems, we
can remove them from the sources.

M  +2    -2    core/app/main/main.cpp
M  +1    -1    core/data/CMakeLists.txt
M  +28   -35   core/utilities/setup/downloader/filesdownloader.cpp

https://invent.kde.org/graphics/digikam/commit/1519c40809cd3c6f20c63f73efea874cd39895cd

Created attachment 133385 [details]
attachment-13167-0.html

I uncommented the yolo lines in the code and the proper files also
downloaded.  Re-checked face detection and all worked great!  

Off to a good start!  Now to just add the detection method selector and
life is good!

Rob


On Mon, 2020-11-16 at 07:06 +0000, Maik Qualmann wrote:
> https://bugs.kde.org/show_bug.cgi?id=426309
> 
> --- Comment #16 from Maik Qualmann <metzpinguin@gmail.com> ---
> Git commit 1519c40809cd3c6f20c63f73efea874cd39895cd by Maik Qualmann.
> Committed on 16/11/2020 at 07:04.
> Pushed by mqualmann into branch 'master'.
> 
> activate the file downloader for testing
> The large binary files contained in the
> source code are no longer installed.
> Later, if there are no problems, we
> can remove them from the sources.
> 
> M  +2    -2    core/app/main/main.cpp
> M  +1    -1    core/data/CMakeLists.txt
> M  +28   -35   core/utilities/setup/downloader/filesdownloader.cpp
> 
> https://invent.kde.org/graphics/digikam/commit/1519c40809cd3c6f20c63f73efea874cd39895cd
>

Git commit 32333a2719bfc0d97d631a84889b56d3f5a9a7ad by Maik Qualmann.
Committed on 16/11/2020 at 21:32.
Pushed by mqualmann into branch 'master'.

add checkbox for YOLO v3 face detection
Related: bug 420128, bug 415559
FIXED-IN: 7.2.0

M  +1    -0    core/app/views/stack/itemiconview_search.cpp
M  +1    -0    core/libs/album/treeview/albumselectiontreeview.cpp
M  +9    -1    core/libs/facesengine/detection/facedetector.cpp
M  +2    -0    core/libs/settings/applicationsettings.cpp
M  +3    -0    core/libs/settings/applicationsettings.h
M  +10   -0    core/libs/settings/applicationsettings_miscs.cpp
M  +3    -0    core/libs/settings/applicationsettings_p.cpp
M  +2    -0    core/libs/settings/applicationsettings_p.h
M  +1    -0    core/utilities/facemanagement/database/facescansettings.cpp
M  +3    -0    core/utilities/facemanagement/database/facescansettings.h
M  +8    -8    core/utilities/facemanagement/threads/facepipeline.cpp
M  +1    -1    core/utilities/facemanagement/threads/facepipeline.h
M  +1    -1    core/utilities/facemanagement/threads/facepipeline_p.h
M  +26   -0    core/utilities/facemanagement/widgets/facescanwidget.cpp
M  +2    -0    core/utilities/facemanagement/widgets/facescanwidget_p.h
M  +2    -1    core/utilities/facemanagement/workers/detectionworker.cpp
M  +1    -1    core/utilities/facemanagement/workers/detectionworker.h
M  +1    -1    core/utilities/facemanagement/workers/recognitionworker.cpp
M  +1    -1    core/utilities/facemanagement/workers/recognitionworker.h
M  +4    -2    core/utilities/maintenance/facesdetector.cpp
M  +1    -2    core/utilities/setup/downloader/filesdownloader.cpp

https://invent.kde.org/graphics/digikam/commit/32333a2719bfc0d97d631a84889b56d3f5a9a7ad

Created attachment 133387 [details]
attachment-22162-0.html

Things look and work great, Maik

Thanks!

On Mon, 2020-11-16 at 21:34 +0000, Maik Qualmann wrote:
> https://bugs.kde.org/show_bug.cgi?id=426309
> 
> Maik Qualmann <metzpinguin@gmail.com> changed:
> 
>            What    |Removed                     |Added
> ---------------------------------------------------------------------
> -------
>          Resolution|---                         |FIXED
>       Latest Commit|                           
> |https://invent.kde.org/grap
>                    |                           
> |hics/digikam/commit/32333a2
>                    |                           
> |719bfc0d97d631a84889b56d3f5
>                    |                            |a9a7ad
>    Version Fixed In|                            |7.2.0
>              Status|REPORTED                    |RESOLVED
> 
> --- Comment #18 from Maik Qualmann <metzpinguin@gmail.com> ---
> Git commit 32333a2719bfc0d97d631a84889b56d3f5a9a7ad by Maik Qualmann.
> Committed on 16/11/2020 at 21:32.
> Pushed by mqualmann into branch 'master'.
> 
> add checkbox for YOLO v3 face detection
> Related: bug 420128, bug 415559
> FIXED-IN: 7.2.0
> 
> M  +1    -0    core/app/views/stack/itemiconview_search.cpp
> M  +1    -0    core/libs/album/treeview/albumselectiontreeview.cpp
> M  +9    -1    core/libs/facesengine/detection/facedetector.cpp
> M  +2    -0    core/libs/settings/applicationsettings.cpp
> M  +3    -0    core/libs/settings/applicationsettings.h
> M  +10   -0    core/libs/settings/applicationsettings_miscs.cpp
> M  +3    -0    core/libs/settings/applicationsettings_p.cpp
> M  +2    -0    core/libs/settings/applicationsettings_p.h
> M  +1    -0   
> core/utilities/facemanagement/database/facescansettings.cpp
> M  +3    -0   
> core/utilities/facemanagement/database/facescansettings.h
> M  +8    -8    core/utilities/facemanagement/threads/facepipeline.cpp
> M  +1    -1    core/utilities/facemanagement/threads/facepipeline.h
> M  +1    -1    core/utilities/facemanagement/threads/facepipeline_p.h
> M  +26   -0   
> core/utilities/facemanagement/widgets/facescanwidget.cpp
> M  +2    -0   
> core/utilities/facemanagement/widgets/facescanwidget_p.h
> M  +2    -1   
> core/utilities/facemanagement/workers/detectionworker.cpp
> M  +1    -1   
> core/utilities/facemanagement/workers/detectionworker.h
> M  +1    -1   
> core/utilities/facemanagement/workers/recognitionworker.cpp
> M  +1    -1   
> core/utilities/facemanagement/workers/recognitionworker.h
> M  +4    -2    core/utilities/maintenance/facesdetector.cpp
> M  +1    -2    core/utilities/setup/downloader/filesdownloader.cpp
> 
> https://invent.kde.org/graphics/digikam/commit/32333a2719bfc0d97d631a84889b56d3f5a9a7ad
>

Many thanks to the fast response and the fast implementation. Incredible! I tested the YOLO model with 7.2.0 beta (build 22-Nov-2020). What an improvement in number of faces detected.

One remark. I assume you don't plan to make the YOLO model default (increase of processing/detection time). So my proposal would be to move the question to download the YOLO model, as soon as you enable the YOLO model and not at application start up. You do it the same way with other functions like panorama stitching, to ask the user for external binaries, as soon as you use the function

I think the one-time download at start is fine. At the moment other necessary data files are still contained in the bundle. But we have already removed them from the sources. This was also a wish of the KDE admins, as our source archive has grown too big. These are files for the standard model or for red eye detection. This means that the downloader will be reactivated soon. The downloader checks offline whether the required files are available and whether the file size is correct. These files are required when digiKam is started and are loaded into the memory. Therefore the download at the beginning to avoid unnecessary restart later.

Maik