Bug 415782 - Face recognition "favors" people with few images: missing faces clustering
Summary: Face recognition "favors" people with few images: missing faces clustering
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Faces-Recognition (show other bugs)
Version: 7.0.0
Platform: Other Linux
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-01-02 00:49 UTC by Daniel
Modified: 2021-04-19 15:03 UTC (History)
14 users (show)

See Also:
Latest Commit:
Version Fixed In: 7.2.0


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel 2020-01-02 00:49:50 UTC
SUMMARY
Currently, the new face recogniotion seems to favor people which are on few pictures heavily to people who are on more pictures. I know this is not really the case, but this what I observe.

For example, I have some persons with only 1-5 pictures, other with 2000+. When the recognition is run, the people with fewer pictures appear very more often as suggestions in the Unconfirmed section for pictures of the person (with 2000+ pictures).


Tested on the current master branch.
Comment 1 Maik Qualmann 2020-01-02 09:37:55 UTC
The "old" training dates are no longer valid. If you have access to the current git/master version, I would suggest building the training data again by activating them in the face settings.

Maik
Comment 2 Daniel 2020-01-02 12:10:45 UTC
(In reply to Maik Qualmann from comment #1)
> The "old" training dates are no longer valid. If you have access to the
> current git/master version, I would suggest building the training data again
> by activating them in the face settings.
> 
> Maik

This is actually what I did. I'm using the git version until commit 

https://invent.kde.org/kde/digikam/commit/7fe8bce623b5ce9be55f19c502999858f652ca9b
Comment 3 morty 2020-01-04 19:38:30 UTC
I can confirm this for 7.0.0-bata1. I have a folder with pictures mainly of our children (which I used to play with the face recognition). While recognition worked quite good in the beginning, it got worse and worse the more faces I tagged. Eventually the algorithm started only tagging faces with people I only tagged once or twice, or not detecting them at all. Although I taggeed faces of a series of very similar pictures.
Is it possible that once I have too many faces of one person that the training gets to fuzzy or something? 
I'm no expert in this, but Picasa did some kind of clustering. Thus when you tagged one picture it suddenly assigend a bunch of other pictures to that person, too.
Comment 4 Daniel 2020-01-04 22:15:55 UTC
(In reply to morty from comment #3)
Did you add some face tags manually when people are shown from behind?
Do the faces change much over time? (My collection spans 20 years for example)

Both is the case for me.
Comment 5 morty 2020-01-05 12:11:54 UTC
(In reply to daniel-other+kdebug from comment #4)
> (In reply to morty from comment #3)
> Did you add some face tags manually when people are shown from behind?
No, only faces detected by DigiKam.

> Do the faces change much over time? (My collection spans 20 years for
> example)
Kind of. Children's faces do change quite a bit in the range of 3 to 6. ;-)
Comment 6 caulier.gilles 2020-01-05 12:21:11 UTC
> (In reply to morty from comment #3)
> Did you add some face tags manually when people are shown from behind?

>>No, only faces detected by DigiKam.

Well do it. How the deep learning can assign auto a name to a face if you don't teach a little bit the neural network.

In Deep Learning the is "to learn". It like at school, you need to teach something before to do something.

Gilles Caulier
Comment 7 morty 2020-01-06 18:26:09 UTC
(Getting OT)

(In reply to caulier.gilles from comment #6)
> > (In reply to morty from comment #3)
> > Did you add some face tags manually when people are shown from behind?
> 
> >>No, only faces detected by DigiKam.
> 
> Well do it. How the deep learning can assign auto a name to a face if you
> don't teach a little bit the neural network.
> 
> In Deep Learning the is "to learn". It like at school, you need to teach
> something before to do something.

Yes.... What does this have to do with marking people seen from behind? I don't expect DigiKam to recognize people based on their heir. ;) Also face _detection_ works fine. The _recognition_ fails the more faces are marked and it thus has more images available to learn from....
Comment 8 Maik Qualmann 2020-01-06 18:44:50 UTC
The number of faces saved in the DB is already limited to the last 20. As a test, we can limit it to 10 or 5.

Maik
Comment 9 morty 2020-01-06 18:55:40 UTC
(In reply to Maik Qualmann from comment #8)
> The number of faces saved in the DB is already limited to the last 20. As a
> test, we can limit it to 10 or 5.

I'm afraid I'm missing some context here. Which 20 faces are saved in the DB and how does that effect the results? 

OT: I updated to Jan 4 2020 and the GUI improved a lot.
Comment 10 Daniel 2020-01-06 18:57:05 UTC
(In reply to Maik Qualmann from comment #8)
> The number of faces saved in the DB is already limited to the last 20. As a
> test, we can limit it to 10 or 5.
> 
> Maik

Maybe there should be at least like 10 images of a person until it is added to the face recognition database? Also it might be useful to make the "last 20" setting you describe available to the user.
Comment 11 Maik Qualmann 2020-01-06 19:01:32 UTC
We are currently storing a person's last 20 confirmed face matrices. Saving all of them would enlarge the database enormously and digiKam would always slow down. We have had in the past because users had thousands of faces from just one person.

Maik
Comment 12 morty 2020-01-06 19:29:49 UTC
(In reply to Maik Qualmann from comment #11)
> We are currently storing a person's last 20 confirmed face matrices. Saving
> all of them would enlarge the database enormously and digiKam would always
> slow down. We have had in the past because users had thousands of faces from
> just one person.


Ok I see. So the quality of the detection depends on the last 20 pictures confirmed? Thus after I confirmed current faces, it unlikely that the next run will recognize the face on an older picture?
If got things right, the face recognition date base currently gets rebuilt with each run, right?
I know, the interface should be kept simple, but maybe understanding why pictures are detected or not could improve by providing more options like how many faces-matrices are used and how they are selected (the last, randomly, folder, time-span, etc). Maybe even being able to show the face-matrices used and being able to deselect them. 
Also, as most people typically have a select number of people with a high likelyness of being on a photo (Family, Fiends) it might be an option to give those haing been tagged often a little boost. Or to spend some extra processing time to better train for this group.

P.s: I have no face-recognition skills; Thus please indulge my ignorance if these suggestions don't make sense. ;)
Comment 13 MarcP 2020-01-08 13:04:26 UTC
I have also noticed that the precision of face recognition did not improve if a person had more pictures. Until that moment, I had tried the face recognition algorithm using a dataset of 85 famous people (http://vis-www.cs.umass.edu/lfw/), mostly portraits centered on their face, looking at the camera, with samples from 16 to 300 pictures each. In that case, tagging about 12 to 40 pictures of each person I achieved around a 80% of true positives (set at an accuracy of 70%).

However, testing the algorithm in my personal database, including around 1500 different people, with some of them having thousands of faces, I had a very different experience. While the face detection works very well (it even detects babies!), recognition rates were below 1% of true positives, where results seem to be completely random.

Maybe the number of people in the collection could be weighted in towards the recognition? Or try to predict which people are in a picture based on similar pictures of the same album or date? Or maybe an advanced option to consider a larger number of faces for each person?
Comment 14 MarcP 2020-01-08 13:09:27 UTC
Or what about preventing recognizing faces for people with a very low sample? (e.g. fewer than 15 or 20 faces)
Comment 15 Maik Qualmann 2020-01-08 19:36:08 UTC
Git commit 03aee05e64d3a2003984943a7a33ea692a9161a3 by Maik Qualmann.
Committed on 08/01/2020 at 19:33.
Pushed by mqualmann into branch 'master'.

leave the last 100 face matrices in the db for a test

M  +2    -2    core/libs/facesengine/facedb/facedb_dnn.cpp

https://invent.kde.org/kde/digikam/commit/03aee05e64d3a2003984943a7a33ea692a9161a3
Comment 16 Maik Qualmann 2020-01-09 17:48:50 UTC
Git commit 9d60f152e77b5201ce6bf23473684a32444bf586 by Maik Qualmann.
Committed on 09/01/2020 at 17:48.
Pushed by mqualmann into branch 'master'.

fix wrong item selection for face recognition
Related: bug 416018, bug 416028, bug 415867, bug 415895

M  +3    -1    NEWS
M  +0    -29   core/utilities/facemanagement/widgets/facescanwidget.cpp
M  +15   -0    core/utilities/maintenance/facesdetector.cpp

https://invent.kde.org/kde/digikam/commit/9d60f152e77b5201ce6bf23473684a32444bf586
Comment 17 Daniel 2020-01-10 23:58:21 UTC
> recognition rates were below 1% of true positives, where results seem to be completely random.
I can confirm that (It recognized about 15 of 600 recognized faces correctly) (correction rate was worse IMHO before changes in 03aee05e64d3a2003984943a7a33ea692a9161a3)
Comment 18 TG 2020-01-23 17:05:15 UTC
(In reply to daniel-other+kdebug from comment #17)
> > recognition rates were below 1% of true positives, where results seem to be completely random.
> I can confirm that (It recognized about 15 of 600 recognized faces
> correctly) (correction rate was worse IMHO before changes in
> 03aee05e64d3a2003984943a7a33ea692a9161a3)

Confirmed here also
testing digikam-7.0.0-beta2-20200121T060840-x86-64 right now
Comment 19 caulier.gilles 2020-01-25 18:02:43 UTC
Nota : Thanh while Google summer of code 2019 has let's a link to a possible clustering method annoted here :

https://invent.kde.org/kde/digikam/blob/master/core/libs/facesengine/TODO#L5

Gilles Caulier
Comment 20 caulier.gilles 2020-01-26 10:30:23 UTC
Nota2 : Thanh while Google summer of code 2019 has let's another instructions for computation improvements here :

https://invent.kde.org/kde/digikam/blob/master/core/utilities/facemanagement/TODO.FACE#L17

Gilles Caulier
Comment 21 Christian 2021-01-18 21:19:49 UTC
(In reply to MarcP from comment #13)
> I have also noticed that the precision of face recognition did not improve
> if a person had more pictures. Until that moment, I had tried the face
> recognition algorithm using a dataset of 85 famous people
> (http://vis-www.cs.umass.edu/lfw/), mostly portraits centered on their face,
> looking at the camera, with samples from 16 to 300 pictures each. In that
> case, tagging about 12 to 40 pictures of each person I achieved around a 80%
> of true positives (set at an accuracy of 70%).
> 
> However, testing the algorithm in my personal database, including around
> 1500 different people, with some of them having thousands of faces, I had a
> very different experience. While the face detection works very well (it even
> detects babies!), recognition rates were below 1% of true positives, where
> results seem to be completely random.
> 
> Maybe the number of people in the collection could be weighted in towards
> the recognition? Or try to predict which people are in a picture based on
> similar pictures of the same album or date? Or maybe an advanced option to
> consider a larger number of faces for each person?

Hey, have the very same experience: recognition rate below 1% with random results, proposed persons pre-dominantly those ones I have only one picture for in the training data - and never the ones with >100 different pictures. Has there been a solution meanwhile or a parameter somewhere, where I could at least restrict automated recognition for persons with >10 pictures?
Comment 22 Maik Qualmann 2021-01-18 21:52:04 UTC
As already said in your bug 431797 report, use the latest version that is very easy to use under Windows, as we provide current snapshots. The face recognition works very well, I estimate the error rate here at < 5%. It is required that digiKam detected the faces. Existing faces from other programs (Picasa) or faces drawn manually are problematic because the face is usually larger. We will try to improve this as part of the GSoC 2021. With a current digikam, it is important to rebuild the training database.

Maik