Bug 444160

How many faces did you learn per person?

Maik

Obviously it varies some have 100+ faces, and some only have a few samples.  However, I still have 20,000+ unmatched faces.  Even though they may not match a known person they should be auto categorized into groups of different unknown faces.

I agree with Jonathan. Instead of listing unknown and unconfirmed faces as an endless list of photos, similar faces should be grouped like e.g. Picasa. In digikam there is one "album" which contains unknown faces while in Picasa there are one album for each unknown face.
One example can be seen here: https://spekxvision.files.wordpress.com/2016/05/052316_0930_facialrecog1.png

Are there any progress on this issue and especially what is mentioned in this comment "Even though they may not match a known person they should be auto categorized into groups of different unknown faces." Grouping unknowns by face would make the training a whole lot easier.

Git commit 5258bccf8eca4ae28ae5075438c1a85bf895d4c5 by Maik Qualmann.
Committed on 02/06/2022 at 17:03.
Pushed by mqualmann into branch 'qt5-maintenance'.

fix wrong accuracy value in face recognition from maintenance dialog
Related: bug 436544, bug 432537, bug 431797, bug 436727

M  +1    -1    core/utilities/facemanagement/database/facescansettings.cpp
M  +3    -1    core/utilities/maintenance/maintenancemngr.cpp

https://invent.kde.org/graphics/digikam/commit/5258bccf8eca4ae28ae5075438c1a85bf895d4c5

@Jonathan

digiKam 8.0.0 is out. This entry still valid with this release ?

Best regards

Gilles Caulier

@Jonathan,


This problem still reproducible with the new digiKam 8.2.0 pre-release Windows
installer available at usual place:

https://files.kde.org/digikam/

This new bundle is based on last Qt framework 5.15.11 and KDE framework 5.110.

Thanks in advance

Gilles Caulier

I have the same problem.

The suggestions of face recognition output many wrong results. Around 20-40% of all suggestions are right. This makes the manual rework quite tedious (the benefit of using a face recognition evaporates). Digikam suggests faces, which are completely different from each other. For example a young boy gets assigned to the same person like the ones from a 80 yo lady. It makes no sense at all. On the other side images which are completely similar to already labelled ones, are not suggested, but are put in the group "unknown". So yeah, I can confirm, this recognition is guess work. Face detection works quite good (it sometimes detects objects as people, but the failure rate is here is acceptable), but face recognition malperforms.

Some additions:
- Deleting all databases and rebuilding them did not help
- Retraining the model (maintenance/...) did not help
- Changing the accuracy slider from the default 70% to 90% did not help (maybe it changed the quantity, but not quality).

I use digikam 8.1.0.

Sidenote: I noted that if I did labeled just a few pictures (~10) for a person, then the suggestions are way better but also much less. So if I label 10 pictures to a person X, the algortihm returns lets sa 5-20 pictures, which are actually that person. But for a person Y, if I label > 100 more pictures, the suggestion become really bad. Isn't it the opposite, that AI get better, the more you give?
You might say: Then I gave many bad labelled images to person Y. But no, the pictures for person Y are in a good quality, similar facial expression, so the variance is not too high.

Created attachment 162319 [details]
Wrong image recognition

An example: If I go on the suggestions for myself as a person in digikam. Around 50-100 different people are suggested there, several flowers,  a dog, a package of burger king and dumbledore is assigned to me.

In case of the flowers and the package of burger king: Ok the image detection failed. But why are they assigned to me? Its ok to have them in the category "unknown", but they have nothing to do with me (considering the accuracy is set to 90%).

Moreover, the 50-100 people look totally different than me. Nevertheless, they all tag-suggested as me.

Yeah there is something really wrong with the face recognition, it is simply not working as it should - not exactly a hard thing to do any more either, the tech has been out for a very long time.  So not only are we met with 4 days or more of waiting to detect faces, then we're met with bad results.  It's quicker to do them manually, even for 90,000 images and that's embarrassing.

I suspect, clicking minus (not this person) and tick (yes this person), is not actually training the model as it should be.  That's what it 'feels' like to me, but could be wrong.  This is still happening on 8.3.0.

I have access to Windows, Linux and Mac and various drives and an Nvidia GPU if anyone wants me to help in any way.

On my wish list, we would get GPU acceleration working first, which seems counter intuitive if it gives bad results, but the reasoning is we don't have to wait days and redo it all the time just to find out it didn't work.

Then after GPU we would focus on quality face matching problems.  Unless of course it's a simple as 'oh the tick and minus aren't actually training it', then yeah, probably pays to do that.

I also have a large number of scanned negatives that are 300MB tiffs at 4800dpi.  Obviously they're slow.  The faces do detect though.  But perhaps that screws up the recognition system, I don't know.

Sadly, in my relatively new, but fairly exhausting usage (exhausting as in time spent this weekend), face detection really isn't a working feature.  Hopefully we can get it fixed.

I can confirm this behaviour with the 04/02/2024 10:33 8.3.0 snapshot on Windows 11. Face *detection* across ~13,000 photos works well. However, after I manually identify ~14 new people from Unknown and assign faces to them (at least 5 faces for 8 of the people), the recognition step proceeds to incorrectly recognise hundreds of faces as just one person. The vast majority of these are completely different people that don't resemble the assigned person at all. I have not seen it ever assign faces to any other person.

Not to pile on, but I agree with all this.  The misidentifies are really outrageously not even close oftentimes (most of the time).  I have tried every setting and variation I can think of and nothing seems to affect the accuracy.  I have a large number of faces, too, and I have read that for some reason a large number of faces can affect things negatively, but I agree, the more faces I give digiKam, the better it should get.  

I understand that the documentation tells you the accuracy can take a hit if you have low-quality pictures or faces with sunglasses and so on, but if I've identified 100 pictures of a person, and 80 of the pictures are pretty good and 10 are fantastic, and the rest are bad, low resolution or wearing sunglasses or whatever, the program ought to be able to examine at all the pictures that I've identified as a certain person and compare all of them and figure for the ones that it doesn't see as similar that this must be an odd or bad or for whatever reason not a picture representative of the person and ignore it.  The user shouldn't have to not identify a person as a person just because their face is partially obscured, and the program should certainly be able to recognize a low-pixel photo and ignore the identification if it's not helpful.

I also agree that the program should group unknown people that you haven't identified yet.  The program should be able to tell that a group of photos of the same person are most likely all the same person without me having to tell it what the person's name is  If the program were to give matching but unknown faces a generic name like Unknown Person 001 or something, then I could just go in and delete or ignore or deselect the misidentifies and just rename the tag for all the rest, rather than me having to (possibly) sort through all my unknowns and find enough examples of this person before I can hope for the rest of them to be identified.

Not coming down on you guys, honestly!  I love this program and the innumerable things it does really really well!  Just face recognition for  some reason isn't one of them.  Yet!

Hi Michael,

Another entry which can be fixed with the new SFace DNN model used with the recognition workflow.

Best

Gilles Caulier

Thanks Gilles,
Yes, the accuracy rate of the new system in 8.5.0 is at least an order of magnitude better than before.

Jonathan, the new face detection system (YuNet) is up to 30 times faster than YOLO.  The new recognition model (SFace) is at least 10x more accurate than the model used before (OpenFace).

Depending on the settings selected by you and the speed of your computer, the new system could process 40,000 images in about 6 hours.

I just got a hold of digikam 8.5 october 8th build.
I cleared the existing matches and it rescan, and it still isn't good.

My nephew has 20,000+ unconfirmed faces which don't match in the slightest.
I turned the accuracy setting up significantly from the default.
I am waiting for the scan to complete, but it is taking way longer than the 6 hours suggested.  Although, digikam claims I have more like 120,000 faces.

There are long periods of time where the GUI lags horribly even though I had a 16 thread cpu, and lots of ram.
1,700 and counting wild misses on about 60,000 photos.(like car tires)
My C drive is nvme, the software drive is SATA SSD, and a different SATA SSD for the database.
Searching name lags
building thumbnails really lags
Scrolling while its building thumb nails even more lag.
When transitioning from one screen to another significant lag.
Double clicking a photo thumbnail to open the photo sometimes lags significantly.
I don't understand how google picasa can run without lagging and digikam can't.

Hi Jonathan,
Can you please tell us what settings you’re using? Which face detection model, detection accuracy %, face size setting,  face recognition model, and face recognition accuracy %?

Cheers,
Mike

My face recognition is at 70
OpenFace

YuNet
Medium

Hi Jonathan,
First, please make sure you have the latest build from yesterday.  

Try these settings:
Detection accuracy: 60%
Model: YuNet
Face Size: Large
Recognition Accuracy: 60%
Model: SFace
Uncheck Work on all processor cores

The last setting is important to reduce lag.  Processing will be faster if it's checked, but digiKam will try to use all the CPU it can, which will cause lag in the UI.  If you want to run a scan while you're away from the computer then checking Work on all processor cores is good.

We're doing our best to tune the models, and your help is greatly appreciated.  Let us know how well digiKam did at detecting and recognizing the faces.

Cheers,
Mike

(In reply to Michael Miller from comment #18)
> Hi Jonathan,
> First, please make sure you have the latest build from yesterday.  
> 
> Try these settings:
> Detection accuracy: 60%
> Model: YuNet
> Face Size: Large
> Recognition Accuracy: 60%
> Model: SFace
> Uncheck Work on all processor cores
> 
> The last setting is important to reduce lag.  Processing will be faster if
> it's checked, but digiKam will try to use all the CPU it can, which will
> cause lag in the UI.  If you want to run a scan while you're away from the
> computer then checking Work on all processor cores is good.
> 
> We're doing our best to tune the models, and your help is greatly
> appreciated.  Let us know how well digiKam did at detecting and recognizing
> the faces.
> 
> Cheers,
> Mike

If there is anything we all can do to help with the training part (as I know that is quite painful), happy to do so with our local gear.  In my case 4090. Or perhaps we need to build up an online a/b user training system that we can all jump into and help. Topic for somewhere else I'm sure but here I'm sure it will get at least a few of the right eyes.

Another thing that is a problem is I will be clicking the check mark to indicate that the name is correct, and suddenly without warning the whole display will scroll up or down any where to 5-100 rows.  This is completely disorientating, and lows productivity.  If I click ignore it doesn't seem to have this behavior.

I did the clear faces of unconfirmed, and I am scanning at the reccommed settings.  The scan has been running over night, and is now at 37% after more that 8 hours of running.

At the current settings the number of wrong faces to face matches is too high.

Even set to 100% matching for face recognition it couldn't even match the faces correctly.  There are so many not even close matches it is funny.

Hi Jonathan,
Let's start from the beginning.

You should be using YuNet to find the faces, 65% accuracy, and medium or large face size.

After the faces are found, you should be using SFace to recognize (match) the faces.  Because you have an existing face database, it's extremely important that you retrain the face matching model.  This is done in Tools->Maintenance->Detect and recognize faces->Clear and rebuild all training data.

After the training has been rebuilt, you can run "Recognize faces".  I recommend about 60% accuracy.

Did you retrain the face model before you did recognition, but after you changed the recognition model to SFace?

Cheers,
Mike

The process is running to clear the training data ...please stand by

Hi Jonathan,
When was the last time you downloaded digiKam?  There was an issue in a build from last week.  The issue was even if you selected SFace in the settings, it would still use OpenFace.  Please download the latest version from https://files.kde.org/digikam/ before trying again.

Cheers,
Mike

Has something specifically changed to make face detection / matching work properly now?  Because if there is nothing specific that can be pointed to, I don't see how downloading new versions is going to fix this problem.  Way back when I tried someone said the same thing, (try the latest beta / version), it didn't work then either.  I have been watching this thread in case it's ever fixed, but haven't heard anything yet.

Has there been new code / models / logic that would justify a new version as a solution?  Of course we all download the latest version anyway, but just worried that it would misrepresent itself as a fix.

I'm not even sure if anyone has actually accepted there is even a problem at this point, but there are threads that show a lot of people are experiencing it.

Hi Jonathan,
Yes, many, many things have changed.  First, we've introduced 2 new models in 8.5.0.  YuNet is now the preferred face detection model.  It is much faster than YOLO and SSD.  Second, we have a new face feature extraction model called SFace.  SFace is faster and more accurate than OpenFace for extracting the face vectors from the thumbnail (which should be generated by YuNet).  Next, we've significantly modified the KNN classifier to reduce both false positives and false negatives. While the new classifier is slightly slower, the performance enhancements of the detector and extractor make the overall process much faster and more accurate. Finally, there are several other minor changes in the face pipeline to improve performance.

I've been working on the models every day for several weeks now, and updates are introduced in the daily builds.  I've been tuning the models against the well-known Labeled Faces in the Wild dataset.  I have a good baseline to use for measuring my results.  Results are better every day.
https://vis-www.cs.umass.edu/lfw/

With YuNet, SFace, and the updated classifier, I'm getting detection accuracy scores of about 91% to 97%, and recognition scores of 92% to 96%.  The scores will vary based on the settings, which are driven by your tolerance for false-positives and false-negatives.

Overall, the new models are showing much better t-SNE clusters when data dimensionality is reduced so it can be plotted in 2-dimensional space.  This is a significant improvement over the aging SDD or YOLO detection models and OpenFace feature extractor with pseudo KDTree classifier where there was marginal clustering at best.

The most important pieces are:
1. you are using YuNet for face detection
2. you are using SFace for recognition
3. You have re-trained the face DB after changing those settings

Cheers,
Mike

I am currently in the process of a rescan, but due to quantity it is likely that it will take another 16 hours.
Then I will do another training purge
then I will do another round of face recognition
I have a build date of 10-14-2024, if that is not new enough let me know ASAP before I waste another 24 hours scanning.

Hi Jonathan,
Please download a new build.  I made some updates to the models yesterday.

Cheers,
Mike

Also, to speed things up, try using "Extra Large" as the face size settings.  "Large" is good, too.  "Large" will be the default starting tomorrow.

Cheers,
Mike

(In reply to Michael Miller from comment #28)
> Hi Jonathan,
> Yes, many, many things have changed.  First, we've introduced 2 new models
> in 8.5.0.  YuNet is now the preferred face detection model.  It is much
> faster than YOLO and SSD.  Second, we have a new face feature extraction
> model called SFace.  SFace is faster and more accurate than OpenFace for
> extracting the face vectors from the thumbnail (which should be generated by
> YuNet).  Next, we've significantly modified the KNN classifier to reduce
> both false positives and false negatives. While the new classifier is
> slightly slower, the performance enhancements of the detector and extractor
> make the overall process much faster and more accurate. Finally, there are
> several other minor changes in the face pipeline to improve performance.
> 
> I've been working on the models every day for several weeks now, and updates
> are introduced in the daily builds.  I've been tuning the models against the
> well-known Labeled Faces in the Wild dataset.  I have a good baseline to use
> for measuring my results.  Results are better every day.
> https://vis-www.cs.umass.edu/lfw/
> 
> With YuNet, SFace, and the updated classifier, I'm getting detection
> accuracy scores of about 91% to 97%, and recognition scores of 92% to 96%. 
> The scores will vary based on the settings, which are driven by your
> tolerance for false-positives and false-negatives.
> 
> Overall, the new models are showing much better t-SNE clusters when data
> dimensionality is reduced so it can be plotted in 2-dimensional space.  This
> is a significant improvement over the aging SDD or YOLO detection models and
> OpenFace feature extractor with pseudo KDTree classifier where there was
> marginal clustering at best.
> 
> The most important pieces are:
> 1. you are using YuNet for face detection
> 2. you are using SFace for recognition
> 3. You have re-trained the face DB after changing those settings
> 
> Cheers,
> Mike

That sounds really good.  I have a huge collection, do you want me to help you test / build anything?

(In reply to atoms from comment #32)

> That sounds really good.  I have a huge collection, do you want me to help
> you test / build anything?

Hi Atoms,
Yes!  Tuning the models isn't an exact science.  While the numbers are solid against a single dataset or two (LWF and my library of 23k+ images and 18k+ faces), every library and machine is different.  The more feedback we get the better we can deliver good software.

Download the latest build from https://files.kde.org/digikam/.  Steps to follow for converting to the new models.
1. Start digiKam.  Make sure you download the new models when prompted.  They are not packaged with the bundle.
2. Go to the people sidebar and change the settings at the bottom.  To start, use these settings:
Detection Accuracy: 60% (7 starting in a day or two.  I'm updating the UI)
Model: YuNet
Face Size: Large
Recognition Accuracy: 60% (7 starting in a day or two.  I'm updating the UI)
Model: SFace

When you change from OpenFace to SFace for recognition you should get prompted to retrain your database.  Click OK and let it run.  It may take several minutes or more to complete.

3. Check if your computer is OpenCL capable by going to "Help->Components Information->OpenCV Configuration->OpenCL availability" (optional, but will make it faster)
4. If your computer is OpenCL capable, go to Preferences/Configuration->Miscellaneous->System and make sure "Disable hardware acceleration OpenCL" is unchecked. (optional, but will make it faster)
5. Run face recognition and detection and let us know the results!

Cheers,
Mike

Hi Jonathan,
There is a bug in the code I just discovered.  Face detection is ok.  Face recognition is not working right now.  I'll let you know as soon as I fix the bug.

Cheers,
Mike

(In reply to Michael Miller from comment #33)
> (In reply to atoms from comment #32)
> 
> > That sounds really good.  I have a huge collection, do you want me to help
> > you test / build anything?
> 
> Hi Atoms,
> Yes!  Tuning the models isn't an exact science.  While the numbers are solid
> against a single dataset or two (LWF and my library of 23k+ images and 18k+
> faces), every library and machine is different.  The more feedback we get
> the better we can deliver good software.
> 
> Download the latest build from https://files.kde.org/digikam/.  Steps to
> follow for converting to the new models.
> 1. Start digiKam.  Make sure you download the new models when prompted. 
> They are not packaged with the bundle.
> 2. Go to the people sidebar and change the settings at the bottom.  To
> start, use these settings:
> Detection Accuracy: 60% (7 starting in a day or two.  I'm updating the UI)
> Model: YuNet
> Face Size: Large
> Recognition Accuracy: 60% (7 starting in a day or two.  I'm updating the UI)
> Model: SFace
> 
> When you change from OpenFace to SFace for recognition you should get
> prompted to retrain your database.  Click OK and let it run.  It may take
> several minutes or more to complete.
> 
> 3. Check if your computer is OpenCL capable by going to "Help->Components
> Information->OpenCV Configuration->OpenCL availability" (optional, but will
> make it faster)
> 4. If your computer is OpenCL capable, go to
> Preferences/Configuration->Miscellaneous->System and make sure "Disable
> hardware acceleration OpenCL" is unchecked. (optional, but will make it
> faster)
> 5. Run face recognition and detection and let us know the results!
> 
> Cheers,
> Mike

OK, what platform do you prefer, I can do M1 Pro Mac, Linux probably any version but currently running opensuse or windows if I must :D

I'm using an M1 MacBook Pro, so we have that covered.  Linux would be good, especially if it has an NVidia CUDA capable GPU. 

It's really whatever you want.  The code is all the same for each platform.

Cheers,
Mike

Cool, yeah there is a 4090 in the linux machine.

I only have a AMD CPU so CUDA is useless to me.

(In reply to Jonathan from comment #38)
> I only have a AMD CPU so CUDA is useless to me.

Hi Jonanathan,
That's ok.  AMD can use OpenCL which will still allow for GPU offloading and speedup.   Follow these steps to make sure it's enabled if possible.
1. Go to Preferences/Configuration->Miscellaneous->System and make sure "Disable hardware acceleration OpenCL" is unchecked. (optional, but will make it faster)
2. Restart digiKam
3. Check if your computer is OpenCL capable by going to "Help->Components Information->OpenCV Configuration->OpenCL availability" (optional, but will make it faster)
4. If your computer is NOT OpenCL capable, go to Preferences/Configuration->Miscellaneous->System and make sure "Disable hardware acceleration OpenCL" is checked to turn off calls to OpenCL.  Restart digiKam if you changed this setting.

Cheers,
Mike

There is a project to provide CUDA on non-Nvidia GPUs.

https://github.com/vosen/ZLUDA

Maik

Look also this comment :

https://bugs.kde.org/show_bug.cgi?id=495014#c3

Typically, to be able to use opencl with opencv in digiKam, Mesa package need to be installed.

Gilles Caulier

(In reply to caulier.gilles from comment #41)
> Look also this comment :
> 
> https://bugs.kde.org/show_bug.cgi?id=495014#c3
> 
> Typically, to be able to use opencl with opencv in digiKam, Mesa package
> need to be installed.
> 
> Gilles Caulier

Most major Windows video drivers should include OpenCL.  If not, I would first see if there are updated drivers from the manufacturer.  If not, you can try this.
https://apps.microsoft.com/detail/9nqpsl29bfff?hl=en-US&gl=US

Cheers,
Mike

I believe I have OpenCL but even though its on, it doesn't seem to help as just face detection alone took around 16 hours.
I have a 7900xtx, and if it was working it should have been done ages ago.

(In reply to Jonathan from comment #43)
> I believe I have OpenCL but even though its on, it doesn't seem to help as
> just face detection alone took around 16 hours.
> I have a 7900xtx, and if it was working it should have been done ages ago.

16 hours is a very long time, even for 40,000 images and not using GPU acceleration.  I have about 24,000 in my library (mostly .cr2 or .arw raw), and scanning everything takes about 2.5 hours. 

What kind of storage are you using? SSD built-in?  USB hard disk?  Network drive?

Also, I checked in the changes to fix the recognition bug.  You should be able to download a version with the fix in a day or two.

Cheers,
Mike

So I installed the newest version from today, and cleared the training data.
The recognition seems to be better.  The false positives seemed to drop significantly.
Digikam doesn't seem to group similar faces if an existing name hasn't been setup.

Can digikam see and read the google picasa face tags?  I'm not sure it does because I should have more matches.

Unfortunately I still have 99,000 unmatched faces.  Many are generic crowd faces, but I'm sure there are plenty of faces that could be matched.  It's going to take awhile to review them.

What are the possibilities of have muiltiple names per face.  For example William shatner is also James T Kirk

(In reply to Jonathan from comment #45)
> So I installed the newest version from today, and cleared the training data.
> The recognition seems to be better.  The false positives seemed to drop
> significantly.
> Digikam doesn't seem to group similar faces if an existing name hasn't been
> setup.
> 
> Can digikam see and read the google picasa face tags?  I'm not sure it does
> because I should have more matches.
> 
> Unfortunately I still have 99,000 unmatched faces.  Many are generic crowd
> faces, but I'm sure there are plenty of faces that could be matched.  It's
> going to take awhile to review them.

Hi Jonathan,
I'm happy face matching is working better for you.  Because of the additive nature of the system, you should run recognition after you approve new face suggestions.  digikam should find more faces that match.  If you keep the recognition accuracy above 7 you shouldn't get any false-positives, or at least very, very few.  This should help sort through the faces you know.

Digikam groups the face thumbnails based on the folder and image the face was found it.  With the new recognition models, you should only need to manually tag or two faces with a name and digikam will find the rest.  Keep in mind you may have to run recognition a few times, approving the suggestions after each time to match all the faces.

I'll check about using the Picasa tags.  It will really depend on the "box" they Google uses for the face, and how easy it will be to translate that to the digikam box.  I probably won't have a chance to look into it until this weekend at the earliest.

Unfortunately, we can't use 2 names for the same tag.  That would require a very significant rewrite of huge portions of the tagging and face recognition engine.  You could always add a second tag to image.  Maybe tag the image with "Star Trek".  Then you can search for all "Star Trek" images that also have "William Shatner". 

Cheers,
Mike

Hi Jonathan,
How is working for you now?  Please let us know the results, and if you think we can close this issue.

Cheers,
Mike

Hi Jonathan,
Any feedback for us?  How is it working?  How can we improve?

Cheers,
Mike

Git commit c003f95327d1239378241226140f9098b989e52e by Maik Qualmann, on behalf of Michael Miller.
Committed on 02/11/2024 at 22:56.
Pushed by mqualmann into branch 'master'.

closing tickets
Related: bug 469329, bug 431797, bug 415782, bug 472031, bug 464266, bug 423113, bug 436544
FIXED IN: 8.5.0

M  +8    -8    NEWS

https://invent.kde.org/graphics/digikam/-/commit/c003f95327d1239378241226140f9098b989e52e