Bug 146288 - Face detection / recognition for digikam
Summary: Face detection / recognition for digikam
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Faces-Workflow (show other bugs)
Version: 1.3.0
Platform: Gentoo Packages Linux
: NOR wishlist
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-06-02 14:49 UTC by Christian Weiske
Modified: 2022-01-22 22:17 UTC (History)
39 users (show)

See Also:
Latest Commit:
Version Fixed In: 2.0.0


Attachments
people tag widget screenshot. (700.64 KB, image/png)
2010-01-09 23:09 UTC, caulier.gilles
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Christian Weiske 2007-06-02 14:49:33 UTC
Version:            (using KDE KDE 3.5.5)
Installed from:    Gentoo Packages

Reaching my personal 10.000 photos barrier in the next weeks, I face a problem:
Tagging photos by hand is not doable anymore since it takes too much time. While setting tags for a whole album (france, wedding, 2007) is easy and fast to do, tagging each of the 200+ photos in an album with the people on it is impossible.

When searching for photos, you often try to find a person - how it looked at a time, or generally need all photos of a certain person. Tags help here, but not all photos are tagged through the reason above. What I would like to see is a plugin that automatically detects faces and allows one to drag a tag from the tag tree on, or just add a tag to the detected face.

There are some face detection and recognition libraries out there that could be used, one well-known is the Intel's OpenCV lib [1]. libFace [2] is another one that specializes in face rec/det and uses intel's lib internally.

Gnome's F-Spot will get this feature trough the summer of code project [3, 4]. If it gets that, it'll be my personal killer feature to use fspot.

It would be very cool if digkam would get such a feature. And by the way, thanks for making digikam. My girlfriend and I use it since years, and it's been unvaluable for managing our photo collection.


References:
1 http://sourceforge.net/projects/opencvlibrary
2 http://libface.sourceforge.net/
3 http://idea.zanestate.edu/archives/2007/04/face-recognition-coming-to-f-spot/
4 http://bugzilla.gnome.org/show_bug.cgi?id=324391

Mailing list thread:
http://mail.kde.org/pipermail/digikam-users/2007-June/003642.html
Comment 1 Stephan Olbrich 2007-06-03 12:55:15 UTC
*** This bug has been confirmed by popular vote. ***
Comment 2 caulier.gilles 2007-06-03 13:37:44 UTC
Hey,  i have never seen a wish confirmed after 2 days than it have been open...

Marcel, what do you think about Face detection. For me it sound a killer feature...

Gilles Caulier
Comment 3 Marcel Wiesweg 2007-06-03 18:38:38 UTC
Yes it sounds very interesting.
We would need to find out what the workflow is (training of the face recognition?), and how well the library works.
Then it's about writing an easy to use wrapper class for the library.
And finally integrating it to digikam, perhaps in the image search or in some yet to be written tagging workflow?
Comment 4 caulier.gilles 2007-06-03 18:50:00 UTC
Marcel,

... or during downloading from camera...

Gilles
Comment 5 Arnd Baecker 2007-06-03 20:44:14 UTC
Well, this will be a long road until success:
I had a quick look at installing opencv (which fortunately exists
in debian etch as version 0.9.7-4).
Then libface needs some tweaking in the header files so that opencv is found,
but compiles fine after that. (I can post more details on that if needed).
Then one has to look in the examples directory of libface-0.1,
and compile
constructmat.cpp  facedetect.cpp  recognize.cpp  train.cpp
However, it is not clear from the (essentially lacking, unless
I missed something) documentation how to use these:
Presumably it is train followed by recognize or facedetect.
For train one has to supply a file which contains names and 
images of faces to be trained.
So far I only managed to get a segmentation fault...
(note to self: don't use my own pictures, but those of models,
the code seems to be picky ... ;-)

Then there is ellip-mask.jpg which provides a mask, maybe(?) used
for the input images. Whether the algorithm is able to detect
the region of interest with a face to be recognized is not clear to me.
Presumably all this will require some more digging in the original opencv
documentation.

Also note that the libface project does not appear that active, 
after an initial check-in 6 weeks ago nothing happened...

OK, I will stop at this point, maybe someone else has more luck ...
Comment 6 caulier.gilles 2007-06-03 20:54:21 UTC
Arnd, 

I'm not surprised by this feedback (:=))). I know very well the world of experimental algorithms study (i'm work in a French scientific research center)

With this experience, i'm always aware about it. It's always an instable world witch need a long time to be stabilized.

Also, if the project of this shared library is not really active, we need to wait is something will be done during Google Summer Code to improve this situation.

My viewpoint is this : Face Detection is an interressing feature of course, but we have more important task to do before KDE4/Qt4 or DB update port for ex. (:=)))

So we wait and see...

Gilles

Comment 7 Arnd Baecker 2007-06-03 21:17:42 UTC
> So we wait and see...


Yes, there is a lot more concrete stuff...
Apart from the technical side of the KDE4 transition,
there are also non-trivial workflow aspects
to be addressed wrt tagging (e.g
http://bugs.kde.org/show_bug.cgi?id=114465)
and improving searching/tag-filtering ...
Comment 8 Christian Weiske 2007-06-03 21:25:43 UTC
What I understood so far about opencv:
- There are two things: Face detection and face recognition. The first is to actually detect the position and region of a face, or all the faces on the photo. The second is to recognize the face at the detected region and say who it is. This needs a training set.

The simple / first thing I think should be implemented is face detection and face outline display in digikam. The user OR a program would say where the the faces are (e.g. user could select a frame as done for selection now), and digikam would store this information somewhere in the database. Further, a tag shall be associatable to that selected region (that would not only allow faces, but other details to be selected - as flickr allows it). This way, you can use a person name/tag and tag photo regions.

The second is face recognition. With the things implemented in the first point, it should be possible to extract the faces/regions from already tagged photos and train the face recognitioner. Untagged regions could then be sent to the recognition engine/script and be tagged automatically after.


So in short: Digikam first needs a way to tag/describe regions of an image. I think rectangles should be fine, no need to make it complicated and support circles or even AND OR and whatnot. After this, even outside scripts can be used to facedetect and recognition.
Comment 9 Arnd Baecker 2007-06-03 21:44:04 UTC
My impression is that first the algorithmic side of both steps
- face detection
- face recognition
has to be working first (well, it did not for me in the first try...),
before the integration into digikam can be done.
All can be tested in detail separately
and this should then also clarify the needs on the digikam side.

Your suggestion to describe regions of an image and
associate those with a tag is indeed something of use
independent of the face detection.
This should have many applications (--> separate B.K.O entry?)

Best, Arnd
Comment 10 Christian Weiske 2007-06-03 21:59:55 UTC
Arndt, I don't think so. OpenCV already provides a working algorithm set, so digikam needs the infrastructure to use the results of that algorithms - that is, mark a region of a photo as "face", and to say that this region/face is someone - tag it. Only then detection/recog are useful. You don't need fuel until your car has wheels.
Comment 11 Arnd Baecker 2007-06-03 22:27:48 UTC
So obviously I missed this then - there is a fully working code based on OpenCV
to which you supply images (eg. just containing the face part) to train
and then do the recognition for new images? Do you have a pointer?
But what is the role of libface then? Is it needed? Did you manage to 
get it working?
Comment 12 Christian Weiske 2007-06-03 22:53:14 UTC
Arnd, it's my fault. OpenCV is a lib that can be used if one wants to do face detect/recognition, and libface would be the one that could be used by digikam. But if it crashes regularly, it doesn't help. Are there other libs out there that could be used, or even a program via cmdline?
Comment 13 Christian Weiske 2007-06-03 23:04:25 UTC
http://openbio.sourceforge.net/ is pretty old, but it has the cmdline programs
Comment 14 Colin Guthrie 2007-06-04 01:38:36 UTC
In Reply to Comment #9:
> Your suggestion to describe regions of an image and
> associate those with a tag is indeed something of use
> independent of the face detection.
> This should have many applications (--> separate B.K.O entry?)

Yes this would be very good. I think a new BKO is needed. From my point of view this would help for my Sync plugin (yeah I know I've made 0 progress lately!) when integrating with e.g. Facebook - Tags of regions to represent your friends etc. I'd imagine an extension of the current tags concept to store always a region (defaults to the whole image) of an item. Smaller regions can be defined that represent e.g. faces or other special notes of interest. 

I can see how Face Detection could automate the assignment of these people tags.

I wonder however if such a capability could be included as a kipi plugin? What more would the API need to do to enable this. We are always discussing when we'll break the API anyway so perhaps the requirements of face detection should help mould the API changes?

Col
Comment 15 Christian Weiske 2007-06-04 08:16:54 UTC
I opened a new bug #146337 for region tagging
Comment 16 Colin Guthrie 2007-06-04 12:47:31 UTC
To clarify: when I said:
> I wonder however if such a capability could be included as
> a kipi plugin? What more would the API need to do to enable
> this. We are always discussing when we'll break the API anyway
> so perhaps the requirements of face detection should help mould
> the API changes?

I was referring to the face detection stuff, rather than the region tagging stuff. Should be obvious if you think about it, but wanted to clarify :)
Comment 17 Christian Weiske 2007-09-26 19:04:44 UTC
Two interesting links for face recognition:
http://www.iis.fraunhofer.de/bf/bv/kognitiv/biom/dd.jsp
(Fraunhofer Institut provides a demo application to detect faces. Perhaps someone could talk to them and convince them to open-source the algorithm? Page is in German)

http://www.riya.com
Is a visual search engine where you can upload your pictures, unfortunately they do not allow/provide offline use.
Comment 18 Christian Weiske 2007-11-06 17:35:20 UTC
It seems the f-spot code is ready:
http://apart-dev.blogspot.com/
Comment 19 caulier.gilles 2007-11-06 18:18:30 UTC
ok, face detection is fun... 

Christian, we will be busy on KDE4 for few months (XMP support + new Database), but i'm interressed to take a look later about this code...

So i will be happy to have more users feedback about this code especially statistics to detect the same face on a collection of picture (to apply a tag automaticly)...

Thanks in advance

Gilles
Comment 20 Arnd Baecker 2008-06-20 15:00:20 UTC
Just to add two more links about face-recognition (discussed in the German 
magazine c't 13, pp 168, 2008):
http://ivt.sourceforge.net/
http://www.face-rec.org/

Ivt  is a GPL library which employs a USB webcam (etc.)   
for face detection and face recognition.
Internally it uses OpenCV, however I don't know whether anything
additional is contained which might be useful for our situation with still images. 
Comment 21 caulier.gilles 2008-07-07 09:35:52 UTC
Arnd,

Andi, know OpenCV library. he develop a new kipi-plugin to fix automatically red-eyes. He can give more info about if necessary.

Gilles
Comment 22 Michael Skiba 2008-07-18 21:51:40 UTC
Here's a collection of additional links to programs which deals with this topics - note however that this page is dated 2005. But maybe there's something valuable..
http://pages.cpsc.ucalgary.ca/~hanlen/vision/facelinks.html
Comment 23 caulier.gilles 2008-12-06 15:25:17 UTC
This file is definitively for KDE4

Gilles Caulier
Comment 24 Julien Narboux 2009-01-06 19:55:14 UTC
A page which explains how to use opencv for face detection: http://opencv.willowgarage.com/wiki/FaceDetection
Comment 25 Andi Clemens 2009-01-06 20:09:58 UTC
Face Detection is not that hard, and there are ready-to-use classifiers as well in the OpenCV package.
But unfortunately those classifiers are only working very precisely when using for frontal faces. If the head is turned to the left or right or if hair hides some parts of the face, detection fails.

The much harder part is face recognition and I don't know how to make it learn the classifier that fast. As I trained the classifier for eyes (RemoveRedEyes plugin), it took two weeks to extract the features. The classifier isn't even that good, I might need to train another.

I really don't know how Picasa is going to do this.

Andi
Comment 26 Andi Clemens 2009-01-06 20:12:05 UTC
(In reply to comment #25)
> I really don't know how Picasa is going to do this.

iPhoto I mean...
Comment 27 Stibbons 2009-01-07 13:41:37 UTC
I do agree this could be a great addition to digikam software.
Currently, iPhoto and Picasa (web) are doing this.

- Go or you picasa web album and click on "Person" tab.
- all faces are detected (I suspect a script is run periodically to update the cache).
- you access to a serie of batch of photo of the same person and you just enter his/her name.

I would suggest, for this feature to be as killing as possible:
- First, perform a face detection on each photo in the entier library or a smaller part. Then, sort and display first the photos on the person on the most shots. This is psychologically important and have a strong impact since the user will think "whow! it is able to find all the photo with my mum, whoaa!!!". There is also lot of single photo (faces too different), but displayed last it have less impact and this can be entered latter
- of course if the scan already have been performed, the result should be cached/saved somewhere so we don't have to perform a new scan each time.
- In each batch of photo, ask to enter his/her name and tag the photo accordingly (please, use metadata tag written in the file itself, accordingly to a rule like "Persons/Name of the person")
- When adding photo, automatically performs face detection and display the most relevant name, or allow to enter his/her name
Comment 28 Otso Helenius 2009-01-07 13:49:47 UTC
This is a real killer feature and I urge including it into digikam.
Comment 29 Alexander Balzer 2009-01-07 15:14:08 UTC
Fspot has it, Picasa has it already, iPhoto has it now.
When digikam is last in implementing users will switch.
I think this is a feature people would switch for!
Comment 30 Stephane List 2009-01-28 16:05:07 UTC
Hi,

You can find an example of howto integrate opencv mechanism into a QT application on my new webpage :
http://slist.lilotux.net/linux/qmotion/

I'm currently working on a similar project for face detection and face recognition.

Don't hesitate to get in touch with me for more informations.

Stephane
Comment 31 Andi Clemens 2009-01-28 16:48:23 UTC
We are already using OpenCV in one of the KIPI plugins.
Face detection in general should not be the problem, but recognition seems to be hard.
What do you plan to solve this problem?

If you don't want to post here, you can sent my a private mail. I used OpenCV for my diploma thesis, it's been a while but I think at the moment I'm one of the devs with the most experience in OpenCV.

Andi
Comment 32 caulier.gilles 2009-01-30 18:39:59 UTC
To all,

This is my vision for Faces auto tagging...

If you use Facebook, you can mark people face over images. this is simple and great. I think it's the first step to do: added a small to tool to mark face manually in digiKam. For this stage, i know somebody in Mandriva who have work on this subject...

The second step, is to implement a tool to use current faces database done manually to be abble to find automatically similar faces over the whole collection of image. This can be done also during image downloading from camera...

For this second step, OpenCv can be used to parse image contents and detect face. Of course we need another algorthim somewhere to create faces fingerprints and compare faces one by one. For me it's sond similar than Haar stuff, but using more precision and details.

Perhaps, OpenCv provide something like this. Here i don't know.

What do you think about ?

Gilles
Comment 33 Alexander Balzer 2009-03-13 13:40:03 UTC
Maybe somebody can translate this, I found it quite interesting.
http://www.heise.de/tr/Gesichtserkennung-Clever-oder-unheimlich--/artikel/134269/0/0

The topic face detection/recognition is very popular at the moment.
Comment 34 Sherwood Botsford 2009-04-05 18:39:47 UTC
Could this be implemented as a KIPI plugin, thus making it useful for more than just digikam?

For the recognition part, it would help if an album could be given a group of people.  If I'm at a family reunion, then I've got a different set of faces than when I take my grade 10 class on a canoe trip.  Have a more restricted set of faces will increase the accuracy of the recognition software.

Peopletags is another case of album specific tagging -- I don't want to see my inlaws in the tags on my canoe trip.  I don't want to see my grade 10's when working on my family tree.
Comment 35 Enrico Ros 2009-04-08 15:50:57 UTC
I saw this GSoC 2009 proposal:
http://socghop.appspot.com/document/show/user/hoheinzollern/digikam_face_rec

The proposal sounds well formed and the use case is really solid!
I hope this gets selected for the summer of code because Digikam will have a very distinctive feature and will get a lot of hype then! ;-)

Enrico Ros
Comment 36 caulier.gilles 2009-04-08 16:22:53 UTC
But who have posted this subject ? Andi, Marcel, Arnd, Mik ?

It's not me...

Gilles Caulier
Comment 37 Andi Clemens 2009-04-08 17:01:16 UTC
No, I have not posted anything...
Comment 38 Alessandro Bruni 2009-04-08 17:15:00 UTC
Hello, it's me.

I'm a student who made a proposal through the GSoC app. I tried to contact you - Gilles - by mail last week.
If you have any comment and want to mentor me let me know!

Alessandro Bruni
Comment 39 caulier.gilles 2009-04-08 23:08:51 UTC
Alessandro,

I can mentor you, but i'm not a specialist of OpenCV. Andi know very well.

Marcel can help you about all database questions.

I can guide you for the rest.

I'm very overloaded currently. I give evening classes for webmaster in my computer science school, and it take a while to prepare. This is why i'm late to manage my pending mail. Sorry.

Gilles Caulier
Comment 40 Alessandro Bruni 2009-04-08 23:26:23 UTC
Thank you for the answer Gilles :-)
I just wanted to get some feedback as we're getting to the end of the period of acceptance, so time is precious.
Good work and don't worry.

So! Andi, would you like to mentor me? ;-) If you want you should become a mentor in the socghop.appspot.com application and comment my proposal.

My regards,
Alessandro
Comment 41 Andi Clemens 2009-04-09 09:41:06 UTC
(In reply to comment #39) 
> I can mentor you, but i'm not a specialist of OpenCV. Andi know very well.

Gilles,

I'm not a specialist in OpenCV, too. I used it once (this is two years ago), but not very indeep. I can tell you how to use the built-in face detection, but you can read about this on a lot of webpages, too. Actually this is just 10 lines of code.

(In reply to comment #40)
> So! Andi, would you like to mentor me? ;-) If you want you should become a
> mentor in the socghop.appspot.com application and comment my proposal.

To be honest, I don't know if I am the right guy for this. I've been with digikam for 8 months now, but still have not done big things with the source code. I'm not so familiar with the code (without the indexer of Eclipse CDT I wouldn't find anything ;-)). I sure could try to assist you with OpenCV, but maybe someone else should be the mentor. By the way: even if you use OpenCV, you have to program the face recognition by yourself, there is no such feature in the library, at least I haven't seen one. So just relying on OpenCV doesn't help here :-) You must find your own ways to recognize faces.

Andi
Comment 42 Alessandro Bruni 2009-04-10 22:25:36 UTC
I come back again with some -fresh new- old info:
http://gsoc-wiki.osuosl.org/index.php/Finding_mentors
Here it's said that you can both mentor me: maybe Gilles should be the primary mentor for my project and Andi a more technically skilled mentor (for the openCV library).
If you agree we should notice it to the kde team now, as we're reaching the end of the interim period:
http://socghop.appspot.com/document/show/program/google/gsoc2009/timeline

Alessandro
Comment 43 Alessandro Bruni 2009-04-10 22:50:55 UTC
Also created a new section in kde wiki's GSoC page:

http://techbase.kde.org/Projects/Summer_of_Code/2009/Ideas#Project:_Face_Recognition
Comment 44 Andi Clemens 2009-04-12 21:50:15 UTC
(In reply to comment #42)
> Here it's said that you can both mentor me: maybe Gilles should be the primary
> mentor for my project and Andi a more technically skilled mentor (for the
> openCV library).

As I said before, I can try to help you with OpenCV, but I'm also no expert. But at least I own a book :-)
Comment 45 Alessandro Bruni 2009-04-12 22:23:46 UTC
(In reply to comment #44)
> As I said before, I can try to help you with OpenCV, but I'm also no expert.
> But at least I own a book :-)

Any knowledge of the library is preciuos and good for this project ;-)

That would be a starting point. But I think that the mentor should know better the architecture of the program, so how to act effectively to implement this feature in the best way. For what I saw theese days getting in contact to digikam's developers Gilles is the only one - afaik - who has such knowledge, so I would ask him to be my mentor. But if you, Gilles, think you cannot give me support theese months I would ask Andi to be my mentor. I also talked with Arnd theese days in IRC and I know he has a good knowledge of the project.

Help! I really don't know what to do and rely on your hands for the start of this project. I think this would be a great occasion for me and you, not to mention the end users! That's a killer feature imho!

Alessandro
Comment 46 Enrico Ros 2009-04-13 00:02:40 UTC
Is the project still on hold because of mentor lack? This can't happen! Me and my friends can't wait for the next summer of code to have this feature on Digikam ;-)

Is anybody here registered as a KDE mentor on the SoC 2009 webapp? I'd to vote for this project but I'm not a mentor.. so.. :-/
Comment 47 Marcel Wiesweg 2009-04-13 13:11:11 UTC
Well I think Gilles said he can mentor you in comment #39.
This is about the question who is formally listed as a mentor with Google.
I am not registered there, this whole thing passed me by once again while I was abroad the last weeks.

Gilles, Andi and me all have the deep codebase insight that is needed to help get you started, the one more in this area the other in that. You don't get more from anyone else ;-)

I have no problem helping you should you get accepted. (That is true for any new contributor but so few come up and ask code questions :-( )
Comment 48 Lydia Pintscher 2009-04-13 15:23:27 UTC
Mentor and project selection for KDE for GSoC 2009 has been completed already. Results will not be made public before April 20th. No point in discussion this here further.
Thanks for understanding.
Comment 49 caulier.gilles 2009-04-13 15:51:17 UTC
I would to be clear here about GOsS...

I don't like this. I don't understand what a GOsC mentor need to do exactly, ho to start... etc... And i don't want to read a huge collection of paper from google to only guide a student with Google rules. Sound like very complicated and unadapted to open source project !

To resume my viewpoint : It's a waste of time for me...

I know how to follow students in real job. I do it everyday. I'm also a teacher and i like to share informations, and see working progress in a project.

But seriously, GOsC add rules witch are definitively not adapted to open source project.

In open-source, why to add GOsC constraints ?  It's already difficult to follow standard contributors, check/review patches, guide all peoples who want to help. Time is precious and i don't want to lost time with GOsC, to follow students release dates, write papers, etc... with no guaranty of results...

And i have a real life outside this computer.

How i work : when time permit ! Look this entry for ex:

https://bugs.kde.org/show_bug.cgi?id=149485

Here a student, outside GOsC, works on new Liquid Rescale tool for digiKam image editor. And it's work very well. Progressively, the job is done on the right way. There is no advantage for me to use GOsC to follow this guy.

And i work on open-source since a very long time now. I have a very good experience on project management. so, i'm sorry, but GOsC is not an advantage for me...

Gilles Caulier.
Comment 50 Andi Clemens 2009-04-13 15:55:10 UTC
I guess the advantage is on the student's side: He can earn a lot of money with it... :-)

Andi
Comment 51 Dotan Cohen 2009-04-13 18:39:56 UTC
Can this bug be either cleaned up, or refiled? The whole GSoC discussion should have been on the mailing list, not cluttering up bugzilla. With Gilles and Andi's permission I would file a new bug which summarizes the useful comments in this bug with none of the 15 OT comments.
Comment 52 caulier.gilles 2009-06-19 12:53:36 UTC
*** Bug 194401 has been marked as a duplicate of this bug. ***
Comment 53 Claudio Noguera 2009-06-29 12:32:20 UTC
I definitely think this should be implemented. To me there are three steps for developing this:

1. Prepare the database so that it can save where the face is in a picture. It could also be extended to enable region tagging (like facebook or flickr have).
2. Implement face detection.
3. Implement face recognition.

The advantage of this approach is that each of the three steps will be seen as an improvement by the final user. Another advantage is that the whole face detection thing will be easier to implement if it can be divided in incremental steps.

An this is briefly how I think each of the steps could be solved:
1 --> add a "position" field to the imagetags table in the database, it would be empty if it is a global tag.
2 --> train a boosting model based on several haar features. Once trained, digikam should just calculate the haar features and apply the model. This is fast, and notice that no classifier training is done in the computer's machine. Picasa does this. The facedetect example in the openCV library uses too little features, that's why it doesn't work properly (there are too many false positives).
3 --> I think this is a clustering problem, but I have never tried to solve it. I guess it is an easier problem than face detection.

I have some experience in machine learning and computer vision, unfortunately no experience in C++ developing. Anyway, I'd like to help in getting this implemented.
Comment 54 Alex 2009-07-07 11:23:24 UTC
Are there any good classifiers out there that can be trained on a generic set, and then have good enough accuracy to perform well without any additional training and improvement? Surely the classifier has to be trained on the specific dataset of images first (users database, maybe not the whole, but a selected few images), and then continuasly improved by the user feedback as it is used. Mac's iPhoto has this feature and even with ongoing improvement it still gets the a lot of false positives, that are then weeded out by the user. This way it is not limited by the accuracy of any model that is trained elsewhere. 

An obvious case where classifier might perform poorly, is when faces are partialy ocluded by some object. It might recognise it as another person or not recognise it at all. In which case it has to be told "No this is not object A, it is object B". I have also seen that using a number of Haar features can still yield a non face recognised as a face by similarity. So the classifier needs to be told to ignore that somehow.

An additional comment is that, as I understood from description above, the database field caters for a position of a face, not multiple faces. May I suggest to use a list of positions where faces are and what they correspond to. So have a separate classifier table, that has internal representation of people feature set and internal id of a person, which can be then tagged to the face position and id of the person. 


E.g. <coordinates of the bounding box (top right and bottom left)>#id of the person: <125,125>,<175,175>#3

Admin question: has this been submited to the GSoC 2009, because I couldn't see it in the project list? Is anyone working on this? Otherwise I will throw my name in the hat to help out with this :)
Comment 55 caulier.gilles 2009-07-07 11:35:37 UTC
>Admin question: has this been submited to the GSoC 2009, because I couldn't see
>it in the project list? Is anyone working on this? Otherwise I will throw my
>name in the hat to help out with this :)

Nobody work on this project for the moment. submitting this subject as GSoC subject is complex and hard to follow. other restrictive subjects have been submitted instead.

Note : personally, I don't like GSoC concept. I don't like to see peoples payed to work on open-source, and followed by mentor not payed as well. 

Gilles Caulier
Comment 56 caulier.gilles 2009-07-07 11:43:03 UTC
Alex,

Feel free to work on this subject. 

In this room you will see 
- Marcel who work on all Database purpose. Talk with him to change DB schema if necessary and to improve DB interface.
- Andi who know OpenCV API and concept (Remove Red Eyes plugin author).

Gilles Caulier
Comment 57 Claudio Noguera 2009-07-07 12:39:14 UTC
I suggested a db modification in the mailing list that Marcel approved because it was backwards compatible. I think it is suitable for the classifier purpose.

It consists of two new tables: faces and imagefaces.
Faces --> faceid, tagid.
Imagefaces --> imageid, faceid, x1, y1, x2, y2

The faces table is used to match a face with a tag in the database. The other one should be directly written by the classifier. Once you have a trained classifier, it'll first will detect faces in an image. The second step is to assign a different faceid to each different face.

I'd really like to be a part of this, but currently I don't have time.

About the performance of the algorithms, don't be too optimistic. It will be very difficult to recognise ocluded faces. Reducing the number of false positives is easier.
Comment 58 Alex 2009-07-07 13:32:56 UTC
One of the features that can be used is a Gabor features. I worked with them little bit this year, and they are used quiet a bit in bioinformatics for feature selection. Maybe worth having a look.

K well lets try and get the first part of this done: Getting to face detection to work. This should be fairly straight forward. I'll just take some time to familiarise myself with the digikam code and framework. As a proof of concepr I suggest we should get the Haar face detection from OpenCV working, unless someone has better suggection. I guess that might already be used in red eyes removal, which I have a look shortly.
Comment 59 Andi Clemens 2009-07-07 13:47:08 UTC
Don't expect much from this plugin! :) It was my first code in C++, always coded Python before.
Also OpenCV is not really integrated here, and this is a KIPI-plugin, so if we really want to use OpenCV in digiKam, we might want to integrate it there, write a DImg::OpenCvLoader or similar and make it more abstract.

My plugin uses a classifier that I trained for single eyes. The face detection of OpenCV is not that good, it only finds faces that are not covered by other objects. Also the faces need to be straight, not in an angled position, otherwise it fails, too.
Comment 60 Alex 2009-07-07 13:48:28 UTC
I been snooping around the idea of the boot model. This seems like a really interesting approach. Here I found an implementation of the AdaBoost in C++.

http://codingplayground.blogspot.com/2009/03/adaboost-improve-your-performance.html
Comment 61 Andi Clemens 2009-07-07 13:55:23 UTC
AdaBoost is what OpenCV uses for learning classifiers. 
But it seems to be very limited, maybe an own implementation would be better here?

Unfortunately I'm not into machine learning that much, so if you have more experience, let the good ideas come :D
Comment 62 Claudio Noguera 2009-07-07 14:19:22 UTC
I tried myself the opencv facedetect example. I used the python version and made a little python script that applied the tag "person" to every picture with a face. It writes directly into the database, it's not connected with digikam at all.
Two things about this detector:
- Increase the minimum size of a face. I changed it to a minimum of 100x100. That way it takes about 0.6 seconds per picture (my pictures are on average about 7 MPx).
- A lot of false possitives.

Due to the amount of false positives, I think it is worth implementing our own face detection method. However, as a first step, it would be really nice to have this one as a base so we can build from it.

About the opencv approach. It is a cascade classifier built with a boosting technique on a set of haar features. The problem is that the set of haar features is too little, so we'll need a way to specify Haar features and calculate them. To my knowledge, openCV's predefined functions only calculate the reduced set of features used by its classifier.
There are other techniques to detect faces, however this one seems the most appropriate.
Comment 63 Alex 2009-07-07 14:24:26 UTC
Well from what I understand, AdaBoost is more like a framework to combine multiple weak classifiers. The idea being many weak classifiers, maybe be geared for different parts, like eyes, lips etc, and combination of them is better than a single strong classifier. For AdaBoost the problem becomes of selecting what weak classifiers to use e.g. Haar-like features, maybe something with wavelets, Gabor features, histograms etc.

I guess this is something to keep in mind when plug-in is at that stage.
Comment 64 Alex 2009-07-30 20:18:37 UTC
I have been thinking about this for a while, and working with OpenCV face detection seems to be quiet good, even if it not 100% accurate.

However, I am still struggling with a way to do face recognition. K a simple way, like using histogram is quiet easy, but what I have in mind is this:

The training of the classifier for the face recognition has to be done continually, e.i. the more it works the better it gets. Learns from itself. But I am struggling to find how that could be done. OpenCV has a haartraining and it would be nice if one could use it to update with the fresh set of the positive and negative data, that has been aquired in between last time the training was run and now.  Then use current classifier with this data to produce new one, a better version of the old one. However, I don;t know wheather that is possible in current OpenCV haartraining. 

Any other thought on how it could be done?
Comment 65 Dante Ashton 2009-07-31 04:15:16 UTC
Face detection algos are difficult to train. I'd love to have this functioning for KDM as well...

As far as it's development goes, we ARE talking about a trainable system. Might I advise it's 'crowd-sourced'? IE, everyone interested uses it, scans themselves in, then this could be sent back to the devs...it would take a weight off them.

In linking with the rest of the system, could we have it so it works accross mutiple accounts, and could even link to KAddressBook? That would be brilliant beyond doubt...
Comment 66 Alexander Balzer 2009-08-10 16:04:20 UTC
I found something which might be worth a look.

"The Kolmogorov project provides tools for machine perception and tutorials for machine learning, machine perception, and statistics. It supports fast face detection in images and information on Hidden Markov models, Gabor filters, Multivariate Logistic Regression, and Stochastic Differential Equations. It includes MPISearch, a library for detecting objects in images very quickly (real-time in some cases)."

http://freshmeat.net/projects/kolmogorov
Comment 67 Dotan Cohen 2009-08-10 16:28:06 UTC
> Might
> I advise it's 'crowd-sourced'? IE, everyone interested uses it, scans
> themselves in, then this could be sent back to the devs...it would take a
> weight off them.

I do not see how that would help, but I would gladly submit tagged photos to help.
Comment 68 Alexander Balzer 2009-08-12 14:09:29 UTC
Just found a similar request for kphotoalbum

https://bugs.kde.org/show_bug.cgi?id=170801
Comment 69 Alex 2009-08-12 22:45:12 UTC
http://freshmeat.net/projects/kolmogorov - Seems to have been moved somewhere.

I had an idea. What if we use the classifier that comes with OpenCV as a base, get the rectangle where the face is. Then use this much smaller image to identify the face. We can use those small images to train classifier in OpenCV. In effect current face detect classifier in OpenCV is as general as it can be, to recognise a generic face, so what we will be creating is a highly specific classifier tailored for each face in the training data, each face in our gallery.

Now, what is needed is to improve general OpenCV classifier locally, because for example, it won't recognise some faces which are altered in spacial domain, e.g. turned 90 deg left/right. I tested this on my machine with an image of Lena (popular test image), which it happely recognises in its original form, but rejects if rotated 90 deg. I had a look at the source code for the haartraining and it needs to be modified slightly because it creates a fresh classifier from scratch everytime, where as what we want is one that takes an existing classifier, then goes through the same motion to update it internally.


Advantages of using OpenCV, as a start, is because it already has lots of features in its bank. However, if that doesn't work well then we can think of an alternative way. I hope to get my hands on some image processing books and papers once I am back at university from holidays. :)
Comment 70 Julien Narboux 2009-09-15 09:34:06 UTC
Another option which could be considered to have facial recognition in Digikam could be to use the Polar Rose web service (http://www.polarrose.com/).

I have just noticed that there is a module to perform facial recognition using Polar Rose inside Gallery 3 open source gallery system (http://gallery.menalto.com/).

A discussion about this module and how it works is here :
http://gallery.menalto.com/node/86320

The module is here :
http://github.com/gallery/gallery3-contrib/tree/master/modules/polar_rose

It seems that the Polar Rose service can download a rss feed of pictures and return facial recognition information for it.

Of course this raises concerns because pictures have to be uploaded to a third party...

Julien
Comment 71 caulier.gilles 2009-09-15 09:42:03 UTC
If Polar Rose algorithm and method is open-source, perhaps we can back-port code to digiKam core as well and implement something around to interface with database. It will be faster an it will run off line.

Where is the code exactly ? Which language is used to implement Polar Rose tool ?
Do you know a paper which explain how it's work ? Do you a review of tool ? Do you know a comparison with others method ?

Gilles Caulier
Comment 72 Julien Narboux 2009-09-15 10:12:23 UTC
Sorry Gilles,

The Polar Tool is not open source, it is a closed web service.

Julien
Comment 73 Alexander Balzer 2009-09-23 16:18:52 UTC
The offline version of Picasa now also supports face detection and recognition.
Maybe it helps to get an idea how the gui could look like.

http://www.heise.de/newsticker/Picasa-erkennt-Gesichter-und-schreibt-Geotags--/meldung/145774
Comment 74 Michael G. Hansen 2009-11-12 19:14:37 UTC
SVN commit 1048077 by mghansen:

Document for discussing ideas for image annotation and face recognition.

CCBUG: 146288
CCBUG: 146337



 AM            ImageAnnotation.odt  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1048077
Comment 75 Sean Reifschneider 2009-11-12 19:26:18 UTC
You may want to point to OpenCV as an engine for doing face detection and
recognition:

   http://opencv.willowgarage.com/wiki/

Sean
Comment 76 Alexander Balzer 2009-12-10 10:44:37 UTC
Just look at the video. I think this is a very good approach of a face detection / recognition implementation:

http://video.golem.de/details/2413/picasa-3.5.html
Comment 77 Alex 2009-12-21 23:02:30 UTC
Hi there, 

I have been working on getting this to work for the project for the last few days.

There are 2 well established ways of doing it (or so it seems anyway). So I have found them on MATLAB website and now trying to port the code and framework in general to C++. So I thought I share how far I got.

I have went with Fisher's face framework first as it seemed more robust then Eigenface (less dependent on light, pose etc. but not perfect still). However, the problem that I have encountered is this:
    It needs QZ algorithm to solve generalised eigen value problem to work. MATLAB has an implementation of it, but I couldn't find an implementation of it for C++ or anything else other than maths software. It's complex and I don't want to spend time on it just yet if I don't have to.

Because of this I have abandoned this approach for Eigenface, at least to have one working no matter how good or bad it is. So I guess I'll get working on that in next few days and see how far i'll get, hopefully will report with more success next time :)


Alex
Comment 78 Alex 2009-12-27 17:22:12 UTC
Merry Christmas everyone.

I think this is an appropriate time to report some success in the project :)

I got v0.01 of my Eigenface Recognition to work and recognise different faces, not recognise others and so on. I have tried it on a very limited set of training data, it worked pretty fast, (there are plenty more optimisations to do anyways), but it works :D

I guess this is the first step on a long road of getting it into production and release. I am going to make a bigger training set in next couple of days (whilst digesting all the left over turkey), if it still works, then I will add more "user friendly" features and would like for others to try out, hopefully break it, so we can make it better. At the moment a lot of things like size of the images and so on are fixed and framework doesn't resize them yet, but all in good time.


Alex
Comment 79 caulier.gilles 2009-12-27 17:34:30 UTC
Great to read...

How do you plan to integrate it to digiKam ?

Do you want to make something internal in a first time ? I recommend it to hack indeep without to be disturbed with external depencies.

In the future, do you want to create an external library ? For exemple in kdegraphics/libs, like we do with libkdcraw/libkexiv2/libkipi 

How your framework can be used ?
Which data are generated ?
Do you have fingerprints to host in database ?
How do you link a face fingerprint with people name ? 
How many time your framework need to recognize a face in a image ?
Can you recognize more than one face at the same time for one image ? 

My tips : I recommend to create a dedicated Database file for that to not change current main digiKam DB schema and not bloat main digiKam database file

All best. 

Gilles Caulier
Comment 80 Alex 2009-12-27 18:54:13 UTC
Hi,

I designed it as an external library, where raw image data, or i guess library path can be passed referencing training data. The principle behind it i this:

 - Normalise training data
 - generate eigenvectors and values for a covariance matrix of the data
 - discard some values that are too low,
 - project training data onto 'facespace' (This is essentially PCA)

So in order to see where input image is, it reshapes it, normalises it and multiplies with pre calculated eigenfaces matrix. Then generates Euclidean distance to see where it is closest to. (This is subject to some values which need to be determined by trial and error, which is bit sucky). So there is no person fingerprint per se, as it is combined into signle matrix for all people, but each column in of the projected image matrix uniquely identifies a person. So all you need is table that matches person name with this ID. Then you need to store mean vector and eigenfaces as well as projected images. As you don't want to be computing them everytime. An interesting bit is how to update the matrix, I was thinking taking a mean image across all images for a person, but I don't know how good it is in practice. An additional feature in digikam itself can be to identify a face manually by drawing a box around it, then use it to add or check with face recognition. This is the nature of all classifier, that they won't pick up certain things (I just been using Apple's iPhoto and that's what they use as well). 

Face detection is fairly straight forward using haarfeatures in opencv. This classifier has been trained extensively and you can't really do better, without reinventing a wheel. So work flow looks something like this:

digikam library -> use haarfeatures to detect faces on image -> use face recognition on face identified by classifier -> output to digikam to confirm

Then update the Eigenfaces, mean vector and Projected matrix to be reused. Hopefully update makes it better as more data used.

The whole thing works with a single face as input for a person, but obviously more variation you have better it should be. 

With regards to databases, apart from having a table for linking coloumn in Projected matrix with a person, I don't see a need to store matrices in it. They are constantly reused and changed, plus they are going to be OpenCV classes, so serialising them backward and forward is unnecessary. I would propose to use a flat file in the background, this is hopefully, doesn't add any more complexity to digikam itself and the whole thing is self contained. 

With regards to face recognition, I don't know how familiar you are with haar classifier in opencv, but its good. It does recognise multiple faces at the same time, its not perfect, but nothing is. I think it has been trained on all open to access image databases. Alternatively you can train your own classifier using the same principle, but there is no easy, straight forward way to update it. I have previously toyed with idea of having a cascade classifier for each person, but there pretty much is no way of updating it once it is trained, unless you retrain the whole thing again.



Hopefully this answers at least some of your questions. Sorry for long post :)

Alex
Comment 81 caulier.gilles 2009-12-27 19:06:20 UTC
Alex, Your comment is fine for me.

My tips :

- make re-entrant code in your library. We will use it in a separate thread.
- in batch mode, we will need a progress indicator. It's very important for end users GUI.
- give library version to make detection with digiKam cmake script easy. We have experience with external digiKam libs

If you have some code available to test/review let's me hear...

Do you want to host your code in KDE repository ? (as kdegraphics/libs)

Marcel,

Do you have any questions relevant of face detection database integration ?

Gilles Caulier
Comment 82 Alex 2009-12-27 19:21:54 UTC
Hi Gilles,

Thanks for the advice and I'll keep all this in mind, but I think this is still early days. There is a lot more testing with bigger libraries to be done and see how it responds to real world images, lots of tweaking to do. I think i will definitely be taking up this offer of more testing from you :)

Once more thorough testing is done, I will try to move the code to KDE repository, so that people can get it easier. I don't believe in making early development code public, as it will inevitable change dramatically very soon, so I will just end up annoying people. 

Anyways, hopefully next post will include some results from testing with different images and perhaps some instruction on how to get hold of this. 


P.S. If there is anyone who is working on similar system now, let me know so we can perhaps unite to combine the efforts.


Alex
Comment 83 Johannes Wienke 2009-12-27 19:33:58 UTC
Maybe it would be a nice idea to put the code into a personal branch of the kde repository right now. This way we can already have a look at it but it wont hurt or annoy anyone. Who will ever depend on code in a private working branch? ;)
Comment 84 Alex 2010-01-09 14:14:03 UTC
A quick update on the status of things. I finally got KDE SVN account but i can't check anything out with my username  :(, but working on it. 

Trying to get some integration with digikam now. I am not sure where to put it in though. I have looked through the code and if I understand it right, then I want to put it some where before the image preview is displayed but after it is loaded. Are all images after they have been loaded with custom loader converted to QImage and then displayed?


Alex
Comment 85 Alex 2010-01-09 14:35:43 UTC
Also a quick question, does digikam use some QCanvas to display images?

The reason is that to annotate image with tags, one doesn't want to draw rectangles and so o on the image itself, but have another layer of widgets on top, for example for face recognition draw a box around and have a label below, which can be edited etc. on a layer above. I don't know if canvas has an ability to do this, unless it can be done differently.

The idea is somewhat similar to panes in Java:
http://java.sun.com/docs/books/tutorial/uiswing/components/rootpane.html


Alex
Comment 86 Marcel Wiesweg 2010-01-09 17:37:27 UTC
Alex, we dont have the infrastructure yet you are talking about here.
Michael has written a proposal how to store tags on image regions, but that is not yet implemented. We also dont have any UI, not at all.

We currently do not use a canvas like QGraphicsView. There is related work to do that in the image editor.
We also dont use QImage, but a custom image container...

What part exactly do you intend to test? Is the workflow fully implemented?
> digikam library -> use haarfeatures to detect faces on image -> use face
> recognition on face identified by classifier -> output to digikam to confirm

I imagine there must be a widget to show boxes (autodetected) and to allow to add boxes (manually), then associate a box with a tag (with a tag, a person, and a face identifier).
Should this for starting be done in a separate window? It would be easier then to play with the UI.

Where and how do you store the results of learning?
Comment 87 Alex 2010-01-09 18:11:41 UTC
I have left the storing of the data to the people that want to use it as far as the library is concerned. In this case, I think it would be appropriate to store in some flat file somewhere, is it is constantly used and changed. Purely because some concerns over storing it in database (see above). The data itself are matrices and vectors of doubles. 

Why I have brought it up, pretty much any of the face recognition has to be supervised (decided by a person whether it is match or not), that is why there is a need for a feedback in the process.

I am going to write a small example where library is used and hopefully outputted for feedback. I think then the overall process will become a lot more clear.



P.S. I think I will try and merge with the existing libface library, last I have looked at it on sourceforge, it was unstable and not maintained anymore for a couple of years. This was the reason why I went to write one in the first place.


Alex
Comment 88 caulier.gilles 2010-01-09 23:09:58 UTC
Created attachment 39728 [details]
people tag widget screenshot.

At this place from KDE repository:

http://websvn.kde.org/trunk/playground/base/nepomuk-kde/peopletag/

You will find nepomuk people tags widget. This is exactly what's you need as widget, we don't care about the rest of nepomuk to implement face detection GUI.

Gilles Caulier
Comment 89 Alex 2010-02-19 12:50:10 UTC
UPDATE:

I was looking at peopletag project but had problem with it compiling altogether. 
I am going to try and get in touch with the developer to add to the people tagger ability to recognise faces through the libface library.

I also will be merging what there is to the libface project on sourceforge, as I think it is the most appropriate place for it.


Progress has been slow as I have a little too much work on my hands lately :( Sorry



Alex
Comment 90 caulier.gilles 2010-02-19 12:53:45 UTC
ALex,

If you need help...

Gilles Caulier
Comment 91 Alexander Balzer 2010-02-19 13:41:05 UTC
@Alex
I have tags of 250 people in ~ 5000 images in digikam. So if you need somebody to test your work. Feel free to contact me.
Comment 92 Jean-Marc Liotier 2010-02-20 21:47:38 UTC
Have you thought about tag storage format ? I have not read any mention of that in this thread, so I suggest using the Fotonotes XML format [1] which is as close as a common image location tagging format as we have. Flickr implementing a derivative of Photonotes [2] is endorsement enough that Fotonotes has the potential. 

[1] http://www.fotonotes.net/fotonotexml.html
[2] http://www.fotonotes.net/#installations

The best thing place to store that data would be in the JPEG header. That way, the embedded location tagging metadata would follow the image wherever it goes, whatever the tool used. I can very well imagine the Gallery Tagface module using that information.
Comment 93 Michael G. Hansen 2010-02-20 21:55:59 UTC
(In reply to comment #92)
> The best thing place to store that data would be in the JPEG header.

Are there any default places defined for other image formats like PNG, TIFF, etc?

Michael
Comment 94 Jean-Marc Liotier 2010-02-21 00:15:02 UTC
(In reply to comment #93)
> Are there any default places defined for other image formats like PNG, TIFF,
> etc?

A great starting point is the part 3 of the XMP Specifications - "Storage in Files" : http://www.adobe.com/devnet/xmp/pdfs/XMPSpecificationPart3.pdf pages 15 to 25. It provides information about how serialized XMP metadata is packaged into XMP Packets and embedded in different file formats.

Like Fotonotes, XMP is formatted with XML so embedding it into image files is exactly the same problem. But no Fotonotes library is in sight.

Sorry for the lack of proofreading in my preceding message - I shouldn't have posted while I was in a hurry to leave...
Comment 95 Aditya Bhatt 2010-03-06 09:08:27 UTC
Hi Alex,
I've been playing with openCV for a few months now. I'm quite familiar with Qt too. I'd like to help and get involved with digikam. I'd like to work on the face recognition thingy. I'm still new to bugzilla and svn, but I'll learn soon enough.

[1] My opinion is that the haar cascades bundled with openCV seem quite good, and using multiple types of cascades for different poses might be a good idea. I think one can call the face detector function with very permissive arguments so that almost all faces and some nonfaces are detected, and then apply some of one's own tricks to discard the nonfaces.

[2] Another thing I'd like to mention is - a guy named Rohan Anil has made a PAM face authentication module in the previous two GSOC's for KDE. I've checked his app and the detection part is quite good. In fact, close to perfect. 

The catch is that Rohan's app is supposed to detect only ONE face in an image. I think that the previous thing I mentioned in [1] can be used to grab anything that looks remotely like a face and feed it to [2].

His face recognition is something I haven't tested much, but I think it would be, well, *security-level* stuff.

The PAM Module's link is here : http://code.google.com/p/pam-face-authentication/

My bitbucket page : http://bitbucket.org/aditya_bhatt/

Please contact me at : adityabhatt1991 at gmail dot com

Regards,
Aditya
Comment 96 Alex 2010-03-07 11:33:01 UTC
@Aditya,
I had a look at the "paper" they have published recently and the module itself and not particularly convinced. For example, the system doesn't learn, which means it is limited to the original raining data. As I understand it, this modules uses eyes for face tracking and authentication. There is no guarantee what the eyes will be like in a photo, e.g. an eye is covered tilted rotated and occluded. 

The module has high rejection rate for security reasons, and that is something is opposite to what we want here. In practical aspects it builds dxd matrix somewhere inside where d is number of pixels in an image. Take an average 7M pixels image and make that a square. What you get is something unfeasable. Or at the very least needs a lot to optimise. Where as working off a 320x220 image from webcam is a different story. It makes no mention on the limitations of the system, rotation in both x-y plane and z axis is not mentioned as well as the image intensity invariance. What is the limit on how big a face it can detect? In the demo they have on website it only detected image when tester moved closer.

Based on what I have just said and more I am very skeptical about this module. However, having said this I think there are some worth while ideas there like MACE filter and Local Binary Pattern. Perhaps interesting things can also be done with eye detection. 

I don't think there is a single system that will do the trick, but instead a number of systems for face detection and recognition put together, working together to reinforce one another will be the best thing in the end.

I don't want to discourage you from trying this approach, so if you write the class based on what they have used, one for face detection and/or one for face recognition feel free to send it to me and I will try to integrate into the library.

By the way since the last post I managed to upload the library to libface (on sourceforge) and this is where it is going to be hosted permanently now. Feel free to download the library and compile it, but be warned the work is still very much in progress :)


Alex
Comment 97 Rohan Anil 2010-03-10 01:20:22 UTC
I dont think MACE Filter would be the way to go. The module i am working on is pam-face-authentication is for authentication that is 1 vs 1 matching. For the design - i had to choose between security vs speed (real-time) vs usability. So adapting it for recognition is out of question. So I dont think its fair to compare it for use in recognition :). And question about face detection - its an implementation available on OpenCV based on a method by Viola and Jones. If you want the performance numbers - http://www.lienhart.de/. He has a paper on it. Also again pam-face-authentication is for realtime authentication - So you have to provide your frontal face to the webcam! :) . I have bigger plans for the authentication system once i perfect my AAM implementation http://code.google.com/p/aam-opencv/, Currently the issue is some manual supervision is required which is a NO-GO with a normal user. It just has to work out of the box. Although the results from AAM-Tracking to Recognition is just phenomenal. 

Now coming to the task at hand. We are looking at the topic of unconstrained face recognition. Best - research for our photo tagging - could be to implement the best performing algorithm from http://vis-www.cs.umass.edu/lfw/ results. We could go with a support vector machine implementation with highly efficient features. But bottle-neck is the time required for training. Can we compromise over training time ? The more pictures you have more training time. Now we should pickup an algorithm which does not require retraining over the entire set for new images. I think i can code this over this/next weekend. But need help with integrating with the code base. If anyone could provide me an api spec of how the class should be like I would like a go at it..


regards
Rohan
Comment 98 Alex 2010-03-10 10:51:43 UTC
I really like the idea of http://vis-www.cs.umass.edu/lfw/ Has a lot of useful information, including ROC curves. :)


I am going to create a mailing list for libface so we can discuss the finer points of face recognition and detection there as opposed to boring people here and leave this thread for digikam/Face recognition issues :)



Alex
Comment 99 Aditya Bhatt 2010-03-19 10:26:15 UTC
Hi, 

I'm Aditya, a 2nd year student doing my Bachelors in Information and Communication Tech.
My favorite fields of work are pattern recognition, image processing and computational math.
I'm mostly self-taught in image processing, and also have some knowledge of neural networks. I'm familiar with PCA for dimensionality reduction.

I'm familiar with svn, git and hg version control.
I have some familiarity with clustering algorithms and eigenfaces. Fisherfaces, I'm learning.

I know how to work with Qt fairly well. I also have a very basic knowledge of KDE libraries - which I hope to improve as soon as my exams (a week later) end.

I've been working with Alex Jironkin on his libface project on sourceforge for a while now, and would like to continue working on it over the summer as part of GSoC 2010. I also want to start digiKam hacking soon to integrate this for tagging as described in the GSoC ideas page. I'm already working on a people tagging widget of my own, although I plan to scrap it and start working on the widget in KDE's svn repo as soon as svn:// port gets unblocked in my college :'(. ( I currently commit to libface over https:// )

I'd love it if Alex can be my mentor. And I'd also like the mentorship of one of the digiKam core devs for integrating this into the digiKam interface, and it's tagging database. It'd be nice to get this merged into digiKam before the end of the summer, and I'd like to work on digiKam too in the future - so I consider this a stepping stone.

Post-GSoC, I'd like to work on KIPI and digiKam's "fuzzy" algorithms, as that is my field of interest.

I look forward to your replies soon :)

Regards,
Aditya
Comment 100 Dotan Cohen 2010-03-19 13:09:26 UTC
It seems that KDE did not get a spot in GSOC this year:
http://socghop.appspot.com/gsoc/program/accepted_orgs/google/gsoc2010
Comment 101 Arnd Baecker 2010-03-19 13:20:55 UTC
> It seems that KDE did not get a spot in GSOC this year:
> http://socghop.appspot.com/gsoc/program/accepted_orgs/google/gsoc2010

Sure? I see KDE listed there (in the second part).
Comment 102 Alex 2010-03-19 13:22:45 UTC
I was going to say, that it would be very suprising if they didn't. I can see them in organisations completed profile lower down the list too.
Comment 103 Dotan Cohen 2010-03-19 14:03:26 UTC
> (in the second part)

Second part?!? Who gets that far?

Sorry for the mistake, I'll go away quietly now...
Comment 104 naresh 2010-03-19 15:25:04 UTC
hi,

I am a student pursuing B.Tech 4th year from Indian Institute of Information Technology- Allahabad, India.

I am keen to get involved in open source project development. I have seen on KDE website the project ideas for GSoC 2010. I would like to work on Face Recognition Project. I took this project because  I have worked on the hand gesture recognition using OpenCV and am currently working on the same area.You can have a look at my work here.

http://www.youtube.com/watch?v=wnKk07-YNFw&feature=player_embedded

I am having good knowledge of C++, Qt  and the classification algorithms.  

I would like to know some more details of the project.

I would be happy to know answers to these following questions.
1. what classification algorithm is to be refined for face detection part. OpenCV provides Haar Classifier using Ada boost learning and it works with almost 99% accuracy. Is it the same algorithm that is to be refined or something else?
2. If it is Haar classifier to be refined. In which part it is to be refined haartraining utility or haar detection part ?
  
Hoping for a positive reply from you as soon as possible.

Thanks
Naresh Reddy S.
IIIT-Allahabad
Comment 105 Alex 2010-03-19 17:06:18 UTC
Haar training is not something of interest, because it doesn't provide ability to update on the fly. Unless you can think of a way that can be done because haartraining learns from scratch everytime. So you need to write updater from scratch and do some clever hacking of retrieving previous states from learning.

So we are interested in refining face detection first. When you say it is 99% good, it also has high rate of Type II errors, identifying something as a face when its not, which is bad. That is what needs refining. This problem arises when you use real photos, because need to specify feature size in the identification. 

Or you can think of a better face detection mechanism


Alex
Comment 106 Alex 2010-03-23 10:10:21 UTC
I seem to be keep shooting people down with ideas. :( Sorry. 

I think we got face detection to work fine now, the area where we need help and fresh ideas is face recognition. So if you have idea or even better an algorithm for face recognition feel free to post it to me or into libface mailing list and we will get on it to be implemented :) 

I guess another area would be deeper integration into KDE itself, like with contacts, I guess this is also can be done in digikam.


Alex
Comment 107 Johannes Wienke 2010-03-23 10:57:01 UTC
At my university we use active appearance models to recognize faces. This approach needs a lot of training data to set up good models but should then scale much better then eg. Eigenfaces. Unfortunately I can only find the diploma thesis about this which is in German:
http://www.airesearch.techfak.uni-bielefeld.de/files/Lang2007-DIP.pdf
Comment 109 Alex 2010-03-23 11:14:56 UTC
Does the whole thing needs to be retrained everytime you want to add more data or can it just be updated.

One of the problems with using Haar features for face recognition now, is because you also need a large training set, and you can't easily add more data to the cascade, in other words it has to be retrained from scratch everytime, which is a big problem.

Also bear in mind this is supervised learning, so there is a reliable input from people to tell what is right and wrong. So only correct data is added to the classifier/cascade.

Thanks for the idea though, would be interesting to have a look at that at some point.


Alex
Comment 110 Johannes Wienke 2010-03-23 11:39:37 UTC
This depends on the classifier you want use. AAM only provides the feature vector. The solution Christian used was a two step procedure:
1. Nearest Neighbour for rejection (trivial update ;) )
2. Classification based on Gaussian Mixture Models. There will certainly exists enough algorithms to update them with new samples.
Comment 111 Alex 2010-04-27 10:21:19 UTC
    Accepted projects for GSOC 2010 were announced last night. I am also happy to say that one of the projects accepted was face recognition project. :)

    Adytia will be doing this project and hopefuly we will have a fully working plugin by the en of the summer.


Alex
Comment 112 caulier.gilles 2010-04-27 10:28:40 UTC
Great that it have been accepted. I'm very impatient to test this feature.

I talk by IRC with Adytia this week end about face detection, and making a plugin is not the good way. It's better to patch digiKam as well, to be able to work directly with digiKam DB. I will attached the conversation to this file, for details.

I plan to organize a coding sprint for digiKam at end of summer in France, Aix en provence. It wil be nice to see libface developpers join us at this event. Note : KDE-ev will sponsorship the event (travel + hotel payed).

Gilles Caulier
Comment 113 caulier.gilles 2010-04-27 10:33:00 UTC
digiKam IRC talk between me and Aditya Bhatt about face detection.

[19:55] <cauliergilles> Adityab: ping ?
[19:55] <Adityab> cauliergilles: Yes?
[19:56] <cauliergilles> Adityab: hi. Just 2 questions
[19:56] <Adityab> cauliergilles: sure
[19:56] <cauliergilles> Adityab: 1. Do you recieve your svn account as
developper ?
[19:57] <cauliergilles> Adityab: 2. in digikam-devel mailing list, do
you recieve my mail about digiKam database problem today (i'm not sure
if this message is passed right because it's long)
[19:57] <Adityab> cauliergilles: No, I'm applying tomorrow. They give
higher priority to GSoC students. And even if I don't get into GSoC,
I'm still applying.
[19:57] <cauliergilles> Adityab: ok. let's me hear if you have some pb
about. I can contact right admin accordingly
[19:58] <cauliergilles> Adityab: and don't forget to add you in coding
sprint oo.org document from svn.
[19:59] <cauliergilles> Adityab: I will need quickly all info from
contributors to book hotel and restaurant at Aix en Provence
[19:59] <Adityab> cauliergilles: Thank you :) Yes, I received the mail
and I read it, but I didn't try to analyse everything as my final
exams are from tomorrow
[20:00] <Adityab> cauliergilles: Yes, I'll update the document soon.
I'll be renewing my passport also, and that will take a max of one
month.
[20:01] <cauliergilles> Adityab: ok. no problem. I just want to be
sure if mail is posted right through KDE server. Normally Marcel must
respond about
[20:01] <cauliergilles> Adityab: great.
[20:01] <cauliergilles> Adityab: currently, i don't share a lots of
time with GoSC, because i change my home, and i very busy.
[20:02] <Adityab> cauliergilles: That's okay :)
[20:02] <cauliergilles> Adityab: i also, contribute a lots in Exiv2
project to finalize Sony Makernotes support.
[20:02] <Adityab> cauliergilles: Thanks :) If you can act as my
referrer to the sysadmins, I'll apply just now.
[20:02] <cauliergilles> Adityab: i'm also teacher at Aix en provence,
in a computer science school
[20:03] <cauliergilles> Adityab: so my free time is limited... (:=)))
[20:03] <Adityab> cauliergilles: Must be a hectic job :)
[20:03] <cauliergilles> Adityab: but i will switch indeep to digiKam
and follow GoSC more seriously in few week
[20:04] <cauliergilles> Adityab: i'm sure that Marcel will follow you
properly. He is serious developer
[20:04] <Adityab> cauliergilles: Yes, he is very helpful. He replied
to my every query
[20:04] <cauliergilles> Adityab: Anyway, if you have question or a
problem don't hesitate to mail me.
[20:05] <cauliergilles> Adityab: i'm not online frequently in this
channel, due my external job
[20:05] <cauliergilles> Adityab: but i read my mail each day
[20:05] <Adityab> cauliergilles: Sure, thanks :)
[20:06] <cauliergilles> Adityab: yesterday, i do a presentation of
digiKam in a LUG at Aix en provence
[20:07] <Adityab> Yes, I saw your update on facebook :)
[20:07] <cauliergilles> Adityab: ah yes.
[20:08] <cauliergilles> Adityab: and a lots of people ask me if face
detection will be implemented soon, to provide the same feature than
Picasa
[20:08] <cauliergilles> Adityab: i said ... yes of course
[20:08] <cauliergilles> Adityab: so people are very impatient to test
this feature
[20:08] <Adityab> cauliergilles: Yes, currently there is no
open-source suite with this feature
[20:09] <Adityab> cauliergilles: Apple, M$, and Google all have it in
their suites
[20:11] <-- allee-k a quitté ce serveur (Read error: Connection reset by peer).
[20:11] <Adityab> cauliergilles: Have you checked libface?
[20:11] <cauliergilles> Adityab: no. i must do it now ?
[20:12] <cauliergilles> Adityab: where is the code ?
[20:12] <-- tro a quitté ce serveur (Read error: Connection timed out).
[20:12] <Adityab> cauliergilles: No, take your time
[20:12] <Adityab> cauliergilles: It's here :
http://sourceforge.net/projects/libface/
[20:12] <cauliergilles> Adityab: i'm not able to judge algorithm, but
to review code, coding style doc, api etc, yes, i can guide you
[20:13] <Adityab> cauliergilles: It's okay, the intefrace for libface
is very friendly
[20:13] <cauliergilles> Adityab: i wil checkout the code and give you
a first feedback
[20:14] <Adityab> cauliergilles: and the API is documented by doxygen too
[20:14] <cauliergilles> Adityab: Q : how do you will use this lib with
digiKam ? directly in core implementation or through a plugin ?
[20:14] <Adityab> cauliergilles: I think a plugin would be a better idea
[20:15] <cauliergilles> Adityab: do you use CMake to compile it. Do
you take a care about portability (Windows especially) ?
[20:16] <cauliergilles> Adityab: i think that a plugin will be
difficult to handle by user, from usuability viewpoint
[20:16] <Adityab> cauliergilles: Yes, we use CMake. It works on Mac
and linux. I haven't tested compilation on windows, but probably it
will need only some on-line change for that
[20:16] <cauliergilles> Adityab: if the goal is to learn to digiKam
face, and later to be able to tags automatically image during
downloading (or with BQM), digiKam core must be patched as well
[20:17] <Adityab> cauliergilles: Then I'll consider core
implementation. I haven't had much time to analyse digikam's
architecture, but my exams will end next thursday
[20:17] <cauliergilles> Adityab: about windows, i already have MinGW
and MSVC compiler ready to test. No need to waste time about for the
moment
[20:18] <cauliergilles> Adityab: just ping me when you want to check
compilation under Win32, i will do it
[20:18] <cauliergilles> Adityab: ok, no problem, take your time
[20:18] <cauliergilles> Adityab: I just want to know which workflow
will be used with face detection
[20:19] <Adityab> cauliergilles: Thanks :) And I think It'd be better
to implement it as the fuzzy search, so we can tell it to generate
face "fingerprints"
[20:19] <cauliergilles> Adityab: also, making batch plugin in kipi is
deprecated now, since Batch Queue Manager is ready to use in digiKam
core
[20:19] <Adityab> cauliergilles: But I'm able to say for sure only
after I will finish exams :(
[20:20] <cauliergilles> Adityab: ok.
[20:20] <Adityab> cauliergilles: Does the KIPI batch uploader also
take care of tags?
[20:20] <Adityab> cauliergilles: That would be useful if autotagged
pictures could be uploaded to facebook/flickr
[20:20] <cauliergilles> Adityab: also, take a look into digiKam core
how we have implemented Haar interface to register and play
fingerprint dedicated to search duplicate items in DB
[20:21] <Adityab> cauliergilles: Okay
[20:21] <cauliergilles> Adityab: and i recomend to register your Face
fingerprnt to digiKam DB. No need to re-invent the wheel (:=)))
[20:22] <Adityab> cauliergilles: You
[20:22] <cauliergilles> Adityab: they are some code ready to use about
Haar stuff from digiKam DB interface
[20:22] <Adityab> re right
[20:22] <Adityab> cauliergilles: You're right, but I think the
eigenfaces database might be too big
[20:22] <cauliergilles> Adityab: about kipi uploader and tags, yes,
kipi api is ready to take tags from kipi host application
[20:23] <-- ciskje a quitté ce serveur (Read error: Connection reset by peer).
[20:23] <Adityab> cauliergilles: Okay.
[20:23] <cauliergilles> Adityab: it's not a problem. just create a
dedicated DB file about
[20:23] <cauliergilles> Adityab: in digiKam we have already a main DB
file and a thumb DB file
[20:23] <Adityab> cauliergilles: That seems good
[20:24] <Adityab> cauliergilles: Should we store the face tags in the
image metadata itself, or a separate file?
[20:24] <Adityab> cauliergilles: picasa uses a separate file
[20:25] <cauliergilles> Adityab:  i'm not sure if Flickr uploader
plugins is able to tags items in flickr iface. But i think yes
[20:25] <cauliergilles> Adityab: i don't kow how to work Picasa about
face detection, but this is my personnal vision for digiKam. Fix me if
i'm wrong :
[20:26] <cauliergilles> Adityab: make face learning GUI, probably in
preview mode
[20:26] <cauliergilles> Adityab: register face fingerprint to a dedicated DB
[20:27] <cauliergilles> Adityab: the goal is to have a list of people
registered in DB, ready to be identified in images
[20:27] <Adityab> cauliergilles: Yes, that I how I think of it too...
[20:28] <cauliergilles> Adityab: of course something like Facebook
face tag editor must be done in preview mode
[20:28] <cauliergilles> Adityab: for that, code from Nepomuk widget
can be used here (do you know ?)
[20:29] <Adityab> cauliergilles: Plus, about libface - the face
detection and recognition is slow at the moment, but don't be
disappointed. I already have faster code in my bitbucket and laptop,
I'll be commiting it to libface after I clean it up
[20:29] <cauliergilles> Adityab: when face DB is ready to use, we can
implement something in Camera GUI to tags face during download from
camera,
[20:29] <Adityab> cauliergilles: Yes, I saw the nepomuk widget
[20:29] <Adityab> cauliergilles: Okay
[20:29] <cauliergilles> Adityab: and in Batch Queue Manager, we can
add a new Batch tool to analyse a set of image to detect face and tag.
[20:30] <Adityab> cauliergilles: right
[20:30] <cauliergilles> Adityab: a important point : libface must be
re-entrant and support multithreading
[20:30] <cauliergilles> Adityab: BQM use thread everywhere
[20:30] <Adityab> cauliergilles: All right. That is something I'll
have to look into
[20:31] <cauliergilles> Adityab: the registration in DB is another
stuff. Marcel know the problem with thread and DB (uncompatibility
with sqlite), but he know the solution using lock
[20:32] <Adityab> cauliergilles: okay. And does digiKam already have
support to share tags with nepomuk?
[20:32] <cauliergilles> Adityab: Note : with Mysql, there is no thread
problem. But it's the future : DB interface is under finalization to
be compatble with Mysql. So we must support Sqlite first, as weel
[20:33] <Adityab> cauliergilles: Yes, I read about the DB support in
the Mailing list
[20:33] <cauliergilles> Adityab: yes, digiKam share tags with nepomuk.
In fact, it's a copy a digiKam DB. In all case, we will _never_ use
nepomuk to replace digiKam DB
[20:34] <cauliergilles> Adityab: nepomuk is synchronized with digiKam
DB, that all
[20:34] <Adityab> cauliergilles: That is good, as even GNOME people use digiKam.
[20:34] <cauliergilles> Adityab: marcel has written this code few month ago
[20:35] <cauliergilles> Adityab: yes a lots of people use digiKam
under Gnome. Look all feedback from ubuntu for ex (:=)))
[20:35] <cauliergilles> Adityab: in fact Gnome people need to install
KDELibs. that all. No need whole KDE4 desktop
[20:35] <Adityab> cauliergilles: But they don't get updated version
all the time, so we also get old Beta feedback :)
[20:36] <cauliergilles> Adityab: yes. it's Ubuntu packager problem,
not digiKam project
[20:36] <cauliergilles> Adityab: with Ubuntu F-spot is the default photo manager
[20:37] <cauliergilles> Adityab: but a lots of people think that
F-spot is not suitable. It crash a lots, and use dot net API from M$.
It's not very clear from Open Source viewpont (;=)))
[20:37] <Adityab> cauliergilles: Yes, but F-Spot is not as
feature-rich, but just my opinion
[20:38] <Adityab> cauliergilles: I think people need to stay away from
mono when something like Qt is there :)
[20:38] <cauliergilles> Adityab: difficult to see if F-Spot is
interresting or not. I tried to test a lots of time, and i can crash
it very quickly (:=)))
[20:38] <Adityab> cauliergilles: By the way, I could not get the
peopletag widget to compile properly. It's in playground and the
developer ( I don't know who ) has abandoned it.
[20:39] <cauliergilles> Adityab: Q Gnome will switch to Mono API in the future ?
[20:40] <Adityab> cauliergilles: Yes, it looks like that. Not
entirely, but they want to make OOP programming the norm for all GNOME
project, it seems
[20:41] <cauliergilles> Adityab: well, Gnome is dead (:=)))
[20:41] <Adityab> cauliergilles: For us :)
[20:41] <-- tro a quitté ce serveur (Ping timeout: 245 seconds).
[20:41] <Adityab> cauliergilles: Okay, I shall talk later, it is late
night here and I have my first exam tomorrow. Nice to talk to you :)
[20:42] <cauliergilles> Adityab: ok. good night

Gilles Caulier
Comment 114 caulier.gilles 2010-12-09 14:19:35 UTC
Implemented in digiKam 2.0.0

Gilles Caulier
Comment 115 simon 2010-12-09 14:33:12 UTC
yay, party on!