| Summary: | Tagged face areas on portait (vertical) oriented images are mispositioned | ||
|---|---|---|---|
| Product: | [Applications] digikam | Reporter: | Alexander Yavorsky <kekcuha> |
| Component: | Faces-Workflow | Assignee: | Digikam Developers <digikam-bugs-null> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | abrownjo, alexandre.belz, andreas.jansson, aseifert, cartman005, caulier.gilles, chandra.kant.eee17, dewvinci, dm.griffiths, iwannaberich, kde-bugs, kde, metzpinguin, philsweeney25, registrations, Struan |
| Priority: | NOR | ||
| Version: | 6.1.0 | ||
| Target Milestone: | --- | ||
| Platform: | Other | ||
| OS: | Linux | ||
| Latest Commit: | https://invent.kde.org/kde/digikam/commit/32bb16fee6b2676b30bf843bf3e083a23084cd3d | Version Fixed In: | 7.0.0 |
| Sentry Crash Report: | |||
| Attachments: |
mispositioned tags
Image sample I am here attaching a photo showing that face tags are working absolutely fine in Digican version 6.0.0 attachment-17016-0.html attachment-14732-0.html attachment-13056-0.html |
||
Can you share some picture processed with Picassa to investigate. Sound like face tags are miss-interpreted in metadata parser while importing. Gilles Caulier Created attachment 102324 [details]
Image sample
Image sample.
Can you reproduce the problem using digiKam Linux AppImage bundle ? The last bundle is available at this url: https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM Gilles Caulier Unfortunately the same with the last AppImage bundle. I have re-read metadata from image, but it doesn't help. I'm also looking to replace Picasa with DigiKam and have hit the same issue. Looking at the import code the issue appears to be that DigiKam expects the coordinates relative to the oriented image and not the unoriented image. https://api.kde.org/extragear-api/graphics-apidocs/digikam/html/dmetadata_8cpp_source.html line 1641 etc. The spec. for MWG states that the region is as per the unoriented image. http://www.metadataworkinggroup.com/specs/ If the DigiKam face rectangle/region is relative to the oriented image then the coordinates need adjusting accordingly. http://u88.n24.queensu.ca/exiftool/forum/index.php/topic,6354.msg32155.html#msg32155 Can you confirm that DigiKam stores it's face rectangles/region relative to the oriented image? *** Bug 378456 has been marked as a duplicate of this bug. *** *** Bug 377628 has been marked as a duplicate of this bug. *** I also use Picasa as the 2nd program. It seems the problem exists with digicam 5.6.0 (Windows). My pictures are not twisted. I use only the Exif orientation tag to display the image in the right orientation. In this case the picasa face-frames are not twisted with the image on portrait oriented images. The frame-position is the position on the untwisted image. *** Bug 395243 has been marked as a duplicate of this bug. *** I just want to report that this problem still exists (5.9.0 win) and how to get around it. The problem seem to be that Picasa always writes the faces areas with respect to the physical orientation without accounting for the EXIF orientation tag. What I did in order to import my Picasa collection into DigiKam was 1- Configure DigiKam to read face tags and do not rotate based on EXIF orientation 2- Added the Picasa tagged directories to the DB (all the face metadata was already embedded on the XMP tags and not on the picasa.ini) 3- Create a couple of searches in DigiKam to filter landscape pictures that had a rotated left or rotated right EXIF tag. 4- On those pictures I did a "Adjust EXIF orientation tag to NORMAL" 5- I manual rotated the picture (in batch). The key to this solution is this last step where DigiKam not only rotates the image but it also rotates the areas for the faces. Instead to test with 5.9.0, it's better to test with 6.0.0 pre-release where few improvement have been done in this area. Files are here : https://files.kde.org/digikam/ Gilles Caulier I tested it with current Digikam 6.0.0-beta3 and the bug is still there: This shows the problem: $ convert -size 400x240 xc:skyblue -fill white -stroke black -draw "rectangle 80, 40 320,200" draw_rect.jpg $ exiftool -n -Orientation=1 -XMP-mwg-rs:RegionAppliedToDimensionsH=240 -XMP-mwg-rs:RegionAppliedToDimensionsW=400 -XMP-mwg-rs:RegionAppliedToDimensionsUnit=pixel -XMP-mwg-rs:RegionAreaUnit=normalized -XMP-mwg-rs:RegionName=John -XMP-mwg-rs:RegionType=Face -XMP-mwg-rs:RegionAreaX=0.5 -XMP-mwg-rs:RegionAreaY=0.5 -XMP-mwg-rs:RegionAreaW=0.62 -XMP-mwg-rs:RegionAreaH=0.7 draw_rect.jpg The location of the face area stays the same if the orientation is changed to something different from top,left: $ exiftool -n -Orientation=8 draw_rect.jpg Created attachment 118486 [details] I am here attaching a photo showing that face tags are working absolutely fine in Digican version 6.0.0 In Bug 372340 ,face tags are working absolutely fine in Digicam 6.0.0 You cannot close this file only you check with one file. I'm wure that nothing have been done around this topic in source code... Maik can you confirm this ? Gilles Caulier Yes i am trying to resolve this issue. I think for this issue -the more aligned faces are the better face recognition results are.So we need to convert the image to greyscale.Then we can rotate face to vertical position so it is aligned perfectly. we can do this by computing height difference between the eyes and when eyes are shut we can use histogram of oriented gradients for nose to get its orientation. First thing first we need to find eyes on picture. we can crop the image to just the face part so the classifier has easier job finding eyes and doesn’t throw false positives. The main idea is to compute vertical and horizontal sobel for nose part and find the angle between them. Then to determine which angle is dominant with help of histogram and use its peak value in finding the best rotation of face. This part can be improved by normalizing the histogram on start and then using just values from one face rotation angle to determine the angle between vertical position and current angle. We can rotate face by this function
void rotate(cv::Mat& src, double angle, cv::Mat& dst)
{
int len = max(src.cols, src.rows);
cv::Point2f pt(len / 2., len / 2.);
cv::Mat r = cv::getRotationMatrix2D(pt, angle, 1.0);
cv::warpAffine(src, dst, r, cv::Size(len, len));
}
So that we can improve face detection by aligning faces.
After having model of faces trained and enhanced image of face we want to recognize. Tricky part here is determining when the face is new (not in training set) and when it is recognized properly. So we can creat some sort of treshold for distance .
int predictedLabel = -1;
double predicted_confidence = 0.0;
model->set("threshold", treshold);
model->predict(croppedImage, predictedLabel, predicted_confidence);
Chandra, this bug is not about face detection, its about meta data only. Region could be anything. The bug is about the wrong placement/highlighting of region meta data, read from non-portrait oriented image files. Chandra, Please stop to spam all face management bugzilla entries, if you don't check in-deep the technical issues. This is not like this that you will retained to GoSC 2019 ! Gilles Caulier Git commit dc7ee21f97cc80b993885487c3c324ebe0d8d2c4 by Maik Qualmann. Committed on 22/03/2019 at 22:11. Pushed by mqualmann into branch 'master'. fix wrong positions of the face areas depending on the orientation FIXED-IN: 6.1.0 M +2 -1 NEWS M +37 -23 core/libs/fileactionmanager/fileworkeriface.cpp M +1 -1 core/libs/fileactionmanager/fileworkeriface.h M +108 -5 core/utilities/facemanagement/facegroup.cpp M +1 -0 core/utilities/facemanagement/facegroup.h https://commits.kde.org/digikam/dc7ee21f97cc80b993885487c3c324ebe0d8d2c4 Git commit f3b5cd5a263b720d470d9eec6e46edd62fbf2ffa by Maik Qualmann. Committed on 29/03/2019 at 20:54. Pushed by mqualmann into branch 'master'. revert the patch for the time being We completely break the behavior to previous versions of digikam. M +6 -6 core/utilities/facemanagement/facegroup.cpp https://commits.kde.org/digikam/f3b5cd5a263b720d470d9eec6e46edd62fbf2ffa *** Bug 406971 has been marked as a duplicate of this bug. *** This problem also occurs for faces tagged by Lightroom (not just Picasa). ahhh I was excited when I saw this on the 6.2 bugfix list "022 ==> 407262 - Face tagging doesn't work with rotation". but its not had an impact on the this issue. I deleted my DB and started a fresh one but a lot of the portrait photos still have the face boxes in the wrong place after reading my Picassa managed photos. I have the same issue ! People correctly detected by Picasa, but Wrongly displayed by Digikam (the square is not correctly located) especially on rotated pictures. (I'm using latest weekly build 6.4) Yes, it is known that digiKam mistreats the face rotation. I had already corrected it. However, since digiKam has been treating it like this since the very beginning, all longtime users are facing a problem. So we first have to write a converter that corrects all images. Maik Ok, thanks Maik. Do you think it can be managed in 7.0 ? *** Bug 413926 has been marked as a duplicate of this bug. *** 7.0.0-beta1 is out with new Face Recognition algorithm based on Deep Learning/Neural Network API from OpenCV https://download.kde.org/unstable/digikam/ Please test and give us a feedback Thanks in advance Gilles Caulier *** Bug 415550 has been marked as a duplicate of this bug. *** Hello Maik, Gilles, can i help on testing a possible new SW based on 7.0 beta ? *** Bug 415941 has been marked as a duplicate of this bug. *** Git commit b94b6abd32d733c6eca7be564584fe1ae2785a2b by Maik Qualmann. Committed on 06/01/2020 at 21:13. Pushed by mqualmann into branch 'master'. we need the orientation from DImg for later M +5 -0 core/libs/dimg/dimg.h M +4 -1 core/libs/dimg/dimg_p.h M +7 -0 core/libs/dimg/dimg_transform.cpp https://invent.kde.org/kde/digikam/commit/b94b6abd32d733c6eca7be564584fe1ae2785a2b (In reply to caulier.gilles from comment #32) > 7.0.0-beta1 is out with new Face Recognition algorithm based on Deep > Learning/Neural Network API from OpenCV > Please test and give us a feedback This issue does not depend on the face recognition mechanism; it is a consequence of improper handling of face rectangles positions in images rotated by the rotation flag in the metadata. Created attachment 125019 [details] attachment-17016-0.html Thank you. I will load beta version and feedback. David ________________________________ From: Alexander Yavorsky <bugzilla_noreply@kde.org> Sent: Thursday, January 9, 2020 3:41:39 PM To: dm.griffiths@hotmail.co.uk <dm.griffiths@hotmail.co.uk> Subject: [digikam] [Bug 372340] Tagged face areas on portait (vertical) oriented images are mispositioned https://bugs.kde.org/show_bug.cgi?id=372340 --- Comment #37 from Alexander Yavorsky <kekcuha@gmail.com> --- (In reply to caulier.gilles from comment #32) > 7.0.0-beta1 is out with new Face Recognition algorithm based on Deep > Learning/Neural Network API from OpenCV > Please test and give us a feedback This issue does not depend on the face recognition mechanism; it is a consequence of improper handling of face rectangles positions in images rotated by the rotation flag in the metadata. -- You are receiving this mail because: You are on the CC list for the bug. The problem has not yet been resolved. The code for this is under development. Maik Created attachment 125042 [details] attachment-14732-0.html Maik Thanks for letting me know Regards David ________________________________ From: Maik Qualmann <bugzilla_noreply@kde.org> Sent: Friday, January 10, 2020 4:42:57 PM To: dm.griffiths@hotmail.co.uk <dm.griffiths@hotmail.co.uk> Subject: [digikam] [Bug 372340] Tagged face areas on portait (vertical) oriented images are mispositioned https://bugs.kde.org/show_bug.cgi?id=372340 --- Comment #39 from Maik Qualmann <metzpinguin@gmail.com> --- The problem has not yet been resolved. The code for this is under development. Maik -- You are receiving this mail because: You are on the CC list for the bug. Git commit 32bb16fee6b2676b30bf843bf3e083a23084cd3d by Maik Qualmann. Committed on 19/01/2020 at 08:37. Pushed by mqualmann into branch 'master'. read and write not aligned face metadata from image FIXED-IN: 7.0.0 M +2 -1 NEWS M +7 -4 core/libs/database/item/scanner/itemscanner_photo.cpp M +3 -29 core/libs/fileactionmanager/metadatahub.cpp M +0 -6 core/libs/fileactionmanager/metadatahub.h M +6 -5 core/utilities/imageeditor/main/imagewindow.cpp https://invent.kde.org/kde/digikam/commit/32bb16fee6b2676b30bf843bf3e083a23084cd3d Note: The metadata must be read from the image again so that the face rectangles are displayed correctly. To do this, the function to clean up the metadata in the database must be activated temporarily in the digiKam metadata settings in order to remove the old face rectangles from the database. Maik Created attachment 125238 [details] attachment-13056-0.html Many thanks. Presumably I need to update my program? David ________________________________ From: Maik Qualmann <bugzilla_noreply@kde.org> Sent: Sunday, January 19, 2020 8:46:00 AM To: dm.griffiths@hotmail.co.uk <dm.griffiths@hotmail.co.uk> Subject: [digikam] [Bug 372340] Tagged face areas on portait (vertical) oriented images are mispositioned https://bugs.kde.org/show_bug.cgi?id=372340 --- Comment #42 from Maik Qualmann <metzpinguin@gmail.com> --- Note: The metadata must be read from the image again so that the face rectangles are displayed correctly. To do this, the function to clean up the metadata in the database must be activated temporarily in the digiKam metadata settings in order to remove the old face rectangles from the database. Maik -- You are receiving this mail because: You are on the CC list for the bug. You can try with Linux AppImage 7.0.0-beta2 available here : https://files.kde.org/digikam/ Gilles Caulier Note : Linux AppImage will be updated with last Maik fix only monday morning, not before. Windows and MacOS are already up-to date. Gilles Caulier I am so glad it could be sorted out. I'll try it as soon as the next built is released. Great news ! :) Thanks to the dev team ! I will also test with latest Windows64 release... One question : what is the difference between - digiKam-7.0.0-beta2-20200120T031953-Win64.exe - digiKam-7.0.0-beta2-20200120T033452-Win64-debug.exe ? debug vs non debug => look in README file, near bundles files : http://www-ftp.lip6.fr/pub/X11/kde-applicationdata/digikam/README Gilles Caulier (In reply to caulier.gilles from comment #44) > You can try with Linux AppImage 7.0.0-beta2 available here : > > https://files.kde.org/digikam/ > > Gilles Caulier Unfortunately beta2 crashes in arch (beta1 works perfectly): walker@ArchLinuxG700 ~ % LANGUAGE=en_EN.UTF-8 ./Applications/digikam-7.0.0-beta2-20200120T081221-x86-64_fcfd171ffeb786048bdf3d4068f8fc39.appimage -- digiKam Linux AppImage Bundle -- Use 'help' as CLI argument to know all available options for digiKam application. -- Notes: to integrate this bundle to your desktop, use AppImageLauncher. -- to enable all debug messages on the console, use 'export QT_LOGGING_RULES="digikam*=true"'. libudev.so.0 /tmp/.mount_digikanSq5Q9/AppRun: line 16: arch: command not found -- Preloading shared libs: unknown: qglx_findConfig: Failed to finding matching FBConfig for QSurfaceFormat(version 2.0, options QFlags<QSurfaceFormat::FormatOption>(), depthBufferSize -1, redBufferSize 1, greenBufferSize 1, blueBufferSize 1, alphaBufferSize -1, stencilBufferSize -1, samples -1, swapBehavior QSurfaceFormat::SingleBuffer, swapInterval 1, colorSpace QSurfaceFormat::DefaultColorSpace, profile QSurfaceFormat::NoProfile) unknown: qglx_findConfig: Failed to finding matching FBConfig for QSurfaceFormat(version 2.0, options QFlags<QSurfaceFormat::FormatOption>(), depthBufferSize -1, redBufferSize 1, greenBufferSize 1, blueBufferSize 1, alphaBufferSize -1, stencilBufferSize -1, samples -1, swapBehavior QSurfaceFormat::SingleBuffer, swapInterval 1, colorSpace QSurfaceFormat::DefaultColorSpace, profile QSurfaceFormat::NoProfile) unknown: Could not initialize GLX /tmp/.mount_digikanSq5Q9/AppRun: line 155: 2360 Aborted (core dumped) digikam $@ Thanks Gilles. I will test using the debug version then. ;) @Alexander, I suspect it crashes in QWebEngine. Which graphics card and drivers do you use? Gilles, is it correct that the AppImage now uses QWebEngine? Maik yes correct... Qtwebengine is used in Appimage and macos PKG now... Gilles Hi Maik, I use 2x laptops with integrated Intel graphics chip. on the one I'm using today, It's running Intel Graphics 520 with driver v26.20.100.6888 FRom digikam, I can also se the versions : digikam version 7.0.0-beta2 CPU cores: 4 Eigen: 3.3.7 Exiv2: 0.27.2 Exiv2 can write to Jp2: Yes Exiv2 can write to Jpeg: Yes Exiv2 can write to Pgf: Yes Exiv2 can write to Png: Yes Exiv2 can write to Tiff: Yes Exiv2 supports XMP metadata: Yes HEIF encoding support: Yes ImageMagick codecs: 7.0.8 KF5: 5.65.0 LensFun: 0.3.95-0 LibCImg: 130 LibJPEG: 90 LibJasper: 2.0.16 LibLCMS: 2090 LibLqr support: No LibPGF: 7.19.03 LibPNG: 1.6.37 LibRaw: 0.20.0 LibTIFF: 4.1.0 Marble: 0.27.20 Parallelized demosaicing: No Qt: 5.14.0 Qt Webkit support: Yes VKontakte support: No AkonadiContact support: No Baloo support: No Calendar support: Yes DBus support: No Database backend: QSQLITE HTML Gallery support: Yes LibAVCodec: 58.54.100 LibAVFormat: 58.29.100 LibAVUtil: 56.31.100 LibGphoto2 support: No LibOpenCV: 3.4.9 LibQtAV: 1.13.0 Media player support: Yes Panorama support: Yes Just tell me if you need a complete Windows report from "System Information". @Alexandre Belz, my question went to @Alexander Yavorsky about the crash. You are on Windows, we still use QWebkit there and are not affected by the possible problem. Maik (In reply to Maik Qualmann from comment #51) > @Alexander, I suspect it crashes in QWebEngine. Which graphics card and > drivers do you use? Gilles, is it correct that the AppImage now uses > QWebEngine? > > Maik I did a little investigation: it looks like the issue is with drivers other than nvidia's proprietary drivers: On a laptop with hybrid graphics beta 2 only starts when the nvidia card is used forced (optirun ./digikam-7.0.0-beta2-...) On a PC with an Nvidia card, it was possible to launch beta2 only after replacing the Nouveau driver with a proprietary Nvidia driver. |
Created attachment 102169 [details] mispositioned tags I use both digikam and picasa software to manage photos. When face tags assinged in picasa for portrait oriented photo in digikam face tags are mipositioned (see attachment).