Summary: | stack underflow crash in Digikam::DImg::load() | ||
---|---|---|---|
Product: | [Applications] digikam | Reporter: | K D Murray <foss> |
Component: | Plugin-DImg-RAW | Assignee: | Digikam Developers <digikam-bugs-null> |
Status: | RESOLVED FIXED | ||
Severity: | crash | CC: | caulier.gilles, metzpinguin |
Priority: | NOR | ||
Version: | 7.2.0 | ||
Target Milestone: | --- | ||
Platform: | Debian unstable | ||
OS: | Linux | ||
Latest Commit: | Version Fixed In: | 7.2.0 | |
Sentry Crash Report: |
Description
K D Murray
2020-09-16 11:55:03 UTC
Well, some QStrings are put on the stack by DRawInfo. Some strings may be significant in size. What does the output of "ulimit -s" show in your system? Maik Hi Maik $ ulimit -s 8192 It would seem that $ ulimit -s 65535 $ digikam makes it not crash. I'll update/reopen this bug report if it starts crashing again. It would be great if digikam could increase the stack size, e.g. using setrlimit on linux if needing a larger stack size is a known issue. I'm happy to provide a patch if this is something you'd like included. Cheers, Kevin I open the bug again, a stack size of 8MB is a normal value for Linux and other operating systems (MacOS) are much less available. We should either create DRawInfo on the heap or find out which string that is passed by libraw is extremely long here. Can you provide the image (029A0172.CR2) that leads to the crash? Maik Git commit 2601c0a8adbd51a3828353e24f5bc169260eb850 by Gilles Caulier. Committed on 19/09/2020 at 10:27. Pushed by cgilles into branch 'master'. Use heap to create DRawInfo instance to prevent stack overflow Related: bug 426175 M +3 -3 core/libs/threadimageio/preview/previewtask.cpp https://invent.kde.org/graphics/digikam/commit/2601c0a8adbd51a3828353e24f5bc169260eb850 Git commit 2ed30b67282e725671e65f5499c6293dc0de7070 by Gilles Caulier. Committed on 19/09/2020 at 10:50. Pushed by cgilles into branch 'master'. Use heap to create DRawInfo instance to prevent stack overflow Related: bug 426175 M +20 -18 core/dplugins/generic/tools/htmlgallery/generator/galleryelementfunctor.cpp https://invent.kde.org/graphics/digikam/commit/2ed30b67282e725671e65f5499c6293dc0de7070 Git commit edf82fbaf85e4131b3e39e4f787e9820a1e273ee by Gilles Caulier. Committed on 19/09/2020 at 10:52. Pushed by cgilles into branch 'master'. Use heap to create DRawInfo instance to prevent stack overflow Related: bug 426175 M +5 -5 core/dplugins/dimg/raw/dimgrawloader.cpp https://invent.kde.org/graphics/digikam/commit/edf82fbaf85e4131b3e39e4f787e9820a1e273ee Git commit f7521ff595caaee5a85fdd75354ac78a753e39dd by Gilles Caulier. Committed on 19/09/2020 at 10:54. Pushed by cgilles into branch 'master'. Use heap to create DRawInfo instance to prevent stack overflow Related: bug 426175 M +80 -80 core/libs/metadataengine/dmetadata/dmetadata_libraw.cpp https://invent.kde.org/graphics/digikam/commit/f7521ff595caaee5a85fdd75354ac78a753e39dd Git commit e662dbc0caea17d673cd84966655fe29fcc50828 by Gilles Caulier. Committed on 19/09/2020 at 10:58. Pushed by cgilles into branch 'master'. Use heap to create DRawInfo instance to prevent stack overflow Related: bug 426175 M +1 -1 core/libs/rawengine/drawdecoder.cpp M +12 -8 core/libs/rawengine/drawdecoder_p.cpp https://invent.kde.org/graphics/digikam/commit/e662dbc0caea17d673cd84966655fe29fcc50828 Git commit 605e6875ca4386291a0b681ec26db9db0b305757 by Gilles Caulier. Committed on 19/09/2020 at 11:18. Pushed by cgilles into branch 'master'. Use heap to create DRawInfo instance to prevent stack overflow Related: bug 426175 M +99 -99 core/libs/dngwriter/dngwriter_convert.cpp M +2 -0 core/libs/dngwriter/dngwriter_p.h https://invent.kde.org/graphics/digikam/commit/605e6875ca4386291a0b681ec26db9db0b305757 Git commit 96bc53a0682968ad3df03c698bf3e9bacb3a665d by Gilles Caulier. Committed on 19/09/2020 at 11:25. Pushed by cgilles into branch 'master'. Use heap to create DRawInfo instance to prevent stack overflow Related: bug 426175 M +2 -1 core/dplugins/dimg/raw/dimgrawloader.cpp M +4 -2 core/dplugins/generic/tools/htmlgallery/generator/galleryelementfunctor.cpp M +2 -1 core/libs/metadataengine/dmetadata/dmetadata_libraw.cpp M +2 -1 core/libs/rawengine/drawdecoder_p.cpp M +2 -1 core/libs/threadimageio/preview/previewtask.cpp M +16 -13 core/tests/rawengine/raw2png.cpp https://invent.kde.org/graphics/digikam/commit/96bc53a0682968ad3df03c698bf3e9bacb3a665d Git commit a1deb248f5e457b5abef17d207cb16daef1237b9 by Gilles Caulier. Committed on 19/09/2020 at 11:36. Pushed by cgilles into branch 'master'. Use heap to create DRawDecoder instance to prevent stack overflow Related: bug 426175 M +1 -1 core/libs/progressmanager/workingwidget.cpp M +3 -2 core/libs/rawengine/drawdecoder.cpp M +1 -0 core/libs/threadimageio/preview/previewtask.cpp M +19 -6 core/tests/multithreading/myactionthread.cpp https://invent.kde.org/graphics/digikam/commit/a1deb248f5e457b5abef17d207cb16daef1237b9 Git commit 0507f9712e4cd76bcbe3aa4cec690ecb25e3fe9e by Gilles Caulier. Committed on 19/09/2020 at 11:42. Pushed by cgilles into branch 'master'. Use heap to create DMetadata instance to prevent stack overflow Related: bug 426175 M +2 -2 core/app/main/digikamapp.cpp M +1 -0 core/app/main/digikamapp_p.h M +11 -10 core/app/views/stack/mapwidgetview.cpp https://invent.kde.org/graphics/digikam/commit/0507f9712e4cd76bcbe3aa4cec690ecb25e3fe9e Kevin, Please check if with my last commits to instantiate DRawInfo and DRawDecoder on heap, your memory allocation problem is fixed. Thanks in advance Gilles Caulier Gilles, Many thanks for the patches. It does seem to have fixed the immediate cause of my crash. However, now with ASAN on, I'm getting a buffer overflow in LibRaw. Crash below, not sure why the line numbers aren't showing, i'm using CMAKE_BUILD_TYPE=Debug. Cheers, Kevin ==948374==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fffa8693892 at pc 0x7ffff2353fbf bp 0x7fffa86937d0 sp 0x7fffa86937c8 READ of size 1 at 0x7fffa8693892 thread T55 (Thread (pooled)) #0 0x7ffff2353fbe in LibRaw::tiff_set(tiff_hdr*, unsigned short*, unsigned short, unsigned short, int, int) [clone .constprop.0] (/home/kevin/.homedir/opt/digikam/lib/x86_64-linux-gnu/libdigikamcore.so.7.2.0+0x15bffbe) #1 0x7ffff2355857 in LibRaw::tiff_head(tiff_hdr*, int) (/home/kevin/.homedir/opt/digikam/lib/x86_64-linux-gnu/libdigikamcore.so.7.2.0+0x15c1857) #2 0x7ffff231acc3 in LibRaw::dcraw_make_mem_thumb(int*) (/home/kevin/.homedir/opt/digikam/lib/x86_64-linux-gnu/libdigikamcore.so.7.2.0+0x1586cc3) #3 0x7ffff237f9dd in Digikam::DRawDecoder::Private::loadEmbeddedPreview(QByteArray&, LibRaw*) (/home/kevin/.homedir/opt/digikam/lib/x86_64-linux-gnu/libdigikamcore.so.7.2.0+0x15eb9dd) #4 0x7ffff236feb4 in Digikam::DRawDecoder::loadEmbeddedPreview(QByteArray&, QString const&) (/home/kevin/.homedir/opt/digikam/lib/x86_64-linux-gnu/libdigikamcore.so.7.2.0+0x15dbeb4) #5 0x7ffff236f3f0 in Digikam::DRawDecoder::loadEmbeddedPreview(QImage&, QString const&) (/home/kevin/.homedir/opt/digikam/lib/x86_64-linux-gnu/libdigikamcore.so.7.2.0+0x15db3f0) #6 0x7ffff212325f in Digikam::ThumbnailCreator::createThumbnail(Digikam::ThumbnailInfo const&, QRect const&) const (/home/kevin/.homedir/opt/digikam/lib/x86_64-linux-gnu/libdigikamcore.so.7.2.0+0x138f25f) #7 0x7ffff2117760 in Digikam::ThumbnailCreator::load(Digikam::ThumbnailIdentifier const&, QRect const&, bool) const (/home/kevin/.homedir/opt/digikam/lib/x86_64-linux-gnu/libdigikamcore.so.7.2.0+0x1383760) #8 0x7ffff211646c in Digikam::ThumbnailCreator::load(Digikam::ThumbnailIdentifier const&) const (/home/kevin/.homedir/opt/digikam/lib/x86_64-linux-gnu/libdigikamcore.so.7.2.0+0x138246c) #9 0x7ffff213906f in Digikam::ThumbnailLoadingTask::execute() (/home/kevin/.homedir/opt/digikam/lib/x86_64-linux-gnu/libdigikamcore.so.7.2.0+0x13a506f) #10 0x7ffff213bee2 in Digikam::LoadSaveThread::run() (/home/kevin/.homedir/opt/digikam/lib/x86_64-linux-gnu/libdigikamcore.so.7.2.0+0x13a7ee2) #11 0x7ffff219e15a in Digikam::DynamicThread::Private::run() (/home/kevin/.homedir/opt/digikam/lib/x86_64-linux-gnu/libdigikamcore.so.7.2.0+0x140a15a) #12 0x7fffefa64691 (/lib/x86_64-linux-gnu/libQt5Core.so.5+0xcc691) #13 0x7fffefa60a00 (/lib/x86_64-linux-gnu/libQt5Core.so.5+0xc8a00) #14 0x7fffef5caea6 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x8ea6) #15 0x7fffef6e7eae in __clone (/lib/x86_64-linux-gnu/libc.so.6+0xfdeae) Address 0x7fffa8693892 is located in stack of thread T55 (Thread (pooled)) at offset 50 in frame #0 0x7ffff235487f in LibRaw::tiff_head(tiff_hdr*, int) (/home/kevin/.homedir/opt/digikam/lib/x86_64-linux-gnu/libdigikamcore.so.7.2.0+0x15c087f) This frame has 2 object(s): [48, 50) 'latref' (line 123) <== Memory access at offset 50 overflows this variable [64, 66) 'lonref' (line 124) HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork (longjmp and C++ exceptions *are* supported) Thread T55 (Thread (pooled)) created by T0 here: #0 0x7ffff76202a2 in __interceptor_pthread_create (/lib/x86_64-linux-gnu/libasan.so.6+0x552a2) #1 0x7fffefa604da in QThread::start(QThread::Priority) (/lib/x86_64-linux-gnu/libQt5Core.so.5+0xc84da) SUMMARY: AddressSanitizer: stack-buffer-overflow (/home/kevin/.homedir/opt/digikam/lib/x86_64-linux-gnu/libdigikamcore.so.7.2.0+0x15bffbe) in LibRaw::tiff_set(tiff_hdr*, unsigned short*, unsigned short, unsigned short, int, int) [clone .constprop.0] Shadow bytes around the buggy address: 0x1000750ca6c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1000750ca6d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1000750ca6e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1000750ca6f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1000750ca700: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1 =>0x1000750ca710: f1 f1[02]f2 02 f3 f3 f3 00 00 00 00 00 00 00 00 0x1000750ca720: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 00 02 f2 f2 0x1000750ca730: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1000750ca740: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1000750ca750: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1000750ca760: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==948374==ABORTING Ha, this part of code is just imported from libraw project as well in digiKam core. The trace is clear : this memory corruption is done when you extract a preview image from Raw, not while Raw demosaicing. For this kind of ASAN report, please : 1/ try to identify which Raw image introduce this memory leak, 2/ Use libraw CLi tool to extract preview image. No need to checkout libraw code outside digiKam, we compile all libraw CLI tool in digiKam tests suite. You will found all CLI tools in your build directory. Right tool is this one with "-e" option : [gilles@pc-gilles rawengine]$ pwd /mnt/devel/GIT/7.x/build/core/tests/rawengine [gilles@pc-gilles rawengine]$ ./simple_dcraw simple_dcraw - LibRaw 0.20.0-Release sample. Emulates dcraw [-D] [-T] [-v] [-e] [-E] 1134 cameras supported Usage: ./simple_dcraw [-D] [-T] [-v] [-e] raw-files.... -4 - 16-bit mode -L - list supported cameras and exit -v - verbose output -T - output TIFF files instead of .pgm/ppm -e - extract thumbnails (same as dcraw -e in separate run) [gilles@pc-gilles rawengine]$ 3/ Look if ASAN dysfunction still exists 4/ Report this upstream problem to libraw team : https://github.com/LibRaw/LibRaw/issues Thanks in advance Gilles Caulier OK, thanks Gilles, will do.
Any hints you have to easily try step 1?
> 1/ try to identify which Raw image introduce this memory leak,
Cheers,
Kevin
Git commit 0815e2d88a9d3c69bca6f6fb8e0500a04c01d863 by Gilles Caulier. Committed on 19/09/2020 at 14:03. Pushed by cgilles into branch 'master'. More debug trace to identfy raw file processed with preview extraction M +3 -1 core/libs/rawengine/drawdecoder.cpp https://invent.kde.org/graphics/digikam/commit/0815e2d88a9d3c69bca6f6fb8e0500a04c01d863 First it's clear, raw preview is only called with RAW files. If you use one kind of camera to generate RAW, well it's simple. try with some RAW sample from your collection. NEF for ex are based on TIFF/EP with Nikon customization. DNG is a pure TIFF/EP format too. Else, the RAW preview is called to render thumbnails (your case here) or to show JPEG embedded preview when you press F3 in icon view. And yes, RAW container has smaller JPEG version inside. This permit to show quickly the RAW content with to process a complex demosaicing. To identify the file, the Preview engine, which call libraw API must print the file in process on the console. You need to enable all debug traces on the console with this env. variable before to run digiKam : export QT_LOGGING_RULES="digikam*=true" With my last commit you must see something like this: digikam.rawengine: LibRaw: loadEmbeddedPreview from "/mnt/data/photos/GILLES/NEW/HDR/img_1720.cr2" Gilles Caulier OK, so now I get a possibly related issue: Clicking on any CR2 leads to "Failed to load image" message in GUI, and this in console with debug logging on: ``` digikam.general: Try to get preview from "/home/kevin/photos/library/2020/2020-09-15_tidbinbilla/2020-09-15_12-44-37_029A0472.CR2" digikam.general: Preview quality: 2 digikam.dimg: "/home/kevin/photos/library/2020/2020-09-15_tidbinbilla/2020-09-15_12-44-37_029A0472.CR2" : Unknown image format !!! digikam.general: Cannot extract preview for "/home/kevin/photos/library/2020/2020-09-15_tidbinbilla/2020-09-15_12-44-37_029A0472.CR2" digikam.general: Stacked View Mode : 1 ``` Also seems as though it's failing to open or even create thumbnails of any JPEG. Cheers, K Git commit c96ede534820d3037aac1cdb1c65ad3b49db4b03 by Gilles Caulier. Committed on 20/09/2020 at 10:48. Pushed by cgilles into branch 'master'. backport libraw source code from git/master rev. 3f701019d5abb44565229d5036ba0bf41a2d57a3 M +1 -1 core/libs/rawengine/libraw/samples/dcraw_emu.cpp M +24 -29 core/libs/rawengine/libraw/samples/raw-identify.cpp M +1 -1 core/libs/rawengine/libraw/src/decoders/load_mfbacks.cpp M +1 -1 core/libs/rawengine/libraw/src/metadata/canon.cpp M +2 -2 core/libs/rawengine/libraw/src/metadata/ciff.cpp M +11 -1 core/libs/rawengine/libraw/src/metadata/cr3_parser.cpp M +19 -0 core/libs/rawengine/libraw/src/metadata/exif_gps.cpp M +26 -13 core/libs/rawengine/libraw/src/metadata/hasselblad_model.cpp M +11 -8 core/libs/rawengine/libraw/src/metadata/identify.cpp M +6 -4 core/libs/rawengine/libraw/src/metadata/misc_parsers.cpp M +1 -5 core/libs/rawengine/libraw/src/metadata/normalize_model.cpp M +1 -1 core/libs/rawengine/libraw/src/metadata/olympus.cpp M +5 -4 core/libs/rawengine/libraw/src/preprocessing/raw2image.cpp M +3 -3 core/libs/rawengine/libraw/src/tables/cameralist.cpp M +2 -2 core/libs/rawengine/libraw/src/write/file_write.cpp https://invent.kde.org/graphics/digikam/commit/c96ede534820d3037aac1cdb1c65ad3b49db4b03 Kevin, About comment #19, i cannot reproduce this problem with CR2 files: https://i.imgur.com/iFvqybj.jpg ...and JPEG: https://i.imgur.com/X2FB6sj.png Thumbnails, preview, ad loading in editor work as expected... Gilles Caulier Git commit ef51f605528649e9f509bd37d2710409f3bf83f0 by Gilles Caulier. Committed on 20/09/2020 at 21:32. Pushed by cgilles into branch 'master'. Add new compilation option to enable compiler sanitizers ASAN and UBSAN Related: bug 426175 M +1 -0 Mainpage.dox M +6 -1 core/CMakeLists.txt M +3 -1 core/cmake/modules/MacroCompiler.cmake https://invent.kde.org/graphics/digikam/commit/ef51f605528649e9f509bd37d2710409f3bf83f0 Hi Gilles, Do you mind sharing the OS/library versions/etc you use in your screenshots? Maybe my issue is triggered by some broken dependency from debian I can work around. Or are there nightly appimage bundles that would include these fixes? And regarding the asan issue: yes it appears fixed after the patch from upstream. Thanks to you/them for that :) Cheers, Kevin Hi Kevin, I use Linux Mageia7. My dependencies are visible on dialog from the right screen side : https://i.imgur.com/iFvqybj.jpg Else the Nightly AppImage bundle builds are available here : https://files.kde.org/digikam/ I close this file now. Note that i officialy add the ASAN support in digiKam build rules. A new cmake option "ENABLE_SANITIZERS" can be used for that: https://invent.kde.org/graphics/digikam/-/blob/master/Mainpage.dox#L461 Best regards Gilles Caulier Gilles |