Bug 454206 - Slow performance with RAW files on big folders
Summary: Slow performance with RAW files on big folders
Status: CONFIRMED
Alias: None
Product: gwenview
Classification: Applications
Component: general (show other bugs)
Version: 23.04.3
Platform: Other Linux
: NOR normal
Target Milestone: ---
Assignee: Gwenview Bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-05-22 11:58 UTC by aronkvh
Modified: 2024-10-06 18:36 UTC (History)
5 users (show)

See Also:
Latest Commit:
Version Fixed In:
Sentry Crash Report:


Attachments
Patch to use KDCraw over QtRaw (2.71 KB, patch)
2024-07-24 11:48 UTC, Martin Kyral
Details

Note You need to log in before you can comment on or make changes to this bug.
Description aronkvh 2022-05-22 11:58:48 UTC
I have a super-large folder (81GB) with many .arw and png/jpeg/mp4 image files. Images can be opened relatively quickly, but then Gwenview starts scanning all the 9k files in the directory which makes it unresponsible for minutes and sometimes freezes. 
It'd be great if you could browse the previous/next images in a way that only scans a portion of the files at first to avoid the app crashing. 

Hard drive: TOSHIBA DT01ACA300
Operating System: KDE neon 5.24
KDE Plasma Version: 5.24.5
KDE Frameworks Version: 5.93.0
Qt Version: 5.15.3
Kernel Version: 5.13.0-41-generic (64-bit)
Graphics Platform: X11
Processors: 8 × Intel® Core™ i7-3770 CPU @ 3.40GHz
Memory: 27,4 GiB of RAM
Graphics Processor: NVIDIA GeForce RTX 3060/PCIe/SSE2
(windows explorer/Quicklook can handle the folders quicker and with less freezes, so probably not a hardware problem)
Comment 1 Andrej Halveland 2023-08-16 12:35:44 UTC
I have the same issue... loading folders with few hundred RAW photos, or even a single (1) RAW photo takes ages to load. Loading just one RAW image takes about 7 seconds to load, and while it's loading, the Gwenview process has CPU usage of 300%. On the other hand, something like Geeqie loads RAW images (even hundreds of images) basically INSTANTLY. Gwenview is just unusable for RAW image viewing.

Even digiKam is significantly faster than Gwenview, but not as fast as Geeqie.

The RAW photos in question are Canon's .CR2

Loading JPEG images is fine.

SOFTWARE/OS VERSIONS
Operating System: Arch Linux
KDE Plasma Version: 5.27.7
KDE Frameworks Version: 5.108.0
Qt Version: 5.15.10
Kernel Version: 6.5-rc5
Graphics Platform: Wayland
Graphics Drivers: mesa-git (23.3.0_devel.176075.8c62b3938a8.d41d8cd-1)
Processors: AMD Ryzen 9 4900HS
Memory:  24GB of RAM
Graphics Processor: Integrated: Vega 8
Manufacturer: ASUS (Zephyrus G15 GA502IV)
Comment 2 Martin Kyral 2023-11-14 18:12:29 UTC
Well, I just guess but with a grain of knowledge as I implemented the original RAW support in gwenview back in 2013:

 * the original KDCRaw-based implementation used the preview image bundled in the raw file itself. That's fast as it's only reading the already prepared preview data and treating them as jpeg (you can get the preview using dcraw -e file.raw).
 * it seems to me that alternative implementation using Qt Raw plugin and it is being preferred
 * from quick peek at the code I assume, that with Qt Raw plugin the trick with using the preview that's aready present is not used (I am not sure if Qt Raw plugin can do that)
 * demosaicing takes time

I consider this to be a major regression.
Comment 3 Martin Kyral 2023-11-15 15:18:44 UTC
By regression I mean the feature is virtually unusable.
Comment 4 Martin Kyral 2023-11-15 15:48:18 UTC
Another downturn is that while with KDCraw you get the image straight from the camera, with Qt Raw plugin the image looks different, rather dull and underexposed as the demosaicing algorithm and colour science is proprietary to the camera manufacturer and thus hard to replicate. While with full raw converter the user has wide palette of tools and settings to do the colour science and processing to their liking, the number of adjustments provided by gwenview is ZERO and providing them is far out of scope of it.
Comment 5 Mirco Miranda 2023-11-15 17:35:10 UTC
Probably the solution is to insert a configuration in Gwenview to force the use of KDcraw even when the RAW plugin is present. The code to use KDcraw is still present in Gwenview as it is used if, for example, the RAW plugin is not found.
This way everyone does what they want.

Or, completely restore the old code: in this way KDcraw is always used by ignoring the RAW plugin at all.
Comment 6 Martin Kyral 2023-11-15 19:42:45 UTC
The config option seems perfectly fine to me. Let the user decide that's best for them.

The problem is, that KDCraw fucntionality is apparently broken (maybe by kimageformats 5.99) for quite a while. So, the KDCraw code is still present but broken. I tried myself with gwenview patched to prefer KDCraw.

I am looking into the code trying to understand what happens with the data.
Comment 7 Martin Kyral 2023-12-04 08:41:10 UTC
(In reply to Martin Kyral from comment #6)
> The config option seems perfectly fine to me. Let the user decide that's
> best for them.
> 
> The problem is, that KDCraw fucntionality is apparently broken (maybe by
> kimageformats 5.99) for quite a while. So, the KDCraw code is still present
> but broken. I tried myself with gwenview patched to prefer KDCraw.
> 
> I am looking into the code trying to understand what happens with the data.

What happened in kimageformats 5.99? Qt Raw pluging support was implemented. That broke kdcraw pipeline in gwenview, which  has been 'fixed' by forcing gwenview to use the Qt RAw plugin as well, which works with the small caveat that it is by orders of magnitude slower, than what was before, rendering gwenview virtually unsable in some use cases (such as crawling through a folder with large number of raw files, which is very common situation everytime one comes home from a shoot). I suppose, kimageformats Qt Raw plugin code shall be reviewed and fixed so it doesn't break code in gwenview that does not utilize it.
Comment 8 Martin Kyral 2023-12-12 10:35:47 UTC
Just to have a glimpse: I was extracting the embedded hi-res jpeg image using dcraw -e (the method kdcraw uses as well) from 150 raws from 16mpx Olympus. It took under 500ms. Compare it to the Qt Raw plugin which takes several seconds per image.
Comment 9 Martin Kyral 2024-06-11 11:02:18 UTC
After some time I looked at the issue again and got some success: the breakage by the kimageformats update was apparently caused by kimageformats somehow affecting the formatHint value, when I hardcoded the format value to "jpeg" for RAW images processed by the KDCraw pipeline, it works. However, I also needed to backport the patch forcing QtRaw plugin in order to get into the KDCRaw pipeline in the first place. With my rudimentary patch, the RAW preview looks like it used to look: it is fast (loading an image in a split second) and it shows the images as get by the camera. Once again: for most manufacturers, the colour science is proprietary and it gets some tweaking if you want to demosaic the raw data and get acceptable colours (getting the colours as if processed by the camera/manufacturer's software is virtually impossible). 

With this progress, I'd like to ask if it's acceptable/manageable to have the preferred RAW processing pipeline (QtRaw or KDCRaw) configurable by the user. Now, IIUC the switch to KDCRaw pipeline would require recompiling QtImage with QtRAw disabled.
Comment 10 Martin Kyral 2024-06-14 08:10:07 UTC
Build of gwenview patched to prefer KDCRaw (Fedora 40 only):

https://copr.fedorainfracloud.org/coprs/mkyral/gwenview/
Comment 11 Martin Kyral 2024-07-24 11:48:06 UTC
Created attachment 171952 [details]
Patch to use KDCraw over QtRaw

Patch to use KDCraw over QtRaw