I have to compress a dir with ~275k files. The files themselves are quite small, but they're numerous. When right clicking the dir to bring up the context menu, Dolphin hangs. When entering the dir, selecting all files (Ctrl + A) and right clicking the selection to bring up the context menu, Dolphin hangs. Expected: Dolphin doesn't hang This might look a bit like an edge case, but I think KDE software should be efficient enough to handle it, it's a situation that can happen IRL. I also dunno what's the limit to trigger this, maybe a fraction of this would already be enough. To reproduce, for example: #!/bin/bash for i in $(seq 0 275000); do dd if=/dev/urandom of=/tmp/file-gen/file-$i.bin bs=4K count=1 iflag=fullblock done Operating System: openSUSE Tumbleweed 20250130 KDE Plasma Version: 6.2.5 KDE Frameworks Version: 6.10.0 Qt Version: 6.8.1 Kernel Version: 6.13.0-1-default (64-bit) Graphics Platform: Wayland Processors: 16 × AMD Ryzen 7 7840U w/ Radeon 780M Graphics Memory: 30.7 GiB of RAM Graphics Processor: AMD Radeon 780M Manufacturer: Framework Product Name: Laptop 13 (AMD Ryzen 7040Series) System Version: A7
Sorting the dir by file size is also slow and turns Dolphin into "Not Responding" after a few seconds
If you read until this point, now try to tar or zip without compression (zip -r 0) all those files, then open the archive with Ark, and you'll be in for a surprise... tar and zip are quite fast in CLI (tar without gz is basically instant, or takes a couple of seconds; zip takes 10 or 20 seconds here). But when it comes to opening the archives in Ark, it hangs eventually. Archives are 1.1 to 1.4GB in size. I'll file a separate report for that on Ark.
(In reply to Andrea Ippolito from comment #2) > If you read until this point, now try to tar or zip without compression (zip > -r 0) all those files, then open the archive with Ark, and you'll be in for > a surprise... > > tar and zip are quite fast in CLI (tar without gz is basically instant, or > takes a couple of seconds; zip takes 10 or 20 seconds here). But when it > comes to opening the archives in Ark, it hangs eventually. Archives are 1.1 > to 1.4GB in size. > > I'll file a separate report for that on Ark. I mean, technically it doesn't hang, it just takes so long that "KDE" considers it to be not responding. But the status bar actually reports an increase in progress (now I'm at 74% after more than 5', so I'll get there eventually)
(In reply to Andrea Ippolito from comment #3) > (In reply to Andrea Ippolito from comment #2) > > If you read until this point, now try to tar or zip without compression (zip > > -r 0) all those files, then open the archive with Ark, and you'll be in for > > a surprise... > > > > tar and zip are quite fast in CLI (tar without gz is basically instant, or > > takes a couple of seconds; zip takes 10 or 20 seconds here). But when it > > comes to opening the archives in Ark, it hangs eventually. Archives are 1.1 > > to 1.4GB in size. > > > > I'll file a separate report for that on Ark. > > I mean, technically it doesn't hang, it just takes so long that "KDE" > considers it to be not responding. But the status bar actually reports an > increase in progress (now I'm at 74% after more than 5', so I'll get there > eventually) Ark eventually opened, took something like 20 minutes - now as expected it's having a hard time scrolling through the contents, as soon as I drag the scroll handle in the sidebar, it goes all white and Not responding
Created attachment 178009 [details] flamegraph Not sure how I can share the full pic This is while opening the context menu for a dir with 30k files, not 275k like in the report. Still takes a few seconds
Comment on attachment 178009 [details] flamegraph this is while opening the context menu on a directory containing 30k 4KB .bin files generated with: #!/bin/bash dd if=/dev/urandom of=/tmp/file-gen/file-random-0.bin bs=4K count=1 iflag=fullblock for i in $(seq 1 30000); do cp /tmp/file-gen/file-random-0.bin /tmp/file-gen/file-random-$i.bin done
(In reply to Andrea Ippolito from comment #6) > Comment on attachment 178009 [details] > flamegraph Very Nice that's helpful. > > this is while opening the context menu on a directory containing 30k 4KB > .bin files generated with: > > #!/bin/bash > > dd if=/dev/urandom of=/tmp/file-gen/file-random-0.bin bs=4K count=1 > iflag=fullblock > for i in $(seq 1 30000); > do > cp /tmp/file-gen/file-random-0.bin /tmp/file-gen/file-random-$i.bin > done You are missing some debug symbols but that's interesting already. We can see the time is spent by the context menu plugin slideshowfileitemaction which is part of gwenview and not enabled by default IIRC. It is clearly due to: https://invent.kde.org/graphics/gwenview/-/blob/master/app/slideshowfileitemaction.cpp?ref_type=heads#L49 There are no images in the folder and the files are of random type. This can be optimized, it uses `db.mimeTypeForFile` which opens files when it can't determine the mimetype by extension. It should use filename extension only.
(In reply to Méven from comment #7) > (In reply to Andrea Ippolito from comment #6) > > Comment on attachment 178009 [details] > > flamegraph > > Very Nice that's helpful. > > > > this is while opening the context menu on a directory containing 30k 4KB > > .bin files generated with: > > > > #!/bin/bash > > > > dd if=/dev/urandom of=/tmp/file-gen/file-random-0.bin bs=4K count=1 > > iflag=fullblock > > for i in $(seq 1 30000); > > do > > cp /tmp/file-gen/file-random-0.bin /tmp/file-gen/file-random-$i.bin > > done > > You are missing some debug symbols but that's interesting already. > We can see the time is spent by the context menu plugin > slideshowfileitemaction which is part of gwenview and not enabled by default > IIRC. > > It is clearly due to: > https://invent.kde.org/graphics/gwenview/-/blob/master/app/ > slideshowfileitemaction.cpp?ref_type=heads#L49 > > There are no images in the folder and the files are of random type. > This can be optimized, it uses `db.mimeTypeForFile` which opens files when > it can't determine the mimetype by extension. That's not true it uses `QMimeDatabase::MatchExtension`. > It should use filename extension only. The optimization would be to limit the time spent in this directory scanning. If there is 1 image among 10 000 non-image files or ~275k we don't really care. > This might look a bit like an edge case, but I think KDE software should be efficient enough to handle it, it's a situation that can happen IRL. 275K not that often. I am guessing that can happen in some research fields or development, data mining and other cases which we want to enable of course.
(In reply to Méven from comment #7) > (In reply to Andrea Ippolito from comment #6) > > Comment on attachment 178009 [details] > > flamegraph > > Very Nice that's helpful. > > > > this is while opening the context menu on a directory containing 30k 4KB > > .bin files generated with: > > > > #!/bin/bash > > > > dd if=/dev/urandom of=/tmp/file-gen/file-random-0.bin bs=4K count=1 > > iflag=fullblock > > for i in $(seq 1 30000); > > do > > cp /tmp/file-gen/file-random-0.bin /tmp/file-gen/file-random-$i.bin > > done > > You are missing some debug symbols but that's interesting already. > We can see the time is spent by the context menu plugin > slideshowfileitemaction which is part of gwenview and not enabled by default > IIRC. > > It is clearly due to: > https://invent.kde.org/graphics/gwenview/-/blob/master/app/ > slideshowfileitemaction.cpp?ref_type=heads#L49 > > There are no images in the folder and the files are of random type. > This can be optimized, it uses `db.mimeTypeForFile` which opens files when > it can't determine the mimetype by extension. > It should use filename extension only. Hey! That's some quick reaction and investigation, thanks :) Indeed as soon as I disabled that context menu entry (and restarted dolphin), the context menu opened instantly on the directory. This entry must come from my distro (opensuse tumbleweed), because I haven't customized the context menu entries, and also it comes back if I click on the "Defaults" button. I don't know if that improvement on the gwenview plugin is something you can report, or if I have to file a separate bug for that (it's just that a bug filed by me would probably just be a link to some comments we exchanged here, while maybe you can provide more useful info - although these comments are probably enough already to get the ball rolling? Please let me know, np for me to file another bug :) ) Now another point I want to mention though is that disabling that entry still doesn't solve the problem when invoking the context menu on the "select all" file selection, which still takes long. I'm collecting a dump/flamegraph for this as well, and will report back.
Created attachment 178012 [details] flamegraph context menu on large file selection This is to show the context menu on a selection of 5000 flat files (4KB .bin), no sub-dirs Seems to be related to mime types again (triggered by the compress/extract actions)?
(In reply to Andrea Ippolito from comment #10) > Created attachment 178012 [details] > flamegraph context menu on large file selection > > This is to show the context menu on a selection of 5000 flat files (4KB > .bin), no sub-dirs > > Seems to be related to mime types again (triggered by the compress/extract > actions)? Yes that's a different issue in ark in CompressFileItemAction and ExtractFileItemAction that scan all the files. They will probably need some similar work to limit the determineMimeType calls: https://invent.kde.org/utilities/ark/-/blob/master/app/compressfileitemaction.cpp?ref_type=heads https://invent.kde.org/utilities/ark/-/blob/master/app/extractfileitemaction.cpp?ref_type=heads It exposes two issues with the API, the plugins can impact the application (there run in main thread) and those mimetypes have already be determined by dolphin when it loaded the folder, so they could share this information.
(In reply to Andrea Ippolito from comment #10) > Created attachment 178012 [details] > flamegraph context menu on large file selection > > This is to show the context menu on a selection of 5000 flat files (4KB > .bin), no sub-dirs > > Seems to be related to mime types again (triggered by the compress/extract > actions)? Disabling Ark's compress and extract context menu entries improved the situation drastically (a few seconds before VS barely over 1 second now)(In reply to Méven from comment #11) > (In reply to Andrea Ippolito from comment #10) > > Created attachment 178012 [details] > > flamegraph context menu on large file selection > > > > This is to show the context menu on a selection of 5000 flat files (4KB > > .bin), no sub-dirs > > > > Seems to be related to mime types again (triggered by the compress/extract > > actions)? > > Yes that's a different issue in ark in CompressFileItemAction and > ExtractFileItemAction that scan all the files. > They will probably need some similar work to limit the determineMimeType > calls: > https://invent.kde.org/utilities/ark/-/blob/master/app/ > compressfileitemaction.cpp?ref_type=heads > https://invent.kde.org/utilities/ark/-/blob/master/app/extractfileitemaction. > cpp?ref_type=heads > > It exposes two issues with the API, the plugins can impact the application > (there run in main thread) and those mimetypes have already be determined by > dolphin when it loaded the folder, so they could share this information. Cool, thanks! What's the best way to report this to Ark/Gwenview in your opinion?
(In reply to Andrea Ippolito from comment #12) > (In reply to Andrea Ippolito from comment #10) > > Created attachment 178012 [details] > > flamegraph context menu on large file selection > > > > This is to show the context menu on a selection of 5000 flat files (4KB > > .bin), no sub-dirs > > > > Seems to be related to mime types again (triggered by the compress/extract > > actions)? > > Disabling Ark's compress and extract context menu entries improved the > situation drastically (a few seconds before VS barely over 1 second now)(In > reply to Méven from comment #11) > > (In reply to Andrea Ippolito from comment #10) > > > Created attachment 178012 [details] > > > flamegraph context menu on large file selection > > > > > > This is to show the context menu on a selection of 5000 flat files (4KB > > > .bin), no sub-dirs > > > > > > Seems to be related to mime types again (triggered by the compress/extract > > > actions)? > > > > Yes that's a different issue in ark in CompressFileItemAction and > > ExtractFileItemAction that scan all the files. > > They will probably need some similar work to limit the determineMimeType > > calls: > > https://invent.kde.org/utilities/ark/-/blob/master/app/ > > compressfileitemaction.cpp?ref_type=heads > > https://invent.kde.org/utilities/ark/-/blob/master/app/extractfileitemaction. > > cpp?ref_type=heads > > > > It exposes two issues with the API, the plugins can impact the application > > (there run in main thread) and those mimetypes have already be determined by > > dolphin when it loaded the folder, so they could share this information. This is the main issue and it is in KIO / widgets, but might take a long time to fix as it might require an API breakage. > > Cool, thanks! What's the best way to report this to Ark/Gwenview in your > opinion? Just like you did with this bug for dolphin. Please keep a link or a "relates to" reference in the new bugs.
See also https://invent.kde.org/frameworks/kio/-/merge_requests/1813
Thanks Méven, I reported and linked 499595 - will try to find some time to report one for the gwenview plugin as well this week
(In reply to Méven Car from comment #14) > See also https://invent.kde.org/frameworks/kio/-/merge_requests/1813 oh it was See also, completely missed that and used the URL field to link them instead Thanks for fixing :)