Additional observations: renaming a file will trigger the problem again, as will opening the same folder in a split, though it was already open and showing thumbnails. Maybe there's some stable reference to use other than the filename to save effort here? And a better way of linking already generated previews to files to cause less IO when (re)opening a folder?

Also I see it's not just generating a static preview thumbnail, but rather a whole GIF/slideshow preview of the file. Is there a way to make this configurable to only do a single static preview image per file?

> no high CPU/IOwait/Network from kioslave5 (media previews still working)

You may need to explain this expectation in greater detail. How do you reconcile remote-IO-without-wait-time with having to access remote data, as would be the case during preview generation? At face value there's always an unknown amount of wait time, because we don't know network throughput and latency, and then remote drive throughput and latency. Or put simply: the only way not to wait is to not read data; when reading data we need to wait for the IO.

> A naive analysis would seem to suggest there are less invasive and resource intensive ways to generate media previews

Such as?

(In reply to Harald Sitter from comment #3)
> > no high CPU/IOwait/Network from kioslave5 (media previews still working)
> 
> You may need to explain this expectation in greater detail. How do you
> reconcile remote-IO-without-wait-time with having to access remote data, as
> would be the case during preview generation? At face value there's always an
> unknown amount of wait time, because we don't know network throughput and
> latency, and then remote drive throughput and latency. Or put simply: the
> only way not to wait is to not read data; when reading data we need to wait
> for the IO.

I meant the overall system IOwait percentage seems to get rather high (sometimes over 30%) and can sometimes cause other processes to stutter and lag. My main system drive is an NVMe (LVM on LUKS) so it's rather odd that kioslave could be causing that much of an issue. I haven't messed with any kernel parameters or changed the scheduler settings from stock Fedora defaults here. I'll also note that it's not network bandwidth being the issue, but the disk read speed of the remote server (spinning rust, also see high iowait when I checked it during the issue) that causes the local iowait for kioslave.

It also seems weird to me that being bottlenecked on IO to a network filesystem can still cause visible lag for other processes only doing local IO, unless there's a shared IO pipeline for all mounted filesystems? Not sure why that is happening exactly. Either way, reducing the process IO priority could be helpful here (like maybe idle priority) to reduce the impact.

> > A naive analysis would seem to suggest there are less invasive and resource intensive ways to generate media previews
> 
> Such as?

I haven't looked at the code, so my suggestions are based on observed behavior.

- I've already mentioned lowering IO priority so it doesn't push out other processes. I feel that this would be a workload perfectly suited to running at idle priority, since its an ancillary function that shouldn't impair the primary functionality of the program (being a file browser).
- Is there anything we can do to reduce the amount of reading necessary to generate a thumbnail for a video file? Presumably we'd only need to look for the first few keyframes and stop there. Reducing the per-file IO helps immensely with large file counts.
- I also notice that hovering will show a mini preview slideshow, so the program is probably reading a lot further into the file to grab those thumbnails. Maybe this behavior could be made tunable "disable deep thumbnail generation" or something. Maybe it should be off by default for non-local files.
- I mentioned in another comment that renaming or moving a file seems to invalidate the thumbnail and trigger another generation. Could we link the generated thumbnails with a more persistent identifier? File hash may not be useful since that requires reading the file. Maybe the inode? I'm not sure if an equivalent to inodes would even exist for SMB (NFS does expose persistent inodes).
- Is this process descending automatically to child directories to be more thorough? That makes even browsing the level above a large video directory problematic, and it doesn't seem to quit when I'm no longer in the vicinity of that directory tree.

Also, if better profiling, metrics, or debugging would be helpful, let me know what to add.

(In reply to Eddie J Carswell II from comment #4)
> (In reply to Harald Sitter from comment #3)
> > > no high CPU/IOwait/Network from kioslave5 (media previews still working)
> > 
> > You may need to explain this expectation in greater detail. How do you
> > reconcile remote-IO-without-wait-time with having to access remote data, as
> > would be the case during preview generation? At face value there's always an
> > unknown amount of wait time, because we don't know network throughput and
> > latency, and then remote drive throughput and latency. Or put simply: the
> > only way not to wait is to not read data; when reading data we need to wait
> > for the IO.
> 
> I meant the overall system IOwait percentage seems to get rather high
> (sometimes over 30%) and can sometimes cause other processes to stutter and
> lag.

If it is the system io wait that is high, then how can you tell it's kioslave5 causing it?

Also, how are you measuring this?

> My main system drive is an NVMe (LVM on LUKS) so it's rather odd that
> kioslave could be causing that much of an issue. I haven't messed with any
> kernel parameters or changed the scheduler settings from stock Fedora
> defaults here. I'll also note that it's not network bandwidth being the
> issue, but the disk read speed of the remote server (spinning rust, also see
> high iowait when I checked it during the issue) that causes the local iowait
> for kioslave.
> 
> It also seems weird to me that being bottlenecked on IO to a network
> filesystem can still cause visible lag for other processes only doing local
> IO, unless there's a shared IO pipeline for all mounted filesystems? Not
> sure why that is happening exactly.

It is certainly weird, but from what you describe your problem seems to be more with the kernel and its io scheduler than with kioslave5. The solution here really shouldn't be don't-do-IO - that's like saying don't turn your computer on, then you also don't have io wait.

> Either way, reducing the process IO
> priority could be helpful here (like maybe idle priority) to reduce the
> impact.
>
> > > A naive analysis would seem to suggest there are less invasive and resource intensive ways to generate media previews
> > 
> > Such as?
> 
> I haven't looked at the code, so my suggestions are based on observed
> behavior.
> 
> - I've already mentioned lowering IO priority so it doesn't push out other
> processes. I feel that this would be a workload perfectly suited to running
> at idle priority, since its an ancillary function that shouldn't impair the
> primary functionality of the program (being a file browser).

idle priority unfortunately won't work because it is subject to starvation. Users do expect previews to be generated eventually.

> - Is there anything we can do to reduce the amount of reading necessary to
> generate a thumbnail for a video file?

Doesn't look like it. We already read very little.

> - I also notice that hovering will show a mini preview slideshow, so the
> program is probably reading a lot further into the file to grab those
> thumbnails. Maybe this behavior could be made tunable "disable deep
> thumbnail generation" or something. Maybe it should be off by default for
> non-local files.

We jump through the file as a matter of fact, it's fairly cheap. There already is a setting for disabling previews on remotes, the sequence generation isn't nearly expensive enough to justify yet another setting.

> - I mentioned in another comment that renaming or moving a file seems to
> invalidate the thumbnail and trigger another generation. Could we link the
> generated thumbnails with a more persistent identifier? File hash may not be
> useful since that requires reading the file. Maybe the inode? I'm not sure
> if an equivalent to inodes would even exist for SMB (NFS does expose
> persistent inodes).

How do you rename the file? Via dolphin or terminal?

> - Is this process descending automatically to child directories to be more
> thorough? That makes even browsing the level above a large video directory
> problematic, and it doesn't seem to quit when I'm no longer in the vicinity
> of that directory tree.

When you have directory previews enabled, then yes it'd naturally have to create previews for the first couple files in subdirectories. Otherwise not.

(In reply to Harald Sitter from comment #5)
> If it is the system io wait that is high, then how can you tell it's
> kioslave5 causing it?
> 
> Also, how are you measuring this?

The system iowait percentage is normally less than 1% on my system, sometimes reaching higher in the single digits when doing tasks. I only see really high numbers when I do something like write a 10GB+ file with no throttling (dd and friends) or when some process is misbehaving. You can see this value with `top` as `wa` and with `iostat` as `iowait`.

> It is certainly weird, but from what you describe your problem seems to be
> more with the kernel and its io scheduler than with kioslave5. The solution
> here really shouldn't be don't-do-IO - that's like saying don't turn your
> computer on, then you also don't have io wait.

The issue is not so much trying to stop IO, but ensuring it doesn't put undue load (pressure?) on the rest of the system while doing it's job.

> idle priority unfortunately won't work because it is subject to starvation.
> Users do expect previews to be generated eventually.

Then perhaps somewhat higher but still less than normal priority? Like best effort priority 7? I still think it shouldn't have the default priority of best effort 0. Anything lower than the default would ensure it yields to other processes with the default or higher priority, yet won't face starvation like idle might.

> We jump through the file as a matter of fact, it's fairly cheap. There
> already is a setting for disabling previews on remotes, the sequence
> generation isn't nearly expensive enough to justify yet another setting.

Ah, found that setting. It doesn't seem to account for filesystems mounted via fstab unfortunately. Not sure how much extra work it would be to check if we're traversing a mount point not connected via a local block device. And I've found performance using `smb://` in dolphin to be worse than just using an fstab automount, which is why I have it setup that way.

> How do you rename the file? Via dolphin or terminal?

This was done in dolphin, both renaming and moving video files. I haven't tested the behavior with a terminal rename. Would that be useful?

> When you have directory previews enabled, then yes it'd naturally have to
> create previews for the first couple files in subdirectories. Otherwise not.

I don't suppose there's an option I'm missing to disable directory previews so they only generate on files in the current directory? Then I can avoid certain folders when browsing via dolphin to ensure they don't trigger this issue. It would be nice to have a way to filter specific paths (maybe even with wildcards) from preview generation, but that's already requested in bug 406725.

After putting a wrapper around it, it seems to behaving itself quite nicely. Thumbnail generation is somewhat slower, but it doesn't seem problematic anymore. I had it using idle priority in this case, but I could test it with other combinations.

> $  cat /usr/libexec/kf5/kioslave5
> #!/bin/sh
> exec nice -n 19 ionice -c 3 $0.orig "$@"
> $  ll /usr/libexec/kf5/kioslave5*
> -rwxr-xr-x. 1 root root    51 Dec 14 23:23 /usr/libexec/kf5/kioslave5
> -rwxr-xr-x. 1 root root 16088 Oct  9 20:00 /usr/libexec/kf5/kioslave5.orig

We're back to single digit iowait now even with the thumbnail generation running on a large remote directory.

(In reply to Eddie J Carswell II from comment #7)
> We're back to single digit iowait now even with the thumbnail generation
> running on a large remote directory.

You've basically taken away the CPU weight from all KIO IO by doing this.

(In reply to Eddie J Carswell II from comment #6)
> > idle priority unfortunately won't work because it is subject to starvation.
> > Users do expect previews to be generated eventually.
> 
> Then perhaps somewhat higher but still less than normal priority?

After some research I'm even less convinced of this. From what I can tell cifs is not subject to any IO scheduling what with being network based. IO scheduling only comes into play when the request is routed into a block device. cifs communication directly goes out into the network stack and would be subject to the queuing discipline, which seems not interested in IOPRIO. Perhaps this actually makes all perfect sense. We request a PNG to create a preview, cifs is happy to oblige and issues (multiple concurrent?) network requests, the remote is dreadfully slow and iowaiting on its disk, translating to us iowaiting on the network sockets coming back with the data. I can kind of replicate this by cgroup limiting smbd IO, but even with very aggressive limits my system stays responsive. So, I'm rather thinking there is a problem with your kernel/configuration.

> Ah, found that setting. It doesn't seem to account for filesystems mounted
> via fstab unfortunately.

Indeed, you are accessing it through the file:// protocol. That makes it inherently local.

> And I've found performance using `smb://` in dolphin to be worse than just
> using an fstab automount, which is why I have it setup that way.

I have the suspicion that this has to do with why things are misbehaving for you to begin with. smb:// is performing close to the same transfer speed as windows, if it doesn't for you that points at a problem elsewhere.

Perhaps let's get our ducks in a row

What's the output of 

```
tc qdisc show
ls -lah "$(qtpaths --plugin-dir)/kf5/thumbcreator"
```

Also please follow https://community.kde.org/Guidelines_and_HOWTOs/Debugging/Debugging_IOSlaves/Debugging_kio_smb#Reporting_Bugs and include 'Logging with libsmbclient debug output'

Created attachment 164239 [details]
debug log from dolphin

Created attachment 164240 [details]
debug log from dolphin

(In reply to Harald Sitter from comment #8)
> You've basically taken away the CPU weight from all KIO IO by doing this.

Ah I see, I've tweaked it to only impact the thumbnail task:

> $ cat /usr/libexec/kf5/kioslave5
> #!/bin/sh
> [ $2 != "thumbnail" ] && exec $0.orig "$@" || exec nice -n 19 ionice -c 3 $0.orig "$@"

That should minimize any side effects of this hack.

(In reply to Harald Sitter from comment #9)
> After some research I'm even less convinced of this. From what I can tell
> cifs is not subject to any IO scheduling what with being network based. IO
> scheduling only comes into play when the request is routed into a block
> device. cifs communication directly goes out into the network stack and
> would be subject to the queuing discipline, which seems not interested in
> IOPRIO. Perhaps this actually makes all perfect sense. We request a PNG to
> create a preview, cifs is happy to oblige and issues (multiple concurrent?)
> network requests, the remote is dreadfully slow and iowaiting on its disk,
> translating to us iowaiting on the network sockets coming back with the
> data. I can kind of replicate this by cgroup limiting smbd IO, but even with
> very aggressive limits my system stays responsive. So, I'm rather thinking
> there is a problem with your kernel/configuration.

Hmm do you have any ideas on what would be impacting this? I'm running Fedora stock kernel 6.6.6-100.fc38.x86_64 and the scheduling settings are default, as far as I can tell.

> Indeed, you are accessing it through the file:// protocol. That makes it
> inherently local.

I suppose I'll just have to deal with that behavior then, or probably set a file size limit acceptable for local and remote files.

> I have the suspicion that this has to do with why things are misbehaving for
> you to begin with. smb:// is performing close to the same transfer speed as
> windows, if it doesn't for you that points at a problem elsewhere.

Straight transfer speed was not the main issue. Aside from being unable to use the protocol path outside of KDE programs (e.g. terminal), there was also the slower response times enumerating directories, moving/renaming files, and sometimes dolphin would even go unresponsive for a few seconds.

I tested a file transfer with the fstab mount and using smb:// and the throughput was similar. The transfer speed sending a ~1GB video file to the server via mount path was 60s (+20s for server IO to settle), and protocol path was 45s (+40s for server IO to settle). Pulling that file took 22s via mount path and 35s via protocol path (in both cases writing to an NVMe, waiting for all IO to settle between).

To remove the disks from the equation, I mounted a ramdisk at both ends and did the test again. Download from the mount path was 20s, protocol path 30s. Upload to the mount path was 40s, protocol path was 50s. So closer numbers this time. It appears to me that in cases where both ends are nearly idle, they aren't that different, but when the server is under higher IO loads, the mount seems to behave better (more responsive, doesn't put the server under as high an IO load).

> Perhaps let's get our ducks in a row
>
> What's the output of

> $  tc qdisc show
> qdisc noqueue 0: dev lo root refcnt 2
> qdisc noqueue 0: dev wlan0 root refcnt 2
> qdisc fq_codel 0: dev enp10s0u1u2 root refcnt 2 limit 10240p flows > 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn > drop_batch 64
> qdisc noqueue 0: dev virbr0 root refcnt 2
> qdisc noqueue 8005: dev vnet4 root refcnt 2
> $  ls -lah "$(qtpaths --plugin-dir)/kf5/thumbcreator"
> total 752K
> drwxr-xr-x.  2 root root 4.0K Nov 28 06:55 .
> drwxr-xr-x. 21 root root 4.0K Nov 13 06:50 ..
> -rwxr-xr-x.  1 root root  44K Nov 13 19:00 audiothumbnail.so
> -rwxr-xr-x.  1 root root  28K Nov 13 19:00 blenderthumbnail.so
> -rwxr-xr-x.  1 root root  40K Nov 13 19:00 comicbookthumbnail.so
> -rwxr-xr-x.  1 root root  20K Nov 13 19:00 cursorthumbnail.so
> -rwxr-xr-x.  1 root root  32K Nov 13 19:00 djvuthumbnail.so
> -rwxr-xr-x.  1 root root  44K Nov 13 19:00 ebookthumbnail.so
> -rwxr-xr-x.  1 root root  32K Nov 13 19:00 exrthumbnail.so
> -rwxr-xr-x.  1 root root  81K Nov 13 19:00 ffmpegthumbs.so
> -rwxr-xr-x.  1 root root  32K Nov 24 19:00 fontthumbnail.so
> -rwxr-xr-x.  1 root root  77K Nov 13 19:00 gsthumbnail.so
> -rwxr-xr-x.  1 root root  24K Nov 13 19:00 imagethumbnail.so
> -rwxr-xr-x.  1 root root  32K Nov 13 19:00 jpegthumbnail.so
> -rwxr-xr-x.  1 root root  20K Nov 13 19:00 kraorathumbnail.so
> -rwxr-xr-x.  1 root root  41K Nov 13 19:00 mltpreview.so
> -rwxr-xr-x.  1 root root  20K Nov 13 19:00 mobithumbnail.so
> -rwxr-xr-x.  1 root root  28K Nov 13 19:00 opendocumentthumbnail.so
> -rwxr-xr-x.  1 root root  24K Nov 13 19:00 rawthumbnail.so
> -rwxr-xr-x.  1 root root  20K Nov 13 19:00 svgthumbnail.so
> -rwxr-xr-x.  1 root root  32K Nov 13 19:00 textthumbnail.so
> -rwxr-xr-x.  1 root root  32K Nov 13 19:00 windowsexethumbnail.so
> -rwxr-xr-x.  1 root root  32K Nov 13 19:00 windowsimagethumbnail.so

> Also please follow
> https://community.kde.org/Guidelines_and_HOWTOs/Debugging/Debugging_IOSlaves/
> Debugging_kio_smb#Reporting_Bugs and include 'Logging with libsmbclient
> debug output'

I'll attach the stdout and stderr log (apparently the debug output ended up in both places). During this time, I browsed a directory containing a few dozen twitch stream captures. If there's anything in particular you want me to try or a more rigorous test, let me know.

(In reply to Eddie J Carswell II from comment #12)
> I'll attach the stdout and stderr log (apparently the debug output ended up
> in both places). During this time, I browsed a directory containing a few
> dozen twitch stream captures. If there's anything in particular you want me
> to try or a more rigorous test, let me know.

Neglected to mention, with the smb:// protocol path, no thumbnails were generated (as expected), and when I browsed via the mount path after, kioslave5 happily generated thumbnails during this time (without putting undue pressure on the system, thanks to the wrapper). I do notice that the thumbnail generation is a bit slower, but still acceptable.

I don't think we'll figure out why the fedora kernel is being weird on the KDE bugtracker :)

What we can probably do easily is lower the cpu niceness.

A possibly relevant merge request was started @ https://invent.kde.org/network/kio-extras/-/merge_requests/322

Git commit 1b31c8a13c49d10a1d48862e67898c046db9c506 by Harald Sitter.
Committed on 19/12/2023 at 13:29.
Pushed by sitter into branch 'master'.

thumbnail: make nice support mandatory

it's part of POSIX 2001 I think it's save to assume it being supported
in 2023.
on windows we use different api.

M  +0    -3    config-runtime.h.cmake
M  +6    -2    thumbnail/thumbnail.cpp

https://invent.kde.org/network/kio-extras/-/commit/1b31c8a13c49d10a1d48862e67898c046db9c506

Well thanks for looking into this. I'll see if that patch improves things when it ends up in the repos.