Bug 434175 - Move to wastebin functionality is so slow that is unusable when it already has a lot of stuff in it
Summary: Move to wastebin functionality is so slow that is unusable when it already ha...
Status: CONFIRMED
Alias: None
Product: frameworks-kio
Classification: Frameworks and Libraries
Component: Trash (show other bugs)
Version: 5.79.0
Platform: Neon Linux
: HI normal
Target Milestone: ---
Assignee: David Faure
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-03-09 08:15 UTC by Michał Zubkowicz
Modified: 2024-01-01 17:09 UTC (History)
8 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michał Zubkowicz 2021-03-09 08:15:38 UTC
SUMMARY


STEPS TO REPRODUCE
1. Install fresh installation of KDE Neon
2. Select many files - press delete button on keyboard

OBSERVED RESULT
Files are deleted 1 per second on SSD drive

EXPECTED RESULT
Should delete more than 100 per second like in every other file manager. 


SOFTWARE/OS VERSIONS
KDE Neon 5.21

ADDITIONAL INFORMATION
I've tried to disable wastebin size limit in Dolphin settings without a success. Disk speed:
Timing cached reads:   19922 MB in  2.00 seconds = 9974.45 MB/sec
Timing buffered disk reads: 1522 MB in  3.00 seconds = 506.74 MB/sec
Comment 1 Nate Graham 2021-03-09 17:03:58 UTC
Do the files being sent to the trash live in a different partition or disk from the partition or disk containing your user account?
Comment 2 Michał Zubkowicz 2021-03-09 20:14:05 UTC
They are on home partition, but it doesn't matter. On any other drive speed is same slow. 

I've emptied wastebin and now it's much faster. So I suppose it's related to amount of files stored inside.
Comment 3 Nate Graham 2021-03-09 20:20:22 UTC
Interesting. Thanks for the info.
Comment 4 tagwerk19 2021-03-12 09:56:05 UTC
I know you say 'KDE Neon' - but I'll ask this on an off-chance...

Might you be using BTRFS? 

I'm guessing the "different partition" is looking to see if the file is copied rather than mv'd. I'm looked at a 'out of the box' openSuse install, there's a single partition with a collection of BTRFS subvolumes - and a 'mv' from one subvol to another copies.

This is a suspicion, will admit I'm not 100% sure, but I see inode values change with a subvol to subvol 'mv'

I also see a lot of work happening 'under the surface' when deleting large files , it can take many seconds for df to sort out what the "free space" is...
Comment 5 Michał Zubkowicz 2021-03-12 10:22:00 UTC
I'm using encrypted drive (luks) and I have one partition for /home.
Comment 6 tagwerk19 2021-03-13 09:27:42 UTC
Comparing the performance of Neon Testing, Neon Testing with LUKS and openSuse with BTRFS, there's not a lot of difference. Strangely, the "straight" Neon Testing was a little slower than the other two. But this is a "bowl of fruit" comparison - not pretending to compare apples with apples...

I created 100000 files each containing "Hello World", selected them in Dolphin and said "Move to Trash". The process started fast (moving 1000 per minute) but slowed by a factor of 10 to about 100 per minute. It was still going many hours later...

With htop:

*   The bottleneck seems to be trash.so, that was sitting solidly at
    99 to 100% CPU
*   baloo_file varied 3% to 50% CPU. It's obviously having to run to keep up.
*   With LUKS, there were several kworker 'kcryptd' processes so that
    work was being spread across several cores.
Comment 7 tagwerk19 2021-03-27 17:03:13 UTC
I suspect this can be moved to CONFIRMED...
Comment 8 Patrick Silva 2021-05-01 16:29:47 UTC
my bug report about bad performance when sending files to Trash: bug 402704
Comment 9 John van Spaandonk 2021-08-05 07:03:17 UTC
I have this as well for a long time now.
KDE neon.
I keep everything at default and updated, so ext3 filesystem, KDE neon unmodified etc.
Problem is on a local SSD partition.
Moving to thrash takes a long time, it makes it impossible to delete files.
Deleting 75 files from trash also takes forever, waiting > 5 minutes now and still not done.
process monitor shows 3 processes trash.so, all have status "disk sleep".
I see many complaints about this on forum.kde.org as well.
https://forum.kde.org/viewtopic.php?t=140002
Is it possible to give this more focus since this is the absolute basic stuff that really should work?
Comment 10 tagwerk19 2021-08-05 07:55:30 UTC
(In reply to John van Spaandonk from comment #9)
> Deleting 75 files from trash also takes forever, waiting > 5 minutes now and
> still not done.
Could it be a "refresh" issue? Does pressing F5 sort it? 
 
Then maybe have a look under
    .local/share/Trash
Do you have very many files in the files and info directories?

I ask as the forum thread you quote has people talking about deleting .cache which would probably not do "the needed" and may mean all the thumbnails are recreated.
Comment 11 Michał Zubkowicz 2021-08-05 08:02:49 UTC
I can confirm that it's strictly related to number of files in trash. I've put 1 000 000 files in trash and then deleting was very slow. After emptying trash it is fast again.
Comment 12 John van Spaandonk 2021-08-05 08:30:13 UTC
On 8/5/21 10:02 AM, Michał Zubkowicz wrote:
> https://bugs.kde.org/show_bug.cgi?id=434175
>
> --- Comment #11 from Michał Zubkowicz <michal.zubkowicz@gmail.com> ---
> I can confirm that it's strictly related to number of files in trash. I've put
> 1 000 000 files in trash and then deleting was very slow. After emptying trash
> it is fast again.
>
I had only 78 files in the trash, and it did not get beyond deleting the 
first one.
I tried seeing the location of trash by right-clicking it, choosing edit 
and pressing the folder symbol.
This completely froze Dolphin.
I solved everything it by simply restarting KDE!
It might be there is some problem related to a temporary storage that is 
cleaned by restarting KDE.
I do tend to use my computer without rebooting for days, and sometimes a 
week.
Anyway, the expected robustness is not there.
Comment 13 tagwerk19 2021-08-05 08:43:59 UTC
(In reply to John van Spaandonk from comment #12)
> I tried seeing the location of trash by right-clicking it, choosing edit 
> and pressing the folder symbol.
> This completely froze Dolphin.
Doesn't sound right. I think sidestep Dolphin and use the command line.

    cd ~/.local/share/Trash
    ls files
    ls info

I'm guessing there are more than the 78 files there than Dolphin is counting.
Comment 14 John van Spaandonk 2021-08-05 12:12:00 UTC
On 8/5/21 10:43 AM, bugzilla_noreply@kde.org wrote:
> https://bugs.kde.org/show_bug.cgi?id=434175
>
> --- Comment #13 from tagwerk19@innerjoin.org ---
> (In reply to John van Spaandonk from comment #12)
>> I tried seeing the location of trash by right-clicking it, choosing edit
>> and pressing the folder symbol.
>> This completely froze Dolphin.
> Doesn't sound right. I think sidestep Dolphin and use the command line.
>
>      cd ~/.local/share/Trash
>      ls files
>      ls info
>
> I'm guessing there are more than the 78 files there than Dolphin is counting.
>
I did what you suggested, and after removing the 78 files using the 
trash "delete" function in Dolphin,
ls~/.local/share/Trash shows 0 files.
Comment 15 Pedro V 2024-01-01 17:09:50 UTC
I also got to "enjoy" this problem quite a few time as I found KDE's trash function neat to avoid the rare but quite serious mistake of accidentally deleting the wrong file, so it was not uncommon to have a ton of waste in the trash that was likely not needed.
Having something there already is not a requirement though, just trashing tens or hundreds of thousands of files also leads to a slowdown over time.

Started digging into what could be the problem, and while I'm not familiar with the details, I've found some quite interesting answers to even questions I thought would be unrelated here:
https://invent.kde.org/frameworks/kio/-/blob/3942b233a777a4fa9e9ab45c8c3b9da58309822f/src/kioworkers/trash/trashimpl.cpp#L1269

So apparently the trash settings regarding size and time limit are enforced every time a file is trashed which is somewhat sensible, the process is just not too efficient.
Initially I thought that the time limit was implemented with some time-based scheduling as that one doesn't actually need to be checked every time there's a file operation, but then on the other hand something needs to take care of that which might not exist.

Do note that apparently the code assumes that trash directories can just appear ( https://invent.kde.org/frameworks/kio/-/blob/3942b233a777a4fa9e9ab45c8c3b9da58309822f/src/kioworkers/trash/trashimpl.cpp#L686 ), so we'll run with the idea of assuming that the filesystem is our database and we can't have anything faster, also allowing files from the trash to just simply disappear which does have its benefits.
Given that, we have these problems and/or possible ways to improve performance:
- As mentioned earlier, the time limit doesn't really need to be checked every time a file is trashed, that could be moved elsewhere, likely there's just no appropriate time-based scheduler for this currently. Also, the earlier mentioned "hotplug" assumption can throw a wrench into such scheduling
- The `useTimeLimit` and `useSizeLimit` checks lead to separate iterations, essentially going through the trash twice to gather file information
- The filesystem could be also used more efficiently than this, there's no need to keep on rescanning, inotify could be used opportunistically (realistically it should almost always work), and rescanning could be used only as a fallback
- Synchronous I/O operations could be replaced with asynchronous ones getting pumped into a work queue (io_uring has all the required goods for that already), so the slow QD1 userspace<->kernelspace dance could be replaced with a high throughput solution utilizing a deep I/O queue welcome by even fast SSDs

Chances are high that this is a significant amount of work in an area which isn't exactly looking too troublesome for common users as-is, so not expecting a whole lot of magic here to happen any soon.
Given that, I'm recommending the following workaround when trashing a ton of files at once which I just discovered and it seems like it will be neat the next time I'm doing more than just an artificial test:
- Verify that there's enough space in the trash before beginning the operation (assuming the files are on the same mount point as the trash, this is a no-op)
- Start the trashing process
- Open trash settings ("Configure trash settings")
- Remove ticks from "Cleanup:" and "Size:" options, hit "Apply"
- Wait for the large trashing operation to finish which should be significantly faster at this point
- Add back ticks to the earlier unticked options, hit "OK" to apply them and exit settings