Bug 208625 - Temporary files (for previewing files in the trash or remote files) aren't deleted
Summary: Temporary files (for previewing files in the trash or remote files) aren't de...
Status: RESOLVED FIXED
Alias: None
Product: kio
Classification: Frameworks and Libraries
Component: general (show other bugs)
Version: 4.8
Platform: unspecified Linux
: NOR normal
Target Milestone: ---
Assignee: David Faure
URL:
Keywords:
: 238309 272249 352975 (view as bug list)
Depends on:
Blocks:
 
Reported: 2009-09-26 19:10 UTC by Alvaro Aguilera
Modified: 2017-05-14 13:49 UTC (History)
25 users (show)

See Also:
Latest Commit:
Version Fixed In: 5.35


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alvaro Aguilera 2009-09-26 19:10:39 UTC
Version:           1.3 (using 4.3.1 (KDE 4.3.1) "release 165", KDE:KDE4:Factory:Desktop / openSUSE_11.1)
Compiler:          gcc
OS:                Linux (x86_64) release 2.6.31-41-desktop

Dolphin creates lots temporary files on /tmp/kde-user. Some of them have names like dolphinAX7159.tmp,  dolphing13600.tmp.part, etc. I think they are created when copying files from a remote server. The problem is that they aren't deleted afterwards and /tmp/kde-user starts to eat up gigabyte and gigabyte. It would be nice not to use that kind of files at all. The former method of writing to the target file.part and then changing the suffix was a lot better. Moreover, if you use a different partition for /tmp, you need to copy the file twice, with the performance overhead that this implies.
Comment 1 Alvaro Aguilera 2009-09-26 19:26:37 UTC
Way to reproduce this braindamage:

1. rm /tmp/kde-user/dolphin*
2. open dolphin and split the window
3. go to a remote machine via SFTP
4. copy a file from the remote machine to the local one

I did this for a file with a size of 700MB, at 250MB copied this was the situation:

/home/file.part ==> 250MB

/tmp/kde-user/ ===>
0       dolphinB13600.tmp                 
348M    dolphinB13600.tmp.part            
0       dolphinC13600.tmp                 
343M    dolphinC13600.tmp.part            
0       dolphinD13600.tmp                 
347M    dolphinD13600.tmp.part            
0       dolphinE13600.tmp                 
9.7M    dolphinE13600.tmp.part            
0       dolphinH13600.tmp                 
138M    dolphinH13600.tmp.part            
0       dolphinI13600.tmp                 
108M    dolphinI13600.tmp.part            
0       dolphinL13600.tmp                 
97M     dolphinL13600.tmp.part
0       dolphinN13600.tmp
285M    dolphinN13600.tmp.part
0       dolphinb13600.tmp
294M    dolphinb13600.tmp.part
0       dolphind13600.tmp
344M    dolphind13600.tmp.part
0       dolphine13600.tmp
335M    dolphine13600.tmp.part
0       dolphinl13600.tmp
286M    dolphinl13600.tmp.part
0       dolphinm13600.tmp
287M    dolphinm13600.tmp.part
0       dolphino13600.tmp
148M    dolphino13600.tmp.part
0       dolphins13600.tmp
351M    dolphins13600.tmp.part
0       dolphint13600.tmp
293M    dolphint13600.tmp.part
0       dolphinu13600.tmp
96M     dolphinu13600.tmp.part
0       dolphinz13600.tmp
114M    dolphinz13600.tmp.part

yes dolphin was coping the file 18 "additional" times. As a bonus, with a cpu utilization of 50%, even after closing dolphin and stopping the copy.

I don't like to criticize, but if there is some money left at KDE e.V. they should put some dollars dolphin and make it work. At version 4.3.1 it's already time to have a reliable file manager. To be able to copy a file shouldn't be this problematic, not on a "state of the art file manager" like dolphin.
Comment 2 FiNeX 2009-09-26 20:43:32 UTC
This bug looks quite strange, can you reproduce using other protocols like samba or fish? Using smb and fish dolphin creates a lot of temporary files, but all are very small (even zero length).

About the high CPU utilization it should be a problem of the notification plasmoid, would you try to disable the plasma notification and try to copy a big file using sftp?

Thanks!
Comment 3 Alvaro Aguilera 2009-09-26 20:55:09 UTC
I can only access the remote machine through SSH, either with sftp:// or fish:// and the problem happens with both. Unfortunately I can't test it with any other  protocol. I don't think the CPU utilization is the fault of the plasmoid.
Comment 4 FiNeX 2009-09-26 21:04:14 UTC
Would you like to disable the notification protocol for file operation and try it anyway? Thanks.
Comment 5 Alvaro Aguilera 2009-09-26 21:08:05 UTC
what exactly do you want me to disable? the "kded notification item watcher"?
Comment 6 FiNeX 2009-09-26 21:33:37 UTC
Right click on the system tray, and un-select "File transfer and other jobs".  Thanks :-)
Comment 7 Alvaro Aguilera 2009-09-26 21:45:34 UTC
disabling the notification is somewhat better, instead of the 18, only 3 extra copies are created on /tmp/kde-user:

0       dolphinH21511.tmp
0       dolphinK21511.tmp
0       dolphinl21511.tmp
268M    dolphinl21511.tmp.part
0       dolphinr21511.tmp
267M    dolphinr21511.tmp.part
0       dolphinV21511.tmp
265M    dolphinV21511.tmp.part
0       dolphinw21511.tmp
0       dolphinW21511.tmp
0       dolphinx21511.tmp

the cpu-utilization remains high, with several kio-sftp's taking around 20% each.
Comment 8 Dimitriy Ryazantcev 2009-10-08 19:01:26 UTC
same with me. protocol - smb.
KDE4.3.1
Comment 9 Marek 2009-10-25 15:13:28 UTC
Same problem here, dolphin 1.3 (KDE 4.3.2), protocol samba
Comment 10 Dmitry Kholodilov 2009-11-08 17:35:45 UTC
Same problem in KDE 4.3.3 (openSUSE 11.2 RC1) when browsing samba shares. Probably dolphin downloads remote files to create previews. It's not a problem when it downloads 1 file and creates 1 tmp file, but when it tries to create previews for several files at the same time, browsing is inconvenient (cpu load is too high). May be restriction on previews count is needed?
Comment 11 FiNeX 2009-11-08 19:07:11 UTC
In data domenica 08 novembre 2009 17:35:51, Dmitry Kholodilov ha scritto:
>  when it tries to create previews for several files at the same time,
>  browsing is inconvenient (cpu load is too high). May be restriction on
>  previews count is needed?

An alternative could be disabling thumbs on remote filesystem.
Comment 12 Dmitry Kholodilov 2009-11-08 19:14:00 UTC
> >  May be restriction on previews count is needed?

> An alternative could be disabling thumbs on remote filesystem.

Yes, it also refers to bug 211007: https://bugs.kde.org/show_bug.cgi?id=211007
Comment 13 Romain 2009-12-16 15:54:33 UTC
I have a similar problem in which whole directory trees can be found in /tmp/kde-username/dolphinFUBAR.tmp/ , but they don't contain any file ! Only empty directories.

I haven't tracked down the suspect but it's probably due to browsing SMB shares with dolphin (and smb://).

It took at least 20+ minutes) to cleanup /tmp, which caused boot "failures" (delays, actually) and much anguish among my coworkers that share this machine.
Comment 14 alancio 2010-02-05 01:58:43 UTC
I can't believe its been more than 4 months and such a serious bug is still in UNCONFIRMED state.
This reminds me of KDE3, in which some bugs were there for years, and by the time some of them were resolved, KDE3 was not supported anymore.
Comment 15 FiNeX 2010-02-05 10:39:26 UTC
@alancio: the key is that nobody do this for work. If developers have time to spend on this project, they can fix bugs (if they want), otherwise patch are always appreciated :-)

P.S: you could give your vote to this bug in order to underline the severity.

P.P.S: I've finished my available votes :-)

P.P.P.S: I've reproduced this bug too, using RC version of KDE 4.4. Can you reproduce with KDE 4.4 too?
Comment 16 Martin F. 2010-02-05 11:15:34 UTC
I can confirm this bug using the smb protocol with kde 4.3.5. next week i will try to confirm it using kde 4.4!

it already happened two times that dolphin completely filled up my root partition (around 13G temporary dolphin* files), which caused plasma to frequently crash :-) 
sadly, no "disk full" or other warnings appeared - this might be a real problem for users without enough computer knowledge. 


a solution which works for me is to disable the information sidebar.
Comment 17 alancio 2010-02-05 11:21:48 UTC
FiNeX: Yes, I used to do the voting thing, bug reporting, et al with KDE3, but like you said, developers don't really care because they are not paid for this, they do it for fun, and I guess supporting KDE3 is not fun anymore.

I just come here to check if anybody has a useful patch, I'm tired of my bug reporting and votes being ignored, but knock yourself out.
Comment 18 Luca Tomat 2010-04-17 17:19:19 UTC
*** This bug has been confirmed by popular vote. ***
Comment 19 Salvatore 2010-04-17 17:32:27 UTC
I can confirm this bug with kde 4.4.1, my wild guess is that this bug is related with browsing samba shares.
Comment 20 Bartemius Crouch 2010-05-09 21:12:59 UTC
my girlfriend had this problem a few minutes ago on her kubuntu 9.10 install (dolphin 4:4.3.5-0ubuntu1~karmic1).
she was deleting local files, no samba involved at all!
the files she deleted where not saved on the /-filesystem, but mounted at /data.
she rebooted, so i cant provide and more details right now.
Comment 21 Lamarque V. Souza 2010-09-13 03:39:10 UTC
Dolphin creates several empty files here with format /tmp/kde-lamarque/dolphinXXXXXX.tmp when I browse my cell phone using bluetooth (bluedevil), just browsing, I do not even need to open a file.
Comment 22 Marek 2010-09-20 08:46:39 UTC
still present in 4.4.5 :-/
Comment 23 Elián Hanisch 2012-02-14 17:43:23 UTC
Still present in 4.8, I just had to delete 5Gb worth of dolphin temporal files.
Comment 24 Dawit Alemayehu 2013-06-16 23:45:14 UTC
Cannot reproduce this at all. I tried three different remote protocols ftp, sftp and smb with dolphin and it does not create a single temp file under /tmp/kde-<user>/ for me. Same thing with Konqueror. Nothing. Do any of you still see this issue in KDE v4.10 or is this something that was fixed?
Comment 25 david 2013-11-10 17:29:44 UTC
still present for me in KDE SC 4.11.3 arch linux
Comment 26 Elián Hanisch 2014-07-14 21:53:14 UTC
Still present in KDE: 4.13.1

Temporal files can also grow very big and fill the partition.
Comment 27 Volker Siegel 2014-08-28 08:20:16 UTC
Confirming with KDE 4.13.3 on Ubuntu 14.04:

I just got a "no space left on device" for /tmp and had to delete two dolphin temp directories from /tmp/user/1000/kde-username

The two directories were dolphinK11352.tmp and dolphinH11352.tmp - note the related names.
Each of them contained 1.2 GB of files;
They were files have once been put into trash a while ago. 
The directories contained a similar of equal copy of the files.
The trash in dolphin was empty at the time.
Comment 28 Alex 2014-09-09 12:33:26 UTC
Over the last couple of months on my desktop, I accumulated 100+ GB in /tmp/kde-user... i.e. most of my SSD was being used by that single folder.  This is with dolphin/KDE 4.13.2, and many accesses to smb shares.

A good solution would probably be to automatically delete those tmp files in FIFO order, keeping only the newest X MB or GB (where X is a value in settings).  Until we have something like that, I created a cronjob to purge that directory nightly, which should keep it down to just a few GB.
Comment 29 Elián Hanisch 2014-12-18 03:09:51 UTC
I believe I found out how to reproduce this bug, these temp files are generated whenever I visit the trash:/ folder with dolphin, and generates new ones for every time I press F5 in there. If image thumbnails aren't enabled, then they don't take much space, but if you enable it and you have a bunch of images then they become quite large. How much space the temp files take depends of the contents of the trash, since it seems to be copying them over. 

In short,
- delete some files and images (so the trash isn't empty)
- go to trash:/
- enable thumbnails
- see how dolphin creates temp files and never deletes them.

Qt: 4.8.6 KDE Development Platform: 4.13.3
Comment 30 Matt Whitlock 2015-01-23 23:11:02 UTC
This bug is particularly bad for those of us with /tmp as tmpfs. I just discovered I had 1.3 GiB of Dolphin temp files sucking up my RAM. I run without disk-based swap, too — swap only on zram devices — so this is really terrible.
Comment 31 hwk 2015-07-04 07:58:21 UTC
Hi,
I've found the same bug on KDE 4.11.5 on openSUSE 13.1. I've noticed that moving those tmp files to trash:/ through Dolphin itself, loads the CPU for the 70% to 90%. While deleting those files from trash:/ the whole system is really slow, i had to use rm from a terminal window.
Comment 32 Andres Nogueiras 2015-10-07 18:37:54 UTC
openSUSE 13.2 (Harlequin) (x86_64)
3.16.7-24-desktop x86_64 GNU/Linux
Qt: 4.8.6
KDE: 4.14.9
Dolphin: 15.04.0

I can confirm that this behavior too. If open the Trash:/ folder with preview (thumbnails) activated, dolphin starts to create plenty of dirs inside of /tmp/kde-user dir, and in seconds (not minutes!) it fills 4GB. Then all the other applications complains or misbehave, due to the lack of space in /tmp.

I've also reported this misbehavior in other related bugs, because this is something really complicated to address as user. Implies stopping the dolphin file manager (also this happens with konqueror when browsing Trash:/ ) and manually remove the files inside /tmp

And as far as I believe that thumbnais is a very useful feature, resources granted to this process (cpu cycles / thread / processes / tmp space) must be somehow tuned in order not to overload the machine resources.
Comment 33 Serhiy Zahoriya 2017-03-05 22:49:57 UTC
Still 100%-reproducible on Dolphin 16.04.3 (kubuntu backports PPA)

I don't see it as files not being removed but as the files being copied to /tmp

Why would you do that?

Please close this bug as a duplicate: https://bugs.kde.org/show_bug.cgi?id=238309
Comment 34 David Faure 2017-03-06 08:58:41 UTC
OK, let's see.

First, trying to reproduce Alvaro's initial steps (with the path in step 1 adjusted for Kf5) :
1. rm /tmp/dolphin*
2. open dolphin and split the window
3. go to a remote machine via SFTP
4. copy a file from the remote machine to the local one (using drag-n-drop)

I copied a 102 MB file and a 3.7 MB file and no temp file was created, not even during the copy.

Then Dmitry's comment says it might be about previews, I tried enabling previews in the toolbar, works locally, no effect on sftp. This is because Dolphin's preferences dialog says "Skip previews for remote files over 0MB".
If I change that to 10 MB, then indeed I get a preview of the image I uploaded - but still no dolphin temp file left over in /tmp.

Now trying the same with trash:/, preview works, no temp file left over.

Now looking at the code: there is a QTemporaryFile in KIO::PreviewJob, obviously, for remote files where the thumbnail plugin doesn't support KIO. But the temp file is deleted in the next step.

KIO::PreviewJobPrivate::getOrCreateThumbnail: Using temp file "/tmp/dolphin.B12772"
KIO::PreviewJob::slotResult: Deleting temp file "/tmp/dolphin.B12772"
(additional debug output I just added)

So a temp file leak would only happen if the first step happens but not the second step.
Reading the code, I don't see how this can ever happen. There is no code path where we wouldn't get to the second step.

The only course of action I can suggest is that someone who can reproduce the bug applies the patch below to their kio build and reports the results.

http://www.davidfaure.fr/2017/previewjob.cpp.diff
Comment 35 David Faure 2017-03-06 08:59:19 UTC
*** Bug 238309 has been marked as a duplicate of this bug. ***
Comment 36 David Faure 2017-03-06 09:00:33 UTC
*** Bug 352975 has been marked as a duplicate of this bug. ***
Comment 37 David Faure 2017-03-06 09:01:05 UTC
*** Bug 272249 has been marked as a duplicate of this bug. ***
Comment 38 Matt Whitlock 2017-03-06 09:29:08 UTC
(In reply to David Faure from comment #34)
> The only course of action I can suggest is that someone who can reproduce
> the bug applies the patch below to their kio build and reports the results.
> 
> http://www.davidfaure.fr/2017/previewjob.cpp.diff

Okay, I applied it and reproduced the bug. When I browse to "trash:" and enable thumbnails, I get thousands of these log messages:

Using temp file "/tmp/dolphin.c25868"
KIO::TransferJob(0x2bb5350) for "/tmp/dolphin.c25868"
Deleting temp file "/tmp/dolphin.c25868"

After all the log activity subsides, I find that 1.3 GiB are now consumed in two directories inside /tmp. Both of these directory paths are reported in "Deleting temp file" messages (multiple times, presumably because the temp file names are reused). Are you certain that QFile::remove(const QString &) supports recursively removing a non-empty directory?
Comment 39 David Faure 2017-03-06 09:44:26 UTC
Oops, no, it definitely doesn't. Well done, that's the issue. This code was written at the time where only files were getting previewed. Now it's getting triggered for entire directories too. I'm not sure we want that -at all- for non-local directories (trash is non local too, the way it's implemented).

There's a choice between two possible fixes here:
- properly removing temp dirs
- skipping dirs for previews
Comment 40 Elvis Angelaccio 2017-05-06 13:04:23 UTC
(In reply to David Faure from comment #39)
> Oops, no, it definitely doesn't. Well done, that's the issue. This code was
> written at the time where only files were getting previewed. Now it's
> getting triggered for entire directories too. I'm not sure we want that -at
> all- for non-local directories (trash is non local too, the way it's
> implemented).
> 
> There's a choice between two possible fixes here:
> - properly removing temp dirs
> - skipping dirs for previews

I vote for the second fix, better safe than sorry. One can always explicitly enter in the remote directory, to trigger the previews of the files in there.
Comment 41 David Faure 2017-05-14 13:41:47 UTC
Git commit 78c45a1ea0e28a98f34c6d113c807f14700b22d4 by David Faure.
Committed on 14/05/2017 at 13:41.
Pushed by dfaure into branch 'master'.

PreviewJob: clean up empty temp file when get() fails.

(e.g. because it's a directory)

M  +13   -4    src/widgets/previewjob.cpp

https://commits.kde.org/kio/78c45a1ea0e28a98f34c6d113c807f14700b22d4
Comment 42 David Faure 2017-05-14 13:49:58 UTC
Git commit 1620032772465be475ae0746aff63a566ef2a546 by David Faure.
Committed on 14/05/2017 at 13:48.
Pushed by dfaure into branch 'master'.

PreviewJob: skip remote directories. Too expensive to preview.

For some protocols, file_copy() would end up copying the whole
directory locally!

FIXED-IN: 5.35

M  +6    -0    src/widgets/previewjob.cpp

https://commits.kde.org/kio/1620032772465be475ae0746aff63a566ef2a546