Bug 147948

Summary: copying via fish stalls
Product: [Frameworks and Libraries] kio-extras Reporter: Thorsten Staerk <dev>
Component: FISHAssignee: Plasma Bugs List <plasma-bugs>
Status: REOPENED ---    
Severity: normal CC: alvaro.carroz, andresbajotierra, cpigat242, doc.evans, dogmad, eherenz, fgunni, ivan.daou, kde-2011.08, ketetefid, kishore96, mstu, palceksmuk, sntmail
Priority: NOR    
Version: unspecified   
Target Milestone: ---   
Platform: Compiled Sources   
OS: Linux   
Latest Commit: Version Fixed In:
Attachments: patch, 2 lines affected

Description Thorsten Staerk 2007-07-17 11:18:53 UTC
Version:            (using KDE Devel)
Installed from:    Compiled sources
OS:                Linux

When I try to copy my digital camera's content ( 1.3 GB, about 1300 files ) via fish to another computer, the copy process stalls after about 200 files, probably forever.
The disk space is not exhausted (60% used).
How can we trace this problem down ?
Comment 1 Thorsten Staerk 2007-12-21 12:03:20 UTC
The problem also occurs if I download the files first. So I just copy from a LOCAL folder to a fish://-drive and it stalls.
Comment 2 Thorsten Staerk 2007-12-21 12:10:58 UTC
Obviously, either kio_fish is lost or kio_file blocks on a socket. The following is the situation when stalling:

scorpio:~ # ps -A | grep kio
 4393 ?        00:00:01 kio_file
scorpio:~ # strace -p 4393
Process 4393 attached - interrupt to quit
select(4, [3], NULL, NULL, NULL       

Comment 3 Thorsten Staerk 2007-12-21 12:37:37 UTC
With the same files, with scp, copying works.
Comment 4 Thorsten Staerk 2008-01-02 15:53:20 UTC
I tried:
- copy a 4.1 GB file from /mnt to fish://root@localhost
=> worked
- copy 506 files (940.5MB) from /mnt/sdb1/root/camera to fish://root@localhost
=> stalled at the 7th file
Comment 5 Paweł Findeisen 2008-10-11 00:03:06 UTC
I have problem with coping from fish to local too, but my KDE version is 4.1.1
Comment 6 Thorsten Staerk 2008-10-11 07:52:43 UTC
In http://www.kdedevelopers.org/node/3683 dfaure describes how he made kio much faster
Comment 7 Thorsten Staerk 2008-10-15 18:10:50 UTC
increasing version from 3.5 to 4.0 because 4.1 is not available
Comment 8 Thorsten Staerk 2008-10-15 18:47:59 UTC
when uploading with two kio_fish processes, sometimes both stall. Then I do a 
 killall kio_fish
and get a message that uploading failed. I choose skip there and then the upload continues.
Comment 9 Paweł Findeisen 2008-10-23 21:29:20 UTC
File transfer from fish:// stalls even if I'm copying single big file,
so it seems that dfaure patch will not fix this.
The problem is fully repeatable in my configuration (file is 1.2 GB, transfer stalls at about 25%).

Comment 10 Paweł Findeisen 2008-10-27 21:38:40 UTC
When file download from fish is stalled the ssh connection is still open, and the kio_fish process busy waits for something to appear on descriptor 5 (/dev/ptmx).
The healthy kio_fish process would copy data from descriptor 5 (ptmx) to descriptor 3 (socket).
Comment 11 Paweł Findeisen 2008-10-27 21:54:06 UTC
While kio client waits for input, the server side process (perl) waits to write next file chunk to stdout (blocked in "write" call).
Comment 12 Frank Sagurna 2009-01-02 01:54:20 UTC
I can confirm this bug. Sometimes a file is copyable, sometimes it stalls. commandline scp works, so bug has to be in kde/kio_slave.
killall kio_fish cancels the copy process.
Comment 13 Frank Sagurna 2009-01-02 01:54:54 UTC
Forgot to mention i am on KDE 4.2beta2 here (Kubuntu packages)
Comment 14 Thorsten Staerk 2009-01-03 11:19:01 UTC
ok, here is a procedure to reproduce, maybe we can turn this into a test case:
1. do the following as root:
 cd
 mkdir kiotest
 cd kiotest   
 for i in $(seq 1 1 1000); do touch file$i; done
2. point your konqueror to /root. Copy the folder kiotest
3. point your konqueror to fish://root@localhost/kiotest. 
4. paste the folder kiotest

Expected result: The folder ist copied.
Actual result: The copy process stalls at about the 4th file.
Comment 15 Thorsten Staerk 2009-01-03 12:47:32 UTC
Konqueror's debugging output points to apps/lib/konq/konq_operations.cpp:void KonqOperations::copy
Comment 16 Thorsten Staerk 2009-01-03 12:51:16 UTC
Next step is to look at http://www.staerk.de/thorsten/index.php/Using_a_kioslave_in_KDE_4 where I describe how to use a kioslave. Then we write a test case involving the command KIO::copy to simulate the behavior of KonqOperations::copy 
Comment 17 Thorsten Staerk 2009-01-04 19:00:31 UTC
*** Bug 173024 has been marked as a duplicate of this bug. ***
Comment 18 Thorsten Staerk 2009-01-04 19:02:42 UTC
I no longer see this behavior with KDE from trunk. See also Bug 179638.
Comment 19 Thorsten Staerk 2009-01-04 19:52:01 UTC
I see this behavior again, here is a test case. CMakeLists.txt:
PROJECT( browser )
FIND_PACKAGE(KDE4 REQUIRED)
INCLUDE_DIRECTORIES( ${KDE4_INCLUDES} . )


SET(kde4startSources main.cpp browser.cpp )


KDE4_ADD_EXECUTABLE(browser ${kde4startSources} )

TARGET_LINK_LIBRARIES(browser ${KDE4_KDEUI_LIBS} ${KDE4_KPARTS_LIBS} )

install(TARGETS browser  DESTINATION ${BIN_INSTALL_DIR} )
Comment 20 Thorsten Staerk 2009-01-04 19:52:56 UTC
browser.cpp:
#include <kio/scheduler.h>       
#include <kurl.h>                
#include <kio/jobclasses.h>      
#include <kdebug.h>              
#include <browser.h>             
#include <kio/copyjob.h>         

class TransferJob;

Browser::Browser() : QWidget(NULL)
{                                 
  slotButtonClicked();            
}                                 

void Browser::slotButtonClicked()
{                                
  kDebug() << "entering function";
  // creating a kioslave          
  kDebug() << "getting via fish*************************************************************";                                                                  
  KUrl::List selectedUrls;                                                      
  selectedUrls.push_back(KUrl("/tmp/fishtest.txt"));                            
  selectedUrls.push_back(KUrl("/tmp/fishtest1.txt"));                           
  selectedUrls.push_back(KUrl("/tmp/fishtest2.txt"));                           
  selectedUrls.push_back(KUrl("/tmp/fishtest3.txt"));                           
  selectedUrls.push_back(KUrl("/tmp/fishtest4.txt"));                           
  selectedUrls.push_back(KUrl("/tmp/fishtest5.txt"));                           
  selectedUrls.push_back(KUrl("/tmp/fishtest6.txt"));                           
  selectedUrls.push_back(KUrl("/tmp/fishtest7.txt"));                           
  selectedUrls.push_back(KUrl("/tmp/fishtest8.txt"));                           
  selectedUrls.push_back(KUrl("/tmp/fishtest9.txt"));                           
  selectedUrls.push_back(KUrl("/tmp/fishtest10.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest11.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest12.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest13.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest14.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest15.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest16.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest17.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest18.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest19.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest20.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest21.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest22.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest23.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest24.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest25.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest26.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest27.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest28.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest29.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest30.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest31.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest32.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest33.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest34.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest35.txt"));                          
  selectedUrls.push_back(KUrl("/tmp/fishtest36.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest37.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest38.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest39.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest40.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest41.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest42.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest43.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest44.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest45.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest46.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest47.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest48.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest49.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest50.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest51.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest52.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest53.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest54.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest55.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest56.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest57.txt"));
  selectedUrls.push_back(KUrl("/tmp/fishtest58.txt"));
  KUrl destUrl("fish://root@localhost/tmp/test");
  KIO::CopyJob* job0 = KIO::copy( selectedUrls, destUrl );
  job0->start();
}

void Browser::dataishere(KIO::Job *,const QByteArray & data )
{
  static int counter=0;
  kDebug() << ++counter << " data is here*************************************************************";
  kDebug() << data;
}
Comment 21 Thorsten Staerk 2009-01-04 19:53:29 UTC
browser.h:
#ifndef KDE4START_H__
#define KDE4START_H__

#include <kmainwindow.h>
#include <kio/scheduler.h>
#include <kurl.h>
#include <kio/jobclasses.h>

class Browser : public QWidget
{
  Q_OBJECT
  public:
    Browser();
  public slots:
    void slotButtonClicked();
    void dataishere(KIO::Job *,const QByteArray &);
};
#endif
Comment 22 Thorsten Staerk 2009-01-04 19:54:08 UTC
main.cpp:
#include <QString>
#include <kapplication.h>
#include <kaboutdata.h>
#include <klocalizedstring.h>
#include <kmessagebox.h>
#include <kcmdlineargs.h>
#include <KMainWindow>
#include <browser.h>

int main (int argc, char *argv[])
{
  // KAboutData (const QByteArray &appName, const QByteArray &catalogName, const KLocalizedString &programName, const QByteArray &version,
  const QByteArray& ba=QByteArray("test");
  const KLocalizedString name=ki18n("myName");
  KAboutData aboutData( ba, ba, name, ba, name);
  KCmdLineArgs::init( argc, argv, &aboutData );
  KApplication khello;

  Browser *mw = new Browser();
  mw->show();
  khello.exec();
}
Comment 23 Thorsten Staerk 2009-01-04 20:03:18 UTC
*** Bug 179638 has been marked as a duplicate of this bug. ***
Comment 24 Dotan Cohen 2009-01-05 13:44:57 UTC
*** Bug 176912 has been marked as a duplicate of this bug. ***
Comment 25 Dotan Cohen 2009-01-05 13:46:05 UTC
These attachments to one of the dupes shows the stalled dialog:
http://bugs.kde.org/attachment.cgi?id=27963
http://bugs.kde.org/attachment.cgi?id=27964
http://bugs.kde.org/attachment.cgi?id=27965

I can also confirm this bug in KDE 4.2 beta.
Comment 26 Thorsten Staerk 2009-01-06 12:53:45 UTC
Created attachment 29964 [details]
patch, 2 lines affected

here is my patch, I am very proud of it
Comment 27 Thorsten Staerk 2009-01-06 14:52:02 UTC
SVN commit 906635 by tstaerk:

Make copy via fish reliable. If FD_ISSET(childFd,&wfds) is true and
FD_ISSET(childFd,&rfds) is true, do not handle both in one loop or the
process will stall.
CCBUGS:147948 


 M  +2 -2      fish.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=906635
Comment 28 Thorsten Staerk 2009-01-06 23:26:35 UTC
I meant BUGS, not CCBUGS.
Comment 29 palcek smuk 2009-01-09 19:27:31 UTC
is this going to be backported to 3.5? :)
Comment 30 Thorsten Staerk 2009-01-09 22:14:27 UTC
This fix will hit the user with KDE 4.3. There will not be any more KDE 3.5 release, but I plan to write a fix that you can apply on KDE 3.5.
Comment 31 Frank Sagurna 2009-01-10 00:35:09 UTC
Shouldnt it hit 4.2 as it is only a bugfix (and important), and no feature?
Comment 32 Thorsten Staerk 2009-01-10 10:27:19 UTC
(In reply to comment #31)
> Shouldnt it hit 4.2 as it is only a bugfix (and important), and no feature?
> 

Only if you verify this bug solves the problem for you as well. So far I am the only person who has observed this bug fixing anything. Set this bug to verified when you see it fixes your problem.
Comment 33 Thorsten Staerk 2009-01-10 14:15:48 UTC
Got confirmation from Will that this code change does fix the problem.
Comment 34 Thorsten Staerk 2009-01-11 15:18:03 UTC
This will hit the user with KDE 4.2: http://websvn.kde.org/branches/KDE/4.2/kdebase/runtime/kioslave/fish/fish.cpp?view=log
Comment 35 Frank Sagurna 2009-01-11 15:37:36 UTC
tstaerk: Thanks alot.
Comment 36 Thorsten Staerk 2009-01-15 09:32:07 UTC
SVN commit 911280 by tstaerk:

backporting 906635: Make copy via fish reliable.
This will be in KDE 4.1.5.
CCBUGS:147948


 M  +2 -2      fish.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=911280
Comment 37 Thorsten Staerk 2009-01-15 14:52:53 UTC
SVN commit 911492 by tstaerk:

Backport 911280: make fish copy reliable.
This will be in the KDE 3.5 branch. It was too late for 3.5.10.
CCBUGS:147948


 M  +2 -2      fish.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=911492
Comment 38 Paweł Findeisen 2009-02-14 20:01:37 UTC
Fish transfer still hangs quite often, while scp works without disruptions...
I'm using kde 4.2.0 shipped with Fedora 9.
Comment 39 Thorsten Staerk 2009-02-14 22:44:21 UTC
(In reply to comment #38)
> Fish transfer still hangs quite often, while scp works without disruptions...
> I'm using kde 4.2.0 shipped with Fedora 9.
> 

OK, where can I download an installable ISO of FC 9 so I can reproduce?
Comment 40 Frank Sagurna 2009-02-15 11:21:26 UTC
The bug still existst in kubuntu intrepid with kde 4.2, too.
Comment 41 Thorsten Staerk 2009-02-15 11:45:03 UTC
(In reply to comment #40)
> The bug still existst in kubuntu intrepid with kde 4.2, too.
> 
On http://www.staerk.de/thorsten/index.php/147948 I give a complete testcase incl. instructions how to kompile it and how to run it. Does this stall?
Comment 42 Thorsten Staerk 2009-02-15 12:24:26 UTC
In order to reproduce it correctly - how did you install KDE 4.2 on Kubuntu? I did a
 sudo apt-get install kubuntu-desktop
on ubuntu 8.10 and got KDE 4.1.
Comment 43 Frank Sagurna 2009-02-15 12:50:53 UTC
I use the repo from 
## KDE 4.2 
deb http://ppa.launchpad.net/kubuntu-experimental/ubuntu intrepid main
Comment 44 Thorsten Staerk 2009-02-15 15:26:09 UTC
Frank and I have met on irc and analyzed this. After installing the gpg key according to http://www.kubuntu.org/news/kde-4.2 we were able to kompile and use my testcase. For me, Kubuntu 8.10 x32, there were no errors. For Frank, Kubuntu 8.10 x64, there were the same errors that I experienced before my fix. However, my fix has also been positively tested on x64 (SUSE) Linux.
Comment 45 Thorsten Staerk 2009-02-15 15:36:55 UTC
(In reply to comment #38)
> Fish transfer still hangs quite often, while scp works without disruptions...
> I'm using kde 4.2.0 shipped with Fedora 9.
> 
I took the time to download FC 9 (x64) and install it into a virtual machine. Now, how do I choose KDE for startup? I only get gnome started.
Comment 46 Thorsten Staerk 2009-03-01 10:46:17 UTC
I found out that the problem is in fish.pl. The line
    exec('/usr/bin/file','-i','-b','-L',$fn);
does not always return on high load. My fix is a workaround on the application level that re-triggers a select and gets things going again.
Comment 47 Thorsten Staerk 2009-03-01 12:03:52 UTC
I have solved the problem on site of fish.pl. The problem is that fork needs some retries to deliver a pid, here is the patch for fish.pl:
-    return '' if (!defined $pid);                                                                                                                                           
+print DEBUG "Entering mimetype4";                                                                                                                                           
+    while (!defined $pid) $pid=fork();
Comment 48 Thorsten Staerk 2009-03-23 08:00:53 UTC
(In reply to comment #45)
> (In reply to comment #38)
> > Fish transfer still hangs quite often, while scp works without disruptions...
> > I'm using kde 4.2.0 shipped with Fedora 9.
> > 
> I took the time to download FC 9 (x64) and install it into a virtual machine.
> Now, how do I choose KDE for startup? I only get gnome started.

OK, I asked you to tell me how to download fedora - you did not. No problem, I found out. I took the time to set up a virtual machine running Fedora. Then I asked you how to make it start KDE by default - no answer from your site. OK, and now I find out it is KDE 4.0 that ships with Fedora 9, not KDE 4.2. So, how do you get KDE 4.2 shipped by Fedora 9?
Comment 49 Thorsten Staerk 2009-03-29 11:54:56 UTC
I just verified this bug is distribution-independant. It also happens with Ubuntu 8.04, kubunut-desktop with KDE 3.5.10.
Comment 50 Dario Andres 2009-07-21 04:52:48 UTC
@tstaerk: should the report be reopened as it seems it is not really fixed? Thanks
Comment 51 Thorsten Staerk 2009-07-21 06:01:58 UTC
There are many bugs around fish. All have in common that nobody gives a way to reproduce them. Before re-opening, please give one (and a description what the problem is).
Comment 52 Paweł Findeisen 2009-07-21 23:04:13 UTC
(In reply to comment #48)

Sorry for no replies. I didn't received notifications about your comments
(in case you're still interested: 'yum update kdebase' as root).
Comment 53 Alvaro Carroz 2009-07-22 01:55:38 UTC
I have this bug happen to me using KDE 4.2.4 (64-bit) on sidux 2009-02. When transferring a large (over 4GB) file, after copying near half the file it stalls. It never comes back or completes, eventually it times out. I tested this using a path with a symlink and one without. Both scp and sftp (this one from the dolphin window) work fine.
Comment 54 Vsevolod Krishchenko 2009-07-29 14:43:12 UTC
The way to reproduce with some degree of success (still not 100% chance, sorry).

I use slow network (100Mbit) and a quite slow server (Epia via, 25% CPU average load). 

1. Create a few mid-sized files (20-30 300-1000Mb work for me) on server.

2. Copy a half of them via fish copy-past to local client. Immediately copy another half to create second kio process (I suppose that it increases chance of failure). 

3. Press "i" widget and wait -- speed will fall to 0B/s and one of the processes will halt in case of "success".

All my attempts to use fish fail in this case, scp works fine.
Comment 55 Dotan Cohen 2009-09-05 13:24:35 UTC
As Vsevolod provided reproduction instructions and I am still experiencing this issue in KDE 4.3.1 I am reopening the bug.

@Vsevolod: can you provide access to your slow server with the files already on it so that KDE devs can simply connect to it and test? Thanks!
Comment 56 Nicolas L. 2010-08-04 10:23:15 UTC
*** Bug 183594 has been marked as a duplicate of this bug. ***
Comment 57 Ivan D 2012-02-26 18:48:45 UTC
Hello, I think I have the same problem while transferring files from an ubuntu server to my computer running KDE 4.7.4 on gentoo.

Way to reproduce (at my home) : 
1/ Connect with fish to my other computer
2/ Transfer a big file (600MB is fine) from distant computer to mine
3/ The transfer reaches 100% but never finishes. It hangs forever. If I stop it with kde jobs, the file is incomplete with a .part extension to its name and is not valid.

Scp or sftp are working just fine.
I would be glad to give more information if you can indicate me a way to do that (debug, trace or other).

Thx
Comment 58 Kete Tefid 2012-10-03 17:11:50 UTC
I exactly have the same problem here.
Confirmed with Gentoo kde 4.8.5. I had to scp all the files using command line.
Comment 59 cthx 2012-11-26 16:16:02 UTC
The same problem with big files, archlinux x86_64, KDE 4.9.3
Comment 60 Kete Tefid 2013-02-28 10:15:47 UTC
Is there any update on this?
The finest and biggest DE should not have such a bug open for a long time. Copying through ssh by KDE is considered a main functionality whereas it is broken. I always have to do it by a terminal. 
For reproducing the results, even you do not need a slow conncetion. On a 1Gb network, it easily happens: Just copy a large file.
Even worse, if you tried to copy a lot of small to medium-sized files, it would definitely stall again.
The users have always appreciated your valuable work and would even do more, if you could kindly kill this super-annoying bug.
Comment 61 Christian Herenz 2013-05-31 19:29:33 UTC
I am also suffering from this! Copying small files via fish works fine - but when copying a large number of big files (>2 GB per file) fish eventually stalls.

Since some of the machines I have to work with do only have ssh login, fish is the only option to copy my files - but with kio-fish (i.e. in krusader) that does not work. My fallback solution is "mc", whose fish implementation seems to be better than kio's.