Version: (using KDE Devel) Installed from: Compiled sources OS: Linux When I try to copy my digital camera's content ( 1.3 GB, about 1300 files ) via fish to another computer, the copy process stalls after about 200 files, probably forever. The disk space is not exhausted (60% used). How can we trace this problem down ?
The problem also occurs if I download the files first. So I just copy from a LOCAL folder to a fish://-drive and it stalls.
Obviously, either kio_fish is lost or kio_file blocks on a socket. The following is the situation when stalling: scorpio:~ # ps -A | grep kio 4393 ? 00:00:01 kio_file scorpio:~ # strace -p 4393 Process 4393 attached - interrupt to quit select(4, [3], NULL, NULL, NULL
With the same files, with scp, copying works.
I tried: - copy a 4.1 GB file from /mnt to fish://root@localhost => worked - copy 506 files (940.5MB) from /mnt/sdb1/root/camera to fish://root@localhost => stalled at the 7th file
I have problem with coping from fish to local too, but my KDE version is 4.1.1
In http://www.kdedevelopers.org/node/3683 dfaure describes how he made kio much faster
increasing version from 3.5 to 4.0 because 4.1 is not available
when uploading with two kio_fish processes, sometimes both stall. Then I do a killall kio_fish and get a message that uploading failed. I choose skip there and then the upload continues.
File transfer from fish:// stalls even if I'm copying single big file, so it seems that dfaure patch will not fix this. The problem is fully repeatable in my configuration (file is 1.2 GB, transfer stalls at about 25%).
When file download from fish is stalled the ssh connection is still open, and the kio_fish process busy waits for something to appear on descriptor 5 (/dev/ptmx). The healthy kio_fish process would copy data from descriptor 5 (ptmx) to descriptor 3 (socket).
While kio client waits for input, the server side process (perl) waits to write next file chunk to stdout (blocked in "write" call).
I can confirm this bug. Sometimes a file is copyable, sometimes it stalls. commandline scp works, so bug has to be in kde/kio_slave. killall kio_fish cancels the copy process.
Forgot to mention i am on KDE 4.2beta2 here (Kubuntu packages)
ok, here is a procedure to reproduce, maybe we can turn this into a test case: 1. do the following as root: cd mkdir kiotest cd kiotest for i in $(seq 1 1 1000); do touch file$i; done 2. point your konqueror to /root. Copy the folder kiotest 3. point your konqueror to fish://root@localhost/kiotest. 4. paste the folder kiotest Expected result: The folder ist copied. Actual result: The copy process stalls at about the 4th file.
Konqueror's debugging output points to apps/lib/konq/konq_operations.cpp:void KonqOperations::copy
Next step is to look at http://www.staerk.de/thorsten/index.php/Using_a_kioslave_in_KDE_4 where I describe how to use a kioslave. Then we write a test case involving the command KIO::copy to simulate the behavior of KonqOperations::copy
*** Bug 173024 has been marked as a duplicate of this bug. ***
I no longer see this behavior with KDE from trunk. See also Bug 179638.
I see this behavior again, here is a test case. CMakeLists.txt: PROJECT( browser ) FIND_PACKAGE(KDE4 REQUIRED) INCLUDE_DIRECTORIES( ${KDE4_INCLUDES} . ) SET(kde4startSources main.cpp browser.cpp ) KDE4_ADD_EXECUTABLE(browser ${kde4startSources} ) TARGET_LINK_LIBRARIES(browser ${KDE4_KDEUI_LIBS} ${KDE4_KPARTS_LIBS} ) install(TARGETS browser DESTINATION ${BIN_INSTALL_DIR} )
browser.cpp: #include <kio/scheduler.h> #include <kurl.h> #include <kio/jobclasses.h> #include <kdebug.h> #include <browser.h> #include <kio/copyjob.h> class TransferJob; Browser::Browser() : QWidget(NULL) { slotButtonClicked(); } void Browser::slotButtonClicked() { kDebug() << "entering function"; // creating a kioslave kDebug() << "getting via fish*************************************************************"; KUrl::List selectedUrls; selectedUrls.push_back(KUrl("/tmp/fishtest.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest1.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest2.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest3.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest4.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest5.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest6.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest7.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest8.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest9.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest10.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest11.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest12.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest13.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest14.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest15.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest16.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest17.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest18.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest19.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest20.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest21.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest22.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest23.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest24.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest25.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest26.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest27.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest28.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest29.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest30.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest31.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest32.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest33.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest34.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest35.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest36.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest37.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest38.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest39.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest40.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest41.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest42.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest43.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest44.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest45.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest46.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest47.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest48.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest49.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest50.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest51.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest52.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest53.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest54.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest55.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest56.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest57.txt")); selectedUrls.push_back(KUrl("/tmp/fishtest58.txt")); KUrl destUrl("fish://root@localhost/tmp/test"); KIO::CopyJob* job0 = KIO::copy( selectedUrls, destUrl ); job0->start(); } void Browser::dataishere(KIO::Job *,const QByteArray & data ) { static int counter=0; kDebug() << ++counter << " data is here*************************************************************"; kDebug() << data; }
browser.h: #ifndef KDE4START_H__ #define KDE4START_H__ #include <kmainwindow.h> #include <kio/scheduler.h> #include <kurl.h> #include <kio/jobclasses.h> class Browser : public QWidget { Q_OBJECT public: Browser(); public slots: void slotButtonClicked(); void dataishere(KIO::Job *,const QByteArray &); }; #endif
main.cpp: #include <QString> #include <kapplication.h> #include <kaboutdata.h> #include <klocalizedstring.h> #include <kmessagebox.h> #include <kcmdlineargs.h> #include <KMainWindow> #include <browser.h> int main (int argc, char *argv[]) { // KAboutData (const QByteArray &appName, const QByteArray &catalogName, const KLocalizedString &programName, const QByteArray &version, const QByteArray& ba=QByteArray("test"); const KLocalizedString name=ki18n("myName"); KAboutData aboutData( ba, ba, name, ba, name); KCmdLineArgs::init( argc, argv, &aboutData ); KApplication khello; Browser *mw = new Browser(); mw->show(); khello.exec(); }
*** Bug 179638 has been marked as a duplicate of this bug. ***
*** Bug 176912 has been marked as a duplicate of this bug. ***
These attachments to one of the dupes shows the stalled dialog: http://bugs.kde.org/attachment.cgi?id=27963 http://bugs.kde.org/attachment.cgi?id=27964 http://bugs.kde.org/attachment.cgi?id=27965 I can also confirm this bug in KDE 4.2 beta.
Created attachment 29964 [details] patch, 2 lines affected here is my patch, I am very proud of it
SVN commit 906635 by tstaerk: Make copy via fish reliable. If FD_ISSET(childFd,&wfds) is true and FD_ISSET(childFd,&rfds) is true, do not handle both in one loop or the process will stall. CCBUGS:147948 M +2 -2 fish.cpp WebSVN link: http://websvn.kde.org/?view=rev&revision=906635
I meant BUGS, not CCBUGS.
is this going to be backported to 3.5? :)
This fix will hit the user with KDE 4.3. There will not be any more KDE 3.5 release, but I plan to write a fix that you can apply on KDE 3.5.
Shouldnt it hit 4.2 as it is only a bugfix (and important), and no feature?
(In reply to comment #31) > Shouldnt it hit 4.2 as it is only a bugfix (and important), and no feature? > Only if you verify this bug solves the problem for you as well. So far I am the only person who has observed this bug fixing anything. Set this bug to verified when you see it fixes your problem.
Got confirmation from Will that this code change does fix the problem.
This will hit the user with KDE 4.2: http://websvn.kde.org/branches/KDE/4.2/kdebase/runtime/kioslave/fish/fish.cpp?view=log
tstaerk: Thanks alot.
SVN commit 911280 by tstaerk: backporting 906635: Make copy via fish reliable. This will be in KDE 4.1.5. CCBUGS:147948 M +2 -2 fish.cpp WebSVN link: http://websvn.kde.org/?view=rev&revision=911280
SVN commit 911492 by tstaerk: Backport 911280: make fish copy reliable. This will be in the KDE 3.5 branch. It was too late for 3.5.10. CCBUGS:147948 M +2 -2 fish.cpp WebSVN link: http://websvn.kde.org/?view=rev&revision=911492
Fish transfer still hangs quite often, while scp works without disruptions... I'm using kde 4.2.0 shipped with Fedora 9.
(In reply to comment #38) > Fish transfer still hangs quite often, while scp works without disruptions... > I'm using kde 4.2.0 shipped with Fedora 9. > OK, where can I download an installable ISO of FC 9 so I can reproduce?
The bug still existst in kubuntu intrepid with kde 4.2, too.
(In reply to comment #40) > The bug still existst in kubuntu intrepid with kde 4.2, too. > On http://www.staerk.de/thorsten/index.php/147948 I give a complete testcase incl. instructions how to kompile it and how to run it. Does this stall?
In order to reproduce it correctly - how did you install KDE 4.2 on Kubuntu? I did a sudo apt-get install kubuntu-desktop on ubuntu 8.10 and got KDE 4.1.
I use the repo from ## KDE 4.2 deb http://ppa.launchpad.net/kubuntu-experimental/ubuntu intrepid main
Frank and I have met on irc and analyzed this. After installing the gpg key according to http://www.kubuntu.org/news/kde-4.2 we were able to kompile and use my testcase. For me, Kubuntu 8.10 x32, there were no errors. For Frank, Kubuntu 8.10 x64, there were the same errors that I experienced before my fix. However, my fix has also been positively tested on x64 (SUSE) Linux.
(In reply to comment #38) > Fish transfer still hangs quite often, while scp works without disruptions... > I'm using kde 4.2.0 shipped with Fedora 9. > I took the time to download FC 9 (x64) and install it into a virtual machine. Now, how do I choose KDE for startup? I only get gnome started.
I found out that the problem is in fish.pl. The line exec('/usr/bin/file','-i','-b','-L',$fn); does not always return on high load. My fix is a workaround on the application level that re-triggers a select and gets things going again.
I have solved the problem on site of fish.pl. The problem is that fork needs some retries to deliver a pid, here is the patch for fish.pl: - return '' if (!defined $pid); +print DEBUG "Entering mimetype4"; + while (!defined $pid) $pid=fork();
(In reply to comment #45) > (In reply to comment #38) > > Fish transfer still hangs quite often, while scp works without disruptions... > > I'm using kde 4.2.0 shipped with Fedora 9. > > > I took the time to download FC 9 (x64) and install it into a virtual machine. > Now, how do I choose KDE for startup? I only get gnome started. OK, I asked you to tell me how to download fedora - you did not. No problem, I found out. I took the time to set up a virtual machine running Fedora. Then I asked you how to make it start KDE by default - no answer from your site. OK, and now I find out it is KDE 4.0 that ships with Fedora 9, not KDE 4.2. So, how do you get KDE 4.2 shipped by Fedora 9?
I just verified this bug is distribution-independant. It also happens with Ubuntu 8.04, kubunut-desktop with KDE 3.5.10.
@tstaerk: should the report be reopened as it seems it is not really fixed? Thanks
There are many bugs around fish. All have in common that nobody gives a way to reproduce them. Before re-opening, please give one (and a description what the problem is).
(In reply to comment #48) Sorry for no replies. I didn't received notifications about your comments (in case you're still interested: 'yum update kdebase' as root).
I have this bug happen to me using KDE 4.2.4 (64-bit) on sidux 2009-02. When transferring a large (over 4GB) file, after copying near half the file it stalls. It never comes back or completes, eventually it times out. I tested this using a path with a symlink and one without. Both scp and sftp (this one from the dolphin window) work fine.
The way to reproduce with some degree of success (still not 100% chance, sorry). I use slow network (100Mbit) and a quite slow server (Epia via, 25% CPU average load). 1. Create a few mid-sized files (20-30 300-1000Mb work for me) on server. 2. Copy a half of them via fish copy-past to local client. Immediately copy another half to create second kio process (I suppose that it increases chance of failure). 3. Press "i" widget and wait -- speed will fall to 0B/s and one of the processes will halt in case of "success". All my attempts to use fish fail in this case, scp works fine.
As Vsevolod provided reproduction instructions and I am still experiencing this issue in KDE 4.3.1 I am reopening the bug. @Vsevolod: can you provide access to your slow server with the files already on it so that KDE devs can simply connect to it and test? Thanks!
*** Bug 183594 has been marked as a duplicate of this bug. ***
Hello, I think I have the same problem while transferring files from an ubuntu server to my computer running KDE 4.7.4 on gentoo. Way to reproduce (at my home) : 1/ Connect with fish to my other computer 2/ Transfer a big file (600MB is fine) from distant computer to mine 3/ The transfer reaches 100% but never finishes. It hangs forever. If I stop it with kde jobs, the file is incomplete with a .part extension to its name and is not valid. Scp or sftp are working just fine. I would be glad to give more information if you can indicate me a way to do that (debug, trace or other). Thx
I exactly have the same problem here. Confirmed with Gentoo kde 4.8.5. I had to scp all the files using command line.
The same problem with big files, archlinux x86_64, KDE 4.9.3
Is there any update on this? The finest and biggest DE should not have such a bug open for a long time. Copying through ssh by KDE is considered a main functionality whereas it is broken. I always have to do it by a terminal. For reproducing the results, even you do not need a slow conncetion. On a 1Gb network, it easily happens: Just copy a large file. Even worse, if you tried to copy a lot of small to medium-sized files, it would definitely stall again. The users have always appreciated your valuable work and would even do more, if you could kindly kill this super-annoying bug.
I am also suffering from this! Copying small files via fish works fine - but when copying a large number of big files (>2 GB per file) fish eventually stalls. Since some of the machines I have to work with do only have ssh login, fish is the only option to copy my files - but with kio-fish (i.e. in krusader) that does not work. My fallback solution is "mc", whose fish implementation seems to be better than kio's.