Bug 112753

I'm not experiencing any crashes with this torrent.

Does this only happen with this torrent ? Or do other torrents show the same behavior ?

Run ktorrent from the commandline in debug mode :

ktorrent --debug

When it crashes, send me the log file (should be in ~/.kde/share/apps/ktorrent)

SVN commit 462330 by guisson:

Changes :
- Use 64-bit fseek and ftell functions (Fixes 112852 and 112753)

BUG: 112753
BUG: 112852



 M  +5 -2      file.cpp  


--- trunk/extragear/network/ktorrent/libutil/file.cpp #462329:462330
@@ -17,8 +17,10 @@
  *   Free Software Foundation, Inc.,                                       *
  *   51 Franklin Steet, Fifth Floor, Boston, MA 02110-1301, USA.             *
  ***************************************************************************/
+
 #include <string.h>
 #include <errno.h>
+#include <stdio.h>
 #include "file.h"
 
 namespace bt
@@ -75,6 +77,7 @@
 
 	Uint64 File::seek(SeekPos from,Int64 num)
 	{
+	//	printf("sizeof(off_t) = %i\n",sizeof(__off64_t));
 		if (!fptr)
 			return 0;
 		
@@ -87,8 +90,8 @@
 			default:
 				break;
 		}
-		fseeko(fptr,num,p);
-		return ftello(fptr);
+		fseeko64(fptr,num,p);
+		return ftello64(fptr);
 	}
 
 	bool File::eof() const

Created attachment 12740 [details]
example debug sessions

According to my experience, this problem is not resolved. I (still) have
the same problem, downloading torrent
http://torrent.unix-ag.uni-kl.de/torrents/KNOPPIX_V4.0.2DVD-2005-09-23-EN.torrent.

ktorrent 1.1rc1 keeps aborting silently. I patched the source a la
comment #2, but I saw no change. I also use SuSE 9.3. I rebuilt ktorrent
with 'configure --enable-debug=full', started the thing and connected
gdb to it. I saw two problems. First, several times I drop into the
debugger, caused by a SIGPIPE. This is probably only a debugging
nuissance, as ktorrent continues with no problem after a 'continue' from
the debugger. The second problem is a breakpoint, forced by a SIGXFSZ,
or a 'file size limit exceeded'. Of course, there is NO file size limit.
A stack backtrace shows that apparently the stack is corrupted. The
attachment shows the basic build stuff for ktorrent, an example of a
SIGPIPE breakpint, and an example of a SIGXFSZ.

SIGXFSZ seems to be emitted when the file size limit get exceeded, so there must be a limit.

Can you run :
ulimit -a

And see what it says

Gonna reopen it

And make sure you run ulimit on the partition the files get stored.

josn@enterprise:~> ulimit -a
core file size        (blocks, -c) 0
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) 32
max memory size       (kbytes, -m) 604350
open files                    (-n) 1024
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) 5632
virtual memory        (kbytes, -v) 1294960

One extra remark, that I forgot: the 'Time Left' value is jumping everywhere,
from minutes to over 24 hours. I'm sure I dont have the bandwidth to get a over 3GB DVD in a few minutes ;-). Btw, I have 6 Mbps down, but never saw a Down Speed of 100 kb/s or more yet (assuming kilo-bits is displayed, not -bytes).

We display kilobytes per sec.

This is weird there is no file size limit and SIGXFSZ still gets raised. I will  look into this when I get home.

I'm getting it to crash similarly. Glibc claims to have detected double frees,
hence crash.

To make it finish downloads now I just run it on a 'while true; do' loop with
--nofork and --nocrashhandler switches. Output is something along the lines of:

*** glibc detected *** free(): invalid pointer: 0x00002aaab0597010 ***
./runtorrent.sh: line 6:  1484 Aborted                 /opt/kde3/bin/ktorrent --nofork --nocrashhandler
*** glibc detected *** free(): invalid pointer: 0x00002aaab1434010 ***
./runtorrent.sh: line 6:  1852 Aborted                 /opt/kde3/bin/ktorrent --nofork --nocrashhandler
*** glibc detected *** double free or corruption (out): 0x00002aaab018f010 ***
./runtorrent.sh: line 6:  1920 Aborted                 /opt/kde3/bin/ktorrent --nofork --nocrashhandler
./runtorrent.sh: line 6:  2347 Segmentation fault      /opt/kde3/bin/ktorrent --nofork --nocrashhandler
*** glibc detected *** double free or corruption (out): 0x00002aaab081c010 ***
./runtorrent.sh: line 6:  2766 Aborted                 /opt/kde3/bin/ktorrent --nofork --nocrashhandler

SUSE 10.0 RC1 & ktorrent 1.1RC1. Will post cores if needed (once the download
has finished..).

Post those cores, they could be very usefull.

Same problem here..
Also SUSE 93, trying to download Knopix DVD 4.02

I have downloaded 2048Mb so far -- I think there must be some stupid memory limit somewhere,,

It is not only the GamesKnoppix that crashes the ktorrent application, the Knoppix DVD also causes the program to die silently. Use this link for testing:

http://torrent.unix-ag.uni-kl.de/torrents/KNOPPIX_V4.0.2DVD-2005-09-23-EN.torrent

Mostly it takes only some minutes until the crash occurs. Just before the crash the application freezes (no more updates in the ktorrent window), the traffic goes down to zero, the CPU load rises (one CPU of the Hyperthreading CPU has 100% load permanently). The memory consumption does not change, the process is not reported as zombie. In this state the program can be terminated with a "kill -HUP" signal. A bit later it would crash (if not killed). It is typical that the crash occurs mostly after ramping up the speed of the download (with DSL this can be 400 to 500 kilobytes per second). It looks a bit as if it would choke on the incoming data. With lower data rates crashes are less likely. It doesn't matter if I set a download rate limit (I tried high and low values). There is no influence from the window state (minimized window/restored windows) and there is no influence from the display of status (shows the block maps) / files (lists files and selections). Also there is no influence from the selection (cursor sits in the line of the Knoppix torrent or any other line).

The number of chunks cannot be the problem, the OpenSuSE torrents have about the same number of chunks. If I remove the Knoppix torrent, the application is stable again. The file size should be no problem either, the crash happens just minutes after the start of the download after maybe 10 or 20 MB.

With six OpenSuSE torrents alone I had ktorrent running for days with no problems.

I downloaded some 100 MB of that knoppix DVD, with speeds > 500 KB/s, doesn't crash. What version are you running Adolf ?

Hmmm ... I did not want to believe it at first, but more and more it looks like this is a simple numeric overflow at exactly 2 GB file size. The other things I observed could be classified as side-effects. I guess that with the speed ramping up to the full bandwidth there simply is a higher probability that a package gets requested and arrives which should be placed behind the 2 GB limit. The fact that it sometimes takes only minutes for a crash to occur and then half an hour or longer supports my latest observations.

The file for the Knoppix DVD should be about 3.12 GB in size when completed, but even after 1.20 GB have been downloaded (after some crashes and restarts) the ISO file is still 2,147,483,467 bytes (which is exactly 2 GB minus 1 Byte!) in size and it doesn't grow anymore. With so much data already downloaded the file should have (nearly) reached the full size. I cannot believe that at this point there had been no a single packet behind the 2 GB "barrier", this would not be statistically sound and certainly adverse to the idea of scattered transfers.

The progress bar depicting the chunks downloaded so far shows only chunks in roughly the first two thirds but absolutely none in the area that is supposed to be behind the 2 GB "limit".

All this would explain why I can download the 3 GB+ torrents from OpenSuSE, because they do not contain files larger than 700 MB. But the GamesKnoppix DVD is larger and there crashes occur just as with the standard Knoppix DVD.

My version reads "KTorrent 1.1rc1 (Using KDE 3.4.2 level "b" SUSE 10.0).
I'm running a SuSE 10.0 from the commercial box with all updates from SuSE in place. In the package manager I get a version level "1.1rc1-5", for this package there had not been an update yet.

By restarting ktorrent over and over again I got as far as 7915 chunks out of 12760 chunks. The status for this torrent shows all the downloaded chunks to be located in the left two thirds. And the iso file still has exactly 2 GB minus 1 byte in size, which is a bit strange as the download has reached 1.93 GB already.

2 GB of file size, that is the critical issue.

I fixed some issues with large sizes (> 2 GB, I basicly switched to 64 bit integers) in 1.1, so it would be best if you compiled the 1.1 version from source, and try it with that version. Or even better the latest code from SVN.

After the 1.1 .rpm for SuSE 9.3 was released, I upgraded, and indeed, the crashing is over. It is now possible to download a single 8G+ file without any crash. But there are still other problems, at least with large files. I downloaded the SuSE10 DL DVD (http://static.thepiratebay.org/downloadtorrent/3393343.torrent/SUSE_Linux_10.0_DVD_Dual_Layer.3393343.TPB.torrent), but the DVD didnt work. Checking afterwards, the md5sum did NOT match the one published by others. I downloaded the same thing again, and again got an incorrect checksum; and different from the previous one. On the last download (and maybe the first one too) the disk was full at some time. Maybe that caused the problem. Disk full seems not to get noticed by ktorrent: everything keeps running as normal, without any error messages, but the download speed falls back to zero. After making room, it simply starts again.

Jos, can you run ktcachecheck on the directory where the temporary files get stored ? (normally ~/.kde/share/apps/ktorrent/torX , replacing X by the right number) ktcachecheck should be installed as a part of ktorrent package.

Created attachment 13200 [details]
output of ktcachecheck

Attached is the output of ktcachecheck. There are *LOTS* of failed chunks.
Questions:
- apparently ktorrent does not check these checksums itself after downloading.
Shouldnt that be done, and when bad, set the chunk to 'not downloaded', or
simply abort the download ?
- am I now (for a long time already) seeding many erroneous chucks?
- is it somehow possible to invalidate the failed chunks, and retry the
download?
In case you want me to check anything more, I will keep the data for some time,
but probably not for long.

KT does check the checksum, and if chunks fail they do not get saved. The problem occurs when we write things to disk. This can be very helpfull to solve the problem.

Sorry for the delay. I performed some additional tests with the 1.1rc1 version. My interest was to find out if the tracker could be part of the crash problems. So I downloaded some groups of files that were managed by the same tracker as the large files. The result is clear now. The tracker has absolutely no influence on the stability. Ktorrent is running for days now without any problem.

Now there is good news for ktorrent version 1.1.0. I downloaded the RPM for SuSE 10.0 from your download site and installed it. The first thing I did is to start a Knoppix DVD download and check the file size in the cache directory. It now correctly allocated a file of about 3 GB size (so the 2 GB limit is gone) and started the download. The block display shows that blocks with high numbers (which are in the area >2GB for sure) are being retrieved.

The download will be running for some hours (at least). After the download is complete I will report the outcome.

Btw, Adolf don't forget to run ktcachecheck when the download is finished.

Hi Joris,
the download of two Knoppix DVDs with 3,12 GB each has completed without any errors. The ktcachecheck program reported no errors for both downloads. All chunks had been retrieved, none had been missed or excluded.

Good to here that, when I get my new PC, I will be able to solve Jos's problem (I currently do not have the disk space to store 8 GB isos)

This has been fixed for quite a while