Bug 168062

Created attachment 26560 [details]
patch to calculate directory completion percentages

This patch lets you see how much of the torrent is completed on a directory by
directory basis.

Created attachment 26587 [details]
updated patch

-One traversal of tree instead of two when initializing
-Make sure don't update percentages if they've not yet changed
-Don't calculate percent for root directory, just use torrent percent.
-Don't bother saving the number of chunks downloaded for files, only
directories

Note: Any cpu/mem usage increase before was not noticeable even on my very slow
computer, and this theoretically reduces it much more.

There is one serious problem with this patch :

You assume that all files in a directory are adjacent to each other chunk wise, but there is no such requirement in the bittorrent specification. So it is quite possible to have two files in the same directory and one file goes from chunk 0 to 20 and the second one goes from 50 to 100.

So what we need is not end_chunks pair but a list of those, and that would complicate things considerably. Maybe another approach is better.

So you're saying you could have a torrent like this?

topdir/subdir1/file1 0-5
topdir/subdir1/file2 10-15
topdir/subdir2/file1 5-10
topdir/subdir2/file2 15-20

How exactly would the program creating the torrent end up doing that? I can't think of a logical way. I can see this happening:

topdir/subdir1/file1 5-10
topdir/subdir1/file2 0-5
topdir/subdir2/file1 15-20
topdir/subdir2/file2 10-15

depending on the sorting, which the endchunks method handles, but truly out of tree order? All the torrents I've tried this on have worked as expected. I certainly believe you're right about the spec, but can you find a torrent that's actually like that?

Anyway, I can rewrite it so it creates a bitset of which chunks are in the directory, intersects that with the chunks you already have, and uses num_on of the intersect divided by num_on of the chunks that are in the directory. It's definitely more operations but it should be OK.

At least my slow computer is a good test for the cpu usage of this. I'll find a big many directory, many file torrent to test this on.

Created attachment 26698 [details]
updated patch: use bitset to track chunks in directory

This patch uses a bitset per node to track the chunks that are in a directory.
It compares that bitset to the downloaded chunks to get a completion
percentage.

In addition, I've made some other slight improvements. It should detect early
if the percentage should be 100% or 0%. It should also handle completion
percentages of directories with excluded or deleted files correctly. The
excluded and seeded bitsets are also or'd only when necessary to reduce the
amount that has to be done for every directory percentage node that has to be
updated.

There does seem to be a slight delay when it's initializing all of the nodes on
a slow computer. Hopefully a few people can test it with torrents with lots of
chunks and lots of files in subdirectories to see what the speed is for them.

OK, I will try the patch out tomorrow

I have modified the patch somewhat, caching the percentages, and only calculating them when they are updated.

SVN commit 844327 by guisson:

Added modified patch from Adam Forsyth showing the directory percentages in the file tree

BUG: 168062


 M  +1 -1      libbtcore/interfaces/torrentfileinterface.h  
 M  +3 -4      libbtcore/torrent/torrentfile.cpp  
 M  +11 -0     libbtcore/util/bitset.cpp  
 M  +6 -0      libbtcore/util/bitset.h  
 M  +72 -14    libktcore/torrent/torrentfiletreemodel.cpp  
 M  +10 -3     libktcore/torrent/torrentfiletreemodel.h  
 M  +2 -2      plugins/infowidget/iwfilelistmodel.cpp  
 M  +1 -1      plugins/infowidget/iwfilelistmodel.h  
 M  +39 -3     plugins/infowidget/iwfiletreemodel.cpp  
 M  +1 -1      plugins/infowidget/iwfiletreemodel.h  


WebSVN link: http://websvn.kde.org/?view=rev&revision=844327

Since I've used your patch subdirectories show up as 0.00% complete.

THis is fixed now : 

http://websvn.kde.org/?view=rev&revision=844844

It doesn't handle the percentages correctly when there are excluded / only seed chunks. You need to logical or the downloadedChunksBitset with those bitsets and use the resultant bitset for the percentage calculations.

It calculates the percentage for the root node. This is wasted work logical or'ing bitsets, as you should just use the completion percentage of the torrent.

If the files are excluded then their chunks will be set to false in the downloaded chunks bitset (with the exception of border chunks, if they are downloaded they will be set to true), so it is normal to have a small percentage of an excluded file, if it borders a non excluded file.

If they are only seed then we still have the chunks which are downloaded, and so we need to include those in the percentage.

So downloaded chunks bitset is exactly what we need.

If you have a directory with two files of the same size, one completely downloaded and the other excluded / deleted, the directory is 100% downloaded, not 50% downloaded. If the file we're downloading is 75% complete, the directory is 75% complete. You need to only look at chunks that are not marked excluded / only seed.

Here's the basic logic from my patch:

havechunks = tc->excludedChunksBitSet();
havechunks.orBitSet(tc->onlySeedChunksBitSet());
BitSet chunks(n->getChunks(false));
chunks -= havechunks;
BitSet donthavechunks(chunks);
donthavechunks -= tc->downloadedChunksBitSet();
percent = 100.0f * (float)(chunks.numOnBits() - donthavechunks.numOnBits()) / chunks.numOnBits();

So you take the chunks you have, and remove any chunks flagged as only seed / exlcuded. (If you will never have excluded chunks, you can take that one line out.) Then you take that set of chunks, and compare it to the set of chunks you still want to download (which is that set minus the downloaded chunks set). That gives you correct percentages when some files in a directory are excluded / only seed.

You simply don't want to consider those files as being part of the torrent at all: otherwise, when we're done downloading, the percentages won't all be 100%, and while we're downloading they won't accurately reflect our progress on the torrent.

And if the purpose of caching the percentages as well as the bitsets was to save work, then you shouldn't be doing the extra work for the root node.

SVN commit 848197 by guisson:

Take only seed chunks into account for file percentages

BUG: 168062


 M  +7 -1      libbtcore/torrent/torrentfile.cpp  
 M  +12 -5     plugins/infowidget/iwfiletreemodel.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=848197