Bug 158816 - Use chunk checksums in Metalinks to repair downloads
Summary: Use chunk checksums in Metalinks to repair downloads
Status: RESOLVED FIXED
Alias: None
Product: kget
Classification: Applications
Component: metalink (show other bugs)
Version: unspecified
Platform: openSUSE Linux
: NOR wishlist
Target Milestone: ---
Assignee: KGet authors
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2008-03-05 19:24 UTC by Ant Bryan
Modified: 2009-09-10 20:28 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ant Bryan 2008-03-05 19:24:26 UTC
Version:            (using KDE 4.0.1)
Installed from:    SuSE RPMs
OS:                Linux

Metalinks contain mirror, checksum, & other information for downloads in an XML format. Most Metalinks also include chunk checksums (aka partial file checksums, as opposed to full file checksums) for segments of a file. These chunk checksums can be used to determine if a segment of the download is corrupted. If so, the corrupted chunk can be requested again.

For large downloads, where Metalinks are frequently used, this can save users considerable time and hassle, along with saved bandwidth.

This is one of the most important Metalink features, I hope KGet can support it.

Example Metalinks with chunk checksums can be found at http://download.packages.ro/metalink/openoffice/

The <pieces> section here contains an example of chunk checksums, some of which have been excised (...) to save space.

    <verification>
     <hash type="md5">74c20c8c4f5df40958278a148feec93b</hash>
     <hash type="sha1">0145902fe029c6b6e98ceb4d70715367f0bf462a</hash>
     <hash type="ed2k">af9ae7c710d769abee63d273429a40d7</hash>
    
     <pieces length="1048576" type="sha1">
      <hash piece="0">4a183d5f17bc8db275f7ad8be821432275d34d9e</hash>
      <hash piece="1">7dcc2812b6f3422f3bf5ec9a4bbd76444b1d072c</hash>
      <hash piece="2">a1e4b602c151a464a77f81ac27d41712cfa6ba3f</hash>
      ...
      <hash piece="136">b7c9cb2306c42b825d0bae212ae89ebd1f25fa44</hash>
      <hash piece="137">58c5a413f32702e5b3ad75568f60d608e287406e</hash>
     </pieces>   

    </verification>
Comment 1 Lukas Appelhans 2008-03-05 19:33:29 UTC
Good idea =)
That will especially useful for multisourcedownloads for metalink.

Thx

Lukas
Comment 2 Ant Bryan 2008-03-05 22:17:46 UTC
yes, it would definitely be great for multisource downloads!

enough distributions use Metalink for ISO downloads, and errors in a 600MB to 3 GB file can be extremely frustrating. errors can be repaired w/ aria2 & metalinks (or by torrents, rsync) but we want things to be as easy for the user as possible.

for instance, he's an openSUSE page on Metalinks: http://en.opensuse.org/Metalink

most of the page covers how to repair a download, doing things that many users would rather not do. the information to repair the file is already in the .metalink, so hopefully more Metalink clients will make things easier on the user :)

I think if KGet supported repairing downloads w/ Metalinks, it would be the preferred download method for ISOs, instead of plain FTP/HTTP, for distributions. I mean, who wouldn't want simpler, error proof downloads?
Comment 3 gerlos 2009-04-18 16:02:33 UTC
Yes, it would be great!
Comment 4 Matthias Fuchs 2009-09-10 20:28:06 UTC
In trunk now, though at the moment the checks are only done when the download finished and then (if needed) a repair starts.