Bug 158816

Summary: Use chunk checksums in Metalinks to repair downloads
Product: [Applications] kget Reporter: Ant Bryan <anthonybryan>
Component: metalinkAssignee: KGet authors <kget>
Status: RESOLVED FIXED    
Severity: wishlist CC: gerlosgm, mat69
Priority: NOR    
Version: unspecified   
Target Milestone: ---   
Platform: openSUSE   
OS: Linux   
Latest Commit: Version Fixed In:

Description Ant Bryan 2008-03-05 19:24:26 UTC 
Version:            (using KDE 4.0.1)
Installed from:    SuSE RPMs
OS:                Linux

Metalinks contain mirror, checksum, & other information for downloads in an XML format. Most Metalinks also include chunk checksums (aka partial file checksums, as opposed to full file checksums) for segments of a file. These chunk checksums can be used to determine if a segment of the download is corrupted. If so, the corrupted chunk can be requested again.

For large downloads, where Metalinks are frequently used, this can save users considerable time and hassle, along with saved bandwidth.

This is one of the most important Metalink features, I hope KGet can support it.

Example Metalinks with chunk checksums can be found at http://download.packages.ro/metalink/openoffice/

The <pieces> section here contains an example of chunk checksums, some of which have been excised (...) to save space.

    <verification>
     <hash type="md5">74c20c8c4f5df40958278a148feec93b</hash>
     <hash type="sha1">0145902fe029c6b6e98ceb4d70715367f0bf462a</hash>
     <hash type="ed2k">af9ae7c710d769abee63d273429a40d7</hash>
    
     <pieces length="1048576" type="sha1">
      <hash piece="0">4a183d5f17bc8db275f7ad8be821432275d34d9e</hash>
      <hash piece="1">7dcc2812b6f3422f3bf5ec9a4bbd76444b1d072c</hash>
      <hash piece="2">a1e4b602c151a464a77f81ac27d41712cfa6ba3f</hash>
      ...
      <hash piece="136">b7c9cb2306c42b825d0bae212ae89ebd1f25fa44</hash>
      <hash piece="137">58c5a413f32702e5b3ad75568f60d608e287406e</hash>
     </pieces>   

    </verification>
Comment 1 Lukas Appelhans 2008-03-05 19:33:29 UTC 
Good idea =)
That will especially useful for multisourcedownloads for metalink.

Thx

Lukas
Comment 2 Ant Bryan 2008-03-05 22:17:46 UTC 
yes, it would definitely be great for multisource downloads!

enough distributions use Metalink for ISO downloads, and errors in a 600MB to 3 GB file can be extremely frustrating. errors can be repaired w/ aria2 & metalinks (or by torrents, rsync) but we want things to be as easy for the user as possible.

for instance, he's an openSUSE page on Metalinks: http://en.opensuse.org/Metalink

most of the page covers how to repair a download, doing things that many users would rather not do. the information to repair the file is already in the .metalink, so hopefully more Metalink clients will make things easier on the user :)

I think if KGet supported repairing downloads w/ Metalinks, it would be the preferred download method for ISOs, instead of plain FTP/HTTP, for distributions. I mean, who wouldn't want simpler, error proof downloads?
Comment 3 gerlos 2009-04-18 16:02:33 UTC 
Yes, it would be great!
Comment 4 Matthias Fuchs 2009-09-10 20:28:06 UTC 
In trunk now, though at the moment the checks are only done when the download finished and then (if needed) a repair starts.