Bug 83832 - Verifying written data is very slow
Summary: Verifying written data is very slow
Status: RESOLVED FIXED
Alias: None
Product: k3b
Classification: Applications
Component: general (show other bugs)
Version: unspecified
Platform: Compiled Sources Linux
: NOR normal with 20 votes (vote)
Target Milestone: ---
Assignee: Sebastian Trueg
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2004-06-22 23:22 UTC by Simon Munton
Modified: 2005-07-22 14:23 UTC (History)
0 users

See Also:
Latest Commit:
Version Fixed In:


Attachments
patch solves the very slow CD verification after write is finished (853 bytes, text/x-diff)
2005-04-14 00:30 UTC, R. Sijrier
Details
Update to previous attachment, contextual now (I hope) (1.31 KB, patch)
2005-04-14 13:23 UTC, R. Sijrier
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Simon Munton 2004-06-22 23:22:41 UTC
Version:           0.11.11 (using KDE KDE 3.2.3)
Installed from:    Compiled From Sources
OS:                Linux

The verify phase of burning takes a long time to complete.

Writing a 698MB CD at 16x speed took approximately 5 min 30 sec. Verifying the data took 12 min 30 sec.

Immediately mounting the cd in the cdwriter, without reloading it, and calculating the md5sums of all the files took 2 min 19 sec. The md5sum of the files on hard disk took 18 secs.

It would seem that the verify should be capable of completing in around 3 minutes instead of the 12 minutes it does take.
Comment 1 Simon Munton 2004-06-24 23:03:22 UTC
It looks like its down to my wretched Samsung SW-248F writer again. I tried using a Plextor writer and the verify completed much faster.

Some more information: I tried changing the size of the buffer used for the md5 calculation in k3bmd5job.cpp to see what effect it would have.
Times given are how long it took to verify 247MB of data with various buffer sizes.

BUFFERSIZE = 2048*10;                   322 seconds
BUFFERSIZE = 2048*16;                   239 seconds
BUFFERSIZE = 2048*32;                   167 seconds
BUFFERSIZE = 2048*64;                   99 seconds

Increasing the buffersize had a dramatic effect on the time to verify. BUT, when BUFFERSIZE was greater than 2048*32, there were a lot of debug messages about falling back to stdlib read:
k3b: (K3bCdDevice::CdDevice) /dev/sr0: READ 10 failed!
k3b: (K3bIso9660) falling back to stdlib read

It would appear that using the stdlib read on my Samsung writer is the fastest way to do the verify.
As a test, I changed K3bIso9660::read() to only read using the stlib read and set the md5 buffer size to 2048*10; The verify in this case took 99 seconds.

Not sure what to suggest as a fix. Maybe an advanced configuration option that forces stdlib read to be used? Or automatically based on the writer being used? Or even detecting the verify was slower than writing and either setting an option automatically or giving the user a hint?
Comment 2 Alberto Lusiani 2005-01-10 11:01:12 UTC
I have used k3b 0.11.18 with the BenQ DV1620 burner, on a DVD+RW, and found verify extremely slow. I have made the following observations, which hold
whatever is the amount of data written to the DVD, even for very small DVDs:

* the verify progress bar fills up quickly, quicker than DVD writing untill
  it reaches exactly 50%

* after 50%, the progress bar fills up very slowly, and the burner LED flashes
  at about 20 Hz, while before it was basically always on

* if I burn a single file of sizes between 10 Mb to 700 Mb, K3b verify works
  as described above, however running MD5sum on the file completes quickly
  and without errors, at a speed that looks similar to the speed of the 1st 50%
  of k3b verify.  So it does not look like there are read errors on the DVD.

I have yet to look at the code, but it seems to me that it must be something
that can be cured by SW, although it might depend on my specific HW.
Comment 3 Alberto Lusiani 2005-01-17 11:06:44 UTC
This is an update of my former additional comment.  I have done some quick investigations and have come to the same conclusions of simon@munton.demon.co.uk (above).  For my burner (BenQ DW1620), when writing a data DVD:

- verify is extremely slow (about a factor 10 slower than writing) when reading back from the DVD;

- verify is much faster when using the stdlib read() function rather than K3bCdDevice::read10() in K3bIso9660::read().

In K3bIso9660::read(), K3bCdDevice::read10() is invoked, with 10 retries in case of errors. In case of 10 errors in a row, stdlib lseek() and read() are invoked.  I found that there is no error, but the operation is very slow.  Perhaps the BenQ burner does not do buffering when accessed with K3bCdDevice::read10(), and does it when accessed with the stdlib read().

Setting "retries" to 0 rather than to 10, and removing the KDE debug statement reporting failure for K3bCdDevice::read10(), is a simple fix to get verifying complete in a time similar to writing, for my configuration (Duron 1GHz and BenQ DW1620).

In the code, there are comments saying that "retries" might eventually become user-configurable.  This would provide a useful handle to cure slow verify and I advise adopting this solution.
Comment 4 R. Sijrier 2005-04-13 21:23:33 UTC
Hi,

Same problem here, though the md5sum calculation is relatively fast _before_ burning.
Verification _after_ burning is exceptionally slow, it's also still weird the md5sum is calculated _again_ for the image on disk first.

I looked at top, and to my surprise both kicker and xorg where consuming 70% of cpu recourses!
Could it be a (the) problem that the kicker applet updating routine is looping  (or something similar) and thus consuming as much cpu as it can get?

Both kicker and xorg returned to normal cpu usage (almost zero) right after the md5sum calculations were finished. 
The k3b kicker icon (or is it an applet) was _very_ flashy during verification and started to be normal again also after verification was finished....

Hope this is of some help.
Comment 5 R. Sijrier 2005-04-13 21:56:33 UTC
Investigated a bit more.

It seems the k3bjobprogresssytemtray gets a signal for each processed piece of data by the md5sum calculation, that is every 2 Kbytes of processed data.

On a CD of 700 MB, the systemtray icon get's updated 358.400 times, which takes some time you know ;-)
(Qt perhaps reduces this number of repaints, but still...)

Maybe I'm wrong, trying to compile k3b now, and see if I can make a fix for this...
If I have something working, I'll try to post a patch

Best,

Remon
Comment 6 R. Sijrier 2005-04-14 00:30:35 UTC
Created attachment 10615 [details]
patch solves the very slow CD verification after write is finished

The attached patch should solve the problem.
Please try it out!

Is there a reason why the data reads are done with only 2048 bytes at a time?
Perhaps some speed improvements can be made here too :-)

Best,

Remon
Comment 7 Sebastian Trueg 2005-04-14 10:35:29 UTC
thanks a lot. :)
could you please provide a patch with context information instead. That would make it easier to apply to both cvs and 0.11.x
Comment 8 R. Sijrier 2005-04-14 12:49:19 UTC
Sure, if I know how to do it ;-)

I'm not used to this stuff, so actually I don't understand what you want to have..





 
Comment 9 R. Sijrier 2005-04-14 13:23:36 UTC
Created attachment 10623 [details]
Update to previous attachment, contextual now (I hope)
Comment 10 Sebastian Trueg 2005-04-16 10:59:12 UTC
thanks. patch is applied. :)
Comment 11 Sebastian Trueg 2005-04-16 10:59:25 UTC
*** Bug has been marked as fixed ***.
Comment 12 Alberto Lusiani 2005-07-22 14:23:01 UTC
I have just tried k32 0.12.2 on Scientific Linux 3 (clone of RHEL 3), with a Duron 1000 and a BenQ DV1620 DVD recorder.  For me verify is still very slow,
about 10 times slower than writing.  It must be specific to my HW configuration, as I do not observe the same behaviour with other systems (Athlon 2400+ with NEC 3500 or LG 4163 burners).  For my case, it would be nice to have an option to use 
the stdlib read() routine rather than K3bCdDevice::read10() without recompiling.