Bug 257241 - corrupt DVD/Blu-Ray image if files > 4GB
Summary: corrupt DVD/Blu-Ray image if files > 4GB
Status: RESOLVED FIXED
Alias: None
Product: k3b
Classification: Applications
Component: Data Project (show other bugs)
Version: 2.0.1
Platform: Slackware Linux
: NOR major
Target Milestone: ---
Assignee: k3b developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-11-18 13:51 UTC by Stéphane Berthelot
Modified: 2018-11-21 19:00 UTC (History)
8 users (show)

See Also:
Latest Commit:
Version Fixed In:
Sentry Crash Report:


Attachments
mkisofs test case. (696 bytes, application/x-xz)
2013-02-03 19:40 UTC, dE
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Stéphane Berthelot 2010-11-18 13:51:26 UTC
I cloned the bug #221638 to reopen it. Please see my first comments on the original bug.

Moreover I've spent some time to test and figured out it was the "-sort" option that makes the bug appear. K3b in my case produces an empty tmp file for use with the -sort option. Not using sort when the list is empty would be a quick and efficient workaround until the bug is fixed in mkisofs.

I used k3b 2.0.1 on Slackware64-current with crdtools/mkisofs 3.0.0

Original non-working k3b command line:
/usr/bin/mkisofs -gui -graft-points -volid "Vol 1" -volset "" -appid "K3B THE CD KREATOR (C) 1998-2010 SEBASTIAN TRUEG AND MICHAL MALEK" -publisher "" -preparer "" -sysid "LINUX" -volset-size 1 -volset-seqno 1 -sort k3bmj3024.tmp -rational-rock -hide-list k3bkL3024.tmp -no-cache-inodes -udf -full-iso9660-filenames -iso-level 3 -path-list k3beo3024.tmp

Working tested command line (on a shell, copying the temp files from k3b) :
/usr/bin/mkisofs -gui -graft-points -volid "Vol 1" -volset "" -appid "K3B THE CD KREATOR (C) 1998-2010 SEBASTIAN TRUEG AND MICHAL MALEK" -publisher "" -preparer "" -sysid "LINUX" -volset-size 1 -volset-seqno 1 -rational-rock -hide-list k3bkL3024.tmp -no-cache-inodes -udf -full-iso9660-filenames -iso-level 3 -path-list k3beo3024.tmp

The result with non working command line is all files below 4G are corrupted, others are ok.
On working command lines all files are correct (tested with MD5/SHA1/SFV tools)

I think this bug should be reported to mkisofs/cdrtools team so I'll try to do it.
But k3b may implement an easy workaround that is to not use "-sort" option when sort list is empty (in my case the file k3bmj3024.tmp was 0 byte ...)
A better approach would be to disable sorting files (not used often I think) when files > 4GB are found on project and flawed mkisofs version ...


+++ This bug was initially created as a clone of Bug #221638 +++

Version:           1.69.0 (using KDE 4.2.4)
OS:                Linux
Installed from:    Slackware Packages

When I create a DVD image containing a file which is bigger than 4 GB, then mount the image with "mount -o loop", most of the files in the image are corrupted. I have used the "Very Large Files (UDF)" filesystem option.

But if I use mkisofs directly, the files in the image are correct (checked with md5sum). The command I used is based on the command that k3b itself runs:

/usr/bin/mkisofs -volid 'foo' -rational-rock -joliet -joliet-long -no-cache-inodes -udf -full-iso9660-filenames -iso-level 3 -o bar.iso /path/to/files/*

The version of mkisofs is:
mkisofs 2.01.01a57 (x86_64-unknown-linux-gnu) Copyright (C) 1993-1997 Eric Youngdale (C) 1997-2008 Jörg Schilling

The version of k3b is 1.69.0alpha4, however I have produced it with k3b 1.0.5 as well. Perhaps the problem is in mkisofs or the way it is used.

Please let me know if more information is required.
Comment 1 joerg.schilling 2010-11-23 16:45:21 UTC
The support code for files > 4 GB was added to mkisofs three years ago and so far nobody did report a problem.

To the end of last week I received this bug report and I created a preliminary test version at: ftp://ftp.berlios.de/pub/cdrecord/alpha/

please report whether the problem is still present.
Comment 2 Stéphane Berthelot 2010-11-23 17:58:37 UTC
I confirm that the bug is fixed in cdrtools 3.01a01-pre

I still advise k3b team to according to cdrtools version checked :
- if >= 3.01a01 pre, allow files > 4GB + sort option
- if <= 3.00 , disallow sorting when files > 4GB are found in project and do not output -sort command line argument
Comment 3 joerg.schilling 2010-11-25 12:01:26 UTC
Please use the final 3.01a01 release as it fixes another bug that is only present with UDF enhancements.

ftp://ftp.berlios.de/pub/cdrecord/alpha/cdrtools-3.01a01.tar.bz2
Comment 4 Itai 2012-11-05 13:39:44 UTC
Still have the problem as of the latest on OpenSUSE 12.2 64bits. Yesterday tried several attempts even smaller files are corrupted in the presence of a single 4GB+ file.

The only workaround that worked once so far is to make a purely UDF disk and ignore the warnings about other directory structures being needed.
Comment 5 joerg.schilling 2012-11-06 14:50:03 UTC
A while ago, somebody claimed a similar bug and send a complete instruction to repeat the problem.

I did reeat the creation call and checked the created filesystem image sector by sector and could not find any problem.

I thus assume that you are using an OS that causes the problem. Please make a bugreport against the OS in question.
Comment 6 Itai 2012-11-07 13:42:19 UTC
There are a lot of potential interactions which could cause this, so I would not be inclined to dismiss it as an OS bug without further investigation. In any case, the software is closer to the problem, so it would be easier to solve or identify on this side.

For the record, this occurs on my system consistently over 3 different OS versions and at least 6 kernels. Since 11.2, every version above had the problem. That is when I got the Blu-Ray, so I have no prior data.

The drive in question is a Pioneer DB-RW BDR-205 (SATA) and my motherboard has an AMD 785G chipset to name the most likely components to be causing this interaction.

The problem is thankfully reproducible too on a rewritable disk, so I am happy to run any diagnostics suggested to  identify the root cause of the problem.
Comment 7 dE 2012-12-20 16:53:52 UTC
(In reply to comment #2)
> I confirm that the bug is fixed in cdrtools 3.01a01-pre
> 
> I still advise k3b team to according to cdrtools version checked :
> - if >= 3.01a01 pre, allow files > 4GB + sort option
> - if <= 3.00 , disallow sorting when files > 4GB are found in project and do
> not output -sort command line argument

I confirm this problem is fixed for a01-pre, but comes back in a different (opposite actually) way with 3.01_alpha06 at least, as stated by a comment here -- 

https://bugs.gentoo.org/show_bug.cgi?id=394553#c20

As of the original bug report, I confirm that with 3.0.
Comment 8 dE 2013-01-10 16:58:25 UTC
But a01-pre has a differnet problem.
Comment 9 dE 2013-01-10 17:04:12 UTC
Is removing the sort option that hard?
Comment 10 joerg.schilling 2013-01-15 14:26:23 UTC
Again: up to now, not a single problem tat looks similar to the problem mentioned here
could be repeated since 30.1a01. I should also mention, that there was no change to
mkisofs since 3.01a01 that could affect large file behavior.

If you still see problems, I recommend you to send information that allows to repeat the
problem.

Note that will not create an image and try to mount it but rather create an image and
verify the content of the image by hand. This avoids from being fooled by bugs in the OS:

If the ISO image is OK but the OS lists problems, you will need to make a bug report
against the OS filesystem drivers.
Comment 11 dE 2013-02-02 05:40:47 UTC
I've a test case, but I checked it by mounting on Linux. I'll attach it soon.

This bug persists on BSD, Linux and Windows alike and everyone agrees that this's an mkisofs problem.

In the mean time I moved to squashfs, tar or 7z

http://delogics.blogspot.com/2013/01/burning-raw-filesfilesystemsarchivestar.html

You've to admit that iso* FS is broken by design; who'd design a FS which only accepts a few handful characters as filename and have strange restrictions on them which makes it feel like an FS designed for Windows.

On top of that they try to fix the horribly broken FS by adding 'extensions' and 'levels' which further complicate the issue.

So I think squashfs is a much better option; it has inbuilt powerful xz compression and officially supports sorting also with almost all unix attributes.
Comment 12 joerg.schilling 2013-02-02 14:49:56 UTC
I am sorry to see that you still missunderstand the problem.

It may be that you believe that there is a problem in mkisofs, but as long as you do not send an instruction on how to repeat your problem, we have no more than an unproven claim.

Once you send instructions on how to repeat your problem, I am able to check whether there is a real problem in the created image or not.

Whatever you do instead does not help for your problem - sorry.

Note that you may send the needed file meta data using "star" without a need to give real file content. Anything you do instead of writing down how to repeat the problem does not help you with your problem.


Note that while iso-9660, UDF and similar are standardized filesystems with no known problems, squashfs is just a vendor unique non-standard fs. I cannot recommend to use it.
Comment 13 dE 2013-02-03 19:40:14 UTC
This problem does not persist for alpha 11, but with 3.00.

Test case attached.
Comment 14 dE 2013-02-03 19:40:48 UTC
Created attachment 76886 [details]
mkisofs test case.
Comment 15 joerg.schilling 2013-02-04 14:05:02 UTC
Thank you for verifying that the problem has been fixed with cdrtools-3.01a01 (from November 25 2010) already.

Note that this is what I wrote in my first remark from, November 23 2010 already which I made to announce that the preliminary fix  a few days before 3.01a01 was released.

The related code did not change since then, so the problem is finally fixed since November 25 2010, all  releades since 3.01a01 (including the currently recent 3.01a11) work with respect to large files and sorting.

Your testcase is a simplified version from what I did use while fixing the bug.

It seems that the main problem is to tell people to upgrade to a recent enough version that includes the fix. It seems that it would be a good idea if k3b could warn users in case they are using a too old version of mkisofs.
Comment 16 dE 2013-02-06 18:40:56 UTC
(In reply to comment #15)
> Thank you for verifying that the problem has been fixed with
> cdrtools-3.01a01 (from November 25 2010) already.
> 

No, it's fixed with alpha 11. Alpha 10 had the same problem, and Alpha 01 has a different issue which I don't remember.
Comment 17 joerg.schilling 2013-02-07 10:57:27 UTC
It seems that you have been confused while testig.

Let me list the recent changes in mkisofs:

3.01a11 no change at all in mkisofs, so 3.01a10 is identical
3.01a10 Revert a change to dvd_file.c::uniq() -> only hits together with "-dvd-video": no effect for your case
3.01a09 Fix a bug with adding another session with multi-session and Rock Ridge: no effect for your case
3.01a08 Fix a bug related to modification date in the PVD: no effect for your case
3.01a07 No change at all in mkisofs, so 3.01a06 is identical
3.01a06 No change at all in mkisofs, so 3.01a05 is identical
3.01a05 Changes related to multi-boot and support for UEFI boot: no effect for your case
3.01a04 Only typo fixes in man page and comments: no effect for your case
3.01a03 Man page enhanced: no effect for your case
3.01a02 Introducing gettext(): no effect for your case
3.01a01 A bug was fixed with -sort and files > 4 GB.

As you see, 3.01a01 is the only version that is related to your problem.

I have no idea what you tested, but if you tested 3.01a01 ... 3.01a11, you did not observe any difference with repect to large files together with -sort.
Comment 18 dE 2013-02-07 15:26:34 UTC
As of the current time, Gentoo only provides alpha 11 ebuild, so it was alpha 11.

I downloaded alpha 01 separately to test a few (may be 1) months back.

I can second this by various people complaining in --

https://bugs.gentoo.org/show_bug.cgi?id=394553

And one of them still complains now.
Comment 19 joerg.schilling 2013-02-07 15:44:03 UTC
To make things easier to verify, I recommend to run "isodebug" on the ISO FS image file.

"isodebug" prints information on the mkisofs version that has been used and on the command line (file names are simplified to prevent leaking secret imformation about the computer.

isodebug test.iso
ISO-9660 image includes checksum signature for correct inode numbers.
ISO-9660 image created at Thu Feb  7 16:40:03 2013

Cmdline: '3.01a11 -o test.iso OBJ'

This allows you to verify that you did not use a different mkisofs version by accident (as a result of PATH or similar...).
Comment 20 Piotr Mitas 2013-02-07 15:54:16 UTC
(In reply to comment #5)
> A while ago, somebody claimed a similar bug and send a complete instruction
> to repeat the problem.

That was probably me.

> I did reeat the creation call and checked the created filesystem image
> sector by sector and could not find any problem.
> 
> I thus assume that you are using an OS that causes the problem. Please make
> a bugreport against the OS in question.

 You told me that you couldn't reproduce the bug on Solaris. I downloaded a Solaris live DVD and succeeded in reproducing this issue. The difference was that it didn't error out outright, but read garbage data from the disc, evidenced by a changed md5 sum and garbage characters printed to console while cat-ing a file that was supposed to be filled with 0s. I sent you a reply email with these findings and you didn't respond. 

Note that I can reliably reproduce this issue on Linux, Windows and Solaris. I doubt all 3 of those systems have the same bug in their UDF implementations.

To make it clear, this bug is still present in alpha11. I can create a broken image by doing
mkisofs -iso-level 3 -udf -sort emptyfile -o badimage.iso badfile
where badfile is larger than 4GiB and emptyfile is an empty file or a valid sort file.

Also remember that it only shows when mounting the image as UDF, the file can be read correctly with mount -t iso9660.
Comment 21 joerg.schilling 2013-02-07 21:48:38 UTC
Thank you for pointing to a UDF problem that is verifyable on Solaris. This helped to
finally understand the problem.

Maybe you now understand why I was trying to ask for a confirmation after I detected
a consistency problem with the previous claimes on when a problem is visible.

Mkisofs is a program that creates complex filesystem images with usually more than one 
logical filesystem inside. Without knowing the exact contraints related to a problem, it
is impossible to understand whether a reported problem exists at all and what it causing
the problem.

It seems that some tests may have mounted different filesystem types than expected 
or reported and that may of course crate very confusing observations.

Now the real cause for the problem from  Piotr Mitas can be repeated and a course fix
exists. I expect testable new code next week.
Comment 22 dE 2013-02-08 05:22:39 UTC
(In reply to comment #21)
>... 
> It seems that some tests may have mounted different filesystem types than
> expected 
> or reported and that may of course crate very confusing observations.
> ...

And now I cant reproduce this bug with alpha 10 even.

What version do you want me to test with?
Comment 23 joerg.schilling 2013-02-08 10:16:59 UTC
If you cannot reproduce it, you seem to have a problem that is caused by the fact that
you may mount different filesystemtypes from a hybrid filesystem.

For Solaris, you need to manually decide between UDFS and hsfs(iso-9660)/Rock-Ridge/Joliet
in case the filesystem is not mounted by the automounter.

In such cases, the hsfs filesystem module uses an internal ranking in order to decide on
which filesystem type to mount and the hsfs module sets up (modifies) the mount flags
in order to allow to see the mounted filesystem type from "mount -p".

I have no idea on how this works on Linux....

In any case, the basic structure on Linux has been copied from Solaris and for this reason there also is a distinction between a UDF module and a module for the iso-9660 based filesystems.
Comment 24 dE 2013-02-09 04:31:40 UTC
There is.

I mounted the FS as iso9660, where, strangely, it could access the 4+GB file, and everything was ok.
Comment 25 joerg.schilling 2013-02-09 10:04:35 UTC
This is themain problem here:

A bug has been reported in 2010 and this bug has been fixed at that time.

Later, another unrelated (beacuse located in a different filesystem) bug is detected and incorrectly reported to the previous problem.

Testing either with iso-9660 or udf thus gives different results and creates the impression of an inconsistent bug report that does not match the source. Note that the iso-9660 bug was finallx fixed long ago and that the udf problem always exists.
Comment 26 dE 2013-02-09 14:25:52 UTC
I'm still sure this problem still persists, but hard to reproduce.

I'll keep using k3b, and report if I find something.
Comment 27 joerg.schilling 2013-02-09 14:56:29 UTC
There is a new test prerelease at:

ftp://ftp.berlios.de/pub/cdrecord/alpha/cdrtools-3.01a12-pre.tar.bz2

Please test and report
This version fixes two UDF related bugs. A final beta version from that will be created next week.
Comment 28 dE 2013-02-10 04:23:24 UTC
Got it! Found the bug in alpha 11.

isodebug /mnt/bad_hdd/test.iso 
ISO-9660 image created at Sun Feb 10 09:30:06 2013

Cmdline: '3.01a11 -gui -graft-points -volid K3b data project -volset  -appid K3B THE CD KREATOR (C) 1998-2010 SEBASTIAN TRUEG AND MICHAL MALEK -publisher  -preparer  -sysid LINUX -volset-size 1 -volset-seqno 1 -sort .../k3bnv6839.tmp -no-cache-inodes -udf -allow-leading-dots -full-iso9660-filenames -relaxed-filenames -allow-lowercase -no-iso-translate -allow-multidot -max-iso9660-filenames -iso-level 3 -path-list .../k3blc6839.tmp'

dmesg gives -- 

[33399.565453] attempt to access beyond end of device
[33399.565456] loop1: rw=0, want=16305864, limit=16305860
[33399.565458] attempt to access beyond end of device
[33399.565459] loop1: rw=0, want=16305864, limit=16305860

On accessing the file.

I'll reproduce the case with zero files and attach the test case.

Also testing in alpha 12.
Comment 29 dE 2013-02-10 04:48:18 UTC
But it's fixed in alpha12-pre.
Comment 30 dE 2013-02-10 04:49:07 UTC
However, the image's isodebug says -- 

Cmdline: '3.01a11 -gui -graft-points -volid K3b data project -volset  -appid K3B THE CD KREATOR (C) 1998-2010 SEBASTIAN TRUEG AND MICHAL MALEK -publisher  -preparer  -sysid LINUX -volset-size 1 -volset-seqno 1 -sort .../k3bS31538.tmp -no-cache-inodes -udf -allow-leading-dots -full-iso9660-filenames -relaxed-filenames -allow-lowercase -no-iso-translate -allow-multidot -max-iso9660-filenames -iso-level 3 -path-list .../k3bU31538.tmp'

Alhpa 11?
Comment 31 Piotr Mitas 2013-02-10 13:02:27 UTC
(In reply to comment #27)
> There is a new test prerelease at:
> 
> ftp://ftp.berlios.de/pub/cdrecord/alpha/cdrtools-3.01a12-pre.tar.bz2
> 
> Please test and report
> This version fixes two UDF related bugs. A final beta version from that will
> be created next week.

I tested it and indeed it seems to be fixed.
Comment 32 joerg.schilling 2013-02-10 13:04:58 UTC
Fixing bugs begins with knowing the contraints to reproduce a problem and
then to analyze the ptoblem.

Sine a few days, I know that the problem was specific to UDF. Testing in this area
finally verified a consistent and reproducable problem.

A big problem was that even on Solaris, the bug that caused superfluous UDF
directory entries was not flagged by the kernel.

It seems that we need either a new "udfinfo" or UDF extensions in isoinfo.

BTW: the new "version" is not a new version but a development snapshot. 
A real version will appear after some more testing.
Comment 33 dE 2013-02-11 04:31:14 UTC
Finally!

Thank you for the fix!
Comment 34 joerg.schilling 2013-02-11 10:47:51 UTC
With this bugfix and the tests I did last weekend, I am now sure that UDF ist mostly OK.

What still needs to be done is:

gid is wrong for symlinks

sometimes ctime is wrong - this does not look good because the alternate time value is not explainable

hardlinks and linkcounts need to be implemented.

When the things above are done (note that before spring 2007, UDF was similar to -r only with simplified metadata) UDF should become as usable as Rock Ridge is since autumn 2006 -> 100% verified.
Comment 35 joerg.schilling 2013-02-12 10:18:29 UTC
Hi,

yesterday the final cdrtools-3.01a12 have been publised.

I am currently working on enhancing the UDF part of the hybrid image generation in mkisofs in order to support all useful meta data and more file types. The related code will appear soon in 3.01a13.

Note that the UDF part in mkisofs was originally only designed to support DVD-Video.

If you like to get developer related information or discuss things, I recommend to
check the cdrtools related mailing lists at:

http://developer.berlios.de/mail/?group_id=5

and to subscribe to the lists of interest.
Comment 36 dE 2013-02-14 13:35:55 UTC
I found a bug in pre alpha 12. One of the files was corrupt.

I'll try to reproduce it with alpha 12.
Comment 37 joerg.schilling 2013-02-14 13:53:51 UTC
The releases are identical execept for the version string.

This evening, there will be a new development snapshot in:

ftp.berlios.de/pub/schily/

that supports more meta data with UDF. The code has been verified
by creating a backup copy from a 10 GB Solaris installation root Filesystem.

There have been no differences except that Solaris and Linux ignore the
inode number from UDF and compute an own number that is granted to be
within 32 bits and that breaks hard links.

BTW: the best was to compare complete file trees including all meta data
is to use star -diff -v ...

As before: a bug exists, in case it can be repeated and thus verified...
Comment 38 joerg.schilling 2013-02-19 16:42:38 UTC
Note that if you believe that there is a bug in alpha12, this must be a different bug
and we thus need new instructions on how to repeat your problem.

If you don't confirm a new problem, I will release the code snapshot that was released in

ftp://ftp.berlios.de/pub/schily/schily-2013-02-15.tar.bz2

as cdrtools-3.01a13 in a few days.
Comment 39 dE 2013-02-20 09:52:33 UTC
No problems yet.

Thanks for the fix!! Finally!
Comment 40 Andrew Crouthamel 2018-11-12 02:45:03 UTC
Dear Bug Submitter,

This bug has been stagnant for a long time. Could you help us out and re-test if the bug is valid in the latest version? I am setting the status to NEEDSINFO pending your response, please change the Status back to REPORTED when you respond.

Thank you for helping us make KDE software even better for everyone!
Comment 41 Andrew Crouthamel 2018-11-21 04:25:03 UTC
Dear Bug Submitter,

This is a reminder that this bug has been stagnant for a long time. Could you help us out and re-test if the bug is valid in the latest version? This bug will be moved back to REPORTED Status for manual review later, which may take a while. If you are able to, please lend us a hand.

Thank you for helping us make KDE software even better for everyone!
Comment 42 joerg.schilling 2018-11-21 11:15:19 UTC
As reported in comment #39 already, the bug in question has been fixed
5.5 years ago with cdrtools-3.01a13.

Since the current cdrtools version is 3.02a10, it is safe to check, whether
the installed software reports at least the string "3.02*".

Please however be careful: Some unfriendly Linux distros still ship the
dead fork called "cdrkit" that results in a source state from May 2004.
So you should warn your users in case that old software has been installed.

Make sure to use a PATH that finds /opt/schily/bin/mkisofs before
/usr/bin/mkisofs and inform your users that recent software is at this
location:

Bi-weekly snapshots (recommended):
http://sourceforge.net/projects/schilytools/files/

Less frequent regular release:
http://sourceforge.net/projects/cdrtools/files/alpha

So called "stable" release:
http://sourceforge.net/projects/cdrtools/files/
Comment 43 Andrew Crouthamel 2018-11-21 19:00:19 UTC
Thanks for the update!