I cloned the bug #221638 to reopen it. Please see my first comments on the original bug. Moreover I've spent some time to test and figured out it was the "-sort" option that makes the bug appear. K3b in my case produces an empty tmp file for use with the -sort option. Not using sort when the list is empty would be a quick and efficient workaround until the bug is fixed in mkisofs. I used k3b 2.0.1 on Slackware64-current with crdtools/mkisofs 3.0.0 Original non-working k3b command line: /usr/bin/mkisofs -gui -graft-points -volid "Vol 1" -volset "" -appid "K3B THE CD KREATOR (C) 1998-2010 SEBASTIAN TRUEG AND MICHAL MALEK" -publisher "" -preparer "" -sysid "LINUX" -volset-size 1 -volset-seqno 1 -sort k3bmj3024.tmp -rational-rock -hide-list k3bkL3024.tmp -no-cache-inodes -udf -full-iso9660-filenames -iso-level 3 -path-list k3beo3024.tmp Working tested command line (on a shell, copying the temp files from k3b) : /usr/bin/mkisofs -gui -graft-points -volid "Vol 1" -volset "" -appid "K3B THE CD KREATOR (C) 1998-2010 SEBASTIAN TRUEG AND MICHAL MALEK" -publisher "" -preparer "" -sysid "LINUX" -volset-size 1 -volset-seqno 1 -rational-rock -hide-list k3bkL3024.tmp -no-cache-inodes -udf -full-iso9660-filenames -iso-level 3 -path-list k3beo3024.tmp The result with non working command line is all files below 4G are corrupted, others are ok. On working command lines all files are correct (tested with MD5/SHA1/SFV tools) I think this bug should be reported to mkisofs/cdrtools team so I'll try to do it. But k3b may implement an easy workaround that is to not use "-sort" option when sort list is empty (in my case the file k3bmj3024.tmp was 0 byte ...) A better approach would be to disable sorting files (not used often I think) when files > 4GB are found on project and flawed mkisofs version ... +++ This bug was initially created as a clone of Bug #221638 +++ Version: 1.69.0 (using KDE 4.2.4) OS: Linux Installed from: Slackware Packages When I create a DVD image containing a file which is bigger than 4 GB, then mount the image with "mount -o loop", most of the files in the image are corrupted. I have used the "Very Large Files (UDF)" filesystem option. But if I use mkisofs directly, the files in the image are correct (checked with md5sum). The command I used is based on the command that k3b itself runs: /usr/bin/mkisofs -volid 'foo' -rational-rock -joliet -joliet-long -no-cache-inodes -udf -full-iso9660-filenames -iso-level 3 -o bar.iso /path/to/files/* The version of mkisofs is: mkisofs 2.01.01a57 (x86_64-unknown-linux-gnu) Copyright (C) 1993-1997 Eric Youngdale (C) 1997-2008 Jörg Schilling The version of k3b is 1.69.0alpha4, however I have produced it with k3b 1.0.5 as well. Perhaps the problem is in mkisofs or the way it is used. Please let me know if more information is required.
The support code for files > 4 GB was added to mkisofs three years ago and so far nobody did report a problem. To the end of last week I received this bug report and I created a preliminary test version at: ftp://ftp.berlios.de/pub/cdrecord/alpha/ please report whether the problem is still present.
I confirm that the bug is fixed in cdrtools 3.01a01-pre I still advise k3b team to according to cdrtools version checked : - if >= 3.01a01 pre, allow files > 4GB + sort option - if <= 3.00 , disallow sorting when files > 4GB are found in project and do not output -sort command line argument
Please use the final 3.01a01 release as it fixes another bug that is only present with UDF enhancements. ftp://ftp.berlios.de/pub/cdrecord/alpha/cdrtools-3.01a01.tar.bz2
Still have the problem as of the latest on OpenSUSE 12.2 64bits. Yesterday tried several attempts even smaller files are corrupted in the presence of a single 4GB+ file. The only workaround that worked once so far is to make a purely UDF disk and ignore the warnings about other directory structures being needed.
A while ago, somebody claimed a similar bug and send a complete instruction to repeat the problem. I did reeat the creation call and checked the created filesystem image sector by sector and could not find any problem. I thus assume that you are using an OS that causes the problem. Please make a bugreport against the OS in question.
There are a lot of potential interactions which could cause this, so I would not be inclined to dismiss it as an OS bug without further investigation. In any case, the software is closer to the problem, so it would be easier to solve or identify on this side. For the record, this occurs on my system consistently over 3 different OS versions and at least 6 kernels. Since 11.2, every version above had the problem. That is when I got the Blu-Ray, so I have no prior data. The drive in question is a Pioneer DB-RW BDR-205 (SATA) and my motherboard has an AMD 785G chipset to name the most likely components to be causing this interaction. The problem is thankfully reproducible too on a rewritable disk, so I am happy to run any diagnostics suggested to identify the root cause of the problem.
(In reply to comment #2) > I confirm that the bug is fixed in cdrtools 3.01a01-pre > > I still advise k3b team to according to cdrtools version checked : > - if >= 3.01a01 pre, allow files > 4GB + sort option > - if <= 3.00 , disallow sorting when files > 4GB are found in project and do > not output -sort command line argument I confirm this problem is fixed for a01-pre, but comes back in a different (opposite actually) way with 3.01_alpha06 at least, as stated by a comment here -- https://bugs.gentoo.org/show_bug.cgi?id=394553#c20 As of the original bug report, I confirm that with 3.0.
But a01-pre has a differnet problem.
Is removing the sort option that hard?
Again: up to now, not a single problem tat looks similar to the problem mentioned here could be repeated since 30.1a01. I should also mention, that there was no change to mkisofs since 3.01a01 that could affect large file behavior. If you still see problems, I recommend you to send information that allows to repeat the problem. Note that will not create an image and try to mount it but rather create an image and verify the content of the image by hand. This avoids from being fooled by bugs in the OS: If the ISO image is OK but the OS lists problems, you will need to make a bug report against the OS filesystem drivers.
I've a test case, but I checked it by mounting on Linux. I'll attach it soon. This bug persists on BSD, Linux and Windows alike and everyone agrees that this's an mkisofs problem. In the mean time I moved to squashfs, tar or 7z http://delogics.blogspot.com/2013/01/burning-raw-filesfilesystemsarchivestar.html You've to admit that iso* FS is broken by design; who'd design a FS which only accepts a few handful characters as filename and have strange restrictions on them which makes it feel like an FS designed for Windows. On top of that they try to fix the horribly broken FS by adding 'extensions' and 'levels' which further complicate the issue. So I think squashfs is a much better option; it has inbuilt powerful xz compression and officially supports sorting also with almost all unix attributes.
I am sorry to see that you still missunderstand the problem. It may be that you believe that there is a problem in mkisofs, but as long as you do not send an instruction on how to repeat your problem, we have no more than an unproven claim. Once you send instructions on how to repeat your problem, I am able to check whether there is a real problem in the created image or not. Whatever you do instead does not help for your problem - sorry. Note that you may send the needed file meta data using "star" without a need to give real file content. Anything you do instead of writing down how to repeat the problem does not help you with your problem. Note that while iso-9660, UDF and similar are standardized filesystems with no known problems, squashfs is just a vendor unique non-standard fs. I cannot recommend to use it.
This problem does not persist for alpha 11, but with 3.00. Test case attached.
Created attachment 76886 [details] mkisofs test case.
Thank you for verifying that the problem has been fixed with cdrtools-3.01a01 (from November 25 2010) already. Note that this is what I wrote in my first remark from, November 23 2010 already which I made to announce that the preliminary fix a few days before 3.01a01 was released. The related code did not change since then, so the problem is finally fixed since November 25 2010, all releades since 3.01a01 (including the currently recent 3.01a11) work with respect to large files and sorting. Your testcase is a simplified version from what I did use while fixing the bug. It seems that the main problem is to tell people to upgrade to a recent enough version that includes the fix. It seems that it would be a good idea if k3b could warn users in case they are using a too old version of mkisofs.
(In reply to comment #15) > Thank you for verifying that the problem has been fixed with > cdrtools-3.01a01 (from November 25 2010) already. > No, it's fixed with alpha 11. Alpha 10 had the same problem, and Alpha 01 has a different issue which I don't remember.
It seems that you have been confused while testig. Let me list the recent changes in mkisofs: 3.01a11 no change at all in mkisofs, so 3.01a10 is identical 3.01a10 Revert a change to dvd_file.c::uniq() -> only hits together with "-dvd-video": no effect for your case 3.01a09 Fix a bug with adding another session with multi-session and Rock Ridge: no effect for your case 3.01a08 Fix a bug related to modification date in the PVD: no effect for your case 3.01a07 No change at all in mkisofs, so 3.01a06 is identical 3.01a06 No change at all in mkisofs, so 3.01a05 is identical 3.01a05 Changes related to multi-boot and support for UEFI boot: no effect for your case 3.01a04 Only typo fixes in man page and comments: no effect for your case 3.01a03 Man page enhanced: no effect for your case 3.01a02 Introducing gettext(): no effect for your case 3.01a01 A bug was fixed with -sort and files > 4 GB. As you see, 3.01a01 is the only version that is related to your problem. I have no idea what you tested, but if you tested 3.01a01 ... 3.01a11, you did not observe any difference with repect to large files together with -sort.
As of the current time, Gentoo only provides alpha 11 ebuild, so it was alpha 11. I downloaded alpha 01 separately to test a few (may be 1) months back. I can second this by various people complaining in -- https://bugs.gentoo.org/show_bug.cgi?id=394553 And one of them still complains now.
To make things easier to verify, I recommend to run "isodebug" on the ISO FS image file. "isodebug" prints information on the mkisofs version that has been used and on the command line (file names are simplified to prevent leaking secret imformation about the computer. isodebug test.iso ISO-9660 image includes checksum signature for correct inode numbers. ISO-9660 image created at Thu Feb 7 16:40:03 2013 Cmdline: '3.01a11 -o test.iso OBJ' This allows you to verify that you did not use a different mkisofs version by accident (as a result of PATH or similar...).
(In reply to comment #5) > A while ago, somebody claimed a similar bug and send a complete instruction > to repeat the problem. That was probably me. > I did reeat the creation call and checked the created filesystem image > sector by sector and could not find any problem. > > I thus assume that you are using an OS that causes the problem. Please make > a bugreport against the OS in question. You told me that you couldn't reproduce the bug on Solaris. I downloaded a Solaris live DVD and succeeded in reproducing this issue. The difference was that it didn't error out outright, but read garbage data from the disc, evidenced by a changed md5 sum and garbage characters printed to console while cat-ing a file that was supposed to be filled with 0s. I sent you a reply email with these findings and you didn't respond. Note that I can reliably reproduce this issue on Linux, Windows and Solaris. I doubt all 3 of those systems have the same bug in their UDF implementations. To make it clear, this bug is still present in alpha11. I can create a broken image by doing mkisofs -iso-level 3 -udf -sort emptyfile -o badimage.iso badfile where badfile is larger than 4GiB and emptyfile is an empty file or a valid sort file. Also remember that it only shows when mounting the image as UDF, the file can be read correctly with mount -t iso9660.
Thank you for pointing to a UDF problem that is verifyable on Solaris. This helped to finally understand the problem. Maybe you now understand why I was trying to ask for a confirmation after I detected a consistency problem with the previous claimes on when a problem is visible. Mkisofs is a program that creates complex filesystem images with usually more than one logical filesystem inside. Without knowing the exact contraints related to a problem, it is impossible to understand whether a reported problem exists at all and what it causing the problem. It seems that some tests may have mounted different filesystem types than expected or reported and that may of course crate very confusing observations. Now the real cause for the problem from Piotr Mitas can be repeated and a course fix exists. I expect testable new code next week.
(In reply to comment #21) >... > It seems that some tests may have mounted different filesystem types than > expected > or reported and that may of course crate very confusing observations. > ... And now I cant reproduce this bug with alpha 10 even. What version do you want me to test with?
If you cannot reproduce it, you seem to have a problem that is caused by the fact that you may mount different filesystemtypes from a hybrid filesystem. For Solaris, you need to manually decide between UDFS and hsfs(iso-9660)/Rock-Ridge/Joliet in case the filesystem is not mounted by the automounter. In such cases, the hsfs filesystem module uses an internal ranking in order to decide on which filesystem type to mount and the hsfs module sets up (modifies) the mount flags in order to allow to see the mounted filesystem type from "mount -p". I have no idea on how this works on Linux.... In any case, the basic structure on Linux has been copied from Solaris and for this reason there also is a distinction between a UDF module and a module for the iso-9660 based filesystems.
There is. I mounted the FS as iso9660, where, strangely, it could access the 4+GB file, and everything was ok.
This is themain problem here: A bug has been reported in 2010 and this bug has been fixed at that time. Later, another unrelated (beacuse located in a different filesystem) bug is detected and incorrectly reported to the previous problem. Testing either with iso-9660 or udf thus gives different results and creates the impression of an inconsistent bug report that does not match the source. Note that the iso-9660 bug was finallx fixed long ago and that the udf problem always exists.
I'm still sure this problem still persists, but hard to reproduce. I'll keep using k3b, and report if I find something.
There is a new test prerelease at: ftp://ftp.berlios.de/pub/cdrecord/alpha/cdrtools-3.01a12-pre.tar.bz2 Please test and report This version fixes two UDF related bugs. A final beta version from that will be created next week.
Got it! Found the bug in alpha 11. isodebug /mnt/bad_hdd/test.iso ISO-9660 image created at Sun Feb 10 09:30:06 2013 Cmdline: '3.01a11 -gui -graft-points -volid K3b data project -volset -appid K3B THE CD KREATOR (C) 1998-2010 SEBASTIAN TRUEG AND MICHAL MALEK -publisher -preparer -sysid LINUX -volset-size 1 -volset-seqno 1 -sort .../k3bnv6839.tmp -no-cache-inodes -udf -allow-leading-dots -full-iso9660-filenames -relaxed-filenames -allow-lowercase -no-iso-translate -allow-multidot -max-iso9660-filenames -iso-level 3 -path-list .../k3blc6839.tmp' dmesg gives -- [33399.565453] attempt to access beyond end of device [33399.565456] loop1: rw=0, want=16305864, limit=16305860 [33399.565458] attempt to access beyond end of device [33399.565459] loop1: rw=0, want=16305864, limit=16305860 On accessing the file. I'll reproduce the case with zero files and attach the test case. Also testing in alpha 12.
But it's fixed in alpha12-pre.
However, the image's isodebug says -- Cmdline: '3.01a11 -gui -graft-points -volid K3b data project -volset -appid K3B THE CD KREATOR (C) 1998-2010 SEBASTIAN TRUEG AND MICHAL MALEK -publisher -preparer -sysid LINUX -volset-size 1 -volset-seqno 1 -sort .../k3bS31538.tmp -no-cache-inodes -udf -allow-leading-dots -full-iso9660-filenames -relaxed-filenames -allow-lowercase -no-iso-translate -allow-multidot -max-iso9660-filenames -iso-level 3 -path-list .../k3bU31538.tmp' Alhpa 11?
(In reply to comment #27) > There is a new test prerelease at: > > ftp://ftp.berlios.de/pub/cdrecord/alpha/cdrtools-3.01a12-pre.tar.bz2 > > Please test and report > This version fixes two UDF related bugs. A final beta version from that will > be created next week. I tested it and indeed it seems to be fixed.
Fixing bugs begins with knowing the contraints to reproduce a problem and then to analyze the ptoblem. Sine a few days, I know that the problem was specific to UDF. Testing in this area finally verified a consistent and reproducable problem. A big problem was that even on Solaris, the bug that caused superfluous UDF directory entries was not flagged by the kernel. It seems that we need either a new "udfinfo" or UDF extensions in isoinfo. BTW: the new "version" is not a new version but a development snapshot. A real version will appear after some more testing.
Finally! Thank you for the fix!
With this bugfix and the tests I did last weekend, I am now sure that UDF ist mostly OK. What still needs to be done is: gid is wrong for symlinks sometimes ctime is wrong - this does not look good because the alternate time value is not explainable hardlinks and linkcounts need to be implemented. When the things above are done (note that before spring 2007, UDF was similar to -r only with simplified metadata) UDF should become as usable as Rock Ridge is since autumn 2006 -> 100% verified.
Hi, yesterday the final cdrtools-3.01a12 have been publised. I am currently working on enhancing the UDF part of the hybrid image generation in mkisofs in order to support all useful meta data and more file types. The related code will appear soon in 3.01a13. Note that the UDF part in mkisofs was originally only designed to support DVD-Video. If you like to get developer related information or discuss things, I recommend to check the cdrtools related mailing lists at: http://developer.berlios.de/mail/?group_id=5 and to subscribe to the lists of interest.
I found a bug in pre alpha 12. One of the files was corrupt. I'll try to reproduce it with alpha 12.
The releases are identical execept for the version string. This evening, there will be a new development snapshot in: ftp.berlios.de/pub/schily/ that supports more meta data with UDF. The code has been verified by creating a backup copy from a 10 GB Solaris installation root Filesystem. There have been no differences except that Solaris and Linux ignore the inode number from UDF and compute an own number that is granted to be within 32 bits and that breaks hard links. BTW: the best was to compare complete file trees including all meta data is to use star -diff -v ... As before: a bug exists, in case it can be repeated and thus verified...
Note that if you believe that there is a bug in alpha12, this must be a different bug and we thus need new instructions on how to repeat your problem. If you don't confirm a new problem, I will release the code snapshot that was released in ftp://ftp.berlios.de/pub/schily/schily-2013-02-15.tar.bz2 as cdrtools-3.01a13 in a few days.
No problems yet. Thanks for the fix!! Finally!
Dear Bug Submitter, This bug has been stagnant for a long time. Could you help us out and re-test if the bug is valid in the latest version? I am setting the status to NEEDSINFO pending your response, please change the Status back to REPORTED when you respond. Thank you for helping us make KDE software even better for everyone!
Dear Bug Submitter, This is a reminder that this bug has been stagnant for a long time. Could you help us out and re-test if the bug is valid in the latest version? This bug will be moved back to REPORTED Status for manual review later, which may take a while. If you are able to, please lend us a hand. Thank you for helping us make KDE software even better for everyone!
As reported in comment #39 already, the bug in question has been fixed 5.5 years ago with cdrtools-3.01a13. Since the current cdrtools version is 3.02a10, it is safe to check, whether the installed software reports at least the string "3.02*". Please however be careful: Some unfriendly Linux distros still ship the dead fork called "cdrkit" that results in a source state from May 2004. So you should warn your users in case that old software has been installed. Make sure to use a PATH that finds /opt/schily/bin/mkisofs before /usr/bin/mkisofs and inform your users that recent software is at this location: Bi-weekly snapshots (recommended): http://sourceforge.net/projects/schilytools/files/ Less frequent regular release: http://sourceforge.net/projects/cdrtools/files/alpha So called "stable" release: http://sourceforge.net/projects/cdrtools/files/
Thanks for the update!