SUMMARY I was performing a backup which includes three VM images - one ~1GB .iso, two .qcow2 files, one 16GB and one 84GB in size. On viewing the resulting archive from Kbackup, the other files all exist, including two of the largest files (1GB and 16GB) but the 84GB file is nowhere to be found. I watched Kbackup compress the file for some two hours (see https://bugs.kde.org/show_bug.cgi?id=479204) and there is no error in the output suggesting something went wrong, but the file is simply not there. Looking at my previous backups which should include this file, it is not present in any of them. So apparently I've never had a backup of this file, and it's the primary reason I'm making this backup. I thought that compressing 100GB of data down to a 6GB archive was impressive, now I see how such a high ratio was achieved :D Jokes aside, this is pretty bad. If there's anything I can do to assist, please do let me know, I would like to help if I can. STEPS TO REPRODUCE 1. Backup large files 2. Open archive 3. Note missing files OBSERVED RESULT File not included in backup archive EXPECTED RESULT File included in backup archive SOFTWARE/OS VERSIONS OpenSUSE Tumbleweed KDE Plasma Version: 5.27.10 KDE Frameworks Version: 5.113.0 Qt Version: 5.15.11 ADDITIONAL INFORMATION System has 32GB of RAM. I don't see much memory usage, but obviously there is some limit to the maximum file size that will be included, and I thought perhaps it might be memory? That's just a random guess, though, I have no indication that is the case. The output from Kbackup, anonymized (replaced my username) and abbreviated (cut out lots of lines of small (<1KB) files), to show that there are no errors reported: ...Lots of small files... /home/username/Documents/_NoCoW/_VM_Images /home/username/Documents/_NoCoW/_VM_Images/Live_Install.qcow2 (16.8 GiB) ...compressing file /home/username/Documents/_NoCoW/_VM_Images/Live_Install.qcow2 /home/username/Documents/_NoCoW/_VM_Images/obs-server.x86_64-qcow2_unmodified.qcow2 (84.1 GiB) ...compressing file /home/username/Documents/_NoCoW/_VM_Images/obs-server.x86_64-qcow2_unmodified.qcow2 /home/username/Documents/_NoCoW/_VM_Images/openSUSE-Tumbleweed-KDE-Live-x86_64-Snapshot20230902-Media.iso (908.6 MiB) ...compressing file /home/username/Documents/_NoCoW/_VM_Images/openSUSE-Tumbleweed-KDE-Live-x86_64-Snapshot20230902-Media.iso ...Lots of small files... ...finished slice /run/media/username/Sandisk Pro/Kbackup/kbackup libvirt_2023.12.31-06.53.35_1.tar -- Filtered Files: 0 -- Backup successfully finished -- PS Happy New Year!
Thinking about this some more, it occurred to me that there are really two separate issues to address here: 1) The missing file 2) The log stating a success when it wasn't This reminds me of that TV show "air crash investigations" where there's never a single thing that goes wrong to cause a disaster, it has to be a chain of failures. This is that. If either one of those two things had happened without the other, it wouldn't have been so bad - I'd either have the file, or I'd have the knowledge that something went wrong and maybe I don't have the file. But the way it is, I am led to believe I have the file, and I don't, and that's a recipe for a disaster. Sorry to double the workload here, but I thought it was an important realisation that this bug is really two and needs two things fixed...
Hi. Thanks for your report - jour style is fun to read :-) I tried to reproduce this issue here, but using a smaller file with ca. 65GB and some other smaller files. The backup completed here after more than 5(!) hours... But the backup archive did correctly include all files, also the large one! Also checking the source and I can't see how a file would not be included in the tar file without an error given. Just to be sure: How did you check the content of the archive after the backup? Does "tar -tvf <backup.tar>" also not show the large file?
(In reply to Martin Koller from comment #2) > Hi. Thanks for your report - jour style is fun to read :-) > > I tried to reproduce this issue here, but using a smaller file with ca. 65GB > and some other smaller files. > The backup completed here after more than 5(!) hours... > But the backup archive did correctly include all files, also the large one! > > Also checking the source and I can't see how a file would not be included in > the tar file without an error given. > > Just to be sure: How did you check the content of the archive after the > backup? > Does "tar -tvf <backup.tar>" also not show the large file? Hello again Martin and thanks once again for looking into this for me. I really didn't expect a quick reply at this time of year, I hope that I have not interrupted your celebrations, and that you are enjoying the festive season. I looked for the file using dolphin at first, and then I tried ark. It was a better idea to try it with tar, thanks for finding the right arguments for me, you saved this dummy from having to read the manpage :D Unfortunately, I don't see the file listed there, either. There's obviously something 'special' about my PC that is causing this, but I have no idea what it might be. The specifics of the operation are that might be unique are that I am running Kbackup with kdesu (so I get root access to files for backup), the file is stored on a BTRFS file system and has the C attribute set, in order to disable copy-on-write (which I read as a recommendation for VM images and databases), and I am backing up to an external USB 3.1 (aka 3.2 Gen2x1 - the 10GBps variety) NVMe hard drive. I might try a few experiments and see if I can get it to include the file, and that should give us a hint. I will try without compression, and try backing up to a different destination (the same internal NVMe disk - there is enough space, and a different USB-connected spinning-rust type drive), and try without kdesu (This file doesn't need root access), and try backing up only this file (rather than using the profile I am using presently).... Hopefully one or more of those will work, and give us a hint about why it failed. I will return with some more info after I do the experiments.
Dear Bug Submitter, This bug has been in NEEDSINFO status with no change for at least 15 days. Please provide the requested information as soon as possible and set the bug status as REPORTED. Due to regular bug tracker maintenance, if the bug is still in NEEDSINFO status with no change in 30 days the bug will be closed as RESOLVED > WORKSFORME due to lack of needed information. For more information about our bug triaging procedures please read the wiki located here: https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging If you have already provided the requested information, please mark the bug as REPORTED so that the KDE team knows that the bug is ready to be confirmed. Thank you for helping us make KDE software even better for everyone!
> This bug has been in NEEDSINFO status with no change for at least > 15 days. I'm sorry for the delay here. I have several bugs open for my machine (I seem to have a talent for finding unusual broken stuff) and on top of that, I am rather ill and disabled, and it can seriously mess with my ability to meet deadlines. I've had this tab open the entire time, and it is on my to-do list, unfortunately as I mentioned that's a long list, and I have trouble getting to it even if it's short. As I am quite positive this is a real issue, in order to appease the bot, I am going to change the status on this case before providing the info, otherwise it'll end up getting closed and I'll just end up making a mess filing a new bug and referring back to this one, when I am able to get said info.