Bug 421565 - KArchive creates broken archives (KTar and KZip)
Summary: KArchive creates broken archives (KTar and KZip)
Status: CONFIRMED
Alias: None
Product: frameworks-karchive
Classification: Frameworks and Libraries
Component: general (show other bugs)
Version: 5.98.0
Platform: Other Linux
: NOR critical
Target Milestone: ---
Assignee: David Faure
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-05-15 13:19 UTC by hadmut
Modified: 2023-02-19 14:05 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In:
fritzibaby: Brainstorm+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description hadmut 2020-05-15 13:19:00 UTC
Hi, 

just experienced with both AppImages 19.12.3 and 20.04.0b running on Ubuntu 18.04:

If a video project contains large video clips, kdenlive silently creates broken unreadable archive files, when using the archive function and selecting to generate a compressed archive file (.tar.gz). 

I've just used kdenlive to create a video from recordings made with a professional video recorder, where video files (mp4) have sizes between 8 and 13 GByte.

When using the archive function from kdenlive, kdenlive creates a tar.gz file and reports success, no problems, but then the generated archive files are unreadable by kdenlive itself. 

Even the regular tar command line tool can't read them. When trying to tar ztf FILENAME, it lists the contained files until the huge files come, and then aborts with 

...
videos/Aufnahme1.mp4
tar: Skipping to next header
tar: Exiting with failure status due to previous errors

It's not related to the gzip compression, it's a problem of the tar archive itself.

Maybe this is related to the fact that the older/original versions of tar file formats had a file size limit of 8GB and could not handle larger files. 

However, this is extremely dangerous, since it does not just crash kdenlive or create bad videos, it can cause the loss of the raw files, the sources.
Comment 1 hadmut 2020-05-16 12:15:53 UTC
I just tested 20.04.1b released this morning, where the release notes tell that the archiving file bug had been fixed, but I still see the same problem.
Comment 2 emohr 2020-06-28 18:07:51 UTC
I think it's similar to issue https://invent.kde.org/multimedia/kdenlive/-/issues/535 that Kdenlive uses the tar standard with all its limitations. 

How big would be the tar file (assuming that MP4 would not be more compressed)?
Comment 3 hadmut 2020-06-28 18:14:37 UTC
Well, there's several revisions of tar to overcome this problem, but I don't think tar is the appropriate format for this task here, owners and permissions are not really useful here. 

It might be an option to use zip instead.
Comment 4 farid 2020-06-29 01:45:39 UTC
Probably it is also more compatible with windows, since it probably doesn't support tar.
Comment 5 hadmut 2020-08-15 12:12:37 UTC
Is there any chance to get this fixed or at least removed or give some warning?


It is really a nasty trap to offer an archiving function that might silently destroy the project (and thus important and expensive work), which people could rely on.
Comment 6 hadmut 2020-10-20 21:00:35 UTC
Bug still exists in 20.08.2
Comment 7 Jean-Baptiste Mardelle 2020-10-21 21:59:17 UTC
Git commit 1989120b658c661cc34456e235cf77c367c17f8a by Jean-Baptiste Mardelle.
Committed on 21/10/2020 at 21:58.
Pushed by mardelle into branch 'release/20.08'.

Project archiving: check after each file if archiving works, add option to use zip instead of tar.gz

M  +41   -15   src/project/dialogs/archivewidget.cpp
M  +1    -0    src/project/dialogs/archivewidget.h
M  +27   -8    src/ui/archivewidget_ui.ui

https://invent.kde.org/multimedia/kdenlive/commit/1989120b658c661cc34456e235cf77c367c17f8a
Comment 8 Jean-Baptiste Mardelle 2020-10-21 22:01:48 UTC
Git commit 9e077005c7922458eba95ccd6f4194ca9e3f055d by Jean-Baptiste Mardelle.
Committed on 21/10/2020 at 22:01.
Pushed by mardelle into branch 'master'.

Project archiving: check after each file if archiving works, add option to use zip instead of tar.gz

M  +40   -14   src/project/dialogs/archivewidget.cpp
M  +1    -0    src/project/dialogs/archivewidget.h
M  +26   -7    src/ui/archivewidget_ui.ui

https://invent.kde.org/multimedia/kdenlive/commit/9e077005c7922458eba95ccd6f4194ca9e3f055d
Comment 9 farid 2020-11-02 19:02:24 UTC
Hello

Could you please test if this is fixed with latest changes?

Thanks
Comment 10 hadmut 2020-11-02 19:09:36 UTC
Is there a new appimage, or would I have to compile?
Comment 11 farid 2020-11-02 19:13:46 UTC
Could you test with the daily appimage build. (It might not work at the moment with Ubuntu prior to 20.04 without qml-module-qtquick-shapes).
Comment 12 hadmut 2020-11-03 19:53:23 UTC
I tried to run

kdenlive-20.11.70-1504519-x86_64.appimage, 

but on both Ubuntu 18.04 and 20.04 I run into 

qrc:/qml/timeline.qml:1376:19: Type Track unavailable
qrc:/qml/Track.qml:284:9: Type Clip unavailable
qrc:/qml/Clip.qml:23:1: module "QtQuick.Shapes" is not installed


even after installing 

sudo apt install qml-module-qtquick-shapes

on the 20.04 machine (not available under 18.04). 



At least ad hoc I don't see how to run the current appimage.
Comment 13 Bug Janitor Service 2020-11-18 04:33:56 UTC
Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least
15 days. Please provide the requested information as soon as
possible and set the bug status as REPORTED. Due to regular bug
tracker maintenance, if the bug is still in NEEDSINFO status with
no change in 30 days the bug will be closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

If you have already provided the requested information, please
mark the bug as REPORTED so that the KDE team knows that the bug is
ready to be confirmed.

Thank you for helping us make KDE software even better for everyone!
Comment 14 hadmut 2020-11-18 17:44:11 UTC
That's why I had responded. 

The daily versions of the appimage just didn't run on my computers.
Comment 15 hadmut 2020-11-18 18:44:20 UTC
Nope, 

not working

kdenlive-21.03.70-5cc266c-x86_64.appimage runs on 20.04, but it tells "Project was successfully archived, while the generated .tar.gz is not readible. 



zip doesn't work either. Again, it says "sucessfully archived", and unzip -l can even list the contents, but not unpack:


% unzip HeuteShowPresseFreiheit2.zip 
Archive:  HeuteShowPresseFreiheit2.zip
warning [HeuteShowPresseFreiheit2.zip]:  21474836480 extra bytes at beginning or within zipfile
  (attempting to process anyway)
file #1:  bad zipfile offset (local header sig):  21474836480
  (attempting to re-compensate)
   creating: videos/
error: invalid zip file with overlapped components (possible zip bomb)
Comment 16 hadmut 2021-01-14 23:18:43 UTC
Both formats (.tar.gz and zip) still not working in 20.12.0 AppImage
Comment 17 Julius Künzel 2021-03-04 00:33:11 UTC
I can not reproduce. For me it works fine (master build)
Comment 18 hadmut 2021-03-04 01:25:13 UTC
So how big was the largest video file in your project?
Comment 19 Julius Künzel 2022-03-21 14:25:30 UTC
I reproduced this with two files: 9,0 GB and 10,7 GB.

It is very likely that this is an upstream issue in KArchive. This report is probably related BUG 329579

Kdenlive: 22.03.80
MLT: 7.5.0
Qt: 5.15.3 (built against 5.15.3 x86_64-little_endian-lp64)
Frameworks: 5.91.0
System: KDE neon User - 5.24
Kernel: linux 5.11.0-46-generic
CPU: x86_64
Comment 20 Julius Künzel 2022-09-18 16:41:09 UTC
This was initially reported to Kdenlive, but I made some tests and I am pretty sure this is a KArchive bug.

If you pack a large *.mp4 file into a KTar Archive, the archive will be broken. As far as I can see, large means bigger than about 8GB. Smaller files do not lead to broken archives.

I tried to archive the same files with Ark and everything worked fine, the archive is not corrupted.

Here is some code that reproduces it: 

```
    std::unique_ptr<KArchive> archive;
    archive = std::make_unique<KTar>(QStringLiteral("/path/to/archive.tar.gz"), QStringLiteral("application/x-gzip"));
    archive->open(QIODevice::WriteOnly);

    QString sourceFile = QStringLiteral("/path/to/EightDotSixGB.mp4");
    QString destinationFile = QStringLiteral("EightSixGB.mp4");

    bool success = archive->addLocalFile(sourceFile, destinationFile);
    // sucess  is true, but archive broken
    archive->close();
```

Broken means Ark show a warning when opening the archive. It is possible to open in read only mode, but if you extract the *.mp4 only part of it can be played.

Extracting the archive with KArchive does not work. It fails on opening:

```
    QString archiveName = QStringLiteral("/path/to/archive.tar.gz");

    KArchive *archive = new KTar(archiveName);

    if (!archive->isOpen()) {
        if (!archive->open(QIODevice::ReadOnly)) {
            qDebug() << archive->errorString();
            // we always end up here with archive->errorString()
            // "Could not read tar header"
            return;
        }
    }

    QDir dir(archiveName.remove(QStringLiteral(".tar.gz")) + QDir::separator());
    if (!dir.mkpath(QStringLiteral("."))) {
        return;
    }

    archive->directory()->copyTo(archiveName + QDir::separator());
    archive->close();
```

KZip does also not work in a similar way. I don't know if it is the same cause or not.  If you try to read a *.zip archive with the above code, archive->errorString() is "Invalid ZIP file. Unrecognized header at offset 643690023" which seems somehow related to BUG: 450597

KF Version: 5.98
Comment 21 Julius Künzel 2023-02-19 14:05:00 UTC
https://invent.kde.org/frameworks/karchive/-/commit/7bf346ed6535176c64ff48e4c28a815565df4485#54ca2e369191acf40811aaf9b8fb8a0465e4a57f_773_779 fixes this for KTar. I does not allow to create archives in that case at all. It would of course be better to support even large files, but at least it does not create broken archives anymore.