Bug 216672 - Opening large (>800MB) archives (tar,zip...) in Dolphin.
Summary: Opening large (>800MB) archives (tar,zip...) in Dolphin.
Status: RESOLVED FIXED
Alias: None
Product: kdelibs
Classification: Unclassified
Component: kdecore (show other bugs)
Version: unspecified
Platform: Gentoo Packages Linux
: NOR normal (vote)
Target Milestone: ---
Assignee: kdelibs bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-11-29 15:51 UTC by id87
Modified: 2010-10-19 19:54 UTC (History)
3 users (show)

See Also:
Latest Commit:
Version Fixed In: 4.5.3


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description id87 2009-11-29 15:51:44 UTC
Version:            (using KDE 4.3.3)
Compiler:          gcc version 4.4.2 (Gentoo 4.4.2 p1.0) 
OS:                Linux
Installed from:    Gentoo Packages

First of my bug is my English. Sorry for that.

This is first time when I'm reporting bug, so maybe I'm wrong.
When I'm opening a zip or tgz big archive (for example 854MB), for a little while nothing happend and then emerge an error "Could not open the file, probably due to an unsupported file format."

File format is *.tgz.
Smaller archives are open with no errors.

Thanks for response and Have a nice day.
Comment 1 Raphael Kubo da Costa 2009-12-13 01:23:34 UTC
Hi. When you click the file in Dolphin, what program is launched to open it?
Comment 2 Raphael Kubo da Costa 2009-12-29 04:08:31 UTC
The reporter seems to have disappeared. Waiting for more information.
Comment 3 id87 2010-03-28 14:11:13 UTC
Thank You for respond.

I'm sorry. I forgot about this. 
The problem existing still. Sorry but I don't understand your question at all.
I'm opening tar,zip in Dolphin. Larger archives do not opeining in Dolphin.

I will put log from Dolphin [from terminal] when He can open archive and when He can't:


TGZ Archve 112 MB (CAN OPEN):


dolphin(5652)/kio (KDirListerCache) KDirListerCache::slotResult: finished listing KUrl("file:///mnt/filmy&inne/Gentoo_BACKUPs_NEW/x86")
dolphin(5652)/kio (KIOJob) KIO::SlaveInterface::dispatch: error  104   "Nie można utworzyć miniaturki dla katalogu"
dolphin(5652)/kio (KDirListerCache) KDirListerCache::forgetDirs: DolphinDirLister(0xb7fbf0)  item moved into cache:  KUrl("file:///mnt/filmy&inne/Gentoo_BACKUPs_NEW/x86")
dolphin(5652)/kio (KDirListerCache) KDirListerCache::listDir: Listing directory: KUrl("tar:/mnt/filmy&inne/Gentoo_BACKUPs_NEW/x86/Home-New-EEE-Backup.tgz")
dolphin(5652)/kio (Slave) KIO::Slave::createSlave: createSlave "tar" for KUrl("tar:/mnt/filmy&inne/Gentoo_BACKUPs_NEW/x86/Home-New-EEE-Backup.tgz")
dolphin(5652)/kio (KIOConnection) KIO::ConnectionServer::listenForRemote: Listening on  "local:/tmp/ksocket-karol/dolphinnT5652.slave-socket"
dolphin(5652)/kio (Slave) KIO::Slave::createSlave: createSlave "tar" for KUrl("tar:/")
dolphin(5652)/kio (KIOConnection) KIO::ConnectionServer::listenForRemote: Listening on  "local:/tmp/ksocket-karol/dolphinmu5652.slave-socket"
dolphin(5652)/kio (Slave) KIO::Slave::createSlave: createSlave "tar" for KUrl("tar:/mnt")
dolphin(5652)/kio (KIOConnection) KIO::ConnectionServer::listenForRemote: Listening on  "local:/tmp/ksocket-karol/dolphinQV5652.slave-socket"
dolphin(5652)/kio (Slave) KIO::Slave::createSlave: createSlave "tar" for KUrl("tar:/mnt/filmy&inne")
dolphin(5652)/kio (KIOConnection) KIO::ConnectionServer::listenForRemote: Listening on  "local:/tmp/ksocket-karol/dolphinyB5652.slave-socket"
dolphin(5652)/kio (Slave) KIO::Slave::createSlave: createSlave "tar" for KUrl("tar:/mnt/filmy&inne/Gentoo_BACKUPs_NEW")
dolphin(5652)/kio (KIOConnection) KIO::ConnectionServer::listenForRemote: Listening on  "local:/tmp/ksocket-karol/dolphinTJ5652.slave-socket"
dolphin(5652)/kio (Slave) KIO::Slave::createSlave: createSlave "tar" for KUrl("tar:/mnt/filmy&inne/Gentoo_BACKUPs_NEW/x86")
dolphin(5652)/kio (KIOConnection) KIO::ConnectionServer::listenForRemote: Listening on  "local:/tmp/ksocket-karol/dolphinJX5652.slave-socket"
dolphin(5652)/kio (Slave) KIO::Slave::createSlave: createSlave "tar" for KUrl("tar:/mnt/filmy&inne/Gentoo_BACKUPs_NEW/x86/Home-New-EEE-Backup.tgz")
dolphin(5652)/kio (KIOConnection) KIO::ConnectionServer::listenForRemote: Listening on  "local:/tmp/ksocket-karol/dolphinwx5652.slave-socket"
dolphin(5652)/kio (Slave) KIO::Slave::createSlave: createSlave "tar" for KUrl("tar:/mnt/filmy&inne/Gentoo_BACKUPs_NEW/x86/Home-New-EEE-Backup.tgz")
dolphin(5652)/kio (KIOConnection) KIO::ConnectionServer::listenForRemote: Listening on  "local:/tmp/ksocket-karol/dolphinLG5652.slave-socket"
dolphin(5652)/kio (KIOJob) KIO::SlaveInterface::dispatch: error  111   "tar:/"
dolphin(5652)/kio (KDirListerCache) KDirListerCache::slotResult: finished listing KUrl("tar:/mnt/filmy&inne/Gentoo_BACKUPs_NEW/x86/Home-New-EEE-Backup.tgz")


*****


TGZ Archive 873 MB (CAN'T OPEN):


dolphin(5682)/kio (KDirListerCache) KDirListerCache::forgetDirs: DolphinDirLister(0x14433e0)  item moved into cache:  KUrl("file:///mnt/filmy&inne/Gentoo_BACKUPs_NEW/x64")
dolphin(5682)/kio (KDirListerCache) KDirListerCache::listDir: Listing directory: KUrl("tar:/mnt/filmy&inne/Gentoo_BACKUPs_NEW/x64/Home_Fresh.tgz")
dolphin(5682)/kio (Slave) KIO::Slave::createSlave: createSlave "tar" for KUrl("tar:/mnt/filmy&inne/Gentoo_BACKUPs_NEW/x64/Home_Fresh.tgz")
dolphin(5682)/kio (KIOConnection) KIO::ConnectionServer::listenForRemote: Listening on  "local:/tmp/ksocket-karol/dolphinPR5682.slave-socket"
dolphin(5682)/kio (Slave) KIO::Slave::createSlave: createSlave "tar" for KUrl("tar:/")
dolphin(5682)/kio (KIOConnection) KIO::ConnectionServer::listenForRemote: Listening on  "local:/tmp/ksocket-karol/dolphinFn5682.slave-socket"
dolphin(5682)/kio (Slave) KIO::Slave::createSlave: createSlave "tar" for KUrl("tar:/mnt")
dolphin(5682)/kio (KIOConnection) KIO::ConnectionServer::listenForRemote: Listening on  "local:/tmp/ksocket-karol/dolphinmu5682.slave-socket"
dolphin(5682)/kio (Slave) KIO::Slave::createSlave: createSlave "tar" for KUrl("tar:/mnt/filmy&inne")
dolphin(5682)/kio (KIOConnection) KIO::ConnectionServer::listenForRemote: Listening on  "local:/tmp/ksocket-karol/dolphinQV5682.slave-socket"
dolphin(5682)/kio (Slave) KIO::Slave::createSlave: createSlave "tar" for KUrl("tar:/mnt/filmy&inne/Gentoo_BACKUPs_NEW")
dolphin(5682)/kio (KIOConnection) KIO::ConnectionServer::listenForRemote: Listening on  "local:/tmp/ksocket-karol/dolphinyB5682.slave-socket"
dolphin(5682)/kio (Slave) KIO::Slave::createSlave: createSlave "tar" for KUrl("tar:/mnt/filmy&inne/Gentoo_BACKUPs_NEW/x64")
dolphin(5682)/kio (KIOConnection) KIO::ConnectionServer::listenForRemote: Listening on  "local:/tmp/ksocket-karol/dolphinTJ5682.slave-socket"
dolphin(5682)/kio (Slave) KIO::Slave::createSlave: createSlave "tar" for KUrl("tar:/mnt/filmy&inne/Gentoo_BACKUPs_NEW/x64/Home_Fresh.tgz")
dolphin(5682)/kio (KIOConnection) KIO::ConnectionServer::listenForRemote: Listening on  "local:/tmp/ksocket-karol/dolphinJX5682.slave-socket"
dolphin(5682)/kio (Slave) KIO::Slave::createSlave: createSlave "tar" for KUrl("tar:/mnt/filmy&inne/Gentoo_BACKUPs_NEW/x64/Home_Fresh.tgz")
dolphin(5682)/kio (KIOConnection) KIO::ConnectionServer::listenForRemote: Listening on  "local:/tmp/ksocket-karol/dolphinwx5682.slave-socket"
dolphin(5682)/kio (KIOJob) KIO::SlaveInterface::dispatch: error  111   "tar:/"
dolphin(5682)/kio (KIOJob) KIO::SlaveInterface::dispatch: error  163   "Nie można otworzyć pliku, prawdopodobnie z powodu nieobsługiwanego formatu pliku.
tar:/mnt/filmy&inne/Gentoo_BACKUPs_NEW/x64/Home_Fresh.tgz"
dolphin(5682)/kio (KIOJob) KIO::SlaveInterface::dispatch: error  163   "Nie można otworzyć pliku, prawdopodobnie z powodu nieobsługiwanego formatu pliku.
tar:/mnt/filmy&inne/Gentoo_BACKUPs_NEW/x64/Home_Fresh.tgz"
dolphin(5682)/kio (KIOJob) KIO::SlaveInterface::dispatch: error  163   "Nie można otworzyć pliku, prawdopodobnie z powodu nieobsługiwanego formatu pliku.
tar:/mnt/filmy&inne/Gentoo_BACKUPs_NEW/x64/Home_Fresh.tgz"
dolphin(5682)/kio (KDirListerCache) KDirListerCache::slotResult: finished listing KUrl("tar:/mnt/filmy&inne/Gentoo_BACKUPs_NEW/x64/Home_Fresh.tgz")

"Nie można otworzyć pliku, prawdopodobnie z powodu nieobsługiwanego formatu pliku.
tar:/mnt/filmy&inne/Gentoo_BACKUPs_NEW/x64/Home_Fresh.tgz" - This means that Dolphin can not open file, due to unsupported file format. 

But file format is the same, only larger.

I hope this helps you.
Comment 4 Raphael Kubo da Costa 2010-03-29 17:41:42 UTC
Right, I'm reopening to investigate further. Thanks.
Comment 5 Raphael Kubo da Costa 2010-03-29 17:44:25 UTC
Actually, you don't seem to be running Ark at all.

Can you go to System Settings/Advanced/File Associations, type "tgz", click in "application" and then "x-compressed-tar", then click "Embedding" and tell what's chosen in the "Left Click Action" box?
Comment 6 id87 2010-03-30 17:56:40 UTC
I did screenshot : http://img534.imageshack.us/img534/7796/tarsett.png

Thanks for helping.
Comment 7 Raphael Kubo da Costa 2010-03-30 20:08:00 UTC
Thanks

* What applications are shown in the other tab in "Application Preference Order"?
* If you click in "application", which of the two options is checked for you?
Comment 8 id87 2010-03-30 20:25:53 UTC
*In "Application Preference Order" is "none" (unactive) I can click Add button.
This is for: x-tar, x-compressed-tar,x-lzma-compressed-tar,xtarz,x-xz-compressed-tar.

*When I click "application" I have checked "Show file in separate viewer"

Generalizing - I have just gentoo-x86 with kde-base-startkde-4.4.1
Comment 9 Raphael Kubo da Costa 2010-06-25 19:55:25 UTC
Sorry for taking so long to come back to this report, it got lost between my inbox and real life.

So do you still experience this issue? From what you've said, apparently you don't have Ark installed, so this may be a bug in kdelibs itself.
Comment 10 Vicente Salvador 2010-10-14 14:07:31 UTC
Problem is still present at KDE 4.4.3 and is not related to dolphin. It's related to kdelibs because trying to open a large zip file on both: dolphin & konqueror, results on same error.

The URL opened is: zip:/tmp/myfile.zip

And the message obtained is "Could not open the file, probably due to an unsupported file format. zip:/tmp/myfile.zip

This message is the same on dolphin and konqueror.

Tried on KDE 4.4.3 on 32-bits system.
Comment 11 Raphael Kubo da Costa 2010-10-14 15:13:50 UTC
Vicente, can you create zip files which show that behaviour at will? Can you give me some steps to follow so that I can try to reproduce it here?
Comment 12 Vicente Salvador 2010-10-14 16:05:51 UTC
Of course. As it's a large one, you can download it from http://www.nabutek.com/MDV2010.11.zip

Download to filesystem and double click with dolphin. I've also tested with x86_64 architecture, and also fails with same error message.

P.D. I'm uploading the file just now. I'll send you an e-mail when upload finished.
Comment 13 Raphael Kubo da Costa 2010-10-14 17:10:24 UTC
Hmm, I can already see the file you're uploading is quite big. Isn't there a way for me to recreate it here (either the file itself or some similar archive which causes the same problem)?
Comment 14 Vicente Salvador 2010-10-14 19:47:04 UTC
Of course,

Take a DVD ISO from your prefered distribution: Mandriva, OpenSuse, Ubuntu, etc.

Create a folder and put the iso file inside this folder. F.E 

MDV2010.10/MYISOFILE.ISO

With dolphin, right click on the folder and select "compress to MDV2010.10.zip"

When finished, try to doubleclick on the created file and it fails!!!!
Comment 15 Raphael Kubo da Costa 2010-10-14 21:04:36 UTC
I was finally able to reproduce it by zipping a 3.4G ISO file on this 32-bit system.

At first glance, the problem seems to be in KZip; more precisely, Q_ASSERT( success ) is being hit. Uncommenting out some debugging lines, I get this with "kziptest list MYFILE.zip":

kziptest(22931)/KZip KZip::openArchive: before interesting dev->pos():  72
kziptest(22931)/KZip KZip::openArchive: compr_size: 0
kziptest(22931)/KZip KZip::openArchive: dev->pos() + compr_size: 72
kziptest(22931)/KZip KZip::openArchive: dev->pos() + compr_size (with cast): 72
kziptest(22931)/KZip KZip::openArchive: after interesting dev->pos():  72
kziptest(22931)/KZip KZip::openArchive: dev->at was successful...
kziptest(22931)/KZip KZip::openArchive: before interesting dev->pos():  173
kziptest(22931)/KZip KZip::openArchive: compr_size: -805515071
kziptest(22931)/KZip KZip::openArchive: dev->pos() + compr_size: -805514898
kziptest(22931)/KZip KZip::openArchive: dev->pos() + compr_size (with cast): -805514898

Reassigning to kdelibs.
Comment 16 Raphael Kubo da Costa 2010-10-14 21:05:01 UTC
Oh, and CC'ing dfaure to the party :)
Comment 17 Raphael Kubo da Costa 2010-10-14 21:43:00 UTC
Actually, an easier way to reproduce this is to use dd to create a reasonably big file (mine ended up being ~3.1GB) from /dev/urandom, and then compress it (zip foo.zip foo.ext). My compressed file ended up being 3.1GB too, and kziptest failed as well.
Comment 18 David Faure 2010-10-16 00:42:40 UTC
SVN commit 1186343 by dfaure:

Fix handling of large .zip files, the size determination ended up negative because of the implicit sign expansion
when doing ... | (uchar)buffer[3] << 24   (int is implicitely used). Tested in isolation, but not with .zip files.
Raphael: please re-test and if this fixes all issues I'll backport and close the bug.
CCBUG: 216672


 M  +6 -6      kzip.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1186343
Comment 19 Raphael Kubo da Costa 2010-10-16 04:19:22 UTC
I've created another big, random file (2GB this time) and made sure "kziptest list" crashed with it before svn up'ing. After your patch it did work indeed.

However "kziptest print" still fails as follows because of some calls to QIODevice::read using a negative maxlen argument. The first one can be traced to KLimitedIODevice receiving int arguments in its constructor and then converting them to qint64 (not sure if changing the argument types is a BIC). I'm still trying to trace the second one -- it makes arr.size() return the correct size, but arr.constData() returns nothing. For some reason, the QIODevice::read() call in QIODevice::readAll passes a negative number as maxlen when result.size() returns 0 and is used in the (result.size() - readBytes) expression.
Comment 20 Raphael Kubo da Costa 2010-10-16 05:42:32 UTC
(In reply to comment #19)
> I'm still trying to trace the second one -- it makes arr.size() return the
> correct size, but arr.constData() returns nothing. For some reason, the
> QIODevice::read() call in QIODevice::readAll passes a negative number as maxlen
> when result.size() returns 0 and is used in the (result.size() - readBytes)
> expression.

Well, before the call to QIODevice::read() there's a call to QByteArray::resize, which takes an int. This in turn makes our big qint64 become a negative integer, and result's size is reset to 0, making things fail later. Not really sure what to do here, since KZip::data() returns a QByteArray that is unable to hold all our data.
Comment 21 David Faure 2010-10-17 20:53:25 UTC
Thanks for the investigations. Calling data() on such a huge amount of data is just wrong and shouldn't be done. It would make any standard machine swap to death. I suggest using other testing methods than "kziptest print", such as extracting with KArchiveFile::copyTo, or from kio_archive.
Comment 22 Raphael Kubo da Costa 2010-10-17 21:56:40 UTC
Does copying a file from the zip ioslave via Konqueror and trying to paste it somewhere else count as a test method? If so, it failed here, even though the archive was shown.
Comment 23 Raphael Kubo da Costa 2010-10-17 22:06:49 UTC
Actually, this last problem is solved by fixing KLimitedIODevice's constructor. After that, I think the commits can be backported and the bug report can be fixed.
Comment 24 Raphael Kubo da Costa 2010-10-18 15:21:13 UTC
SVN commit 1187156 by rkcosta:

Take qint64's instead of int's in KLimitedIODevice's constructor.

The class attributes were already qint64, and receiving int's made it fail on huge (2GB+) files.

Ack-ed by dfaure.

CCBUG: 216672


 M  +1 -1      klimitediodevice.cpp  
 M  +1 -1      klimitediodevice_p.h  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1187156
Comment 25 Raphael Kubo da Costa 2010-10-18 15:42:17 UTC
SVN commit 1187160 by rkcosta:

Backport r1186343, by dfaure.

Fix handling of large .zip files, the size determination ended up negative
because of the implicit sign expansion
when doing ... | (uchar)buffer[3] << 24   (int is implicitely used). Tested in
isolation, but not with .zip files.

CCBUG: 216672


 M  +6 -6      kzip.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1187160
Comment 26 Raphael Kubo da Costa 2010-10-18 15:45:29 UTC
SVN commit 1187161 by rkcosta:

Backport r1187156.

Take qint64's instead of int's in KLimitedIODevice's constructor.

The class attributes were already qint64, and receiving int's made it fail on huge (2GB+) files.

CCBUG: 216672


 M  +1 -1      klimitediodevice.cpp  
 M  +1 -1      klimitediodevice_p.h  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1187161
Comment 27 Raphael Kubo da Costa 2010-10-18 15:47:58 UTC
Now that both commits have been backported, the bug can be closed. Thanks!