Bug 375765

Summary:	KIO-gdrive dies unexpectedly when downloading larger files
Product:	[Frameworks and Libraries] kio-gdrive	Reporter:	Aram <aramgrigoryan>
Component:	general	Assignee:	Elvis Angelaccio <elvis.angelaccio>
Status:	RESOLVED FIXED
Severity:	major	CC:	aramgrigoryan, arojas, chgonzalezg, diego.ml, f.alexander.wilms, fabian, lukasgarcya, maxime.haselbauer, mesoptafel, nate, rikmills, trmdi
Priority:	NOR	Keywords:	usability
Version:	unspecified
Target Milestone:	---
Platform:	Kubuntu
OS:	Linux
Latest Commit:	https://commits.kde.org/kio-gdrive/1b710a46d04ac6ea1d9c272be14d2bd2aa49cf54	Version Fixed In:	1.2.5
Sentry Crash Report:

Description Aram 2017-01-30 19:44:13 UTC

Trying to open ~70 MB video file gives an error message "The process for the gdrive protocol died unexpectedly."

System openSUSE leap 42.2, Plasma 5.8, KDE Frameworks 5.26.0, Qt 5.6.1

Comment 1 Elvis Angelaccio 2017-02-03 12:55:06 UTC

Confirmed, I can reproduced it while downloading ~70 and ~220 Mib files. What happens is the following:

1. gdrive dowloads the file
2. when the download is over, gdrive.so freezes one cpu core, for an amount of time which depends on the file size.
3. gdrive.so dies, Dolphin shows the "died unexpectedly" error message.

Comment 2 Elvis Angelaccio 2017-02-03 18:54:37 UTC

Backtrace seems to point to kio.

#0  0x00007fa9ce71fe66 in __memmove_sse2_unaligned_erms () at /usr/lib/libc.so.6
#1  0x00007fa9ceaeeedd in QByteArray::remove(int, int) (this=this@entry=0xb7a270, pos=pos@entry=0, len=len@entry=16384)
    at tools/qbytearray.cpp:2112
#2  0x00007fa9c35c404f in KDEPrivate::AccessManagerReply::readData(char*, long long) (this=0xb7a260, data=<optimized out>, maxSize=<optimized out>) at /tmp/makepkg/kio/src/kio-5.30.0/src/widgets/accessmanagerreply_p.cpp:148
#3  0x00007fa9cebf061a in QIODevice::read(char*, long long) (this=this@entry=0xb7a260, data=0x7fa97b93e028 "\361\037\255\377\327KZ\370\374\333\365\237\347i\351\266}\367\375\243\237\a\266\334\375\236\235\225w\361\373zw\373i\333\323Ϝ\317\357\037ֿ\235\275>=~\331\377\207\347\306\375\273\031\313\337nv\372\177\336\377\244\326\066\357o\247\306\357N\336b\271\276zO\233_n1[\372\177\371@\300\377\b_\332\376\177\254H", maxSize=maxSize@entry=16384) at io/qiodevice.cpp:1067
#4  0x00007fa9cebf0c80 in QIODevice::readAll() (this=this@entry=0xb7a260) at io/qiodevice.cpp:1232
#5  0x00007fa9bd64b518 in KGAPI2::Job::Private::_k_replyReceived(QNetworkReply*) (this=0xa9b3c0, reply=0xb7a260)
    at /tmp/makepkg/libkgapi/src/libkgapi-5.3.1/src/core/job.cpp:107
#6  0x00007fa9bd64e048 in KGAPI2::Job::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) (_o=<optimized out>, _c=<optimized out>, _id=<optimized out>, _a=<optimized out>) at /tmp/makepkg/libkgapi/src/build/src/core/moc_job.cpp:114
#7  0x00007fa9cecf01b9 in QMetaObject::activate(QObject*, int, int, void**) (sender=sender@entry=0xb93e40, signalOffset=<optimized out>, local_signal_index=local_signal_index@entry=2, argv=argv@entry=0x7ffef57923e0) at kernel/qobject.cpp:3740
#8  0x00007fa9cecf0ad7 in QMetaObject::activate(QObject*, QMetaObject const*, int, void**) (sender=sender@entry=0xb93e40, m=m@entry=0x7fa9c5527380 <QNetworkAccessManager::staticMetaObject>, local_signal_index=local_signal_index@entry=2, argv=argv@entry=0x7ffef57923e0)
    at kernel/qobject.cpp:3602
#9  0x00007fa9c5207d42 in QNetworkAccessManager::finished(QNetworkReply*) (this=this@entry=0xb93e40, _t1=<optimized out>)
    at .moc/moc_qnetworkaccessmanager.cpp:388
#10 0x00007fa9c5208854 in QNetworkAccessManagerPrivate::_q_replyFinished() (this=0xb76ed0) at access/qnetworkaccessmanager.cpp:1367
#11 0x00007fa9c520beb1 in QNetworkAccessManager::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) (_o=<optimized out>, _c=<optimized out>, _id=<optimized out>, _a=<optimized out>) at .moc/moc_qnetworkaccessmanager.cpp:192
#12 0x00007fa9cecf01b9 in QMetaObject::activate(QObject*, int, int, void**) (sender=0xb7a260, signalOffset=<optimized out>, local_signal_index=local_signal_index@entry=1, argv=argv@entry=0x0) at kernel/qobject.cpp:3740
#13 0x00007fa9cecf0ad7 in QMetaObject::activate(QObject*, QMetaObject const*, int, void**) (sender=<optimized out>, m=m@entry=0x7fa9c5529ee0 <QNetworkReply::staticMetaObject>, local_signal_index=local_signal_index@entry=1, argv=argv@entry=0x0) at kernel/qobject.cpp:3602
#14 0x00007fa9c52d66a3 in QNetworkReply::finished() (this=<optimized out>) at .moc/moc_qnetworkreply.cpp:369
#15 0x00007fa9c52d6a65 in QNetworkReply::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) (_o=_o@entry=0xb7a260, _c=_c@entry=QMetaObject::InvokeMetaMethod, _id=_id@entry=1, _a=_a@entry=0x7ffef5792710) at .moc/moc_qnetworkreply.cpp:218
---Type <return> to continue, or q <return> to quit---
#16 0x00007fa9ceccfd2e in QMetaMethod::invoke(QObject*, Qt::ConnectionType, QGenericReturnArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument) const (this=this@entry=0x7ffef57928e8, object=object@entry=0xb7a260, connectionType=Qt::DirectConnection, 
    connectionType@entry=4118358384, returnValue=..., val0=..., val1=..., val2=..., val3=..., val4=..., val5=..., val6=..., val7=..., val8=..., val9=...) at kernel/qmetaobject.cpp:2222
#17 0x00007fa9cecd571a in QMetaObject::invokeMethod(QObject*, char const*, Qt::ConnectionType, QGenericReturnArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument) (obj=obj@entry=0xb7a260, member=member@entry=0x7fa9c3665f4f "finished", type=4118358384, 
    type@entry=Qt::AutoConnection, ret=..., val0=..., val1=..., val2=..., val3=..., val4=..., val5=..., val6=..., val7=..., val8=..., val9=...) at kernel/qmetaobject.cpp:1489
#18 0x00007fa9c35c4b92 in QMetaObject::invokeMethod(QObject*, char const*, Qt::ConnectionType, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument) (val9=..., val8=..., val7=..., val6=..., val5=..., val4=..., val3=..., val2=..., val1=..., val0=..., type=Qt::AutoConnection, member=0x7fa9c3665f4f "finished", obj=0xb7a260) at /usr/include/qt/QtCore/qobjectdefs.h:472
#19 0x00007fa9c35c4b92 in KDEPrivate::AccessManagerReply::emitFinished(bool, Qt::ConnectionType) (this=this@entry=0xb7a260, state=state@entry=true, type=type@entry=Qt::AutoConnection) at /tmp/makepkg/kio/src/kio-5.30.0/src/widgets/accessmanagerreply_p.cpp:487
#20 0x00007fa9c35c7390 in KDEPrivate::AccessManagerReply::slotResult(KJob*) (this=0xb7a260, kJob=0xa99100)
    at /tmp/makepkg/kio/src/kio-5.30.0/src/widgets/accessmanagerreply_p.cpp:431
#21 0x00007fa9cecf01b9 in QMetaObject::activate(QObject*, int, int, void**) (sender=0xa99100, signalOffset=<optimized out>, local_signal_index=<optimized out>, argv=<optimized out>) at kernel/qobject.cpp:3740
#22 0x00007fa9ce441632 in KJob::result(KJob*, KJob::QPrivateSignal) () at /usr/lib/libKF5CoreAddons.so.5
#23 0x00007fa9ce442fa1 in KJob::finishJob(bool) () at /usr/lib/libKF5CoreAddons.so.5
#24 0x00007fa9c58c8392 in KIO::SimpleJob::slotFinished() (this=this@entry=0xa99100)
    at /tmp/makepkg/kio/src/kio-5.30.0/src/core/simplejob.cpp:233
#25 0x00007fa9c58d6746 in KIO::TransferJob::slotFinished() (this=0xa99100)
    at /tmp/makepkg/kio/src/kio-5.30.0/src/core/transferjob.cpp:166
#26 0x00007fa9c58d5211 in KIO::TransferJob::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) (_o=<optimized out>, _c=<optimized out>, _id=<optimized out>, _a=<optimized out>) at /tmp/makepkg/kio/src/build/src/core/moc_transferjob.cpp:157
#27 0x00007fa9cecf01b9 in QMetaObject::activate(QObject*, int, int, void**) (sender=sender@entry=0xb357d0, signalOffset=<optimized out>, local_signal_index=local_signal_index@entry=4, argv=argv@entry=0x0) at kernel/qobject.cpp:3740
#28 0x00007fa9cecf0ad7 in QMetaObject::activate(QObject*, QMetaObject const*, int, void**) (sender=sender@entry=0xb357d0, m=m@entry=0x7fa---Type <return> to continue, or q <return> to quit---
9c5b8c7e0 <KIO::SlaveInterface::staticMetaObject>, local_signal_index=local_signal_index@entry=4, argv=argv@entry=0x0)
    at kernel/qobject.cpp:3602
#29 0x00007fa9c589d443 in KIO::SlaveInterface::finished() (this=this@entry=0xb357d0)
    at /tmp/makepkg/kio/src/build/src/core/moc_slaveinterface.cpp:426
#30 0x00007fa9c589f741 in KIO::SlaveInterface::dispatch(int, QByteArray const&) (this=0xb357d0, _cmd=<optimized out>, rawdata=...)
    at /tmp/makepkg/kio/src/kio-5.30.0/src/core/slaveinterface.cpp:160
#31 0x00007fa9c589da40 in KIO::SlaveInterface::dispatch() (this=0xb357d0)
    at /tmp/makepkg/kio/src/kio-5.30.0/src/core/slaveinterface.cpp:89
#32 0x00007fa9c58a2501 in KIO::Slave::gotInput() (this=0xb357d0) at /tmp/makepkg/kio/src/kio-5.30.0/src/core/slave.cpp:399
#33 0x00007fa9c593e5a5 in KIO::Slave::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) (_o=<optimized out>, _c=<optimized out>, _id=<optimized out>, _a=<optimized out>)
    at /tmp/makepkg/kio/src/build/src/core/KF5KIOCore_automoc.dir/moc_slave_HJU3JQIV4SC6KR.cpp:89
#34 0x00007fa9cecf01b9 in QMetaObject::activate(QObject*, int, int, void**) (sender=0xb41160, signalOffset=<optimized out>, local_signal_index=local_signal_index@entry=0, argv=argv@entry=0x0) at kernel/qobject.cpp:3740
#35 0x00007fa9cecf0ad7 in QMetaObject::activate(QObject*, QMetaObject const*, int, void**) (sender=<optimized out>, m=m@entry=0x7fa9c5b8b540 <KIO::Connection::staticMetaObject>, local_signal_index=local_signal_index@entry=0, argv=argv@entry=0x0) at kernel/qobject.cpp:3602
#36 0x00007fa9c583789f in KIO::Connection::readyRead() (this=<optimized out>)
    at /tmp/makepkg/kio/src/build/src/core/moc_connection_p.cpp:144
#37 0x00007fa9c583789f in KIO::ConnectionPrivate::dequeue() (this=0xb40890)
    at /tmp/makepkg/kio/src/kio-5.30.0/src/core/connection.cpp:46
#38 0x00007fa9cecf1019 in QObject::event(QEvent*) (this=0xb41160, e=<optimized out>) at kernel/qobject.cpp:1263
#39 0x00007fa9cd1603dc in QApplicationPrivate::notify_helper(QObject*, QEvent*) (this=<optimized out>, receiver=0xb41160, e=0xb9d250)
    at kernel/qapplication.cpp:3745
#40 0x00007fa9cd167bf1 in QApplication::notify(QObject*, QEvent*) (this=0x7ffef5794060, receiver=0xb41160, e=0xb9d250)
    at kernel/qapplication.cpp:3502
#41 0x00007fa9cecc48b0 in QCoreApplication::notifyInternal2(QObject*, QEvent*) (receiver=0xb41160, event=event@entry=0xb9d250)
    at kernel/qcoreapplication.cpp:988
#42 0x00007fa9cecc703d in QCoreApplication::sendEvent(QObject*, QEvent*) (event=0xb9d250, receiver=<optimized out>)
    at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:231
#43 0x00007fa9cecc703d in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) (receiver=receiver@entry=0x0, event_type=event_type@entry=0, data=0x9d8c10) at kernel/qcoreapplication.cpp:1648
---Type <return> to continue, or q <return> to quit---
#44 0x00007fa9cecc74a8 in QCoreApplication::sendPostedEvents(QObject*, int) (receiver=receiver@entry=0x0, event_type=event_type@entry=0)
    at kernel/qcoreapplication.cpp:1502
#45 0x00007fa9ced190b3 in postEventSourceDispatch(GSource*, GSourceFunc, gpointer) (s=0xa7b150) at kernel/qeventdispatcher_glib.cpp:276
#46 0x00007fa9c911c587 in g_main_context_dispatch () at /usr/lib/libglib-2.0.so.0
#47 0x00007fa9c911c7f0 in  () at /usr/lib/libglib-2.0.so.0
#48 0x00007fa9c911c89c in g_main_context_iteration () at /usr/lib/libglib-2.0.so.0
#49 0x00007fa9ced194bf in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) (this=0xa892a0, flags=...)
    at kernel/qeventdispatcher_glib.cpp:423
#50 0x00007fa9cecc2d0a in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) (this=0x7ffef5793ba0, flags=...)
    at kernel/qeventloop.cpp:212
#51 0x00007fa9bdd1d310 in KIOGDrive::runJob(KGAPI2::Job&, QUrl const&, QString const&) (this=0x7ffef5794010, job=..., url=..., accountId=...) at /home/elvis/dev/kde/src/kio-gdrive/src/kio_gdrive.cpp:604
#52 0x00007fa9bdd1f17b in KIOGDrive::get(QUrl const&) (this=0x7ffef5794010, url=...)
    at /home/elvis/dev/kde/src/kio-gdrive/src/kio_gdrive.cpp:551
#53 0x00007fa9c588546e in KIO::SlaveBase::dispatch(int, QByteArray const&) (this=0x7ffef5794010, command=67, data=...)
    at /tmp/makepkg/kio/src/kio-5.30.0/src/core/slavebase.cpp:1127
#54 0x00007fa9c587ee16 in KIO::SlaveBase::dispatchLoop() (this=0x7ffef5794010)
    at /tmp/makepkg/kio/src/kio-5.30.0/src/core/slavebase.cpp:301
#55 0x00007fa9bdd1b023 in kdemain(int, char**) (argc=4, argv=0xa21070) at /home/elvis/dev/kde/src/kio-gdrive/src/kio_gdrive.cpp:75
#56 0x000000000040a6ed in  ()
#57 0x000000000040c280 in  ()
#58 0x0000000000404c7e in  ()
#59 0x00007fa9ce6bc291 in __libc_start_main () at /usr/lib/libc.so.6
#60 0x00000000004059ca in _start ()

Comment 3 Elvis Angelaccio 2017-04-17 08:42:56 UTC

*** Bug 378848 has been marked as a duplicate of this bug. ***

Comment 4 trmdi 2017-09-25 07:13:17 UTC

Any news about this bug? My machine always freezes when try downloading/uploading a ~80mb file, I have to hold the power button to force shutdown it.

Comment 5 Colin J Thomson 2018-08-20 17:14:49 UTC

*** Bug 397560 has been marked as a duplicate of this bug. ***

Comment 6 Colin J Thomson 2018-09-02 11:24:33 UTC

*** Bug 398150 has been marked as a duplicate of this bug. ***

Comment 7 Fabian Vogt 2018-09-11 13:30:33 UTC

I had a quick look and if the BT is correct, the bottleneck should be fixed with https://phabricator.kde.org/D15426. Please give it a try, I only build-tested it.

Comment 8 Fabian Vogt 2018-09-22 16:23:59 UTC

Git commit d2d52da38016a4b3443c6b9f65e4d54dd8106134 by Fabian Vogt.
Committed on 22/09/2018 at 16:23.
Pushed by fvogt into branch 'master'.

Avoid QByteArray::remove in AccessManagerReply::readData

Summary:
If m_data contains a relatively large amount of data, calls to readData are
really expensive as it copies the remaining data around in memory.

For some reason, readAll() on an AccessManagerReply results in calls to
readData with a maxSize of 16384 until it returns 0. That's pretty much the
worst case scenario, but also the most likely used one.

Avoid the expensive remove operation by only calling remove if it saves
sufficient memory.

Additionally, bail out if maxSize is < 0.

Test Plan:
Ran accessmanagertest, passes.
Downloaded files with kio-gdrive, not stuck in memmove anymore.
It still dies for very large files, but that's a design issue in kio-gdrive
which can't be addressed in KIO.

Reviewers: #frameworks, elvisangelaccio, dfaure, svuorela

Reviewed By: svuorela

Subscribers: svuorela, ngraham, bruns, kde-frameworks-devel

Tags: #frameworks

Differential Revision: https://phabricator.kde.org/D15426

M  +43   -9    src/widgets/accessmanagerreply_p.cpp
M  +1    -0    src/widgets/accessmanagerreply_p.h

https://commits.kde.org/kio/d2d52da38016a4b3443c6b9f65e4d54dd8106134

Comment 9 Fabian Vogt 2018-09-23 10:16:31 UTC

Git commit 1b710a46d04ac6ea1d9c272be14d2bd2aa49cf54 by Fabian Vogt.
Committed on 23/09/2018 at 10:15.
Pushed by fvogt into branch '1.2'.

Send file contents in chunks

Summary:
KIO can only send a maximum of 0xFFFFFF bytes of data.
TransferJob has an additional limit of 14 MiB.

Test Plan:
Downloaded a 132MiB file, no more "slave died unexpectedly".
Size and md5sum match.

Reviewers: elvisangelaccio, dfaure

Subscribers: dfaure, ngraham

Differential Revision: https://phabricator.kde.org/D15448

M  +13   -1    src/kio_gdrive.cpp

https://commits.kde.org/kio-gdrive/1b710a46d04ac6ea1d9c272be14d2bd2aa49cf54

Comment 10 Colin J Thomson 2018-09-30 10:04:17 UTC

Thank you, 
I just built 1.2.5 on this F28 box and can confirm downloading large files from drive is working.
I tested moving and copying various file types from gdrive to my local machine, the largest being a folder ~ 350 MB of photo's with no issues.

Comment 11 trmdi 2018-09-30 10:07:52 UTC

(In reply to Colin J Thomson from comment #10)
> Thank you, 
> I just built 1.2.5 on this F28 box and can confirm downloading large files
> from drive is working.
> I tested moving and copying various file types from gdrive to my local
> machine, the largest being a folder ~ 350 MB of photo's with no issues.

What about uploading?

Comment 12 Aram 2018-09-30 12:53:19 UTC

finally guys, now you can advertise your product, as only now it is functional...
thank you very much !

Comment 13 mes 2018-10-03 12:17:33 UTC

downloading a file 25Mb or larger gives the errormessage "The process for the gdrive protocol died unexpectedly.
uploading a very large file (1.4gb) gives the same error

system: kubuntu 18.04, plasma 5.12.6
kio-gdrive version: 1.2.4-0ubuntu0.1

The error occurs every time during downloading a large file or uploading a very large file. The copying then just stops.

Comment 14 Fabian Vogt 2018-10-03 12:20:15 UTC

(In reply to mes from comment #13)
> downloading a file 25Mb or larger gives the errormessage "The process for
> the gdrive protocol died unexpectedly.
> uploading a very large file (1.4gb) gives the same error

Uploading is a different bug.

> system: kubuntu 18.04, plasma 5.12.6
> kio-gdrive version: 1.2.4-0ubuntu0.1

You need 1.2.5, which is not released yet.

> The error occurs every time during downloading a large file or uploading a
> very large file. The copying then just stops.

Comment 15 Rik Mills 2018-10-28 07:18:59 UTC

Tested with 1.2.5 on Kubuntu 18.04 and 18.10 with a ~230 MB krita 4.1.5  source tar.

In 18.04 on real hardware seemed to complete after some disk and memory thrashing, but no progress shown to give you much clue.

On 18.10 (in a VM, admittedly) same file on multi errors in the end with the "The process for the gdrive protocol died unexpectedly." message.