Bug 113648 - kioslave camera deletes upon move
Summary: kioslave camera deletes upon move
Status: CLOSED FIXED
Alias: None
Product: kio
Classification: Unclassified
Component: general (show other bugs)
Version: 3.5
Platform: openSUSE RPMs Linux
: NOR grave (vote)
Target Milestone: ---
Assignee: David Faure
URL:
Keywords:
: 118418 (view as bug list)
Depends on:
Blocks:
 
Reported: 2005-09-30 23:27 UTC by Thorsten Staerk
Modified: 2008-01-02 15:27 UTC (History)
3 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
xx.pat (2.63 KB, patch)
2007-05-03 23:08 UTC, Marcus Meissner
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Thorsten Staerk 2005-09-30 23:27:43 UTC
Version:            (using KDE KDE 3.4.0)
Installed from:    SuSE RPMs
OS:                Linux

I love konqueror and KDE, but this time, I am annoyed because of data loss.
I came back from vacation and wanted to move my favorite vacation movie from my digicam to my desktop. I used konqueror and its kioslave camera:/ to do this. Konqueror copied the file to the target and deleted it on the source. But on the target, the file size is 0 byte. This happens again and again when I copy big (about 20MB) movies from my digicam. With gphoto2, the files can be copied without problems. Please solve that error. Is there a way to restore my favorite vacation movie ?
Comment 1 Thiago Macieira 2005-10-01 03:09:09 UTC
data loss -> increasing severity

Can you reproduce this problem with other ioslaves than camera:/ ?
Comment 2 Thorsten Staerk 2005-10-01 07:39:44 UTC
No, I cannot reproduce this with other kioslaves.
Comment 3 Nicolas Goutte 2005-10-01 14:14:49 UTC
"data loss" is supposed to be "grave" in Bugzilla.
Comment 4 Thiago Macieira 2005-10-01 16:51:05 UTC
Are you sure it's camera:/ not kamera:/ ?
Comment 5 Thorsten Staerk 2005-10-01 17:44:42 UTC
yes, it is camera:/
Comment 6 Marcus Meissner 2005-10-01 18:00:49 UTC
which camera is it exactly?
Comment 7 Thorsten Staerk 2005-10-02 04:51:03 UTC
I do not think that matters, as stated, gphoto2 gets the movies perfectly. Problem source seems to be the size of 20MB per movie (it works well for smaller stuff).
It is a Kodak EasyShare V550.
Comment 8 Thiago Macieira 2005-10-02 07:03:32 UTC
camera:/ has nothing to do with gphoto2
(kamera:/ does)
Comment 9 Marcus Meissner 2005-10-02 09:51:41 UTC
incorrect, camera:/ is the kioslave based on libgphoto2.

kamera:/ is unknown to me.
Comment 10 Stephan Kulow 2005-10-02 10:05:52 UTC
I just noticed, the camera.protocol from kmobile is not installed. So it's indeed Marcus to blame ;)
Comment 11 Thiago Macieira 2005-10-02 17:33:21 UTC
$ locate camera.protocol
/home/thiago/programs/src/kde3/KDE/kdepim/kmobile/kioslave/camera.protocol

$ locate kamera.protocol
/home/thiago/programs/src/kde3/KDE/kdegraphics/kamera/kioslave/kamera.protocol

Hence the conclusion.
Comment 12 Marcus Meissner 2006-04-11 09:57:39 UTC
hmm, what libgphoto2 (or SUSE ) version? 
Comment 13 Thorsten Staerk 2006-04-11 20:43:45 UTC
SUSE Linux 10.0

scorpio:~ # rpm -qa| grep gphoto
gphoto-2.1.99head-3
libgphoto2-2.1.99.head.0-12
scorpio:~ #
Comment 14 Marcus Meissner 2006-04-18 20:38:21 UTC
hmm.

pretty difficult to find out without having such a kamera :(

if you copy out the images, do they have size 0 too?
Comment 15 Thorsten Staerk 2006-04-19 08:59:05 UTC
yes.
Comment 16 Marcus Meissner 2006-04-26 23:10:49 UTC
can you try normal gphoto2 -P download? does this work?

you could also compile the kamera slave with debug and attach the debugoutput here, so we can see if there are visible problems :(
Comment 17 Thorsten Staerk 2006-04-28 22:07:20 UTC
I have compiled kdelibs with debugging symbols :)
Comment 18 Daniel Hahler 2007-04-27 22:44:26 UTC
I can confirm this. But it has nothing to do with moving. It also happens when you just copy.
I have a Fujifilm Finepix F30 here and it happened yesterday with a movie of 19MB size. Now I have reproduced it with a 200MB movie.

"gphoto2 -P" loads it fine and displays the progress status during copying - in contrast to the kioslave, which stayed all the time at 0% until it silently closed and left the 0 byte file.

Please tell me/us how we can debug this. Should I turn DEBUG on when compiling the Ubuntu kdegraphics package? How should we invoke the ioslave?
Comment 19 Thorsten Staerk 2007-04-29 10:09:27 UTC
I pledge for entirely removing this kioslave. I am still sad because it ate my favorite holiday movie. Better have nothing than a data-eater.
Comment 20 Javier Jardon 2007-04-29 16:27:23 UTC
Bug reported in Ubuntu: https://bugs.launchpad.net/ubuntu/+source/kdegraphics/+bug/111036
Comment 21 Daniel Hahler 2007-04-29 16:49:47 UTC
Marcus, have you tried reproducing it?

The basic problem here seems to be that the original source file gets deleted (when moving), although the copy process before (silently) failed.

Here's what's happening: the camera get initialised and then goes into copy mode. It even seems to copy for about 2.5 minutes (according to the camera's led, which is blinking), but then the "Copy File(s) Progress" window just gets closed and a 0 byte file is left.
Comment 22 Marcus Meissner 2007-04-29 17:12:54 UTC
will have to look at it again. As I tried it it worked. :/
Comment 23 Marcus Meissner 2007-05-02 19:24:45 UTC
i bet it is triggering a timeout. :/
Comment 24 Marcus Meissner 2007-05-02 20:38:45 UTC
confirmed with a Canon IXUS Wireless, libgphoto2 just before 2.3.0
Comment 25 Marcus Meissner 2007-05-02 20:39:08 UTC
*** Bug 118418 has been marked as a duplicate of this bug. ***
Comment 26 Marcus Meissner 2007-05-03 23:01:08 UTC
fixed for 3_5 branch in r660816

the reason is simple;

kio/connection.cpp discards data blobs larger than 16MB silently. We wrote
a data blob of xxxx MB....

I am now splitting up this datablob into 1 MB pieces, just to be on the safe side.
Comment 27 Marcus Meissner 2007-05-03 23:08:28 UTC
Created attachment 20475 [details]
xx.pat
Comment 28 Marcus Meissner 2007-05-03 23:10:18 UTC
committed, untested to TRUNK in r660823
Comment 29 Marcus Meissner 2007-05-03 23:12:34 UTC
so this bug should have happened onlyu for files >= 16MB.

If it happened for smnaller files, please reopen.
Comment 30 Daniel Hahler 2007-05-04 00:46:28 UTC
Thanks a lot for fixing it!
I can confirm that the patch works with a 200MB file.

But: IMHO the really bad thing happening here is, that the source file gets deleted, though the first part in the two-step "move process" (copy+delete) failed - and there had been no error dialog.
Is this a bug with using the kioslave system in this specific case or a more general design flaw? Should we file a new bug about this?
Comment 31 Marcus Meissner 2007-05-04 07:53:15 UTC
it was silently discarded, without any error report.

so I think it is a problem in the KIO slave support in kdelibs.
Comment 32 Marcus Meissner 2007-05-04 07:54:38 UTC
(issue is:

any data blob larger than 16MB is discarded. See connection.cpp:

bool Connection::sendnow( int _cmd, const QByteArray &data )
{   
    if (f_out == 0) {
        return false;
    }

    if (data.size() > 0xffffff)
        return false;



The error is not returning back to data()
Comment 33 David Faure 2007-05-04 11:04:34 UTC
On Friday 04 May 2007, Marcus Meissner wrote:
> any data blob larger than 16MB is discarded. See connection.cpp:

Yes. The kioslave needs to chunk it.
Comment 34 Thorsten Staerk 2007-05-04 11:10:36 UTC
Marcus, you did a great job analyzing this. Thanks. Can we maybe
-    if (data.size() > 0xffffff) 
+    if (data.size() > 0xffffffff) 
I mean, 16 MB is really little for my movies.
 
Comment 35 Marcus Meissner 2007-05-04 11:18:38 UTC
tstaerk: well, some lines below it does %06 printf of the size, so I guess there is a hard limit.

the camera slave is doing chunked writes now, so the issue is mostly addressed.
Comment 36 David Faure 2007-05-04 16:36:26 UTC
> Marcus, you did a great job analyzing this. Thanks. Can we maybe
> -    if (data.size() > 0xffffff) 
> +    if (data.size() > 0xffffffff) 
> I mean, 16 MB is really little for my movies.

The limit in KIO isn't a file size limit - it's a communication limit, i.e. how big should a chunk of data be.
More than 16MB at a time really kills the system, the slave (and the application) have to chunk the 
data they're sending (as documented in the dataReq signal for instance).

Marcussays the slave is doing chunked writes now; closing report.
Comment 37 Thorsten Staerk 2008-01-01 18:41:07 UTC
I cannot verify the solution. camera:/ no longer shows anything for me.
Comment 38 Thorsten Staerk 2008-01-02 14:24:43 UTC
I can now reproduce this bug on my KDE 3.5.5. r660823 is between 3.5.6 and 3.5.7, so, upgrading to test if the patch resolves the issue.
Comment 39 Thorsten Staerk 2008-01-02 14:27:12 UTC
r660823 is in kdegraphics only, to verify, we need to upgrade this. To be able to access camera:/// in konqueror, you need the package kdegraphics installed.
Comment 40 Thorsten Staerk 2008-01-02 15:23:12 UTC
camera:/// shows nothing, system:/media/camera now shows the same as camera:/// before. Continueing, I am now talking about system:/media/camera.
Comment 41 Thorsten Staerk 2008-01-02 15:24:02 UTC
I confirm this bug is fixed, set its state to VERIFIED and thank all participants for their good work.
Comment 42 Thorsten Staerk 2008-01-02 15:27:44 UTC
I set this bug to closed (although I do not understand why this is necessary).