Bug 174586

You use beta1. please try with last beta5 release...

Gilles Caulier

I don't know how to install beta5. Couldn't compie libkipi.

Abyway I notice that digicam 0.9.4 also has that problem. It peaked at 800 megs of used memory. It  was all free when the avi-file was successfully retreived. Since I also hav 1Gig of memory this was pretty annoying and basically locked up my machine.

If problem appear in KDE3 version too, it must be a problem with libgphoto2 used by digiKam to play with non USB Mass Storage camera.

Which libgphoto2 version you use ?
Which camera type/model ?
Which video format ?

Gilles Caulier

camera:
cannon isux 90is 

libghoto2:
2.4.2-0ubuntu2 (I do not know if there are any ubuntu patches)

video format (reported by 'file'):
RIFF (little-endian) data, AVI, 640 x 480, 30.00 fps, video: Motion JPEG, audio: uncompressed PCM (mono, 44100 Hz)

Soren, 

Can you test with last digiKam 0.10.0-beta6 if problem still exist ?

Thanks in advance

Gilles Caulier

sure .... having trouble compiling just now, but I will return.

Can't get digikam 0.10 beta6 to work. After compiling it and trying to run it, it is not even able to show my existing images...

are you updated libkipi, libkexiv2, and libkdcraw from trunk before to compile digiKam ?

Gilles Caulier

Soren,

well, do it. use kdegraphics/libs from trunk as explained here:

http://www.digikam.org/sharedlibs

Gilles Caulier

digikam-0.10 beta7 still has the problem. Any hint on how to pinpoint where the memory is actually allocated?

Does it use only memory or CPU as well? Watch it using a system monitor, top or ksysguard.
Any output on the console?
While digikam is running and consuming memory, attach gdb. To do this, find out the PID of the process and then call "gdb att <PID>". Type the command "thr appl all bt" and post the output here.

Any news on this?

S0ren,

Please take a look in comment #11 and report more info in this thread. thanks in advance

Gilles Caulier

Created attachment 31378 [details]
backtrace of digikam 0.10.0 rc2 using 300 Mb memory

Backtrace of digikam using 300 Mb of memory during a download of a 1600 Mb AVI-file from my Canon IXUS 90IS

Sorry for the long delay, but my camera has been left along for a while.

Anyway top says the following - and this is only the beginning, because the sky is the limit with this. It just keeps on going until killed by OOM.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5086 sgh       20   0  492m 276m  12m T    0 31.5   0:27.45 digikam

Thanks for the backtrace. This indeed hints at a problem in gphoto2: the only running digikam thread is deep inside libgphoto2 called to download an item.

Gilles, I guess we should CC the gphoto2 developer?

Marcel,

Gphoto2 lead developer is Marcus Meissner <marcus at jet dot franken dot de>. He is registered to B.K.O. Just add email as CC is enough...

Gilles

libgphoto2 slurps everything into memory first if you are not using filedescriptor based transfer. So for 1.6GB movie, it will try to allocate 1.6GB at least.

You can work around this by diownloadiung just this movie using
gphoto2 commandline tool, which uses a passtghrough method.

hmm... well I guess this is a no no. The most movie-files is normaly several megabytes pr minute, so I think digikam should indeed use filedescriptor based transfer. But this must mean that everyone has this problem, also Gilles.

Well, no, because i never use PTP driver with my camera : it's too limited (No RAW preview). I always use UMS mode.

So, we need a Canon user here to confirm...

Gilles Caulier

So, Marcus, as I understand this, to have downloading with gphoto2 scale well with large files, we need to use the "filedescriptor based method".

What is this? Is it advisable to use? Any problems or drawbacks?

instead of doing

gp_file_new(&file);
... get file ... 
gp_file_get_data_and_size(file,...)
gp_file_unref (file);
you do:

... open fd writeable ... 
gp_file_new_from_fd(&file, fd);
... get file same as before
gp_file_unref (file);
 (no gp_file_get_data_and_size() )

The "get file" part will already write it to "fd" on the disk.

drawbacks ... hmm, not so much. needs libgphoto2 2.4.0 or newer, but this is the case for all distros these days.

We are currently using gp_file_save but reading the header file this is considered deprecated. I guess new_from_fd is the best choice then.

Created attachment 35254 [details]
gp_file_new_from_fd

See the attached patch. I switched my camera to PTP mode and managed to load the a few pictures but I get errors for about every second image (see error printout below). I dont know if this is due to my patch or any other problem because I never used the PTP mode of my camera.
Anyone here to test the patch with an established testing environment for gphoto2?
Marcus, do you know what may cause this error in my patch from gp_camera_file_get()?

printGphotoErrorDescription: Libgphoto2 error:  I/O-Problem  ( -7 )

from patch:
+    errorCode = gp_file_new_from_fd(&cfile, file.handle());
+
+    if (errorCode != GP_OK)
+    {
+        kDebug(50003) << "Failed to get camera item!" << file.handle();
+        printGphotoErrorDescription(errorCode);
+        gp_file_unref(cfile);

remove the gp_file_unref(cfile) here, cfile is not initialized when errorCode != GP_OK.

Also QFile:handle() ... does it return the fd QFile still continues to own? 

Then try using
+    errorCode = gp_file_new_from_fd(&cfile, dup(file.handle()));
instead since this call hands lifetime control of "fd" over to libgphoto2 (and we would close fd twic otherwise).

Thanks Marcus, I was indeed messing around with file descriptors!
Now everything works with my camera.

SVN commit 995957 by mwiesweg:

Use the file descriptor based method to download a file with gphoto2.
This should solve problems with very large files because they can directly be written to disk.

BUG: 174586

 M  +4 -1      NEWS  
 M  +25 -11    utilities/cameragui/gpcamera.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=995957