Bug 297793

So I just learned that I cannot attach files larger than 1 MB; I don't have a hosting offering, but I can provide the image by eMail if it is of any help to someone.

Thanks alot in advance!

Can you provide an url where an test image can be downloaded ?

Gilles Caulier

Hi,

I have uploaded the image to my FritzBox - ftp://ftpuser@carfesh.dyndns.org
Password is ftpuser - the file is in the folder Acer-FlashDisk-01 and is named P1030314.JPG.

I just took the time to try it with a Ubuntu 11.10 virtual machine (I have the error on Windows) - on Ubuntu, digiKam version 2.5.0 does not present that error, the image is loaded fine. The problem may be Windows related.

Kind regards,
  Torsten

Under linux, preview work fine :

http://www.flickr.com/photos/digikam/6915530148/sizes/o/in/photostream/

Gilles Caulier

Yes, I've noticed on Ubuntu that the preview works fine there, too - maybe one of the components contained in the Windows installation is outdated (Windows itself displays those JPEGs just fine)?

I could not find out how to activate debug logging on Windows - I have read that people were able to narrow down this error by looking at the debug console output, but I could not find any debug related functionality in the Windows installed digiKam.

Can you tes with another images taken with another camera or in different format ? It still reproducible ?

To debug under windows, install M$ DebugView program. Start it and run digiKam. Debug trace will be visible in this application

Gilles Caulier

Other images are displayed fine, yes - all images from my old digital camera are displayed fine, and a bunch of pictures I downloaded from the internet display fine, too.

So I think the images from the Panasonic must have some property that is throwing the display function off (whatever that might be).

I will give DebugView a go tonight, thanks alot for the tipp!

Hi, so I tried to trace this problem using DebugView - no error message is displayed that is related to digiKam (infact I don't see any digiKam messages at all) - I see other tools on the system writing debug messages, like my UMTS stick software, but nothing related to KDE or digiKam.

Are only errors logged here?

Before to run debugview, run kdebugdialog from kde dir. It's a tool to set which debug space must be displayed as trace. Select digikam, kexiv2, kdcraw, kipi.

Rerun debugview and start digiKam 

Gilles Caulier

In order to get the output logging in DebugView, you need to make sure that 'Capture Win32' is ticked in the Capture menu. For some reason this is always unticked for me every time I start DebugView.

I tried this on my Windows XP VM with DK 2.3.0. The first time I tried it, it came up with 'Failed to load image'. However I did not have DebugView loaded at that time. Later when I had installed DebugView and tried it again, this time the thumbnail was there and the full size preview opened normally. I don't use that VM much and I think it may have installed some updates between the two tries. I have now updated that VM to DK 2.5.0 and the provided image is still being shown as normal.

Hi,

so I got DebugView to capture the digiKam output; I'm not really sure what exactly did it, "Capture Win32" was already ticked. The last attempt was on my notebook, where it did not capture, but on my desktop PC it does.

DebugView will show the following output when I click one of the files and the error message appears in digiKam:

[7020] digikam(7020)/digikam (core): Failed to allocate chunk of memory of size 39923712 bad allocation 
[7020] digikam(7020)/digikam (core) void __thiscall Digikam::PreviewLoadingTask::execute(void): Cannot extract preview for  "D:/Users/torsten/Documents/Bilder/20100928-Bilder Notebook/P1010992.JPG" 

I also have some pictures where it will just crash completely, showing a debug dialog with an option to submit online.

So I'm afraid the message is not really helpful, the pathname looks correct, and I can open it in explorer.

Do you have any ideas what I might try?

Thanks alot in advance,
  Torsten

preview is extracted using Exiv2 library in background. I suspect a problem of memory allocation here...

Can you switch preview option in digiKam config panel to use full size image as preview, an try again ?

Gilles Caulier

So I activated the checkmark "Embedded preview loads full-sized images". That was kind of interesting, because it changed the behaviour a little bit (however I begin to get the feeling that the error is intermittent);

I could open one picture by double clicking, DebugView showed

[6748] digikam(6748)/digikam (core): Failed to allocate chunk of memory of size 39923712 bad allocation 

However when I clicked a different picture, it again showed the same two messages:

[6748] digikam(6748)/digikam (core): Failed to allocate chunk of memory of size 39923712 bad allocation 
[6748] digikam(6748)/digikam (core) void __thiscall Digikam::PreviewLoadingTask::execute(void): Cannot extract preview for  "D:/Users/torsten/Documents/Bilder/20091008-Urlaub Portugal/00-entwickeln/P1000077.JPG"

This is strange because it try to allocate 39Mb of memory to store preview image. Do you have enough memory in your machine ?

Which Exiv2 version you use ? Look "Help/Components Info" for details...

Gilles Caulier

Note : in case of full image size preview, it's libjpeg which is used to load image, not Exiv2. Which libjpeg you use on your system ?

Can you load this JPEG image in editor ? (In this case it's a different way to load it) Which messages can you read in debugview in this case ?

Gilles Caulier

The system has 4 GBs of RAM (Windows 7 x64), currently there are 1600MB free in task manager.

I used the installer for v2.5.0 from the SourceForge site - I noticed BTW that kdebugview is not included in that install, I just looked for it because I tried to follow your first set of instructions.

I suppose, since the 39MB are always the same, it's the memory reserved to store the uncompressed image (I normally store images in the camera's maximum density of 10 megapixels). Maybe there's a restriction somewhere on how much memory can be reserved for that in one step? Lower resolution pictures (like from my older 4 megapixel camera) display fine, only when I use those 10MP pictures things get really unstable. digiKam will also need a restart once the error has occurred, no new pictures will open and some dialogs just will crash afterwards, context menus don't show up, ...

I will attach a screenshot of the components info; it lists exiv2, but does not show the exact version.

Using image editor to open the images will display

[3184] digikam(3184)/digikam (core): Failed to allocate chunk of memory of size 39923712 bad allocation 
[3184] digikam(3184)/digikam (core) class Digikam::DImg *__thiscall Digikam::DImgInterface::getImg(void) const: d->image is NULL

Created attachment 70317 [details]
screenshot of components info

Exiv2 is 0.21. not the last

There is no memory restriction from digiKam viewpoint. 39Mb is not huge. I suspect that a malloc failed under windows. Why ???

Perhaps Ananta know a possible issue under windows here...

Gilles Caulier

I found an interesting article discussing some limits regarding malloc: http://forums.codeguru.com/showthread.php?threadid=518947

I can see in Process Monitor that the virtual size of digiKam is about 1,8 GB when that call fails (once the GUI starts, digiKam seems to pull obscene amounts of virtual memory), so it may be that you are reaching the limit of the 2 GB _virtual_ memory size, or that that 2GB space doesn't have enough unfragmented free space for 39MB. 

Physical memory usage of digiKam is just about 110MB however, but that memory usage looks kind of weird. The only other application I have running with that kind of virtual memory size is a SQL server express installation. Is it possible there is a memory leak somewhere?

Since I'm on a 64 bit system, it might help a little bit to compile the application as large address aware to counter that I suppose.

The database is just 20MB in size, so I don't think it's the database itself being kept in memory that is taking up so much.

Interesting - since I've done that in the past for games that do not utilize the full memory, I've applied CFF Explorer's 4GB path to digiKam and voila - for now it works, the virtual size of digiKam is around 2GB now in Process Explorer and I can open images without that malloc failure.

So I'm convinced it's related to the 2GB virtual address space limit that 32 bit application have on Windows.

CFF Explorer can be found here: http://www.ntcore.com/exsuite.php

A nice explanation of those limits can be found in the wikipedia: https://en.wikipedia.org/wiki/Virtual_address_space. 

digiKam is using the 2GB up on my system, making it large address aware increases that limit to 4GB.

This wantmean that digiKam under windows can be only used on 64 bits ?

Gilles Caulier

Hi,

setting that large address aware flag will have no effect on a normal 32 bit Windows, so it is safe to set it in any case, it just will not improve the situation. On a 64 bit system this will allow 4 GB of maximum virtual memory usage (please note that this is not physical memory used, but virtual memory that may be swapped out to the hard disk); on a 32 bit system it depends, normally you will have 2GB available, and only if /3GB is set in the boot.ini this will give the process 3GB. So the flag will not harm other users.

But I think the real question is: does digiKam really need more than 2GB of memory to function? I can see the virtual memory going up quickly during the launch of digiKam (in the splash screen), but the physicial usage remains quite constant; my guess is that something (just as an example let's say the loading of the KIPI plugins) is doing a lot of work during intialization, consuming memory, but not freeing the memory properly on Windows. Once it is started up completely, the number remains quite constant. Few applications use that much memory constantly - an example from my system is a SQL server, VMware Workstation and obviously games. But ACDSee and Lightroom as an example manage to deal with around 512MB-1024MB virtual memory when I start them.

I think setting that large address aware flag should only serve as a workaround until it can be found out why digiKam needs so much memory - I am happy for now because I can start actually using digiKam for work, however it would be bad if in 4 weeks I managed to reach the 4GB limit :)

I will play around with digiKam to see if I find some setting that has an effect on that number so that maybe the area can be narrowed down; for example I just disabled the scan for new files on startup, but that didn't improve the memory use.

Kind regards,
  Torsten

Hi,

so I have played around some more and I'm afraid the problem seems to be related mostly to the number of images in the database, so there's no way for me to influence it except to keep my collections small.

I have observed the behaviour a bit today more, the biggest chunk of memory is allocated while the splash screen shows that the database is being loaded.

I have created some test databases; I have one with around 12.000 images - when I open my large address aware patched digiKam with that database, virtual memory consumption is already around 3,8 GBs right after I start digiKam (without having done any real work - just opening it is enough to use up that memory).

So I'm afraid for large image collections, digiKam right now is not practical. A 32 bit Windows process can never consume more than 4GB of virtual memory, the only way around that limitation is to compile a native 64 bit application. I'm sure there are other things on the top of your priority list.

It would be nice if you could investigate why digiKam allocates that much memory. As mentioned, I suspect some kind of memory leak - but maybe that's really just the way digiKam utilizes memory, I don't know what kind of information about the images you are trying to keep in memory.

For now I'm afraid I have to keep using ACDSee, as I have some collections that are even larger. It's too bad, because I was really amazed when I found at what huge feature set digiKam has; I really would love to begin using it.

Kind regards,
  Torsten

The memory problem does not exist in build 2.3.0 for Windows; sadly there's no 2.4 build to narrow it down any further, but it may very well be that the problem was introduced with a new feature or change in the newer versions; I'm not sure what that could be.

Official digiKam 2.6.0 release is out since few days now :

http://www.digikam.org/drupal/node/656

Please, check if this entry still valid, or update report accordingly.

Thanks in advance.

Gilles Caulier

Hi,

I wanted to verify if this issue still exists. However I cannot start digiKam 2.7.0; I experience the issue described under https://bugs.kde.org/show_bug.cgi?id=305191

Kind regards,
  Torsten

Ok, so after removing an attached USB stick I can start digiKam. The issue still exists in 2.7 - once virtual memory allocation reaches 1,8GB, the GUI will start behaving eratically. 

Oddly enough, no alloc error messages are shown in debugview anymore, but it is still unusable.

Git commit 57b269e2d6dfd8f0c11d482ec12d3e9c126c1cd1 by Ananta Palani.
Committed on 03/02/2013 at 23:04.
Pushed by palani into branch 'master'.

Fix thumbnails not shown in Windows and excessive use of virtual memory / handles which may cause crashes and other unexpected behavior by removing file monitoring and only monitoring directories for changes. These problems might still occur with very large numbers of folders but could not be confirmed. However, medium to large collections should no longer cause a problem.
Related: bug 290962, bug 308310, bug 310252, bug 310865, bug 312422, bug 312999, bug 291917, bug 295445, bug 297686
FIXED-IN: 3.0.0

M  +9    -2    digikam/album/albumwatch.cpp

http://commits.kde.org/digikam/57b269e2d6dfd8f0c11d482ec12d3e9c126c1cd1

I have made a beta compilation of digiKam 3.0.0 for you to test whether the commit I just made fixes your problem:

   https://docs.google.com/file/d/0B4Esh3gAgC9QcW5nS04zQWVsNGs/edit?usp=sharing

Can you give it a try and let me know how it works for you?

Thanks a lot, using the beta the issue I saw in my collection has disappeared, I can browse it fine now, great job!