Bug 353674 - Operations requiring disk access are unreasonably slow
Summary: Operations requiring disk access are unreasonably slow
Status: RESOLVED UPSTREAM
Alias: None
Product: digikam
Classification: Applications
Component: Import-Gphoto2 (show other bugs)
Version: 5.3.0
Platform: Debian stable Linux
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-10-08 03:56 UTC by Nikolaus
Modified: 2022-01-24 20:49 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In: 7.6.0


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Nikolaus 2015-10-08 03:56:18 UTC
When using digikam, all operations that require some disk access seem extremly slow compared. For example, importing 360 files with 2.1 GB using the "Import Images" dialog takes about 6 minutes. A simply "cp -a" using the same source and destination directories, preceded by "echo 3 > /proc/sys/vm/drop_caches" takes a mere 6.53 seconds. 

Reproducible: Always
Comment 1 caulier.gilles 2015-10-08 04:03:56 UTC
A lots of fixes have been done since 4.4.0 release. Please update to last 4.13.0 and try again.

Also check well all option from Setup/Camera/Behaviors panel. Handling metadata options can introduce time latency.

Finally, which kind of camera driver your use ? Gphoto or UMS ?

Gilles Caulier
Comment 2 Nikolaus 2015-10-08 04:32:07 UTC
I'm currently compiling 4.13 but running into some trouble (see mail to digikam-devel).  I don't think I'm using any camera driver, I'm importing the files from a local hard disk.
Comment 3 Nikolaus 2015-10-09 02:59:03 UTC
I've now upgraded to 4.13 and it's indeed faster (~3 minutes instead of 6 minutes). However, it's still extremly slow compared to using "cp" (~6 seconds). 

All checkboxes in "Configure -> Camera -> Behavior" are unchecked.
Comment 4 caulier.gilles 2016-07-06 16:16:11 UTC
This file still valid using last digiKam 5.0.0 ?

Gilles Caulier
Comment 5 bert.vanvreckem 2016-07-25 10:23:08 UTC
(In reply to caulier.gilles from comment #4)
> This file still valid using last digiKam 5.0.0 ?

I see this behaviour with Digikam 5.0.0 on Windows, when importing from an SD card. Importing is extremely slow, and the GUI becomes unresponsive even though the processor (Quad core Intel i5) is only at ~15 - 20%.
Comment 6 caulier.gilles 2016-11-25 06:50:09 UTC
What's about this problem using DK pre-release 5.4.0 bundles ?

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

Gilles Caulier
Comment 7 Nikolaus 2016-12-05 02:19:36 UTC
I don't know about the 5.4 pre-release, but I tried the 5.3.0 appimage and it's even worse. I don't have exact numbers, but just generating thumbnail takes about half a second per image (and images are 2-3 MB).
Comment 8 Nikolaus 2016-12-05 02:20:47 UTC
(I'm a bit hesitant to download and run software from a Google Drive link on a public bugtracker, sorry)
Comment 9 Maik Qualmann 2017-01-30 21:44:48 UTC
Git commit 043459000f01c0420a7bccb6bdd15696683403d7 by Maik Qualmann.
Committed on 30/01/2017 at 21:42.
Pushed by mqualmann into branch 'master'.

use separate camera thumbnail queue for quickly camera response
Related: bug 363937
FIXED-IN: 5.5.0

M  +2    -1    NEWS
M  +10   -9    utilities/importui/backend/cameracontroller.cpp

https://commits.kde.org/digikam/043459000f01c0420a7bccb6bdd15696683403d7
Comment 10 caulier.gilles 2017-01-31 21:21:49 UTC
Nikolaus,

I upload myself the file to GDrive. The AppImage is done on same computer (VM) than official one uploaded to KDE repository (which more complex to upload day by day this kind of file due to restriction and right).

The file uploaded in GDrive is safe. This is the fast way for the team to provide a suitable solution for end users to validate last fix.

I just updated the 64 bits AppImage bundle. Please give us a fresh feedback. It's important...

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

Gilles Caulier
Comment 11 Nikolaus 2017-02-05 21:29:11 UTC
I tried the most recent x64 appimage, and the entire GUI just hangs right after the flash screen has disappeared. Last messages on the console are:

[...]
digikam.general: mimetype =  ""  ext =  "MP4"
digikam.general: Cannot create thumbnail for  "/home/nikratio/lib/Dropbox/Photos/2016/11 (november)/2016-11-13_16_23_16.mp4"
digikam.general: Thumbnail is null for  "/home/nikratio/lib/Dropbox/Photos/2016/11 (november)/2016-11-13_16_23_16.mp4"
digikam.general: Request to get thumbnail for  "/home/nikratio/lib/Dropbox/Photos/2016/11 (november)/2016-11-13_16_23_16.mp4"
digikam.general: Video duration for  "2016-11-13_16_23_16.mp4" is  24363  seconds
digikam.general: Trying to get thumbnail from  "2016-11-13_16_23_16.mp4"  at position  4872
Comment 12 caulier.gilles 2017-02-05 21:56:04 UTC
Run AppImage bundle from a console with "debug" as argument. This will run digiKam into GDB. When digiKam hand up, press CTRL+C. You must take the GDB prompt. enter "bt" to see where digiKam loop...

Gilles Caulier
Comment 13 caulier.gilles 2017-02-05 21:57:02 UTC
My last comment is only valid with 5.5.0 pre version only, available here :

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

Gilles Caulier
Comment 14 Nikolaus 2017-02-05 22:37:39 UTC
I didn't have to press Ctrl-C, it threw me right into the debugger:

digikam.general: Video thumbnail extracted for  "/home/nikratio/lib/Dropbox/Photos/2016/11 (november)/2016-11-13_16_23_16.mp4"  ::  QImage(QSize(256, 144),format=6,depth=32,devicePixelRatio=1,bytesPerLine=1024,byteCount=147456)
digikam.general: scan mode: ScanDeferredFiles
digikam.general: total scan value :  25704
digikam.dbengine: Database is locked. Waited 0
[Switching to Thread 0x7fffe3fff700 (LWP 10144)]
Catchpoint 1 (exception thrown), 0x00007fffee688dbd in __cxa_throw ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt
#0  0x00007fffee688dbd in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00000033b5f68fb5 in Exiv2::ImageFactory::open (path=..., useCurl=<optimized out>)
    at /b/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/src/image.cpp:834
#2  0x00007ffff67d70e1 in Digikam::MetaEngine::load (this=this@entry=0x7fffdc1035e0, 
    filePath=...)
    at /b/dktemp/digikam-development/garbagecollection/core/libs/dmetadata/metaengine.cpp:278
#3  0x00007ffff681f986 in Digikam::DMetadata::load (this=this@entry=0x7fffdc1035e0, 
    filePath=...)
    at /b/dktemp/digikam-development/garbagecollection/core/libs/dmetadata/dmetadata.cpp:96
#4  0x00007fffeea549ba in Digikam::ImageScanner::loadFromDisk (
    this=this@entry=0x7fffe3ffe330)
    at /b/dktemp/digikam-development/garbagecollection/core/libs/database/item/imagescanner.cpp:1568
#5  0x00007fffeea54b50 in Digikam::ImageScanner::newFile (this=this@entry=0x7fffe3ffe330, 
    albumId=174)
    at /b/dktemp/digikam-development/garbagecollection/core/libs/database/item/imagescanner.cpp:289
#6  0x00007fffee996f0e in Digikam::CollectionScanner::scanNewFile (this=this@entry=
    0x7fffe3ffecf0, info=..., albumId=174)
    at /b/dktemp/digikam-development/garbagecollection/core/libs/database/collection/collectionscanner.cpp:1259
#7  0x00007fffee998b2b in Digikam::CollectionScanner::scanAlbum (
    this=this@entry=0x7fffe3ffecf0, location=..., album=...)
    at /b/dktemp/digikam-development/garbagecollection/core/libs/database/collection/collectionscanner.cpp:1087
#8  0x00007fffee99877f in Digikam::CollectionScanner::scanAlbum (
    this=this@entry=0x7fffe3ffecf0, location=..., album=...)
    at /b/dktemp/digikam-development/garbagecollection/core/libs/database/collection/collectionscanner.cpp:1117
[...]
Comment 15 caulier.gilles 2017-02-06 05:56:39 UTC
It's a crash in Exiv2 when image is open for scanning at startup.

Report this problem to Exiv2 bugzilla as UPSTREAM. You will need to identify the file on disk which has the problem (i suspect 2016-11-13_16_23_16.mp4).

Gilles Caulier
Comment 16 Christoph Feck 2017-02-07 02:17:04 UTC
Gilles, according to investigation in bug 375952 libexiv2 raises exceptions to indicate unsupported file formats. I am not sure if it is possible to catch exceptions within a Qt application, but nevertheless informing you, that this is not a regular crash bug.
Comment 17 caulier.gilles 2017-02-07 06:12:28 UTC
We already catch the exception from Exiv2 into digiKam. This is not the problem.

MP4 files are know as problematic in Exiv2 0.25, as this is the last version published include an amount of bugs while scanning video files.

Next Exiv2 0.26 will be better. This is why digiKam AppImage bundle include Exiv2 0.26-svn source code since digiKam 5.4.0.

Gilles Caulier