Bug 223936 - Moving photos between albums is slow
Summary: Moving photos between albums is slow
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Database-Albums (show other bugs)
Version: 1.0.0
Platform: Compiled Sources Linux
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-01-23 16:38 UTC by Frédéric COIFFIER
Modified: 2017-07-25 16:49 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In: 5.7.0


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Frédéric COIFFIER 2010-01-23 16:38:38 UTC
Version:           1.0.0 (using Devel)
Compiler:          gcc (Gentoo 4.4.2 p1.0) 4.4.2
 
OS:                Linux
Installed from:    Compiled sources

It's a recent problem as I don't remember seeing it in pre-1.0 versions (or maybe, it's linked to new KDE 4.4 libraries).

When I move 10-20 (RAW or JPEG) photos in an other album, it takes 5~10s before the view is refreshed. In the past, this action was instantaneous.
Comment 1 Marcel Wiesweg 2010-01-23 18:32:34 UTC
Which process is using CPU?
Comment 2 Frédéric COIFFIER 2010-01-24 12:44:11 UTC
During the first try, I've seen a "virtuoso" process. So, I've disabled the Nepomuk service.
But, now, during the 10s which lasts the moving operation, I don't see any process which take lots of CPU. Even the Digikam process is around ~4% (whereas it takes 11-16% to redraw thumbnails). It seems to wait an asynchronous event...

In the console, I can see:

...
igikam(3036)/digikam (core) Digikam::AlbumManager::slotKioFileMoved: ... digikam(3036)/digikam (core) Digikam::AlbumManager::handleKioNotification: KDirNotify detected file change at ...
digikam(3036)/digikam (core) Digikam::AlbumManager::handleKioNotification: KDirNotify detected file change at "/hdf/photos/2009-06 Vacances dans le Cantal Corrèze/2009-06 Vacances dans le Cantal Corrèze - Rush "                                                                                                                                                                                   
digikam(3036)/digikam (core) Digikam::ImageScanner::addImage: Adding new item ...
digikam(3036)/digikam (core) Digikam::AlbumManager::slotKioFileMoved: ...                                                            
digikam(3036)/digikam (core) Digikam::ImageScanner::copyFromSource: Recognized ... as copied from 28250
...

But there are lots of these error messages :

Error: Directory Unknown with 64366 entries considered invalid; not read.   
...
Error: Directory Unknown with 25212 entries considered invalid; not read. 
...
Error: Directory Unknown with 43693 entries considered invalid; not read.
...
Error: Directory Unknown with 43693 entries considered invalid; not read.


I can send you debug messages of Digikam by mail if you want.
Comment 3 Marcel Wiesweg 2010-01-28 16:11:44 UTC
I have identified a problem here which led to a scan of the complete collection after a copy or move operation, but only for the very first operation after program startup.
Seems you are compiling SVN; can you check if this fixes your problem?
Comment 4 Marcel Wiesweg 2010-01-28 16:13:19 UTC
SVN commit 1081515 by mwiesweg:

Ensure that the initial setup of modification dates of db file directory is
done after the dir watch black list is setup. Otherwise, a complete collection
scan would be triggered for each first use.
Also fix retrieving the directory from the dir watch path, which can be either a
file or a directory path.

CCBUG: 223936

 M  +20 -12    albummanager.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1081515
Comment 5 Frédéric COIFFIER 2010-01-28 17:30:23 UTC
I've compiled the last SVN version but it doesn't change anything.
Around ~10% of the CPU is shared between Digikam, kio_digikamalbum and kio_digikamdate (and ~80% of CPU idle).

I also tried a large amount of RAW photos : If I tried to move 228 photos, only 56 are really moved and the other 172 photos stay in their album.

If you need a log file, traces..., explain me (quickly) how to generate them and I'll send them.
Comment 6 Johannes Wienke 2010-01-28 17:40:16 UTC
First of all start kdebugdialog and enable full logging for all digikam area codes.

Afterwards start digikam from a console, do the file operations and close digikam. Then paste the console log here.

Do you have the same problems when you move the files with dolphin?
Comment 7 Frédéric COIFFIER 2010-01-28 18:03:59 UTC
With dolphin, moving 341 RAW files between directories is instantaneous (less than 1s).

For the trace, I'll do it tonight and I'll send you by mail (because it'll be big).
Comment 8 Marcel Wiesweg 2010-01-28 18:17:28 UTC
Please also enable the code 50004 and provide messages added to your ~/.xsession-errors file.
Comment 9 Marcel Wiesweg 2010-01-30 17:06:40 UTC
A few more questions.

On the console output, there are two main sections: One contains a lot of handleKioNotification statements. The second section mostly contains Digikam::ImageScanner output.
Now, during the 10s while you copy files, when you watch the console "live", which lines are output during the majority of time? Before section 1, in Section 1, Section 2, or after section 2?

Did you disable Nepomuk in System Settings (Advanced tab)? To rule out it's a Nepomuk issue.
Comment 10 Andi Clemens 2010-01-30 17:12:37 UTC
Do you use EXT4? If so, try turning off barriers in /etc/fstab, for example:
LABEL=media /mnt/data ext4 defaults,noatime,barrier=0  0 2

This helped on my computer, EXT4 + Barriers + Sqlite3 is a slow bastard :D
Comment 11 Frédéric COIFFIER 2010-01-30 19:00:57 UTC
Andi, you're right: this bug isn't a regression. It's just that I have change my filesystem to ext4 some weeks ago.
I've made the try with :
mount -o remount,barrier=0 /my_partition
and Digikam is as fast as previously.

Moreover, I succeed to move ~300 photos without problems (with default ext4 options, Digikam didn't move all selected photos).

For me, the problem is really linked to Sqlite3 + ext4. I knew this problem of ext4 thanks to Phoronix's articles: http://feedproxy.google.com/~r/Phoronix/~3/cVGeygVMUC0/vr.php

Marcel, you can close this bug report if you want.
Comment 12 caulier.gilles 2017-07-14 14:30:16 UTC
Here it crash in KIO while processing changes in database. Since 5.0.0 release,
we have replaced the KIO mechanism to play with database by a multi-threaded
interface. This problem will never be reproducible.