Bug 306282

Summary:	Improvements for maintenance tool
Product:	[Applications] digikam	Reporter:	Axel Krebs <axel.krebs>
Component:	ProgressManager-Batch	Assignee:	Digikam Developers <digikam-bugs-null>
Status:	RESOLVED FIXED
Severity:	normal	CC:	caulier.gilles
Priority:	NOR
Version:	2.8.0
Target Milestone:	---
Platform:	Ubuntu
OS:	Linux
Latest Commit:	http://commits.kde.org/digikam/62125c25374b0caf2d6a3812d1d34812ede8a6fd	Version Fixed In:	3.3.0
Sentry Crash Report:

Description Axel Krebs 2012-09-05 07:32:11 UTC

When starting maintenace tool, we can see wonderful surface. Congratulations!

I would like to have more tranparencey in using: as I habve a large poic collection, it takes forever to run all options for the whole collection. 

- Is it possible to interrupt on of the maintenace task indepently?
- when I re-run one of the possible tasks in the maintenance tool, just after finishing run (not seriously, only for check purposes), it looks like maintenance re-start from scrach. Arent't there pic-individual markers (processed or not") This takes time forever...


Reproducible: Always

Steps to Reproduce:
1. seems to be constant since previous version
2. 
3. 



bug?

_and_ wish

Comment 1 caulier.gilles 2013-07-28 08:24:49 UTC

Git commit 62125c25374b0caf2d6a3812d1d34812ede8a6fd by Gilles Caulier.
Committed on 28/07/2013 at 08:00.
Pushed by cgilles into branch 'master'.

Maintenance Manager : complete rewrite of maintenance tools handling.
Instead to use signal provided by tool to know if job is done, use progress manager signal.
Store instance of each tools in d private container instead stack to provide a better check of stage under progress.
Patch ScanController to provide a better feedback to NewItemsFinder, in all scan mode. Add a new method to only call core code to perform
scan WITHOUT to show a progress dialog. In fact progress manager as already all in place to give user feedback about progress...
Patch FaceManagement maintenance tool to indicate that face are detected and recognized.
Move status progress bar with ProgresManager code, as this one is fully relevant.

Marcel, I would to drop fully DProgressDialog code, which is only used now to ScanController, with all first run step to collect all information from image
managed by database. I remember that in 2.x serie, you have patched digiKam to only perform first run scan in background when albumgui is show. I'm right ?
At now, this feature sound fonctionnal some time, but it's not fully reproducible.
If you take a look how Google Picasa work, you will see that main DB referencing run in background when main GUI is displayed. A progress indication is show in statusbar.
User can start to work with collected items in real time and don't need to wait a long time that all referencing is done through a small progress dialog...
Look this entry for ex : 320359.
What do you think about?
Related: bug 295256, bug 320121, bug 295255, bug 297614

CCMAIL: marcel.wiesweg@gmx.de

FIXED-IN: 3.3.0

M  +1    -1    CMakeLists.txt
M  +19   -7    digikam/database/scancontroller.cpp
M  +7    -0    digikam/database/scancontroller.h
M  +1    -1    libs/progressmanager/progressmanager.cpp
R  +1    -0    libs/progressmanager/statusprogressbar.cpp [from: libs/widgets/common/statusprogressbar.cpp - 098% similarity]
R  +1    -0    libs/progressmanager/statusprogressbar.h [from: libs/widgets/common/statusprogressbar.h - 096% similarity]
M  +2    -3    utilities/maintenance/duplicatesfinder.cpp
M  +1    -1    utilities/maintenance/facedetector.cpp
M  +1    -2    utilities/maintenance/fingerprintsgenerator.cpp
M  +1    -1    utilities/maintenance/maintenancedlg.cpp
M  +117  -77   utilities/maintenance/maintenancemngr.cpp
M  +14   -8    utilities/maintenance/maintenancemngr.h
M  +3    -1    utilities/maintenance/maintenancetool.cpp
M  +24   -20   utilities/maintenance/newitemsfinder.cpp
M  +0    -1    utilities/maintenance/newitemsfinder.h
M  +1    -2    utilities/maintenance/thumbsgenerator.cpp

http://commits.kde.org/digikam/62125c25374b0caf2d6a3812d1d34812ede8a6fd

Comment 2 Axel Krebs 2013-07-28 10:46:04 UTC

Gilles (& Marcel):

Thank you these large amount of information.

As much as I understand, we can expect a serious improvement in V 3.3- I
am keen on it!

Last time, my thumbnail is growing to more than 12 GB, and most probably
related to this fact, startinge tim of my digiKam V. 3.2 is rising to
some twenty minutes!!!

One obstacle seem to be clear: the max cpu percentage does _not_ exceed
25% which mean under quadcore, that digiKam does _not_ use my machine
efficiently.

In many motherboards, the OS can force the graphic chip to support the
cpu. Do you intend to achieve this in future?
--

Besides, the database concept of digiKam seems to limit many features of
the program itself.

There would are many causes to "attach" metadata to some expternal file,
as xml could be. If one starts to look for a selected directory,
all xml-files would be loaded instantaneously- no huge and potentially
risky database- files had to be loads at all.

Belonging to one picture, all derived version of a specific pic could be
treated as one single entity, meaning, that versioning of pics would
dealt automatically.

Tagging, face detection could be incorporated into this single metadata
file.

There are many other advantages as
- independence of storage location of directories,
- minimized starting up time
- keeping together pics, versioneing and metadata at selected locations
etc. etc.

What do you think about?


My statistics:
digiKam version 3.2.0
Bilder:
BMP: 				5
GIF: 				114
JP2: 				14
JPEG: 				60
JPG: 				129282
NIKON-NEF: 			55
PCX: 				2
PGF: 				2
PNG: 				2382
RAW-CR2: 			339
RAW-CRW: 			10824
RAW-DNG: 			6
RAW-NEF: 			92976
TIFF: 				1931
XCF: 				5
Gesamt: 			237997
:
Videos:
AVI: 				60
MOV: 				26
MPEG: 				1
Gesamt: 			87
:
Gesamtzahl der Einträge: 	238084
Alben: 				3816
Stichwörter:			 100
Datenbanktreiber: 		QSQLITE


thumbnail db: 			12,3 GB
starting up time:		27 min!!!



Am 28.07.2013 10:24, schrieb Gilles Caulier:
> https://bugs.kde.org/show_bug.cgi?id=306282
> 
> Gilles Caulier <caulier.gilles@gmail.com> changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>              Status|UNCONFIRMED                 |RESOLVED
>    Version Fixed In|                            |3.3.0
>          Resolution|---                         |FIXED
>       Latest Commit|                            |http://commits.kde.org/digi
>                    |                            |kam/62125c25374b0caf2d6a381
>                    |                            |2d1d34812ede8a6fd
> 
> --- Comment #1 from Gilles Caulier <caulier.gilles@gmail.com> ---
> Git commit 62125c25374b0caf2d6a3812d1d34812ede8a6fd by Gilles Caulier.
> Committed on 28/07/2013 at 08:00.
> Pushed by cgilles into branch 'master'.
> 
> Maintenance Manager : complete rewrite of maintenance tools handling.
> Instead to use signal provided by tool to know if job is done, use progress
> manager signal.
> Store instance of each tools in d private container instead stack to provide a
> better check of stage under progress.
> Patch ScanController to provide a better feedback to NewItemsFinder, in all
> scan mode. Add a new method to only call core code to perform
> scan WITHOUT to show a progress dialog. In fact progress manager as already all
> in place to give user feedback about progress...
> Patch FaceManagement maintenance tool to indicate that face are detected and
> recognized.
> Move status progress bar with ProgresManager code, as this one is fully
> relevant.
> 
> Marcel, I would to drop fully DProgressDialog code, which is only used now to
> ScanController, with all first run step to collect all information from image
> managed by database. I remember that in 2.x serie, you have patched digiKam to
> only perform first run scan in background when albumgui is show. I'm right ?
> At now, this feature sound fonctionnal some time, but it's not fully
> reproducible.
> If you take a look how Google Picasa work, you will see that main DB
> referencing run in background when main GUI is displayed. A progress indication
> is show in statusbar.
> User can start to work with collected items in real time and don't need to wait
> a long time that all referencing is done through a small progress dialog...
> Look this entry for ex : 320359.
> What do you think about?
> Related: bug 295256, bug 320121, bug 295255, bug 297614
> 
> CCMAIL: marcel.wiesweg@gmx.de
> 
> FIXED-IN: 3.3.0
> 
> M  +1    -1    CMakeLists.txt
> M  +19   -7    digikam/database/scancontroller.cpp
> M  +7    -0    digikam/database/scancontroller.h
> M  +1    -1    libs/progressmanager/progressmanager.cpp
> R  +1    -0    libs/progressmanager/statusprogressbar.cpp [from:
> libs/widgets/common/statusprogressbar.cpp - 098% similarity]
> R  +1    -0    libs/progressmanager/statusprogressbar.h [from:
> libs/widgets/common/statusprogressbar.h - 096% similarity]
> M  +2    -3    utilities/maintenance/duplicatesfinder.cpp
> M  +1    -1    utilities/maintenance/facedetector.cpp
> M  +1    -2    utilities/maintenance/fingerprintsgenerator.cpp
> M  +1    -1    utilities/maintenance/maintenancedlg.cpp
> M  +117  -77   utilities/maintenance/maintenancemngr.cpp
> M  +14   -8    utilities/maintenance/maintenancemngr.h
> M  +3    -1    utilities/maintenance/maintenancetool.cpp
> M  +24   -20   utilities/maintenance/newitemsfinder.cpp
> M  +0    -1    utilities/maintenance/newitemsfinder.h
> M  +1    -2    utilities/maintenance/thumbsgenerator.cpp
> 
> http://commits.kde.org/digikam/62125c25374b0caf2d6a3812d1d34812ede8a6fd
>

Comment 3 Axel Krebs 2013-07-28 11:24:10 UTC

Dear Gillier, dear Marcel:

the  more I think about, the more I believe: a huge step forward!!

Thank you!


Axel
-------- Original-Nachricht --------
Betreff: [digikam] [Bug 306282] Improvements for maintenance tool
Datum: Sun, 28 Jul 2013 10:46:04 +0000
Von: Axel Krebs <axel.krebs@t-online.de>
Antwort an: bug-control@bugs.kde.org
An: axel.krebs@t-online.de

https://bugs.kde.org/show_bug.cgi?id=306282

--- Comment #2 from Axel Krebs <axel.krebs@t-online.de> ---
Gilles (& Marcel):

Thank you these large amount of information.

As much as I understand, we can expect a serious improvement in V 3.3- I
am keen on it!

Last time, my thumbnail is growing to more than 12 GB, and most probably
related to this fact, startinge tim of my digiKam V. 3.2 is rising to
some twenty minutes!!!

One obstacle seem to be clear: the max cpu percentage does _not_ exceed
25% which mean under quadcore, that digiKam does _not_ use my machine
efficiently.

In many motherboards, the OS can force the graphic chip to support the
cpu. Do you intend to achieve this in future?
--

Besides, the database concept of digiKam seems to limit many features of
the program itself.

There would are many causes to "attach" metadata to some expternal file,
as xml could be. If one starts to look for a selected directory,
all xml-files would be loaded instantaneously- no huge and potentially
risky database- files had to be loads at all.

Belonging to one picture, all derived version of a specific pic could be
treated as one single entity, meaning, that versioning of pics would
dealt automatically.

Tagging, face detection could be incorporated into this single metadata
file.

There are many other advantages as
- independence of storage location of directories,
- minimized starting up time
- keeping together pics, versioneing and metadata at selected locations
etc. etc.

What do you think about?


My statistics:
digiKam version 3.2.0
Bilder:
BMP:                 5
GIF:                 114
JP2:                 14
JPEG:                 60
JPG:                 129282
NIKON-NEF:             55
PCX:                 2
PGF:                 2
PNG:                 2382
RAW-CR2:             339
RAW-CRW:             10824
RAW-DNG:             6
RAW-NEF:             92976
TIFF:                 1931
XCF:                 5
Gesamt:             237997
:
Videos:
AVI:                 60
MOV:                 26
MPEG:                 1
Gesamt:             87
:
Gesamtzahl der Einträge:     238084
Alben:                 3816
Stichwörter:             100
Datenbanktreiber:         QSQLITE


thumbnail db:             12,3 GB
starting up time:        27 min!!!



Am 28.07.2013 10:24, schrieb Gilles Caulier:
> https://bugs.kde.org/show_bug.cgi?id=306282
> 
> Gilles Caulier <caulier.gilles@gmail.com> changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>              Status|UNCONFIRMED                 |RESOLVED
>    Version Fixed In|                            |3.3.0
>          Resolution|---                         |FIXED
>       Latest Commit|                            |http://commits.kde.org/digi
>                    |                            |kam/62125c25374b0caf2d6a381
>                    |                            |2d1d34812ede8a6fd
> 
> --- Comment #1 from Gilles Caulier <caulier.gilles@gmail.com> ---
> Git commit 62125c25374b0caf2d6a3812d1d34812ede8a6fd by Gilles Caulier.
> Committed on 28/07/2013 at 08:00.
> Pushed by cgilles into branch 'master'.
> 
> Maintenance Manager : complete rewrite of maintenance tools handling.
> Instead to use signal provided by tool to know if job is done, use progress
> manager signal.
> Store instance of each tools in d private container instead stack to provide a
> better check of stage under progress.
> Patch ScanController to provide a better feedback to NewItemsFinder, in all
> scan mode. Add a new method to only call core code to perform
> scan WITHOUT to show a progress dialog. In fact progress manager as already all
> in place to give user feedback about progress...
> Patch FaceManagement maintenance tool to indicate that face are detected and
> recognized.
> Move status progress bar with ProgresManager code, as this one is fully
> relevant.
> 
> Marcel, I would to drop fully DProgressDialog code, which is only used now to
> ScanController, with all first run step to collect all information from image
> managed by database. I remember that in 2.x serie, you have patched digiKam to
> only perform first run scan in background when albumgui is show. I'm right ?
> At now, this feature sound fonctionnal some time, but it's not fully
> reproducible.
> If you take a look how Google Picasa work, you will see that main DB
> referencing run in background when main GUI is displayed. A progress indication
> is show in statusbar.
> User can start to work with collected items in real time and don't need to wait
> a long time that all referencing is done through a small progress dialog...
> Look this entry for ex : 320359.
> What do you think about?
> Related: bug 295256, bug 320121, bug 295255, bug 297614
> 
> CCMAIL: marcel.wiesweg@gmx.de
> 
> FIXED-IN: 3.3.0
> 
> M  +1    -1    CMakeLists.txt
> M  +19   -7    digikam/database/scancontroller.cpp
> M  +7    -0    digikam/database/scancontroller.h
> M  +1    -1    libs/progressmanager/progressmanager.cpp
> R  +1    -0    libs/progressmanager/statusprogressbar.cpp [from:
> libs/widgets/common/statusprogressbar.cpp - 098% similarity]
> R  +1    -0    libs/progressmanager/statusprogressbar.h [from:
> libs/widgets/common/statusprogressbar.h - 096% similarity]
> M  +2    -3    utilities/maintenance/duplicatesfinder.cpp
> M  +1    -1    utilities/maintenance/facedetector.cpp
> M  +1    -2    utilities/maintenance/fingerprintsgenerator.cpp
> M  +1    -1    utilities/maintenance/maintenancedlg.cpp
> M  +117  -77   utilities/maintenance/maintenancemngr.cpp
> M  +14   -8    utilities/maintenance/maintenancemngr.h
> M  +3    -1    utilities/maintenance/maintenancetool.cpp
> M  +24   -20   utilities/maintenance/newitemsfinder.cpp
> M  +0    -1    utilities/maintenance/newitemsfinder.h
> M  +1    -2    utilities/maintenance/thumbsgenerator.cpp
> 
> http://commits.kde.org/digikam/62125c25374b0caf2d6a3812d1d34812ede8a6fd
>