Bug 123097

I would like to second this and illustrate it with my usage case: I regularly take pictures on my mobile phone's camera for everyday business usage. The images are stored in the folder 'Images' on the phone's memory card. Next to that, also my music collection is stored on the card as well as all the thumbnails.
Importing images into digiKam takes quite a long while because it browses all folders on the memory card and faithfully lists all the music titles anew. The ability to turn off certain folders and/or turn on specific folders would speed up the process quite a bit. After all it does get tiresome looking through a list of a couple of hundred items where 90% remain unchanged over a long time. Plus it is almost impossible to distinguish thumbnails - which I do not want to download - from the real pictures.
Thank you for your consideration and thank you very much for the excellent work invested into digiKam!
St. Mueller

I also vote for this bug. I'm Using digikam 1.0

my background is: I use a separate partition for the digikam_albums, the partition-mountpoint (here: /digikam_alben) is the root for the album tree.
As I use KDE, this partition also contains a .trash folder that appears in digikam in the album tree.

When moving images or folders to the trash within digikam, digikam again recognises them in the trash folder, thus making trashing files very slow.

It would be nice - and I consider it important - to have a setting to exclude any folder from the digikam-album tree.

*** This bug has been confirmed by popular vote. ***

*** Bug 151317 has been marked as a duplicate of this bug. ***

I just thought I would bump the version number on this bug as this is a feature that is still being asked for by new users.

*** Bug 153423 has been marked as a duplicate of this bug. ***

This is not really import tool related, but more for the general views, right? If so, the title and the component should be changed...

*** Bug 277619 has been marked as a duplicate of this bug. ***

Created attachment 89319 [details]
Patch to ignore @eaDir subdirectories and their content

Synology Diskstation and other NAS systems use these directories for DNLA or some other obscure purposes, these subdirectories contain tumbnails in several resolutions.
In digikam I only want to see the original images, and not the extra three tumbnails that the NAS creates for every picture!

With the patch I provided, Digikam starts in 3minutes, instead of 45 minutes.
I still get thousands of messages like this:

digikam(14625)/digikam (core): Could not find parent with url:  "/page3_files/@eaDir"  for:  "/page3_files/@eaDir/ggp1.png"

Any idea how to prevent those ?

Just some thoughts:

1. The already existing entries from database with the filter should not be removed in my opinion in that place but in maintenance if at all.
2. Blacklist entries should not be hardcoded directly, but provided through some kind of interface/variable (QStringList? Regexp support?), so that other dirs and files can be easily added for exclusion in the future if needed.
3. Is there a need to expose this to the UI somehow? Adding a setting to "add collection" dialog?

And this "Could not find parent with url" error is coming from AlbumManager::scanPAlbums(), which gets its data from AlbumDB::scanAlbums() in libs/database/albumdb.cpp which in turn reads it from the database.

Maik, Marcel,

There is patch here which can be interesting to include and improve.

What do you think about ?

Gilles

@Amilcar Lucas
Is @eaDir still valid at the moment, In other words is your patch still valid?

Some weeks ago, I tested the Amilcar Lucas patch and it was still valid. With reference to  Teemu Rytilahti post, I would like to add a UI interface, where all folders that should be ignored can be added. I think of free text field where users can add a comma separated list with folder names.
What do you think: Is the dialog under Setting -> Configure digiKam a place for this purpose? If yes, could someone point me to the responsible classes for this dialog?

Hi,

Yes, a setup dialog field to configure this feature is right.

I propose to put this new settings into Setup/Database page. There are plenty of freen space on the bottom.

The setup page is hosted in class SetupDatabase :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/utilities/setup/setupdatabase.cpp

The settings is hosted in a manager named ApplicationSettings :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/app/settings/applicationsettings_database.cpp

ApplicationSettings is a singleton accessible everywhere. Take a care to define right default settings.

Gilles Caulier

Hello Gilles,
I have extended the setupdatabase and it is possible to add directory names that should be ignored. The settings are correctly stored in the digikamrc config file.

But now I have a problem when I try to access the settings in the collection scanner. There I include applicationsettings.h and want to access the settings by the following command:
QString ignoredDirectories = ApplicationSettings::instance()->getIgnoredDirectories();

I get the following error:
CMakeFiles/digikamdatabase.dir/__/libs/database/collectionscanner.cpp.o: In function `Digikam::CollectionScanner::CollectionScanner()':
/home/jan/Downloads/digikamGIT/core/libs/database/collectionscanner.cpp:376: undefined reference to `Digikam::ApplicationSettings::instance()'
/home/jan/Downloads/digikamGIT/core/libs/database/collectionscanner.cpp:376: undefined reference to `Digikam::ApplicationSettings::getIgnoredDirectories() const'
collect2: error: ld returned 1 exit status

Could you give me hint, what I make wrong?

Can your share your current patch. It will more easy to investiguate.

It sound like a linking problem between digiKam components. It's strange because all database relevant code are compiled in same component.

Gilles Caulier

Created attachment 95010 [details]
ignore directories patch

Thank's Gilles. Attached you find my current working state.

Hey Gilles,
when I understand your mail to the mailing list correctly, there will no further investigations in digikam version 4.x based on KDE4/Qt4 from this week on. Hence, I suppose its the best way to integrate my current patch in the new Qt5/KF5 version. What do you think?

yes absolutely. No KDE4 version will be planed now. git/master is now Qt5/KF5 code based.

I currently finalize 5.0.0-beta1 code. I will review your patch just after.

Gilles Caulier

Thanks. Then I have to install newer versions of all dependent software packages to compile current dk 5.0 beta version on my PC.

Jan,

Do you have a revised version of your patch working with current implmeentation from git/master based on Qt5 ?

Gilles Caulier

Hi Gilles, at the moment I didn't tested it. But I hope I will find some time to test it with dg 5 during the next 2-3 weeks.

Created attachment 101781 [details]
IgnoreDirectories-Patch dk 5

Hello Gilles,

I found some time to port my fix to new dk version 5. The attached code depends on tag v5.2.0.

It depends on the initial patch from Amilcar Lucas to ignore all @eaDir directories (hard-coded in class collectionscanner.cpp). My part was to configure what directories should be ignored. Therefore I added a text field to the database settings menu. The entries a user make there will be stored in digikamrc.

The last part to complete the task is to replace the hard-coded ignoration of @eaDir by the entries made by users. Therefore, I need access to the database parameters in collectionscanner class:
const DbEngineParameters prm = ApplicationSettings::instance()->getDbEngineParameters();

Unfortunately, I get the same error as in dk 4 version:
CMakeFiles/digikamdatabase_src.dir/collection/collectionscanner.cpp.o: In function `Digikam::CollectionScanner::CollectionScanner()':
/home/jan/Downloads/digikamGIT/core/libs/database/collection/collectionscanner.cpp:363: undefined reference to `Digikam::ApplicationSettings::instance()'
/home/jan/Downloads/digikamGIT/core/libs/database/collection/collectionscanner.cpp:363: undefined reference to `Digikam::ApplicationSettings::getDbEngineParameters() const'
collect2: error: ld returned 1 exit status
make[2]: *** [core/libs/database/libdigikamdatabase.so.5.2.0] Error 1
make[1]: *** [core/libs/database/CMakeFiles/digikamdatabase.dir/all] Error 2
make: *** [all] Error 2

You guessed last year it is a linking problem. Could you please check that?

Created attachment 101805 [details]
Patch on current repository status

The same problem on current status.
The problem seem to be somewhere under the hood. Would be nice, if someone with deeper understanding of dk code could check that.

Created attachment 101955 [details]
updated patch to link fine.

[  9%] Linking CXX shared library libdigikamcore.so
[  9%] Built target digikamcore
[  9%] Linking CXX shared library libdigikamdatabase.so
CMakeFiles/digikamdatabase_src.dir/collection/collectionscanner.cpp.o: In function `Digikam::CollectionScanner::CollectionScanner()':
/home/gilles/Devel/5.x/core/libs/database/collection/collectionscanner.cpp:363: undefined reference to `Digikam::ApplicationSettings::instance()'
/home/gilles/Devel/5.x/core/libs/database/collection/collectionscanner.cpp:363: undefined reference to `Digikam::ApplicationSettings::getDbEngineParameters() const'
collect2: error: ld returned 1 exit status

You try to use symbols defined in libdigiKam into libdigikamdatabase. Symbols are separated due to use libdigikam with showfoto which is database free.

In fact you just want to print as debug statement the DB engine parameters from ApplicationSettings in CollectionScanner. This is not permit at linking time to prevent cycling symbols linking.

To get DbEngineParameters in CollectionScanner, just use DbCoreAcess.parameters() as well.

Note : never use Tabs in source code. This break the coding style.

Note2 : In CollectionScanner, i suppose that all is not yet implemented. The new settings from DbEngineParameters about directories to ignore while scanning collections is just not used. All checks about "@eaDir" still hard-coded as well. Right ?

Gilles Caulier

Hello Gilles,
thanks for your help. I will show your code the next days and complete the fix for this issue.

@Note 1: Ok, I will replace tabs by blanks.

@Note 2: You are right, "@eaDir" is still hard-coded, but with adjusted patch I should be able to use the new settings from DbEngineParameters.

ok,

Also, i see in Setup dialog databse page that some TODO are not yet implemented, especially for the GUI.

Gilles Caulier

Jan,

Any progress here to be able to include your patch in next 5.4.0 release ?

Thanks in advance 

Gilles Caulier

Created attachment 102158 [details]
Update Patch IgnoreDirectories

Hello Gilles,

yes there are progress.
The attached patch supports the requested feature: You are able to add directories (separated by whitespaces in the text field) that you want to ignore from your collection. The settings are stored in DbEngineParameters. All directories that exactly match the configured directory name, will be ignored from the photo collection.

But unfortunately, the configured directories gets ignored only after restarting digikam.
When a user clicks the OK-Button of settings dialog, the DbEngineParameters are updated correctly. But in the CollectionScanner CoreDbAccess().parameters() delivers instead of the actual parameters, the parameters that were valid at digikam startup.
Do you have, any hints to solve this?

Are there plans to implement this patch in the official release? I am using Philip's PPA repository to stay up-to-date with my digikam5 version. It would be cool to have it included there, and avoid manual patching work. I am also having an extremely slow Digikam experience because of @eadir folders on the nfs mounted Synology. Some time ago I just deleted all those system generated folders, but recently had them rebuild. I would much rather have an exclude filter on the collection discovery. Thanks, Arnaud

Arnaud,

Yes, it's planed to include it for next 5.4.0 if Jan complete it in time, which will be the case as 80% of the patch is already done...

Gilles Caulier

Jan,

Any progress to complete the patch before 5.4.0 release planed in December ?

Gilles Caulier

Hi Gilles,
since my last update there is no further progress, but I hope to find time to complete the last 20% till next release.
For what date in December is version 5.4 planned?

5.4.0 is planed before Christmas. Don't forget that i need at least one week before release date to check implmentation, and to lets time to translators to update i18n with new strings introduced with your patch.

Hi Gilles,
when changes in the new ignoredDirectories settings text field are made I trigger now a album rescan to ignore the directories. At this moment the dbEngineParameters contain the newly inserted ignoredDirectories. But the CollectionScanner refers to CoreDbAccess().parameters() which do NOT contain the newly inserted ignoredDirectories.
Could you give me a hint how to update CoreDbAccess().parameters() after performing changes at dbEngineParameters?

Please share the new patch to review the code. It will be more easy.

Gilles

Created attachment 102709 [details]
New version of Patch

Hi Gilles,
see attached the patch. To update the paramters in CoreDbAccess according to the dbEngineParameters, I call setupdatebase.cpp: CoreDbAccess::setParameters(params);

See the complete snippet here:
if (d->databaseWidget->getDbEngineParameters().ignoredDirectories != d->databaseWidget->orgDatabasePrm().ignoredDirectories)
    {
        qCDebug(DIGIKAM_GENERAL_LOG) << "Ignored directories settings changes. Update...";
        DbEngineParameters params = d->databaseWidget->getDbEngineParameters();
        settings->setDbEngineParameters(params);
        settings->saveSettings();
        CoreDbAccess::setParameters(params);
        ScanController::instance()->completeCollectionScanInBackground(false);
    }

But unfortunately, this crashed the hole application:
*** Error in `/opt/kf5/bin/digikam': double free or corruption (fasttop): 0x00007fffc00dcbe0 ***
Program received signal SIGABRT, Aborted.

Do you know a way to update the parameters in CoreDbAccess correctly?

Created attachment 102753 [details]
Patch

Hi Gilles,
I have attached a version without the crash caused by the wrong call of CoreDbAccess::setParameters.
The remaining problem is still that after updating the settings, the CollectionScanner refers to CoreDbAccess().parameters() which do NOT contain the newly inserted ignoredDirectories.

It would be nice to take a look on that and tell me how to update CoreDbAccess after adding new settings.

Hello Gilles,
do you have any hint for me to update CoreDbAccess after updating dbEngineParameters?
It would be great if you could help me to fix this last problem to get it work before dk 5.4 deadline.

Best regards,
Jan

I'm on holidays this week end for 2 weeks. I will have plenty of time to respond. For the moment, until Saturday evening, i'm very busy in my office...

Gilles

Jan to your question from #40.

In SetupDatabase, when you change ignored directories settings, If think you just need to to call AlbumManager::instance()->changeDatabase(params) to update global DB settings before to call ScanContoller.

I will update and polish the patch for testing.

Gilles Caulier

Created attachment 103602 [details]
new patch version ready for testing

Screenshot of Database config pannel with this patch:

https://www.flickr.com/photos/digikam/32441766536/in/dateposted-public/

Gilles Caulier

Jan,

I'm not fully satisfied by this patch.

1/ we must set a default ignore list of files/directories already know as problematic while scanning collections

For example :

- under MacOS, .DS_Store hidden files are created everywhere by Desktop search engines. This bloat disk of course but there is no way that i know to disable this feature.

- Under Linux/Plasma, Dolphin files manager permit to customize icon view order by directory. This generate .directory hidden files.

- Under Linux/Plasma, the trash tool create an hidden directory .Trash-_userid_ with a content of deleted files. This one must be ignored by digiKam.

And i forget certainly some hidden files/directories under other desktops as Gnome or Windows.

I propose to set a list that we know already to ignore. This list must be visible by end user from config panel, like it's already done in View/Mime-Type section for each setting.

Gilles Caulier

Hello Gilles,

thanks for your help and your feedback. With your hint it generally works, thats great!

Now to your point regarding a default ignore list: 
The examples you mentioned all begin with a dot and has been ignored by default anyway. Do I understand you correctly, that you wish to establish a list (as in View/Mime-Type) with ignored directories (e.g. all that begin with a dot) *just to inform the user* about this default feature?
If that is your intension, I will add such a list.

yes exactly.

The new option in setup database must permit to define new combinaison of ignored directory schemes. By default we must already set in database interface some problematic schemes that we know already. 

For type mime default scheme, database set these values CoreDbSchemaUpdater class.

https://cgit.kde.org/digikam.git/tree/libs/database/coredb/coredbschemaupdater.cpp#n460

On SetupMime class, we use this widgets to configure extra mime filters:

https://cgit.kde.org/digikam.git/tree/utilities/setup/album/setupmime.cpp#n108

Gilles Caulier

Created attachment 103660 [details]
Newest version of patch

Hello Gilles,
I added an info pop-up as in the mime config including the explanation that all directories starting with a dot will be ignored and a small list of explicit directory names you mentioned (this list could be extended of course).
I would be glad if you check-in the patch in the repository :)

Best regards,
Jan

I have to admit I have not followed the entire discussion here, but I wanted to throw in my 2 cents...  The fix for this bug/feature request should allow users to exclude MTP mounted smartphone directories, which are very, very, very slow to process.  For example:
"/run/user/1910/gvfs/mtp:host=%5Busb%3A002%2C029%5D/SD card/"
which includes some strange syntax.
Also udisk mounts under /media, which can also be very, very slow to process.  Also FUSE filesystem mounts, which can also have some strange syntax, like the MTP example.

MTP camera is relevant of Import tool, not database scanning process.

By scanning, we talk about the process which scan collections to register items to database.

removal camera are driven by Import tool, which have already some dedicated rules to ignore contents.

Gilles Caulier

Jan,

I stil to not understand how this patch can work in background level to ignore some kind of subdirectories at scan.

You said :

"For example, when you store your Synology NAS (Network Attached Storage), the system creates in every directory a subdirectory @eaDir to store thumbnails."

and i can see this list populated in help widget :

QLatin1String(".DS_Store") << QLatin1String(".directory") << QLatin1String(".Trash-_userid")

... but there is no real code to handle this directory naming scheme.
Where is defined the DEFAULT ignore dir settings in core database implementation with "@eaDir", ".DS_Store", ".directory", ".Trash-_userid" ? As i said in comment #47, the core DB as already a set of type-mime values pre-configured. End users will expect to have the same default rules for directories to ignore at scan time.

Gilles Caulier

Hello Gilles,

I suppose, we have a misunderstanding here. My patch does not introduce new directories that get ignored by default. Even the @eaDir directory gets ignored only, when a user adds it to the new configuration dialog (the directories are saved as part of DbEngineParameters). My approach was that digikam behaves with my patch but without configuring directories in the new dialog in the same way as in the current version without my patch.

Concerning your comment with the info dialog, I thought you want a list including all directories from which we know that they will be ignored. And that’s just all directories starting with a dot. Since, my patch does not introduce additional ignored directories by default, such info dialog with exemplary ignored directories could be skipped.

What do you think?

Well the info dialog is perfect. Just add a default ignore settings user will expected to have in core digiKam, as it's done with mime type settings. We need to be homogeneous here.

Gilles

Hi Gilles,
ok, I will support default settings. I looked a bit deeper in the code defining the default settings for mime types. They are all directly stored in the SQLite DB in the settings table, supported by function setSettings in DbCore class.
Up to now, I store the ignored directories settings in DbEngineParameters.

What do you suppose:
1) Should I leave the ignored directories settings there and add just default ignore directories?
2) Or should I move the settings to the database as its done for mime types.

Regarding the directories starting with a dot: They are ignored anyway and are not touched by my patch. I think that should remain so in the future. Do you agree?

I vote for 2/, for these reasons :

1/ to be homogeneous with other settings.
2/ each collection database file can be customized. The settings is not common and is portable. Remember that DB file to use with digiKam session can be passed on CLI.

Gilles

Yes, I agree that this is the most consistent way.

But what do you mean that "DB file to use with digiKam session can be passed on CLI."?

digikam --help
Usage: digikam [options]
Manage your photographs like a professional, with the power of open source

Options:
  -v, --version               Displays version information.
  -h, --help                  Displays this help.
  --author                    Show author information.
  --license                   Show license information.
  --desktopfile <file name>   The base file name of the desktop entry for this
                              application.
  --download-from <path>      Open camera dialog at <path>
  --download-from-udi <udi>   Open camera dialog for the device with Solid UDI
                              <udi>
  --detect-camera             Automatically detect and open a connected gphoto2
                              camera
  --database-directory <dir>  Start digikam with the SQLite database file found
                              in the directory <dir>

^^^ The last argument permit to change the collection database (DB) on the command line interface (CLI)

Typically, it's the same than to change the database config from digiKam/Setup dialog...

Gilles

Created attachment 104043 [details]
Reworked patch

Hi Gilles,
attached is the reworked patch, which behaves exactly as the mime config now. The settings are directly stored in SQLite DB. The @eaDir is now ignored by default, but users are able to include it by prefixing it with a minus (the same behaviour as in mime config).

Excelent. Thanks a lot Jan.

I will have plenty of time while this week end to review this patch for next 5.5.0 release.

Gilles

Git commit e9bc2261c28fd55cf9f1285f51dbee93f4a3e3c3 by Gilles Caulier.
Committed on 21/02/2017 at 18:53.
Pushed by cgilles into branch 'master'.

Apply patch #104043 from Jan Wolter to ignore some kind of directories at collections
scan and to prevent to bloat database with unwanted/unecessary entries.
FIXED-IN: 5.5.0

M  +3    -1    NEWS
M  +69   -157  libs/database/collection/collectionscanner.cpp
M  +12   -8    libs/database/collection/collectionscanner.h
M  +154  -83   libs/database/coredb/coredb.cpp
M  +12   -4    libs/database/coredb/coredb.h
M  +14   -3    libs/database/coredb/coredbschemaupdater.cpp
M  +5    -3    libs/database/coredb/coredbschemaupdater.h
M  +93   -11   libs/database/utils/dbsettingswidget.cpp
M  +6    -2    libs/database/utils/dbsettingswidget.h
M  +6    -2    utilities/setup/setupdatabase.cpp
M  +2    -2    utilities/setup/setupdatabase.h

https://commits.kde.org/digikam/e9bc2261c28fd55cf9f1285f51dbee93f4a3e3c3