Bug 353331 - can’t limit fuzzy-search to albums [patch]
Summary: can’t limit fuzzy-search to albums [patch]
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Searches-Similarity (show other bugs)
Version: 5.0.0
Platform: Debian unstable Linux
: NOR normal (vote)
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-09-29 18:35 UTC by sylvain.l.sauvage
Modified: 2017-04-11 17:37 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In: 5.6.0


Attachments
New fuzzy sidebar : duplicates search (40.78 KB, image/png)
2017-04-04 15:24 UTC, Mario Frank
Details
New fuzzy sidebar : fuzzy image search (39.37 KB, image/png)
2017-04-04 15:25 UTC, Mario Frank
Details
New fuzzy sidebar : fuzzy sketch search (39.95 KB, image/png)
2017-04-04 15:25 UTC, Mario Frank
Details
Proposed patch for the improvements (91.65 KB, patch)
2017-04-04 15:29 UTC, Mario Frank
Details
Updated patch for improvements (102.20 KB, patch)
2017-04-05 15:05 UTC, Mario Frank
Details
Updated patch for improvements (103.03 KB, patch)
2017-04-11 09:57 UTC, Mario Frank
Details
Updated patch for improvements (103.88 KB, patch)
2017-04-11 13:00 UTC, Mario Frank
Details

Note You need to log in before you can comment on or make changes to this bug.
Description sylvain.l.sauvage 2015-09-29 18:35:53 UTC
When I select a collection to search for duplicates in Fuzzy|Duplicates, results are limited to the images in the root album of the collection. The pictures in the other albums of the same collection are not searched/shown.


Reproducible: Always

Steps to Reproduce:
1.Select a collection in the first drop down menu in Fuzzy|Duplicates
2.Search for duplicates
3.

Actual Results:  
Only images in the root album of the selected collection are searched/shown.

Expected Results:  
Images in the other albums of the selected collection should be searched and shown.

It seems to be similar to https://bugs.kde.org/show_bug.cgi?id=336076 which was closed in 4.1.0. I’m using 4.13.0 home-compiled on a Debian Sid.

Also, the first checkbox is called “Whole albums collection” but the selection below presents my collections (not albums). So the checkbox really means “All collections”, no?

And, yes, there are lots of duplicates in the inner albums of the collection I select. They are shown if I check the “Whole albums collection” checkbox (I’m then submerged with pictures of all the collections and their albums). or use “Find similar” on one of them.
Comment 1 caulier.gilles 2015-09-29 19:53:21 UTC
Do you use the Maintenance tool ?

Gilles Caulier
Comment 2 sylvain.l.sauvage 2015-09-29 20:08:52 UTC
(In reply to Gilles Caulier from comment #1)
> Do you use the Maintenance tool ?

Not on purpose: its settings are the default ones, that is: 
- under Common Options, Whole albums collection and Whole tags collection are checked,
- every thing else is unchecked.
Comment 3 caulier.gilles 2015-09-29 20:31:32 UTC
Just to compare, Can you try to run Duplicates Finder from Maintenance tool with same settings than Fuzzy Searches (Duplicates).

In Maintenance, we have fixed this problem. Perhaps it still present in Fuzzy view.

Gilles Caulier
Comment 4 sylvain.l.sauvage 2015-09-29 20:53:04 UTC
(In reply to Gilles Caulier from comment #3)
> Just to compare, Can you try to run Duplicates Finder from Maintenance tool
> with same settings than Fuzzy Searches (Duplicates).

Okay, it works from Maintenance tool, but I have to select each and all sub-albums (a quick way to select all sub-albums is sorely missing).

> In Maintenance, we have fixed this problem. Perhaps it still present in
> Fuzzy view.

Well, Fuzzy doesn’t present sub-albums, only collections so there is at least vocabulary and UI problems.
Comment 5 caulier.gilles 2015-09-29 21:12:21 UTC
To select parent/children albums recursively from somewhere in collection tree view available in Maintenance tool, through combo box, simply use contextual menu...

Gilles Caulier
Comment 6 sylvain.l.sauvage 2015-09-29 22:23:42 UTC
(In reply to Gilles Caulier from comment #5)
> To select parent/children albums recursively from somewhere in collection
> tree view available in Maintenance tool, through combo box, simply use
> contextual menu...

Oh yes! I dumbly didn’t manage to see it. Thanks.
Comment 7 caulier.gilles 2016-07-06 19:43:35 UTC
Sylvain,

Can we considerate this file fixed for last digiKam 5.0.0 ?

Gilles Caulier
Comment 8 sylvain.l.sauvage 2016-07-06 22:57:12 UTC
(In reply to caulier.gilles from comment #7)
> Can we considerate this file fixed for last digiKam 5.0.0 ?

You do whatever want but if you’re asking me if it now works then the answer is a big no!

Two local collections.
The combobox still presents them (and only them, not their sub-albums) as two “albums.”
If I select one, only the root album is selected. And then the search is only made in the root album.
If I use the context menu, I can add all the children of the selected collections but I still can’t chose among my albums.
If the children are selected (“453 albums selected” shown), I can’t deselect them easily: deselecting one collection only deselects the root album of that collection (“452 albums selected” instead of the expected “255…”).

IOW, that UI is still a mess, nothing changed.
If the only way to search in a selected set of albums is to go through the maintenance tool, then simply suppress that useless combobox.

To be frank, I’m disappointed with 5.0.0.
The entire UI has not improved.
Half the menus are not translated.
Dialogs are always too small, half the elements are hidden (I don’t have that problem with my other Qt/KDE apps).
Hiding the menu bar (Ctrl+M or via the menu) doesn’t work: it’s the status bar that disappear!

I had just compiled digikam 5 to test that bug, I’ll stay with Sid’s 4.14 for now.
Comment 9 caulier.gilles 2016-11-29 11:04:16 UTC
Can you reproduce the problem using digiKam Linux AppImage bundle ? The last
bundle is available at this url:

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

Gilles Caulier
Comment 10 Mario Frank 2017-02-20 13:44:12 UTC
(In reply to sylvain.l.sauvage from comment #8)
> (In reply to caulier.gilles from comment #7)
> > Can we considerate this file fixed for last digiKam 5.0.0 ?
> 
> You do whatever want but if you’re asking me if it now works then the answer
> is a big no!
> 
> Two local collections.
> The combobox still presents them (and only them, not their sub-albums) as
> two “albums.”
> If I select one, only the root album is selected. And then the search is
> only made in the root album.
> If I use the context menu, I can add all the children of the selected
> collections but I still can’t chose among my albums.
> If the children are selected (“453 albums selected” shown), I can’t deselect
> them easily: deselecting one collection only deselects the root album of
> that collection (“452 albums selected” instead of the expected “255…”).
> 
> IOW, that UI is still a mess, nothing changed.
> If the only way to search in a selected set of albums is to go through the
> maintenance tool, then simply suppress that useless combobox.
> 
> To be frank, I’m disappointed with 5.0.0.
> The entire UI has not improved.
> Half the menus are not translated.
> Dialogs are always too small, half the elements are hidden (I don’t have
> that problem with my other Qt/KDE apps).
> Hiding the menu bar (Ctrl+M or via the menu) doesn’t work: it’s the status
> bar that disappear!
> 
> I had just compiled digikam 5 to test that bug, I’ll stay with Sid’s 4.14
> for now.

Sylvain,

I had the same problem as you had. But I only had one collection. I first thought it was a bug in my database. But my database was new.
After some time of useless debugging, I found out, that I first have to double-click on the collection. After doing that, all sub albums of this collection were shown. So far, so good. I added another collection. In fuzzy search, the first collection (and its sub-albums were shown). Additionally, I now had the second collection (but no sub-albums for it). After double-klicking on it, the sub-albums showed up.

Can you please test this with the current state of digiKam (Gilles wrote the location)? I tested this with the current state from git.

Perhaps the missing functionality was just not shown prominently.

Cheers,
Mario
Comment 11 Mario Frank 2017-04-03 09:23:18 UTC
Git commit 5bb8e75151f1a2ca5ade6a8ea7b3c1e349996f54 by Mario Frank.
Committed on 03/04/2017 at 09:16.
Pushed by mfrank into branch 'master'.

Fix some minor glitches in fuzzy search sidebar:
1) In a fresh installation, the root albums in the album selection box in duplicates search are not automatically expanded.
Though this is no bug (expanding is possible via double-click), we now automatically expand the root albums to not confuse users.
if nothing was expanded, yet.
2) In duplicates search, the last similarity thresholds are saved. But after restart, one the range of the max similarity
threshold is 90-100 until the min similarity is changed. This is an ugly glitch. Fixed that.
3) The computation of the similar images had a glitch when max similarity = min similarity (e.g. 50%) The results were only
the ones with exactly 50.00 % but not also results with 50.xy as it should be. Fixed that.
4) The xml structure of the haar search queries for fuzzy and sketch search were not wellformed (the closing "search" tag was missing)
Related: bug 372972

M  +2    -1    NEWS
M  +3    -0    app/utils/searchmodificationhelper.cpp
M  +24   -7    libs/album/albumtreeview.cpp
M  +7    -1    libs/database/haar/haariface.cpp
M  +1    -1    utilities/fuzzysearch/findduplicatesview.cpp

https://commits.kde.org/digikam/5bb8e75151f1a2ca5ade6a8ea7b3c1e349996f54
Comment 12 Mario Frank 2017-04-04 15:24:30 UTC
Created attachment 104871 [details]
New fuzzy sidebar : duplicates search
Comment 13 Mario Frank 2017-04-04 15:25:09 UTC
Created attachment 104872 [details]
New fuzzy sidebar : fuzzy image search
Comment 14 Mario Frank 2017-04-04 15:25:48 UTC
Created attachment 104873 [details]
New fuzzy sidebar : fuzzy sketch search
Comment 15 Mario Frank 2017-04-04 15:29:43 UTC
Created attachment 104874 [details]
Proposed patch for the improvements

This patch adds the following improvements to the fuzzy sidebar:

Duplicates search:
The user can select both tags and albums and apply one of 5 operations to the selected objects:
- "One of" aka union - Search in the set of images that are either in the selected tags or albums
- "Both" aka intersection - Search in the set of images that are both in the selected tags and albums
- "Albums but not tags" aka difference - Search in the set of images that are in the selected albums but not in the selected tags
- "Tags but not albums" aka difference - Search in the set of images that are in the selected tags but not in the selected albums
- "Only selected tab" aka obvious - Search in the selected albums, if albums tab is active and in the selected tags if tags tab is active

Fuzzy image (drop) search:
The user can now restrict the results to be in the specified albums.
Restricting to tags is technically possible but would impose complexity problems potentially leading to a slower search - depending on the tagging activities of the user.

Fuzzy sketch search:
The user can now restrict the results to be in the specified albums.
Restricting to tags is technically possible but would impose complexity problems potentially leading to a slower search - depending on the tagging activities of the user.

As I work on the current master branch, some texts are not localised.
Comment 16 caulier.gilles 2017-04-04 16:07:54 UTC
I don't test the p
Comment 17 caulier.gilles 2017-04-04 16:12:18 UTC
I don't tested the patch yet, but the purpose of tab search restriction can be applied to all place where search on albums/tags can be processed.

Gilles
Comment 18 Mario Frank 2017-04-05 15:05:05 UTC
Created attachment 104892 [details]
Updated patch for improvements

The new patch adds patches for 
https://bugs.kde.org/show_bug.cgi?id=183425
and
https://bugs.kde.org/show_bug.cgi?id=300565
Comment 19 Mario Frank 2017-04-11 09:57:27 UTC
Created attachment 104963 [details]
Updated patch for improvements

The AlbumSelectors class is also used in face scan. Updated calls to this class in order to prevent regression bugs in face scan.
Comment 20 caulier.gilles 2017-04-11 11:10:36 UTC
writer.writeField(QLatin1String("noeffect_targetAlbums"), SearchXml::OneOf);
writer.writeValue(targetAlbums);

... why to call "noeffect_targetAlbums" a property which define the target album to process ? It's ambiguous... Or i miss understand something in your code ?

Gilles
Comment 21 Mario Frank 2017-04-11 12:12:47 UTC
(In reply to caulier.gilles from comment #20)
> writer.writeField(QLatin1String("noeffect_targetAlbums"), SearchXml::OneOf);
> writer.writeValue(targetAlbums);
> 
> ... why to call "noeffect_targetAlbums" a property which define the target
> album to process ? It's ambiguous... Or i miss understand something in your
> code ?
> 
> Gilles

Hey Gilles,

The field is only set for HAAR search, i.e. fuzzy image and sketch search.
The search can be restricted to only return images if they are in one of the target folders.
Do you have a better terminus in mind?

Cheers,
Mario
Comment 22 caulier.gilles 2017-04-11 12:14:14 UTC
Not better terminology for the moment. Just put some comment to clarify it...

Gilles
Comment 23 Mario Frank 2017-04-11 12:47:47 UTC
(In reply to caulier.gilles from comment #22)
> Not better terminology for the moment. Just put some comment to clarify it...
> 
> Gilles

Okay,

I would commit the changes with additional comments.
Is that okay for you?
Comment 24 Mario Frank 2017-04-11 13:00:32 UTC
Created attachment 104965 [details]
Updated patch for improvements

Updated the patch to the current state of the master branch and added some comments concerning target albums.
Comment 25 caulier.gilles 2017-04-11 14:06:46 UTC
yes fine for me. I test the patch and it sound working fine for me.

I will rebuild the bundles this evening. I recommend to post a message to user ML to have some feedback with new bundles including last patches. The possibilies to introduce a
Comment 26 caulier.gilles 2017-04-11 14:07:59 UTC
yes fine for me. I test the patch and it sound working fine for me.

Or perhaps somebody have discovered a dysfunction with this patch. Any feedback ?

I will rebuild the bundles this evening. I recommend to post a message to user ML to have some feedback with new bundles including last patches. The possibilities to introduce a dysfunction is not negligible of course...

Gilles
Comment 27 Mario Frank 2017-04-11 17:31:42 UTC
Git commit 660782be8137d8448ef10e5e944a2dd213d781cc by Mario Frank.
Committed on 11/04/2017 at 17:27.
Pushed by mfrank into branch 'master'.

Added an overall improvement of fuzzy, duplicates and extended search.
1) Users can now restrict the results of fuzzy image/drop/sketch search to be in specific albums.
2) Users can now select both tags and albums in duplicates search and restrict the set of images in which duplicates are searched with logical operations (in selected albums or/and tags, in selected albums but not tags and vice versa and only in
selected objects of current tab)
3) In extended search, the tags selection can also be restricted such that images must be either tagged with at least one of or with all selected tags.
Related: bug 300565, bug 183425

M  +7    -1    NEWS
M  +31   -3    app/utils/searchmodificationhelper.cpp
M  +11   -0    app/utils/searchmodificationhelper.h
A  +307  -0    libs/album/albumselectioncomponent.cpp     [License: UNKNOWN]  *
A  +121  -0    libs/album/albumselectioncomponent.h     [License: GPL (v2+)]
M  +248  -173  libs/album/albumselectors.cpp
M  +34   -19   libs/album/albumselectors.h
M  +7    -0    libs/database/coredb/coredbsearchxml.cpp
M  +1    -0    libs/database/coredb/coredbsearchxml.h
M  +1    -0    libs/database/dbjobs/dbjob.cpp
M  +11   -0    libs/database/dbjobs/dbjobinfo.cpp
M  +4    -0    libs/database/dbjobs/dbjobinfo.h
M  +119  -30   libs/database/haar/haariface.cpp
M  +30   -8    libs/database/haar/haariface.h
M  +32   -10   libs/database/item/imagelister.cpp
M  +44   -0    libs/database/item/imagequerybuilder.cpp
M  +6    -4    libs/settings/applicationsettings.cpp
M  +3    -0    libs/settings/applicationsettings.h
M  +11   -0    libs/settings/applicationsettings_miscs.cpp
M  +1    -0    libs/settings/applicationsettings_p.cpp
M  +2    -0    libs/settings/applicationsettings_p.h
M  +1    -1    utilities/facemanagement/facescandialog.cpp
M  +69   -22   utilities/fuzzysearch/findduplicatesview.cpp
M  +83   -21   utilities/fuzzysearch/fuzzysearchview.cpp
M  +1    -0    utilities/fuzzysearch/fuzzysearchview.h
M  +6    -1    utilities/maintenance/duplicatesfinder.cpp
M  +1    -1    utilities/maintenance/duplicatesfinder.h
M  +4    -4    utilities/maintenance/maintenancedlg.cpp
M  +2    -1    utilities/maintenance/maintenancemngr.cpp
M  +72   -27   utilities/searchwindow/searchfields.cpp
M  +18   -4    utilities/searchwindow/searchfields.h

The files marked with a * at the end have a non valid license. Please read: http://techbase.kde.org/Policies/Licensing_Policy and use the headers which are listed at that page.


https://commits.kde.org/digikam/660782be8137d8448ef10e5e944a2dd213d781cc