Bug 293297 - Digitaglinktree does not process any photos even if tags exists under People category + new option for year subdirectory [patch]
Summary: Digitaglinktree does not process any photos even if tags exists under People ...
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Plugin-Generic-WishForNewTools (show other bugs)
Version: 2.1.1
Platform: Ubuntu Linux
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-02-04 14:27 UTC by cyril.raphanel
Modified: 2013-11-27 14:45 UTC (History)
2 users (show)

See Also:
Latest Commit:
Version Fixed In: 4.0.0


Attachments
-r option enable again for digikam version > .10. (23.40 KB, patch)
2012-02-06 19:04 UTC, cyril.raphanel
Details
New Version 1.8.0 - proposal for patch (24.31 KB, application/octet-stream)
2012-02-07 16:34 UTC, cyril.raphanel
Details
digitaglinktree Version 1.8.1 (25.95 KB, application/octet-stream)
2012-02-08 12:43 UTC, krienke
Details
Fixed digitrablinktree version (26.16 KB, application/octet-stream)
2012-02-13 07:51 UTC, krienke
Details
Manual Page for digitaglinktree (3.44 KB, application/x-gzip)
2012-02-13 07:52 UTC, krienke
Details

Note You need to log in before you can comment on or make changes to this bug.
Description cyril.raphanel 2012-02-04 14:27:48 UTC
Version:           2.1.1 (using KDE 4.7.4) 
OS:                Linux

I have tagged a lot of people under people category.

If I select the "tags" table
********************************************
select * from tags where pid="9" or  id="9"
********************************************
"id", "pid", "name", "icon", "iconkde"
"9", "0", "People", "0", ""
"10", "9", "Unknown", "0", ""
"25", "9", "cyril", "0", ""
"26", "9", "irene", "0", ""
"27", "9", "amelie", "0", ""
"28", "9", "liam", "0", ""
"29", "9", "theo", "0", ""

Tagtree table shows
************************************************
select * from tagstree where pid="9" or  id="9"
************************************************
"id", "pid"
"9", "0"
"10", "9"
"27", "9"
"25", "9"
"26", "9"
"28", "9"
"29", "9"

If I take one of the tags I can find associated images
****************************************
select * from imagetags where tagid="29"
****************************************
"imageid", "tagid"
"12018", "29"
"12020", "29"

When I lauch the script
****************************************************
digitaglinktree -l my_target -d my_digikam_directory
****************************************************
Processed 0 photos.



Reproducible: Always

Steps to Reproduce:
Every time I launch the script

Actual Results:  
Processed 0 photos

Expected Results:  
complete tree for People category tags
Comment 1 cyril.raphanel 2012-02-04 14:34:18 UTC
Script version 
Version: 1.7.0
Comment 2 cyril.raphanel 2012-02-05 12:21:50 UTC
What language is written digitaglinktree?

Where could I find the source code without installing the whole digikam source package?
Comment 4 cyril.raphanel 2012-02-05 12:57:48 UTC
Thanks,

old perl script (last version is 2008) so probably not fully compatible with current sql schema.

Is there an alternative from the user interface (kipi export plugin?) that would fo the same?
Comment 5 caulier.gilles 2012-02-05 13:20:36 UTC
No alternative. script must be adapted...

Gilles Caulier
Comment 6 cyril.raphanel 2012-02-05 14:27:00 UTC
Now I get stuck in my investigation 

Problem seems around this part, it systematically skip all files

###############################################################################
	# $refImages always contains all images from all albums. For Mode -A 
	# in digikam 0.10 it may happen that some albums are not on the same filesystem 
	# like the photoRoot. in this case this album is not cloned to another 
	# directory buth neverless
	# refImages contains also data about photos in this album. But since it is 
	# not cloned, the cloned root # does not exits and so the photo therin does 
	# not exist. Thats what we check here.
        # CR
	print "BEFORE -- ${photoRootDir}$path/$image\n";
	print "PHOTOROOTDIR -- ${photoRootDir}\n";
	print "PATH -- $path\n";
	print "image -- $image\n";
	# RC
	next if( ! -r "${photoRootDir}$path/$image" ); 
        # CR
	print "AFTER -- ${photoRootDir}$path/$image\n";
	# RC
	$count++;
###############################################################################

I can see "BEFORE" log
##############################################
BEFORE -- /photos/thumb/canond30_IMG_1041.jpg
PHOTOROOTDIR -- /photos
PATH -- /thumb
image -- canond30_IMG_1041.jpg
##############################################

but never reach the "AFTER" log

I would need some help from a PERL Guru ....
Comment 7 cyril.raphanel 2012-02-05 14:34:10 UTC
Somehow make sense as "photoRootDir" is not the correct full path here. So if the script check if the file exist it never find it ...

Need to figure out how the "photoRootDir" is compiled/read
Comment 8 cyril.raphanel 2012-02-05 14:52:31 UTC
Is my database data correct?

***************************
select * from  albumroots;
***************************
"id", "label", "status", "type", "identifier", "specificPath"
"1", "photos", "0", "1", "volumeid:?uuid=560e6bbb-a2ae-498f-a335-4a8592005e47", "/photos"

Should "specificPath" contains the exact album path?

My "photos" directory is not under root file system but under an hardrive mounted under "/media/disk1".

Correct complete path of my root for album is "/media/disk1/photos" and not "/photos".

Is it now:
- a bug in digkam data management?
- a bug in the script?
- a configuration setting I should do in digikam?

Thanks
Comment 9 cyril.raphanel 2012-02-05 15:00:28 UTC
I have edited manually the "specificPath" to the real album path "/media/disk1/photos". 

Script is now working perfectly but is it consistent for digikam data model?
Comment 10 cyril.raphanel 2012-02-05 17:07:15 UTC
I originally add the directory as "Local collections" would it make any difference for the "SpecificPath" attribute if I had the directory with "Removable Media" or "Networkd shares"?
Comment 11 cyril.raphanel 2012-02-05 19:06:59 UTC
Reading some related bug
https://bugs.kde.org/show_bug.cgi?id=236730

could it be that simple that the script should include "identifier" information to define the complete path?

how do you do that in perl/bash script?
Comment 12 cyril.raphanel 2012-02-05 19:19:48 UTC
uuid is the one of the hardrive I have my collection
/dev/sdb: LABEL="disk1" UUID="560e6bbb-a2ae-498f-a335-4a8592005e47" TYPE="ext4"

I mount it on "/media/disk1" directory ...

Is there a smarter way to make so it is understood by digikam and the script?
Comment 13 cyril.raphanel 2012-02-06 19:04:32 UTC
Created attachment 68572 [details]
-r option enable again for digikam version > .10.

Following lines commented
    # Now Process all albums found
    # if( defined($opt_r) ){
    #   warn "\n$0: Hint: Album root definitions given by \"-r\" ignored \n",
    #        "for digikam version >= 0.10 . Instead all Albums defined in digikams \n",
    #	    "database are used. Continuing work ... \n\n";
    #}


Following lines added
      # NEW CODE
      # Interprete -r as base for all albums
      $albumRoots{$i}="$opt_r$albumRoots{$i}";
      # END NEW CODE

This has been tested with
digitaglinktree -l target_directory -d digikam_database -r directory_where_partition_including_albums_is_attached
Comment 14 cyril.raphanel 2012-02-06 19:30:21 UTC
Might need more testing and a bit of documentation.

Potential improvement:
Add "date" parameter which would add time directory below each tag
People->paul->2001->image1.jpg
            ->2002->image2.jpg
Comment 15 cyril.raphanel 2012-02-07 16:34:37 UTC
Created attachment 68595 [details]
New Version 1.8.0 - proposal for patch

# 1.7.0 -> 1.8.0 2012/02/07 Cyril Raphanel, cyril.raphanel@gmail.com
# Added (-r) option for new digikam database as root directory for a set of albums below same file system
# Added (-Y) option to add "year" directory below each tag

Tested with 
  -l <tagdir>  path to directory where the tag directory structure with 
               links to the original images should be created 
  -d <dbfile>  full path to the digikam database file 
  -r <rootdir> path to digikams root directory containing all photos.
               For digikam vers >= 0.10 root directory of a set of albums,
               Script has to be used as many time as there is different root 
               	directories.
               Mandatory for earlier digikam versions!
  -Y           Add Year directory below each tag directory 
  -h           Print this help 
  -v           Print scripts version number 
Using Digikam 2.1.1

Not tested with
  -A <ardir>   Create selfcontained archive of photos and tags in directory
               ardir. rootdir and arDir have to be on the *same* filesystem!
  -C           If -A <archdir> was given this option will put hardlinks of all
               photos in the "Photos" directory not only of those with tags.
  -a           Create absolute symbolic links instead of relative ones 
  -H           Use hard links instead of symbolic links in linktree. 
  -f           If there are hierarchical tags (tags that have subtags) 
               create a flat tag directory hierarchy. So directories for
               subtags are at the same directory level like their parent tags
Comment 16 caulier.gilles 2012-02-07 16:41:54 UTC
Cyril,

Thanks for your patch

Krienke,

Can you review it please ?

Gilles Caulier
Comment 17 cyril.raphanel 2012-02-07 18:41:55 UTC
Hi,

It should be also possible to use mysql instead of sqlite.

Only difference seems that result given by only tools is with tab delimiter
mysql -u myuser -p -N -r --execute="mysql query"

I do not have database install with digikam but could try this week.
Comment 18 cyril.raphanel 2012-02-07 18:42:40 UTC
sorry meant "command line tool"
Comment 19 krienke 2012-02-08 07:43:01 UTC
Hi cyril.raphanel@gmail.com,

I looked at your patch and it looks good. Nice year extension. For my photo collection the old 1.7.0 version still works fine. 

I think your trouble originated in the the fact that one of your albumroots was not mounted in / but in /media/. So the database entry for specific path was not an absolute path to your album but relative to the mount point of device with the uuid you posted. 

If this is true, then the solution would be to fix function getAlbumRoots to get things straight automatically. 

Can you please check with your system if this works:

ls -l /dev/disk-by-uuid/560e6bbb-a2ae-498f-a335-4a8592005e47/photos

It should list the contents of your album. If this works one could extract the uuid information for an album root in function getAlbumRoots() and if we find an uuid in column "identifier" we should prepend /dev/disk-by-uuid/<uuid-value> in front of the value of column "specificPath". The result would be a complete path to the albumroot. So option -r would not be needed in your case.
Comment 20 krienke 2012-02-08 09:10:54 UTC
@cyril.raphanel@gmail.com:
The result of of the check I asked you to do:

ls -l /dev/disk-by-uuid/560e6bbb-a2ae-498f-a335-4a8592005e47/photos

is of course a device. It should be the one containing your album.  So in order to find out the path where this device is (possibly) mounted one still has to search eg the contents of /proc/mounts (if available). To the result of this search one has to append "specificPath" and this should then be the albums root.
Comment 21 krienke 2012-02-08 12:42:24 UTC
I created a new version of digitaglinktree that just does what I described above. 
So an albumroot for removable media that is described by a uuid value in digikams database should now automatically be found by digitaglinktree. So not -r /path is needed for this..

Could you please check if it works for you.

Thanks
Rainer
Comment 22 krienke 2012-02-08 12:43:19 UTC
Created attachment 68620 [details]
digitaglinktree Version 1.8.1
Comment 23 cyril.raphanel 2012-02-08 18:26:12 UTC
Works great!
More elegant ... re-enabling "-r" was a quick fix than a real solution :-)

If you do not mind I will still look in your script as there is one feature that would make it perfect for my need. User could define what tag category could be used as drill down strategy.

For example, parameter would be People|Places|Animal|Date will result in:
People
->Peter
--> Places (continue by places)
---> Germany
----> Dates (continue by date)
-----> 2006
------> Animal (continue by animal)
-------> Image_of_Peter_In_Germany_With_A_cat_In_2006
------> All2
-----> All
----> Animal (continue by animal)
----> All
---> France
---> All
--> Dates (continue by date)
--> Animal (continue by date)
--> All
->All

Places
-> "start" again with same logic ....

Looks complex but coding should not be that difficult.
Comment 24 cyril.raphanel 2012-02-08 18:31:55 UTC
somehow, I have stopped in the middle with the -Y option ... My coding skills are a bit rusty!
Comment 25 krienke 2012-02-09 09:48:59 UTC
Good to hear that it works. I will still fix the help text and manual pages.

Your proposal about you example  People|Places|Animal|Date is something I do not really understand yet. The purpose of digitaglinktree is a mapping of tag hierarchies into the filesystem. So basically folder have subfolders if a corresponding tag has a subtag.  

It seems to me that your wish is something different?

Anyhow a discussion about this subject would fit better in a new  feature request because its not related to the bug that has been discussed in this thread.
Comment 26 cyril.raphanel 2012-02-09 19:17:30 UTC
the searching logic is actually hierarchical but i agree this is something that the problem I was originally facing.

I will open another case on the wish list with maybe a proposal if I do not lose my mind in recusivity loops.
Comment 27 cyril.raphanel 2012-02-10 02:51:25 UTC
one additional thing  for "Y" option we could get add "all" directory.
Comment 28 caulier.gilles 2012-02-11 13:13:05 UTC
There is any patch ready to apply on git/master now ?

Best

Gilles Caulier
Comment 29 krienke 2012-02-13 07:51:57 UTC
Created attachment 68749 [details]
Fixed digitrablinktree version
Comment 30 krienke 2012-02-13 07:52:45 UTC
Created attachment 68750 [details]
Manual Page for digitaglinktree
Comment 31 krienke 2012-02-13 07:53:44 UTC
Hi Gilles,

just added 2 new attachments for the script as well as the manual page. Would be nice if you could put it into git master.

Rainer
Comment 32 caulier.gilles 2013-11-26 09:12:09 UTC
Krienke,

files to put in git/master are the last versions ?

Gilles Caulier
Comment 33 krienke 2013-11-26 09:16:54 UTC
No changes since Comment 31 from 2012-02-13).
Comment 34 caulier.gilles 2013-11-26 13:24:56 UTC
Git commit d744bec46402c0d12cb56a239562cdd5c2249b58 by Gilles Caulier.
Committed on 26/11/2013 at 13:22.
Pushed by cgilles into branch 'master'.

update digikamlinktree script and man page to 1.8.2
FIXED-IN: 4.0.0

M  +101  -25   utilities/scripts/digitaglinktree/digitaglinktree
M  +5    -3    utilities/scripts/digitaglinktree/digitaglinktree.1

http://commits.kde.org/digikam/d744bec46402c0d12cb56a239562cdd5c2249b58
Comment 35 krienke 2013-11-27 14:45:57 UTC
Should be fixed in version 1.8.3. You may try the new script version attached to Bug 293832

Rainer