Nepomuk libraries not found even if nepomuk-core-devel-4.12.95 installed.

to comment #1 : this is a system install problem, not digiKam...

Gilles Caulier

I disabled Nepomuk Support in this commit:

http://quickgit.kde.org/?p=digikam.git&a=commitdiff&h=d049efab4869a6d521e7670ad0aeb749b2b46e11&hp=3486e3ed6ac8bfc88d05d8da0ad81ceda2d59580

I think Ballo, will use a different approach and current Nepomuk implementation, can't be patched to work with Ballo. It's basically dead code.

Hello!

I just tryed to install Digikam 4.0 on openSUSE 13.1 with KDE 4.13.1. Unfortunnatelly it's comming with nepomuk dependency. I do not want Nepomuk and try to avoid it at all cost.

So I hope nepomuk dependency will die in further release because it is an absolute showstopper to me!

It is a little bit sad because Nepomuk support is reintegrated at exactly the moment Nepomuk is dead, bud let us be honestly, the dead of Nepomuk is not sad it is a reason to feel happy!

There is a digiKam Cmake option to turn on/off Nepomuk support :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/CMakeLists.txt#L12

By default it's disabled !

Gilles Caulier

So the problem is, that the openSUSE packagers have switched nepomuk on. Hmmmm... okay I will compile it from source.

Thank You!

Hi!

Are there any plans to integrate baloo into digikam?
It would be really cool if Tags and Ratings are shared system wide (digikam, dolphin, searching,  etc.). 

Thanks!

Help is welcome...
We invested considerably in Nepomuk over the years. Motivation to work on Baloo is low from my side.

Marcel,

Veaceslav is in copy of this file, and he maintained Nepomuk code last summer through GSoC 2013.

As he know well current implementation, perhaps he can spare some free time to port code to Baloo interface. As i can see in Baloo wiki page, it's not too difficult :

http://community.kde.org/Baloo#Porting_your_Application_to_use_Baloo

Gilles

Hmm... still a lot to take care, lots of users do not like integration, must make it compile with and without Baloo support, probably on my long term TODO list...

Thanks for your reply! Is the baloo API so different to the Nepomuk API, that it is so hard to replace it?

I think a good system integration is always a killer feature for an application.

Hi Konrad

Good question. If you look at 

http://community.kde.org/Baloo#Porting_your_Application_to_use_Baloo

... it said :

"Baloo is not a drop in replacement for Nepomuk. Applications relying on Nepomuk will have to port to Baloo. The majority of Nepomuk applications just rely on its tags, ratings and comments features. Baloo offers a simple asynchronous API for modifying that file metadata. This metadata is now stored with the extended attributes of the file instead of storing it in a separate database. Baloo also offers a search API which is similar to that of Nepomuk in some ways."

My response : it difficult to estimate time to port current digiKam code based on Nepomuk

At least a small guide/tutorial need to be written by Baloo team to make transition more easy.

Gilles Caulier

My only guess is that, everything related to Nepomuk should be dropped. And ballo sync to be implemented alongside with metadata sync(another option to sync to baloo)

By dropping Nepomuk, you want mean to port current code to Baloo. If i understand well Baloo wiki page, no need to drop all interface in digiKam, just to change Nepomuk calls to Baloo calls

No, it is not changing Nepomuk calls with Baloo, as far as I understand, it only offer an async api to set and retrieve tags, ratings etc... No more services, no more 2 way syncronization...

So, when we write or read image metadata, we can sync and Ballo also... this how I see port to Baloo.

(In reply to comment #12)
> 
> At least a small guide/tutorial need to be written by Baloo team to make
> transition more easy.
> 
> Gilles Caulier

I have put Vishesh Handa on the CC list, I think he is the maintainer of baloo and I hope he reads this and can help ;-)

Hi Vishesh Handa,

Can you give us a simple estimation of work to do about to port current digiKAm Nepomuk interface to use Baloo instead.

All code relevant is here :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/show/utilities/nepomuk

Also, please take a look of analysis from Veaceslav in comments #13 and #15 to see if all is fine...

Thanks in advance

Gilles Caulier

(In reply to comment #17)
> Hi Vishesh Handa,
> 
> Can you give us a simple estimation of work to do about to port current
> digiKAm Nepomuk interface to use Baloo instead.
> 
> All code relevant is here :
> 
> https://projects.kde.org/projects/extragear/graphics/digikam/repository/
> revisions/master/show/utilities/nepomuk
> 
> Also, please take a look of analysis from Veaceslav in comments #13 and #15
> to see if all is fine...
> 
> Thanks in advance
> 
> Gilles Caulier

Hi,

Vishesh has posted about tagging and baloo: http://vhanda.in/blog/2014/07/tagging-your-files/ , does this help?

Sorry about the delay. I finally looked at the code.

With Baloo, we have tried to keep Baloo just as a file indexer for searching. Everything else which Nepomuk used to do have either been thrown away or put in other places. In the case of tags, ratings and comments, they are stored in the extended attributes.

So, if you guys want, you can skip depending on Baloo completely. Lets talk about the code -

dknepomukwrap.cpp - This can be converted into reading and writing the extended attributes.
dknepomukquery.cpp - This is something you can use Baloo for if you want. Or you can scan the files yourself. If you want to use Baloo, I can help with the query. Baloo would just be giving you a list of files which have the xattrs set.
dknepomukwatcher.cpp - Baloo can inform you when a file changes. Or you can use KDirWatch and listen for changes yourself. It's up to you.

On a separate note, one of the things I hated about Nepomuk was the need for these "synchornization agents" to be always running. We had those in PIM and in digikam. Perhaps, now would be a good time to revise that? I'm not aware of the digikam code base, but it might be simpler to just read and write tags from the file system directly.

We have our own database so the demon is needed to sync to the digikam database when digikam is not running, and a file is changed.
Hm, is the modification date touched when the xattr is edited?

(In reply to comment #20)
> We have our own database so the demon is needed to sync to the digikam
> database when digikam is not running, and a file is changed.
> Hm, is the modification date touched when the xattr is edited?

Hi, I edited xattr (user.xdg.comment and user.baloo.rating) with the setfattr command and with a little java app. Result: The modification date is not touched.

(In reply to comment #21)
> (In reply to comment #20)
> > We have our own database so the demon is needed to sync to the digikam
> > database when digikam is not running, and a file is changed.
> > Hm, is the modification date touched when the xattr is edited?
> 
> Hi, I edited xattr (user.xdg.comment and user.baloo.rating) with the
> setfattr command and with a little java app. Result: The modification date
> is not touched.

The mtime does not change, but the ctime does -

vlap:~ $ touch fook
vlap:~ $ stat fook
  File: ‘fook’
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 12h/18d Inode: 13395683    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/ vishesh)   Gid: ( 1000/ vishesh)
Access: 2014-07-11 13:36:24.339823962 +0200
Modify: 2014-07-11 13:36:24.339823962 +0200
Change: 2014-07-11 13:36:24.339823962 +0200
 Birth: -
vlap:~ $
vlap:~ $ setfattr -n "user.baloo.rating" -v "4" fook
vlap:~ $ stat fook
  File: ‘fook’
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 12h/18d Inode: 13395683    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/ vishesh)   Gid: ( 1000/ vishesh)
Access: 2014-07-11 13:36:24.339823962 +0200
Modify: 2014-07-11 13:36:24.339823962 +0200
Change: 2014-07-11 13:36:41.619823391 +0200
 Birth: -

I think you can use QFileInfo::created() to access it.

I decided to simplify a little the entire nepomuk/baloo thing.

Since digiKam database and metadata can be out of sync, I suggest to extend read/write metadata options to also read and write using Baloo.

This will reduce the need of having a Nepomuk service and would be much easier to maintain.

But currently I have 2 problems:
1. digiKam image scanner do not detect changes when I add tag with Dolphin
2. I'm still looking for a tutorial for get/set with Ballo, CMake rules and API (I don't know, google does not return relevant searches for Baloo API or tutorial)

Veaceslav,

>digiKam image scanner do not detect changes when I add tag with Dolphin

Image scanner must be triggered by media change notifier at least. Do you see this kind of information printed on the console when you tag file with Dolphin.

If yes, image scanner must be not able yet to see Baloo event. But are you sure that Dolphin write tags in image metadata when Baloo info are changed ?

>I'm still looking for a tutorial for get/set with Ballo, CMake rules and API (I >don't know, google does not return relevant searches for Baloo API or tutorial)

If Dolphin is already ported to Baloo, you can find probably all relevant info in Dolphin code :

https://projects.kde.org/projects/kde/applications/kde-baseapps/repository/revisions/5707e1e92c9c6ad320dbce5f9eeecb637f3e3312

Gilles Caulier

(In reply to Veaceslav Munteanu from comment #23)

> 2. I'm still looking for a tutorial for get/set with Ballo, CMake rules and
> API (I don't know, google does not return relevant searches for Baloo API or
> tutorial)

Hi,

you don't need baloo for setting or getting the attributes from a file. The important part is, that the attributes are stored in a namespace as defined here: http://www.freedesktop.org/wiki/CommonExtendedAttributes/

baloo uses the following namespaces for attributes:
 - user.xdg.comment
 - user.xdg.tags => tags are separated with , 
 - user.baloo.ratig => rating range is from 0 to 10 (9 is displayed as 4.5 Stars in dolphin)

In "the near future" baloo will detect changes on this attributes on its own, see Bug 337602 for more information.

I think the real cool part is, that every application which uses the freedesktop defined namespaces will be able to interpret those attributes and is not depened to another software or library.

I don't know how to set the attributes with C/C++ but I have found these:
 - http://stackoverflow.com/questions/9979864/system-file-read-write-o-create-custom-metadata-or-attributes-extended
 - http://www.lesbonscomptes.com/pxattr/

I have to agree with Konrad Renner. As a previously said, it might be best NOT to use Baloo. Diadvantages -

* Using Baloo introduces a large number of dependencies
* The API still does not guarantee source compatibility in KF5

The whole point of moving to xattr was also so that other applications can use tags/ratings/comments without having to use Baloo. The primary AIM of Baloo is just to be a good file indexer + searching frameworking.

wait... what?

I already ported both sync digiKam->Baloo and Baloo -> digiKam for tags/comment/rating

This means I must one more time drop everything to trash?

Do Baloo have any signals which tells if an image is updated?(I need this to tell digiKam to rescan metadata...)

Created attachment 88192 [details]
attachment-15435-0.html

On Sat, Aug 9, 2014 at 8:28 PM, Veaceslav Munteanu <
veaceslav.munteanu90@gmail.com> wrote:

>   *Comment # 27 <https://bugs.kde.org/show_bug.cgi?id=332665#c27> on bug
> 332665 <https://bugs.kde.org/show_bug.cgi?id=332665> from Veaceslav
> Munteanu <veaceslav.munteanu90@gmail.com> *
>
> wait... what?
>
> I already ported both sync digiKam->Baloo and Baloo -> digiKam for
> tags/comment/rating
>
> This means I must one more time drop everything to trash?
>
>
No. Definitely not. I just meant that there may be some *minor* changes to
the API. Minor.


> Do Baloo have any signals which tells if an image is updated?(I need this to
> tell digiKam to rescan metadata...)
>
> When someone changes the tag/rating/comment? We have a file monitor.

File monitor is not quite flexible. In old nepomuk implementation we could register a listener for tags, comments and rating and images.

Adding few thousand of urls in filemonitor at each digiKam start-up will not scale at all...

Marcel, Gilles, I sucessfully integrated read/write with metadatahub and imagescanner, but I don't know how to keep baloo <-> digiKam in sync when is off... even when It is on, might be a problem if I don't find a way to get an universal signal...

Should I let the user to sync manually through read/write metadata?

I already implemented read/write and all necessary switches in digiKam.

But extended atributes are very volatile. If I disable write support, each metadata update simply destroy all existing extended attributes...

(In reply to Veaceslav Munteanu from comment #30)
> I already implemented read/write and all necessary switches in digiKam.
> 
> But extended atributes are very volatile. If I disable write support, each
> metadata update simply destroy all existing extended attributes...

Could you please elaborate? The metadata update destroys all existing extended attributes?

yes, when I enable write to baloo, tags, comments and rating is updated. If I disable it, and trigger write metadata, the image is empty: no rating, no comment, no tags..

Vishesh,

Any progress here to help Veaceslav to complete Baloo digiKam interface ?

Thanks in advance

Gilles Caulier

@Veaceslav: Could you give me a specific test case for me to reproduce it and fix it. Perhaps it would be simpler to just look at the code? The code is in baloo/src/file/lib/filemodifyjobtest.cpp. The code is *quite* simple. I'll be happy to answer any questions.

Is there anything else you require? We do not plan to add more facilities in the file monitor for informing clients about what exactly changed. It's too expensive.

Is this bug figed in 4.3.0?

Here message when compiling 4.3.0:

--  Baloo libraries found.................... YES (optional)

Partially.

See Veaceslav comments for details. It still some work to do i think...

Gilles Caulier

So, can it be already enabled at comile time or better to disable baloo in 4.3.0?

@Vishesh, in older Nepomuk Api, I was able to register in ResourceMonitor which type of resource I want to monitor. 

I used to register: comments, tags, ratings and images.

what I really need from Baloo is signalFileMetadatachanged(Kurl file), so I can also trigger it in digiKam.

At least, if filemodifyjob could register a folder path, that would be much better.

If it's to expensive to fire a signal when Baloo register a change.... well there is nothing more I can do to integrate better with digiKam except basic read/write on request.

@nucleo, I don't really understand how packaging with optional dependencies work, but baloo support, even if it's only basic, must be good enough for shipping now. This apply for KDE users. 

Non-KDE users won't be happy with Baloo support enabled :)

(In reply to Veaceslav Munteanu from comment #39)
> @Vishesh, in older Nepomuk Api, I was able to register in ResourceMonitor
> which type of resource I want to monitor. 
> 
> I used to register: comments, tags, ratings and images.
> 
> what I really need from Baloo is signalFileMetadatachanged(Kurl file), so I
> can also trigger it in digiKam.
> 
> At least, if filemodifyjob could register a folder path, that would be much
> better.
> 
> If it's to expensive to fire a signal when Baloo register a change.... well
> there is nothing more I can do to integrate better with digiKam except basic
> read/write on request.

We do fire a change when the file metadata is modified. There is the FileMonitor, but I think I understand what you mean - there is no way to globally watch everything. You could just listen to the dbus code (look at the implementation of the FileMonitor), but that's hackish. Either way, Baloo in Qt4 is now frozen, so the API isn't going to change.

will the kf5 port of digiKam make use of baloo and/or xattr?

The code is unchanged in frameworks implementation. It's just ported to compile under KF5.

Veaceslav, do you have some plan to improve Baloo interface ?

Gilles

(In reply to Gilles Caulier from comment #43)
> The code is unchanged in frameworks implementation. It's just ported to
> compile under KF5.
> 
> Veaceslav, do you have some plan to improve Baloo interface ?
> Gilles

If he doesn't have plans, then please contact me. I can spend the required 30 minutes to port it.

Gilles, what exactly must be done?

Veaceslav,

I don't know exactly. Ask to Vishesh (:=)))...

Gilles

You need to get it to compile with the Qt5 version. Have a look at KFileMetaData UserMetaData. It's a trivial change.

Ok, I will look into it.

Vishesh, how do I commit changes to file?

I am able to retrieve data with KFileMetadata Usermetadata, but when I set them, changes are not reflected to files(Dolphin panel do not show anything...)

(In reply to Veaceslav Munteanu from comment #49)
> Vishesh, how do I commit changes to file?
> 
> I am able to retrieve data with KFileMetadata Usermetadata, but when I set
> them, changes are not reflected to files(Dolphin panel do not show
> anything...)

Check if the xattr have been set via `getfattr -d`.

Thank you, problem was somewhere else. 

Everything is fixed, digiKam successfully ported to baloo.

(In reply to Veaceslav Munteanu from comment #51)
> Thank you, problem was somewhere else. 
> 
> Everything is fixed, digiKam successfully ported to baloo.

Could you possibly not call it the Baloo backend? You're mostly only using KFileMetaData (a soon to be framework). I'm trying to make Baloo to just be a file indexer.

(In reply to Veaceslav Munteanu from comment #51)
> Thank you, problem was somewhere else. 
> 
> Everything is fixed, digiKam successfully ported to baloo.

I just had a look at the code, you're appending "BalooTags/" to each tag. Do you think the user should be aware of Baloo? In Plasma, we have tried our best to remove technology names from the end user.

line 158: Q_FOREACH -> Maybe converting the list to a set in unnecasary?

The old Baloo code is still in comments. Perhaps we can remove that? It's always there in git, if we want it.

The Baloo dependency can be dropped.

If you want to get the list of tags, you can use Baloo::TagListJob, however, please remember that this will NOT give you the list of all tags. Only the tags from the files that are indexed.

>The Baloo dependency can be dropped.

I don't understand why exactly ?

Gilles Caulier

(In reply to Gilles Caulier from comment #54)
> >The Baloo dependency can be dropped.
> 
> I don't understand why exactly ?
> 

Baloo in KF5 is just a file indexer. It goes through your file and creates an index optimized for searching. It also indexes tags, ratings and comments. These tags, ratings and comments are obtained from the extended attributes of the file via KFileMetaData::UserMetadata. This is a os independent wrapper on top of xattr.

Digikam is currently only using KFileMetaData. It is not using Baloo. Therefore, the dependency can be dropped.

You only need to use Baloo is you want to perform some kind of queries on the index.

Veaceslav,

What do you think about to remove Baloo dependency and to let's only KFileMetadata ?

Gilles Caulier

Gilles,  yes we do not use anything except KFileMetadata.

A clean-up will follow next days...

Indeed BalooTags need to remain, due to digiKam storing tags in hierarchical structures, while KFileMetadata is better with simple tags...

(In reply to Veaceslav Munteanu from comment #57)
> Gilles,  yes we do not use anything except KFileMetadata.
> 
> A clean-up will follow next days...
> 
> Indeed BalooTags need to remain, due to digiKam storing tags in hierarchical
> structures, while KFileMetadata is better with simple tags...

I'm not sure I understand. Could you please elaborate on why "BalooTags" needs to remain.

Git commit c07b1e032e1afb65a5a12c5e73a5d6d2f7cef716 by Veaceslav Munteanu.
Committed on 15/03/2015 at 12:38.
Pushed by munteanu into branch 'frameworks'.

M  +13   -16   CMakeLists.txt
M  +1    -1    app/CMakeLists.txt
M  +4    -4    app/main/digikamapp.cpp
M  +2    -2    app/utils/componentsinfo.h
M  +1    -1    app/utils/digikam_config.h.cmake.in
M  +3    -3    libs/database/item/imagescanner.cpp
M  +2    -2    libs/fileactionmanager/metadatahub.cpp
M  +6    -6    libs/settings/applicationsettings.cpp
M  +1    -1    utilities/CMakeLists.txt
M  +1    -5    utilities/baloo/CMakeLists.txt
M  +1    -122  utilities/baloo/baloowrap.cpp
M  +0    -18   utilities/baloo/baloowrap.h
M  +4    -4    utilities/setup/setupmetadata.cpp

http://commits.kde.org/digikam/c07b1e032e1afb65a5a12c5e73a5d6d2f7cef716

Vishesh, because we store only "TagName" in baloo and not the full path "/parent/TagName". If we reread without appending a parent for it, it will polute root tag.

So we assign a special "BalooTags" and who will need tags from now KFileMetadata will find them there... Of course I will think about replacing BalooTags with something else, or maybe to let user choose the prefix through an option in the menu....

Git commit 08162a7e1a19bb1e278bc29b8b4e46f8a252392c by Marko Käning.
Committed on 15/03/2015 at 13:38.
Pushed by kaning into branch 'master'.

Remove baloo dependency and use only KFileMetadata
FIXED-IN: c07b1e032e1afb65a5a12c5e73a5d6d2f7cef716
CCMAIL: veaceslav.munteanu90@gmail.com

M  +0    -1    dependency-data-kf5-qt5

http://commits.kde.org/kde-build-metadata/08162a7e1a19bb1e278bc29b8b4e46f8a252392c

Sorry, I know this bug is now closed, but from a user perspective, I was hoping the developers could clarify the interoperability with Dolphin (which is the tool that I use, presumably with baloo integrated). Below are my assumptions based on this bugzilla thread; please let me know whether or not I correctly understood.

Digikam interoperability with Dolphin: 
* Tags and ratings that are originally generated using Digikam, assuming the user uses the option to save metadata stored in file, will now make use of linux filesystem extended attributes. Consquently, the tags and ratings originally generated in Digikam will be reflected in Dolphin's information panel, and detailed view. Right?
* Will tags, ratings, and comments be allowed to be generated or modified from Dolphin, and then reflected in Digikam?
* Will the list of available tags in Dolphin be synchronized with the list of tags in Digikam?
* If a tag, rating, or comment is applied to a file using Dolphin, when Digikam is started/loaded, will it check the file metadata/ extended attributes against the database and prompt the user to confirm the changes? Corrollary: I can imagine a use-case, or configuration option, where attributes generated outside of Digikam are automatically dropped/ ignored by Digikam.

Thank you very much!

Here are some answers to your questions, Jeff:
1. digiKam has options in settings menu to enable/disable read/write
2. If you change data from digikam and write is enabled, tags, ratings and comments are automatically syncronized
3. If you have read enabled, data from baloo is read when any read metadata operation is triggered
4. There is no way for digiKam to know if data was changed from outside. At least at this moment.
5. If digikam has a tag structure like, aaa/bb/cc/dd, digikam store tags with KFileMetadata as "dd" without it's full path. We can store the full path, but tag itself will be unreadable in Dolphin. Also, because of read after write, we could end up with a lot of duplicate tags in digiKam tag tree. Thats why we store them under a special branch, which is now called "BalooTags" so it won't pollute the whole tag tree.

There is a lot of shortcomings on current implementation, I know, but if someone can come up with better design, please write your suggestions here :)

(In reply to Veaceslav Munteanu from comment #63)
> Here are some answers to your questions, Jeff:
> 1. digiKam has options in settings menu to enable/disable read/write
> 2. If you change data from digikam and write is enabled, tags, ratings and
> comments are automatically syncronized
> 3. If you have read enabled, data from baloo is read when any read metadata
> operation is triggered

"data from xattr is read ..."

> 4. There is no way for digiKam to know if data was changed from outside. At
> least at this moment.

file system monitoring? inotify provides signals on when a file's attributes have been changed. Or rather do a full scan on startup or when the user requests it and check the ctime of the files?

> 5. If digikam has a tag structure like, aaa/bb/cc/dd, digikam store tags
> with KFileMetadata as "dd" without it's full path. We can store the full
> path, but tag itself will be unreadable in Dolphin. Also, because of read
> after write, we could end up with a lot of duplicate tags in digiKam tag
> tree. Thats why we store them under a special branch, which is now called
> "BalooTags" so it won't pollute the whole tag tree.
> 

I've said this before, I really really disapprove of using the name "BalooTags". If you must use something, use "Digikam". At least the user will then know which application these tags come from. Adding the name Baloo gives them nothing of value and in-fact would confuse them even know since I keep telling everyone that Baloo no longer handles tags.

Maybe I should just submit a patch.

Yes, you can commit a patch. The prefix is in the baloowrap.cpp. Set it for the more appropiate value.

Also, when digiKam is running, it might see that through it's file scanner. Scanner must be extended a little to see that changes. But this is only when digiKam is running, complete syncronization is not possible without an additional service.