Bug 161403

please try playground/base/nepomuk-kde/filewatch and tell me about your experiences.

kde4@kiwi:~/filewatch-build$ cmakekde /home/gaston/cuisine/trunk/filewatch/
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Check size of void*
-- Check size of void* - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
CMake Error: Error in cmake code at
/home/gaston/cuisine/trunk/filewatch/kinotify/CMakeLists.txt:14:
Unknown CMake command "kde4_add_library".
-- Configuring done


I tried for an hour copying what the main KDE modules do in order to use "kde4_add_library", without success.

well, to compile it you need the kde4 devel environment. And also it does not work with KDE 4.0.x anymore.

I do have a current kde devel environment, and this is against current kde trunk.

Could it be that I shouldn't run cmake directly on the filewatch/ directory, and should instead run it on some parent directory such as nepomuk-kde.... I'll try.

> Could it be that I shouldn't run cmake directly on the filewatch/
> directory, and should instead run it on some parent directory such as
> nepomuk-kde.... I'll try.


Yes, sorry, I forgot to mention that. :)

Now it builds (sidenote: in nepomuk-kde/pimoshell/resourcepropertydelegate.cpp, I had to comment out the line kDebug() << parent << option << index; because operator<< wasn't defined for QStyleOption).

How do I use it? Should I manually enable something, or just re-login into KDE?

I can't really test because of a separate problem -- so I am very grateful if you see a solution: a few days ago, nepomukservices started eating 100% CPU constantly (and more and more RAM). This makes dolphin completely unusable (response times above 10s) so I had to disable nepomuk, and now when I re-enable it I still have the same problem. So I can't test in Dolphin right now. Maybe you would at least know how to reset nepomuk to its original state before that bug appeared?

> How do I use it? Should I manually enable something, or just re-login into
> KDE?


It will automatically be used after next login, yes.

> I can't really test because of a separate problem -- so I am very grateful
> if you see a solution: a few days ago, nepomukservices started eating 100%
> CPU constantly (and more and more RAM). This makes dolphin completely
> unusable (response times above 10s) so I had to disable nepomuk, and now
> when I re-enable it I still have the same problem. So I can't test in
> Dolphin right now. Maybe you would at least know how to reset nepomuk to
> its original state before that bug appeared?


Is Strigi enabled? Does "ps aux|grep strigi" show the "strigidaemon"?

Strigi wasn't enabled when that bug appeared. Today when I re-enabled Nepomuk I also enabled Strigi, and got the same result. So it seems strigi-independent.

I just re-enabled nepomuk to check: CPU usage went again to 100%, and strigi wasn't running (according to ps aux).

I have only a few files nepomuk-tagged in my home dir.

FYI, here is part of the cmake output in kdesupport:

-- Soprano Components that will be built:                                       
   * Redland storage backend                                                    
   * Raptor RDF parser                                                          
     (including TriG parser)                                                    
   * Raptor RDF serializer                                                      
   * The CLucene-based full-text search index library                           

-- Soprano Components that will NOT be built:
   * Sesame2 storage backend (java-based)    

Could you please debug into the process taking all the cpu to see what it is 
doing?

You mean run it in gdb ? Then what is the appropriate command-line, just "gdb nepomukservices" ? nepomukservices is something that KDE launched for me and I am just not sure how to properly launch it myself. Also, is it OK to run it while nepomuk is disabled in the KCM?

I thought about debugging into the running 100% cpu process.
You can do that by issuing:

gdb nepomukservicestub <pid>

Today I did svn up and rebuilt kde trunk, but in "debugfull" mode (formerly I used "release" mode).

I also did svn up in nepomuk-kde from playground, rebuilt and installed, and re-logged into kde. In systemsettings, I enabled nepomuk but not strigi.

The second problem (nepomuk eating 100% CPU) is gone: nepomuk now only takes negligible CPU time. I don't know if it's because you fixed it, or if it's because this issue only exists in "release" mode (which would be strange!) I am willing to test but wouldn't like to rebuild all of kde again, so I'd need to know what parts to rebuild, if you're interested.

Unfortunately, the problem that I describe in the original text of this bug report, still exists: metadata is lost when moving a file (i tried both moving and copying in dolphin).

Here are the nepomuk-related processes running:

gaston@kiwi:~$ ps aux | grep nepomuk | grep -v grep
gaston   10221  0.0  1.1  67336 11512 ?        Sl   17:41   0:00 /home/kde4/kde/bin/nepomukserver
gaston   10241  0.8  1.4  41760 14636 ?        S    17:41   0:07 /home/kde4/kde/bin/nepomukservicestub nepomukstorage
gaston   10246  0.0  1.0  32612 10696 ?        S    17:41   0:00 /home/kde4/kde/bin/nepomukservicestub nepomukalignment
gaston   10248  0.4  1.3  36792 14408 ?        S    17:41   0:03 /home/kde4/kde/bin/nepomukservicestub nepomukfilewatch
gaston   10249  0.0  0.9  27540  9468 ?        S    17:41   0:00 /home/kde4/kde/bin/nepomukservicestub nepomukontologyloader

I can confirm that Nepomuk-related metadata do not follow the file when it is moved, renamed or deleted.

If you
1. Create a file
2. Associate some comment/tag/rating to it
3. Delete it
4. Create a new file with the same name
you will see the metadata associated to the new file, while it should have been discarded along with the deleted one.

I'm currently experiencing this behavior on Kde4Daily, built against SVN revision 821684.

it should work fine whenever KIO is used to move or delete the files. Moving or deleting on the command line or via gnome tools is not yet handled. I am still open for good suggestions here. Maybe even something which guesses move locations and presents some gui where the user can then choose.
Or even better: a kernel patch that allows us to track all file operations. ;)

Frankly, I believe that this is a very important issue: I don't see myself tagging files if the metadata can be lost at anytime if the files are moved by a non-KDE application.

What is the status of inotify support? Isn't that the kind of kernel support that was needed? (totally uninformed about inotify)

I can confirm that copying or moving tagged files with Dolphin 1.3 under KDE 4.3.0 does _not_ preserve meta information.
I actually have no idea at all how to copy/move files in a way that takes meta informations with them (I tried it with konqueror as well.)
In my case, I tagged a few pictures with gwenview, and then wanted to move the folder they are in. (I tried to copy a single file too, just as a test).

To reproduce the problem:
1. tag a picture in gwenview
2. move the file with dolphin
3. open with gwenview again -> tags gone
( 4. move file back -> tags are there again)

Damnit, correction:
Copy the file, do not move it.
Moving a single file with Dolphin does indeed preserve meta information.
It's the first action I found to actually do that (Moving the whole folder did not). So I actually found a solution for my current problem: Just mark all files in the folder, and move them that way.
That does not mean however, that all those other cases (moving folders, copying files, ...) should not work as well.

please provide the output of "qdbus|grep nepomuk"

$ qdbus|grep nepomuk
 org.kde.nepomuk.services.nepomukstorage
 org.kde.nepomuk.services.nepomukfilewatch
 org.kde.nepomuk.services.nepomukontologyloader
 org.kde.nepomuk.services.nepomukqueryservice

So you want a copy have the same metadata as the original?
I thought about that but came to no good conclusion. So I decided to not do it.
As for moving folders: that should work and I will have a look.

To me, preserving the metadata across copy operations is the "obviously" wanted behavior; I wonder what can be arguments against this?

Just an example, if nothing else, the copy operation could be made to archive important personal files; then the user also wants to keep the metadata.

There is also the consistency-with-other-metadata-systems argument: e.g. tags embedded in music files are preserved when the file is copied, and this has set a standard of what the user expects.

Finally, the user may reasonably assume that moving a file has the same effect as copying it and then deleting the original. (Of course it's very different in the way it works, and moving is much faster, but most users don't know that).

>So you want a copy have the same metadata as the original?

Yes, I do. Ideally, I would want the metadata to be embedded within the file, so that no matter what tool I use to move, copy, backup, ... the file, the metadata is preserved. As long as that is not the case, I must use only kde applications to work with the files.
But at least these applications I expect to treat the metadata as part of the file. So yes, I do want the copy to have the same metadata as the original.

What argues against that?

I agree that metadata needs to be copied with a file copy, everything else does not make much sense to me.

I agree that the meta data should be preserved when copying the file. To me that is expected behaviour.

I just tested this with KDE SC 4.4.82 (4.5 beta 2). The good news is that tags are now persistent: moving and renaming files does not destroy the metadata!

However, while all four commenters above agreed that it is obviously desirable to copy metadata as well when copying files, this is not the case.

Sebastian, thank you for fixing this. But could you please consider to change the behavior and copy metadata as well? [Use case/example: I can't see why I would want the copy of a picture tagged "Frank" to be disassociated from the name since the person is still in the image.]

Should we open a new bug report for this so this one can be closed?

Copying metadata is not trivial as there are multiple ways to copy a file in KDE. It may even be necessary to patch certain applications in order to get the desired behavior.

I saw an idea in another bug report of writing an UUID to the file's xattr, then nepomuk could use the UUID to find the file's metadata. Wouldn't that work?

*** Bug 152777 has been marked as a duplicate of this bug. ***

I too, feel that metadata should follow a file.  If one wants different metadata for two copies of the same file, they should change the name of the 2nd file.

I just encountered this problem recently.  I've:

# nepomukserver --version
Qt: 4.7.3
KDE Development Platform: 4.6.3 (4.6.3)
Nepomuk-server: 0.2


# qdbus|grep nepomuk
 org.kde.digikamnepomukservice-11053
 org.kde.nepomuk.services.digikamnepomukservice
 org.kde.nepomuk.services.nepomukqueryservice
 org.kde.nepomukqueryservice-11057
 org.kde.nepomuk.services.nepomukfilewatch
 org.kde.nepomukfilewatch-11061
 org.kde.nepomuk.services.nepomukbackupsync
 org.kde.nepomukbackupsync-11054
 org.kde.nepomuk.services.nepomukremovablestorageservice
 org.kde.nepomukremovablestorageservice-11059
 org.kde.nepomuk.services.nepomukontologyloader
 org.kde.nepomuk.services.nepomukstorage
 org.kde.nepomukstorage-10737
 org.freedesktop.Akonadi.Agent.akonadi_nepomuk_contact_feeder
 org.kde.akonadi_nepomuk_contact_feeder-10759

I asked about it on the openSUSE kde emaillist first:
http://lists.opensuse.org/opensuse-kde/2011-05/msg00015.html and was pointed to this bug report.

In short, I moved many files with dolphin from folder A to folder B and the tags (metadata) does not move with the file.  After reading this bug report I tried to move only one file, using Dolphin, from folder A to folder B with the same result.

I really hope that this functionality can be made to work.

For the people that think that this bug / feature is important to solve, it might be a good idea to give your votes to this bug report.

Before anyone suggests "fs.inotify.max_user_watches = 524288" to increase the number of folders KDirWatch can monitor for moves, each inode watched is un-swappably pinned into memory - 1 or 4 KB each, i don't remember exactly.

Can someone please remind me (also see comment 27) why xattrs couldn't be used to store metadata in a way that ensures that it follows files?

If xattrs doesn't work, then, doesn't that mean that it would be useful to work with kernel developers to implement something what would work?

To be clear: as time passes, I am less and less interested in a solution that works only for KIO, and more and more thinking that it's vital to find a solution that integrates with the system VFS. Even people like me who are full-time KDE users may use cp/mv commands, and with a KIO-only solution, that would lose metadata. So if xattrs is not good enough then surely kernel developers should be interested in making a better replacement for it? I'm thinking about Linus' claims about how the desktop is where things are really interesting. Time to put his desktop commitment to the test ;-)

@Benoît: it is indeed desired to have a general solution, but this looks like outside the scope of this bug report.  Let's use this bug report to obtain a KIO only solution and have another (new) bug report / feature request to track the general solution that you propose / requires in comment #23.  If you agree (and hopefully you  do) will you open a new bug report requesting the general solution?

*** Bug 304100 has been marked as a duplicate of this bug. ***

Update on the bug -

Tags are preserved when moving files (This has been the case for quite some time now). They are not preserved when copying because it is simply not possible on a technical level. Maybe we could integrate into KIO for file copies, but we have no way of doing it at the kernel level, nor will that ever be possible.

There is already bug 221547 for saving the metadata in the extended attributes.

Marking this bug as CONFIRMED. We still need someone to implement the added integration with KIO for copying.

I think using an UUID in the extended attributes to reference files in the Nepomuk database would make it easier to copy the Nepomuk metadata along with the file. I describe it a bit in bug 221547 comment 2.

At the moment the file is copied, there would be two files with the same UUID and thus referencing the same metadata. That wouldn't be a problem since then Nepomuk could be implement a copy on write scheme: when editing metadata information that is referenced by two files, then a new UUID is generated for the file being edited and the metadata is copied and associated with that UUID.

*** Bug 320164 has been marked as a duplicate of this bug. ***

Thank you for taking the time to file a bug report.

The Nepomuk project is no longer included in the KDE Software Compilation. With Plasma 5, we have replaced most of the underlying technology with Baloo and other components. Hopefully this will have addressed your concern.

We encourage you to try out Plasma 5 (+Baloo) and let us know if your problem persists.