Bug 289330 - Album file watch failing (QKqueueFileSystemWatcherEngine:addPaths: open: Too many open files in system) [patch]
Summary: Album file watch failing (QKqueueFileSystemWatcherEngine:addPaths: open: Too ...
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Database-Scan (show other bugs)
Version: 2.4.0
Platform: MacPorts macOS
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-12-19 01:39 UTC by brad
Modified: 2017-07-25 17:08 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In: 2.6.0


Attachments
small patch to limit 'watched' files on OS X. (971 bytes, patch)
2011-12-24 20:45 UTC, brad
Details
simple patch that should apply smoothly to current source. (876 bytes, patch)
2012-01-11 15:19 UTC, brad
Details

Note You need to log in before you can comment on or make changes to this bug.
Description brad 2011-12-19 01:39:33 UTC
Version:           2.4.0 (using KDE 4.7.3) 
OS:                OS X

It appears that since the commit : 
https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/dcf2ba9d7e92d51b049bdd354f82ddbf75e51a31

digikam does not load images in OS X. The console error that i receive (thousands of times) is the following:

  QKqueueFileSystemWatcherEngine::addPaths: open: Too many open files in system
  QFileSystemWatcher: failed to add paths: /Volumes/Home/Directory/Pictures/Savedpictures/Pic0193.jpg
  QKqueueFileSystemWatcherEngine::addPaths: open: Too many open files in system
  QFileSystemWatcher: failed to add paths: /Volumes/HomeDirectory/Pictures/Savedpictures/Pic0194.jpg


This might be related to the following 'fixed' QT bug:
https://bugreports.qt.nokia.com//browse/QTBUG-2478


Reproducible: Always

Steps to Reproduce:
compile and run digikam from the latest git, then try to load an image.

Actual Results:  
image will not load. GUI will report failed to load image.

Expected Results:  
To see image loaded.
Comment 1 brad 2011-12-24 20:45:54 UTC
Created attachment 67089 [details]
small patch to limit 'watched' files on OS X.

I propose the attached patch to fix this problem on OS X.  OS X limits the number files opened per process to be 255, so if the user has more than 255 files, they cannot open them, which doesn't make digikam all that useful. This patch makes digikam useful again but only watching directories, this limits the user to 255 directories of images. This is not a perfect solution, but for the timebeing this will enable digikam to be usable on OS X. I do not think this bug should be closed since this is just a work-around and not a genuine fix.
Comment 2 caulier.gilles 2011-12-24 21:55:21 UTC
Marcel,

Fine for you to apply this patch. It sound like not too intrusive and without any action under non OSX system...

Gilles Caulier
Comment 3 Marcel Wiesweg 2011-12-26 21:32:42 UTC
Apparently the underlying Qt bug is not fixed? Are you running Qt 4.7 and Mac OS X >= 10.5?
Even with the number limited to 255, digikam would not be able to open more than one other file. I'd rather disable directory watching if the FSEvents API is not available
Comment 4 brad 2011-12-27 01:00:02 UTC
On Dec 26, 2011, at 4:32 PM, Marcel Wiesweg wrote:

> https://bugs.kde.org/show_bug.cgi?id=289330
> 
> 
> 
> 
> 
> --- Comment #3 from Marcel Wiesweg <marcel wiesweg gmx de>  2011-12-26 21:32:42 ---
> Apparently the underlying Qt bug is not fixed? Are you running Qt 4.7 and Mac
> OS X >= 10.5?

Yes i'm running current QT 4.7.4, and OS X 10.6.8. I have a simple test program i can post if you have access to an OS X machine and want to look at.

the qt bug is not fixed. it really seems to be a bug (short fall) of OS X not QT. eitherway, a process can't open more than 255 files without some system manipulation which i wouldn't expect end users to do.

> Even with the number limited to 255, digikam would not be able to open more
> than one other file. I'd rather disable directory watching if the FSEvents API
> is not available
> 
If you'd rather disable file watching, then go for it. I'm not familiar with the file watching code and i wanted to make the smallest change possible to the code to get it running on OS X.
Comment 5 Marcel Wiesweg 2011-12-28 10:26:08 UTC
According to API docs, Qt would be using FSEvents on Mac OS X >= 10.5:
http://en.wikipedia.org/wiki/FSEvents

http://developer.qt.nokia.com/doc/qt-4.8/qfilesystemwatcher.html
"Mac OS X 10.5 and up use a different backend and do not suffer from this issue."

Now looking into the source code of Qt 4.7.3, apparently the creation of the FSEvents backend is commented out (QFileSystemWatcherPrivate::createNativeEngine()). This is still true on Qt master on gitorious. So apparently the API docs are wrong here - the backend does not seem to be used. I would recommend to report this bug to Qt, or try to reopen the bug report mentioned above.
Comment 6 brad 2011-12-29 22:50:55 UTC
(currently i'm compiling qt with fsevents uncommented to see if it will work)

after searching around the net, it appears that QT has gone back and forth from using fvevents to kqueue. it's not clearly to me why, but i think its because of bugs in fsevents on apple's side. 

In the meantime it seems that disabling file waching (or album watching) on OS X is right way to go for digikam until this QT/OS X bug is resolved.
Comment 7 brad 2012-01-11 15:19:09 UTC
Created attachment 67697 [details]
simple patch that should apply smoothly to current source.

Trying to get better at git, creating patch with git format-patch. To apply use git am 0001-disable-filewatching-on-OS-X-but-dir-watching-still-.patch in the digikam-sc/core directory.
Comment 8 brad 2012-01-29 23:40:13 UTC
Here is an update (blog post) on qfilesystemwatcher for QT5. In the comments the author can get over 10k watched files on OS X. I'm try to contact the author to get more info. haven't heard anything yet. 

http://blog.rburchell.com/2012/01/qfilesystemwatcher-internals-in-qt-5.html
Comment 9 g2florette 2012-02-21 23:11:47 UTC
Hi,

Compiling digikam 2.6 from git on MacOSX Lion. I've the "too many open files" problem with my > 10K images collection : digikam isn't able to start. Without any collection, it runs fine...
The patch from Brad, applied like showed in comment #7, fixes the problem : I can use digikam with my whole collection without any problem.
Hope this small report helps...

G.
Comment 10 Jan Gosmann 2012-03-24 16:41:26 UTC
With Digikam 2.5.0 and Mac OS X Lion I had also the "too many open files" problem leading to a crash of digikam at startup. With the patch from comment #7 I can start Digikam.
Comment 11 Jan Gosmann 2012-03-24 16:42:44 UTC
*** Bug 292522 has been marked as a duplicate of this bug. ***
Comment 12 Marcel Wiesweg 2012-04-14 14:44:04 UTC
I will apply the patch by brad. Then I consider this bug fixed; it is not, but the real fix is beyond our scope, in OS X and Qt.
Any better solution is welcome any time.
Comment 13 Marcel Wiesweg 2012-04-14 16:17:52 UTC
Git commit 5e6b607b38076eb226a7fa92c82f0fe476bc9681 by Marcel Wiesweg.
Committed on 14/04/2012 at 17:12.
Pushed by mwiesweg into branch 'master'.

Apply patch by Brad <bkn@ithryn.net> to disable file watch on OS X
until this is properly supported by this platform.

M  +2    -1    NEWS
M  +7    -0    digikam/album/albumwatch.cpp

http://commits.kde.org/digikam/5e6b607b38076eb226a7fa92c82f0fe476bc9681