Hi Ian,

I hate to mention "updated versions" but this should be fixed for kdelibs 4.9.2. It was in the context of fixing OpenBSD support, but in any event a generic (slightly CPU-wasteful) lock primitive was introduced for systems without support for process-shared POSIX synchronization primitives. Unfortunately Mac OS X seems to be one of these systems (or at least they declare support for a certain primitive but it doesn't actually work, I'd have to dig up Till Adam's review request to get the full details).

The new code still prefers POSIX synchronization primitives if they claim to fully implement process sharing. I don't know if this applies to Mac OS X Lion. If you want to be doubly-sure then you can try forcing the spinlock to always be used by adding a "return LOCKTYPE_SPINLOCK;" to the top of the "findBestSharedLock()" function in kdecore/util/kshareddatacache_p.h, recompiling and reinstalling kdelibs, and then removing the *.kcache files to force them to be re-created.

I've heard of other reports (that I can't reproduce) of very heavy cache usage causing problems in some scenarios so I don't want to claim this would completely fix the issue if that's what you're seeing... but at least I'd know which issue it is! ;)

Thanks, Michael.  ATM I depend on Macports to supply kdelibs and the latest port is 4.8.3.  I have a KDE development area set up, so I will attempt to clone and build kdelibs 4.9.x there, hoping the already installed strigi and kde-runtime chains will suffice.  There is an outside chance they might --- or could be persuaded to ;-) --- because the Mac environment  does not usually use more than the bare essentials of the runtime processes (e.g. dbus and kdeinit4).  It will be a couple of days before I have time to try this.

I cloned kdelibs and checked out branch KDE/4.9 and attempted to build. There was a curious cmake error early on. I have cmake version 2.8.9.

-- No backend for KAuth was explicitly specified: probing system to find the best one available
-- Building Apple KAuth backend
CMake Error at kdecore/CMakeLists.txt:393 (set_source_files_properties):
  set_source_files_properties called with incorrect number of arguments.
-- Found Qt-Version 4.8.2 (using /opt/local/bin/qmake)

The context I have at kdecore/CMakeLists.txt:393 is:

    # KSharedDataCache uses exceptions as of 4.8.4, the rest of kdecore
    # doesn't. Try to enable exceptions only for that file, except for
    # enable-final builds where set_source_files_properties doesn't work.
    if(KDE4_ENABLE_FINAL)
        set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${KDE4_ENABLE_EXCEPTIONS}")
    else()
        set_source_files_properties(util/kshareddatacache.cpp
            PROPERTIES COMPILE_FLAGS ${KDE4_ENABLE_EXCEPTIONS})
    endif()

I think you must know something about that ... :-)

Apart from the above, I am getting the usual cruft from Nepomuk and Strigi (wanting more recent versions than I have), so I do not expect to be able to progress much further until the Macports developers do a port of kdelibs 4.9 to Apple OS X.

I can only imagine that KDE4_ENABLE_EXCEPTIONS is not actually defined in FindKDE4Internal.cmake (or whatever it's called that defines that on Linux). *sigh* :)

I wonder if MacPorts already fixes this for earlier KDE 4 versions (and if so whether there's a patch available)

(In reply to comment #4)
> I can only imagine that KDE4_ENABLE_EXCEPTIONS is not actually defined in
> FindKDE4Internal.cmake (or whatever it's called that defines that on Linux).
> 
You are right.  It is because all the places in FindKDE4Internal.cmake where it is
defined are within a conditional block, like "if (CMAKE_COMPILER_IS_GNUCXX)".

Macports uses the clang compilers now and KDE is not clang-ready yet.  I have
struck this problem before, when compiling KDE Games, but have a workaround
for that case - but that did not work when compiling kdelibs.

> I wonder if MacPorts already fixes this for earlier KDE 4 versions (and if
> so whether there's a patch available)

Correct. There *are* patches, but I cannot see any that apply to using clang.

For a while I have been trying to build kdelibs with a GCC compiler, but not
with much success.  I am going to have to shelve this for a few weeks until
I have time to dig deeper into what the problem is.

Is this still a problem these days? I've just committed the missing bits for proper clang support to kdelibs, FWIW.

Sorry to have been so slow on this one. I did install a KDE 4.10 library in September 2013, but then I was away for a while and busy on another project when I got back. I have upgraded again a few weeks ago and now have KDE 4.12.2 and Apple OS X 10.7.5 (still Lion, but updated).

It looks as though the cache is now working fine, but it is hard to tell for certain. My processor is quad-core Intel i7 and the qt4-mac library uses native Apple widgets and graphics, so full SVGZ rendering is very fast. Cache files are definitely being created. I see a blip (< 1 second) when I switch to a new KGoldrunner theme or a new size (requiring full rendering) and no blips after that, also different log messages, so that is why I think the cache is working OK.

Two messages still appear when starting KGoldrunner:
kgoldrunner(419)/KSharedDataCache: Creating shell of a lock! 
kgoldrunner(419)/KSharedDataCache: Unable to setup shared cache lock, although it worked when created. 
and this when switching to a new graphics theme (requiring rendering and caching):
kgoldrunner(419)/KSharedDataCache ensureFileAllocated: This system misses support for posix_fallocate() -- ensure this partition has room for at least 3164256 bytes. 

If these are normal on Apple OS X, please feel free to close this bug.

Raphael: Thanks again for your work on making KDE Clang-ready. I think it has helped with this bug too.

> kgoldrunner(419)/KSharedDataCache: Creating shell of a lock! 
> kgoldrunner(419)/KSharedDataCache: Unable to setup shared cache lock, although it worked when created. 

Well these are worrying, it means the lock-creating factory function was asked to create a lock a runtime that is not actually thought to be supported. If this happens for new *.kcache files (which I get the impression is the case) then that shouldn't be happening.

A fix is easy enough (instead of creating a "shell" lock just use that spinlock fallback we added for OpenBSD) but it is interesting that this is happening in the first place.

(In reply to comment #8)
> > kgoldrunner(419)/KSharedDataCache: Creating shell of a lock! 
> > kgoldrunner(419)/KSharedDataCache: Unable to setup shared cache lock, although it worked when created. 
> 
> Well these are worrying, it means the lock-creating factory function was
> asked to create a lock a runtime that is not actually thought to be
> supported. If this happens for new *.kcache files (which I get the
> impression is the case) then that shouldn't be happening.

Those two messages have been happening all along, for every app that uses KSharedDataCache on Apple OS X, ever since I first reported this problem.

Also, I just now checked Apple OS X's system Console log. There are no messages from OS X or KDE processes coming out there. I checked because, with some other problems we have been investigating recently, the OS X and KDE run-time process log messages have been a help.

> A fix is easy enough (instead of creating a "shell" lock just use that
> spinlock fallback we added for OpenBSD) but it is interesting that this is
> happening in the first place.

(In reply to comment #8)
> but it is interesting that this is happening in the first place.

I agree. Ian, can you put a backtrace in SharedMemory::performInitialSetup() (kshareddatacache.cpp) and see what the call to findBestSharedLock() returns?

If findBestSharedLock() worked fine we basically need to find out what later made shmLock.type become 0 again.

How do I "put a backtrace in" something? What does it mean? This is not a debugging technique I normally use.

Also, on Apple OS X, I am unable to build a debugging version of kdelibs (i.e. kdesrc-build does not run on Apple OS X) and Dr Konqui usually does not work (i.e. if I get a crash, I do not usually get a backtrace). This is part of a raft of problems that affect KDE on Apple OS X, which Marko Kaening and I have been trying to get some help with recently on the kde-devel mailing list.

(In reply to comment #11)
> How do I "put a backtrace in" something? What does it mean? This is not a
> debugging technique I normally use.

It means launching your application in a debugger such as gdb and using the "backtrace" command to tell it to stop when a certain function is reached when your program is executed.

> Also, on Apple OS X, I am unable to build a debugging version of kdelibs
> (i.e. kdesrc-build does not run on Apple OS X) and Dr Konqui usually does
> not work (i.e. if I get a crash, I do not usually get a backtrace). This is
> part of a raft of problems that affect KDE on Apple OS X, which Marko
> Kaening and I have been trying to get some help with recently on the
> kde-devel mailing list.

Hmm, this is more problematic. I'm way behind in those kde-devel threads. Does it mean the build just fails if you call "cmake -DCMAKE_BUILD_TYPE=DebugFull" to build kdelibs?

(In reply to comment #12)
> (In reply to comment #11)
> > How do I "put a backtrace in" something?
> 
> It means launching your application in a debugger such as gdb and using the
> "backtrace" command to tell it to stop when a certain function is reached
> when your program is executed.

Hmmm, I have not used gdb for quite a while - maybe 20 years ... :-)  My experience has been mainly in real-time, multi-user, large databases, games, etc., where gdb changes the timing ...

> > Also, on Apple OS X, I am unable to build a debugging version of kdelibs
> > (i.e. kdesrc-build does not run on Apple OS X) ...
> 
> Does it mean the build just fails if you call "cmake
> -DCMAKE_BUILD_TYPE=DebugFull" to build kdelibs?

No. I usually rely ATM on KDE 4.12.2 built by the MacPorts developers, who take care of all the dependencies and changing versions of dependencies in each KDE release.

When I try to build a recent kdelibs from the command line (e.g. master or KDE/4.13), which I attempted once before on this thread IIRC, I inevitably find that several dependencies have new required versions (usually in the Nepomuk chain, which is very long). I give up at that point. I found it hard enough to chase up those dependencies and build them by hand on Linux, until kdesrc-build came along.  Marko Kaening, who looks after KMyMoney4 on MacPorts, got into similar difficulties a couple of weeks ago.

If there is some way we could short-circuit or amputate the Nepomuk dependencies of kdelibs, we might have a chance. There is no actual need for Nepomuk on Apple OS X. It has a built-in desktop indexer called Spotlight.

Also, it would save a lot of work for the MacPorts developers and a lot of build time, if they could bypass the Nepomuk chain.

And I can see that Marko and I and anyone we can get to help us will need to be able to build kdelibs, etc. standalone if we are to get to the bottom of problems like why Dr Konqui will not run and (in Marko's case) why Help->Report Bug... fails to produce output or accept input on Apple OS X. I would want, at least, to plant extra log messages in the participating KDE processes, to get a better idea of what is going on.

Any ideas?  Perhaps I could try cloning and building KDE/4.12.2 kdelibs with debug and hoping it would pick up the dependencies already installed in MacPorts? Would that be any use to you?

It appears there is an option in kdelibs' CMakeLists.txt to omit Nepomuk, but there is also a comment that it does not work because several "modules" depend on it.

Just on the off-chance, I tried building kdelibs from master ... and it worked! I guess Nepomuk has not changed its dependency version requirements since 4.12.2, so the build was able to use already-installed versions.

I got into a chicken-and-egg problem with gdb. It would not let me set a breakpoint until I had loaded symbol tables, but said it would remember my breakpoint till then. Then when I did "run", it went right past the point of interest in SharedMemory before it had finished loading symbols.

I tried inserting a qDebug() message in SharedMemory (kshareddatacache.cpp), but got no output from it. Also, my test app (KGoldrunner) was severely broken (not responding to keystrokes), so I think I need to build KDE Games libs before going on.

I'll try again another day, but had to stop and fix some bugs for the KDE 4.13 release of the game I have been busy on (Palapeli).

HELP needed!

For the past two weeks or so I have been trying to build a KDE 4.13 software base from source code on Mac OS X, using kdesrc-build.  My objectives are twofold.  One objective is to build with Debug and source so that I can investigate what is happening in KSharedDataCache, as you requested.  The other is to provide a development base from which I and others can investigate and narrow down or fix portability problems that occur with KDE on Mac OS X.

Two such problems are that unsightly graphics glitches occur in several KDE applications and some of the applications run at a snail's pace.  Both are due to Qt4-Mac not using raster graphics as standard and QGraphicsView not using raster graphics even if Qt4-Mac is configured with raster.  I have a possible solution that will enforce raster at the KApplication or KMainWindow level, but cannot test it properly because I cannot build KDE 4.13.

There are two major roadblocks - see https://bugs.kde.org/show_bug.cgi?id=334702 - but I have had no response to that yet.  I guess everybody is too busy with KF5.

I wonder if you could have a quick look at that report, Michael or Rafael, and see if there is an easy workaround, or better still a solution, to what appears to be a bug in the KDE CMake files or macros on Apple OS X.

Thanks very much for your help with the KDE Macros bug, Raphael and Michael.  Sorry to have been so long in getting back to you.  I now have a test setup for KDE 4.13.x working on Apple OS X and have been using it to investigate and fix some serious problems, such as inability to report bugs if an application crashes and unavailability of raster graphics as a default on Apple OS X.

The good (bad?) news is that the troublesome KSharedDataCache messages:

2/07/14 2:05:24.948 PM [0x0-0x4d04d].kbounce: kbounce(627)/KSharedDataCache: Creating shell of a lock! 
2/07/14 2:05:24.948 PM [0x0-0x4d04d].kbounce: kbounce(627)/KSharedDataCache: Unable to setup shared cache lock, although it worked when created. 
2/07/14 2:05:25.865 PM [0x0-0x4d04d].kbounce: kbounce(627)/libkdegames KGameRendererPrivate::_k_setTheme: Setting theme: "The Beach"

still occur in KDE 4.12.5 (the latest available in MacPorts), but do NOT occur in my build from the KDE/4.13 branch, using kdesrc-build:

2/07/14 2:10:45.517 PM [0x0-0x51051].kbounce: kbounce(644) mac_set_dbus_address: set session bus address to "unix:path=/tmp/launch-AgvYYY/unix_domain_listener"
2/07/14 2:10:45.634 PM [0x0-0x51051].kbounce: kbounce(644) KBounceGameWidget::settingsChanged: Settings changed
2/07/14 2:10:45.634 PM [0x0-0x51051].kbounce: kbounce(644)/libkdegames KGameRendererPrivate::_k_setTheme: Setting theme: "The Beach"

So where to now?  Is it a difference in building method?  Seems unlikely.

Probably not a difference in build method. Perhaps a difference in compiler flags? I just checked the source and there were no differences for KSharedDataCache for between 4.12 and 4.13 so it's not a coincidental bugfix.

Perhaps also a difference in debug output, maybe KSharedDataCache is still making the warning but it's not being seen. Or maybe the error was being shown for a buggy cache and it got regenerated finally for some reason (if exceptions are enabled KSharedDataCache will use that to regenerate caches that it identifies as corrupt).

I went into /var/tmp/kdecache-ianw on the command line and deleted all caches there (that is for MacPorts on KDE 4.12.5).  The only error message I got after that, when I ran a few games, was: 4/07/14 2:03:33.650 PM [0x0-0x33033].kbounce: kbounce(843)/KSharedDataCache ensureFileAllocated: This system misses support for posix_fallocate() -- ensure this partition has room for at least 3164256 bytes. 
which I have been getting all along.  No more lock problems and the caches get created OK.

MacPorts has moved along a lot in the last few months.  More use of Clang, new releases of MacPorts itself and we have now patched kdelibs (kapplication.cpp) to ALWAYS use Qt's raster graphics whenever possible.  It was not the default on Qt-Mac 4.8.  That fixes a lot of KDE bugs.

Anyway, Michael, please feel free to close this bug now.

Sounds good, glad to see we could get things fixed up finally for Mac OS X.