Created attachment 144998 [details]
kstars run log

It doesn't actually need to involve a remote ekos.  If I select the default simulators ekos profile and launch it, kstars crashes.

Hoping to reproduce this with more info I tried to compile kstars on macos with rlancaste@'s script, but it fails in sonnet for me:

[141/189] Building CXX object src/plugins/nsspellchecker/CMakeFiles/sonnet_nsspellchecker.dir/nsspellcheckerdict.mm.o
FAILED: src/plugins/nsspellchecker/CMakeFiles/sonnet_nsspellchecker.dir/nsspellcheckerdict.mm.o
/Library/Developer/CommandLineTools/usr/bin/clang++ -DKF_DEPRECATED_WARNINGS_SINCE=0x060000 -DQT_CORE_LIB -DQT_DEPRECATED_WARNINGS_SINCE=0x060000 -DQT_DISABLE_DEPRECATED_BEFORE=0x050f02 -DQT_MAC_USE_COCOA -DQT_NO_CAST_FROM_ASCII -DQT_NO_CAST_FROM_BYTEARRAY -DQT_NO_CAST_TO_ASCII -DQT_NO_DEBUG -DQT_NO_FOREACH -DQT_NO_KEYWORDS -DQT_NO_NARROWING_CONVERSIONS_IN_CONNECT -DQT_NO_URL_CAST_FROM_STRING -DQT_STRICT_ITERATORS -DQT_USE_QSTRINGBUILDER -DSONNETCORE_DEPRECATED_WARNINGS_SINCE=0x0 -DSONNETCORE_DISABLE_DEPRECATED_BEFORE_AND_AT=0x0 -D_DARWIN_C_SOURCE -D_LARGEFILE64_SOURCE -Dsonnet_nsspellchecker_EXPORTS -Isrc/plugins/nsspellchecker -I/Users/castor/AstroRoot/craft-root/build/kde/frameworks/tier1/sonnet/work/sonnet-5.89.0/src/plugins/nsspellchecker -Isrc/plugins/nsspellchecker/sonnet_nsspellchecker_autogen/include -Isrc/core -I/Users/castor/AstroRoot/craft-root/build/kde/frameworks/tier1/sonnet/work/sonnet-5.89.0/src/core -iframework /Users/castor/AstroRoot/craft-root/lib -isystem /Users/castor/AstroRoot/craft-root/lib/QtCore.framework/Headers -isystem /Users/castor/AstroRoot/craft-root/./mkspecs/macx-clang -fdiagnostics-color=always -fno-operator-names -fno-exceptions -Wno-gnu-zero-variadic-macro-arguments -Wall -Wextra -Wcast-align -Wchar-subscripts -Wformat-security -Wno-long-long -Wpointer-arith -Wundef -Wnon-virtual-dtor -Woverloaded-virtual -Werror=return-type -Werror=init-self -Wvla -Wdate-time -pedantic -Wzero-as-null-pointer-constant -fdiagnostics-color=always -O2 -g -DNDEBUG -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk -mmacosx-version-min=10.13 -fPIC -fvisibility=hidden -fvisibility-inlines-hidden -fPIC -std=c++17 -MD -MT src/plugins/nsspellchecker/CMakeFiles/sonnet_nsspellchecker.dir/nsspellcheckerdict.mm.o -MF src/plugins/nsspellchecker/CMakeFiles/sonnet_nsspellchecker.dir/nsspellcheckerdict.mm.o.d -o src/plugins/nsspellchecker/CMakeFiles/sonnet_nsspellchecker.dir/nsspellcheckerdict.mm.o -c /Users/castor/AstroRoot/craft-root/build/kde/frameworks/tier1/sonnet/work/sonnet-5.89.0/src/plugins/nsspellchecker/nsspellcheckerdict.mm
In file included from /Users/castor/AstroRoot/craft-root/build/kde/frameworks/tier1/sonnet/work/sonnet-5.89.0/src/plugins/nsspellchecker/nsspellcheckerdict.mm:11:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/System/Library/Frameworks/AppKit.framework/Headers/AppKit.h:10:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/System/Library/Frameworks/Foundation.framework/Headers/Foundation.h:12:
/Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSBundle.h:91:143: error: function does not return NSString
- (NSAttributedString *)localizedAttributedStringForKey:(NSString *)key value:(nullable NSString *)value table:(nullable NSString *)tableName NS_FORMAT_ARGUMENT(1) NS_REFINED_FOR_SWIFT API_AVAILABLE(macos(12.0), ios(15.0), watchos(8.0), tvos(15.0));
                                                         ~~~~~~~~~~~~~~                                                                       ^                  ~
/Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSObjCRuntime.h:103:48: note: expanded from macro 'NS_FORMAT_ARGUMENT'
        #define NS_FORMAT_ARGUMENT(A) __attribute__ ((format_arg(A)))
                                                      ^          ~
1 error generated.
ninja: build stopped: subcommand failed.
Action: compile for kde/frameworks/tier1/sonnet:5.89.0 FAILED

(In reply to Castor Fu from comment #3)
> Hoping to reproduce this with more info I tried to compile kstars on macos
> with rlancaste@'s script, but it fails in sonnet for me:
> 

This was a problem with macos 12.1's sdk.  Updating to the current command-line-tools seems to have fixed the build break.  Hopefully it will finish compiling in a bit.

I am actually currently in the process making a bunch of changes to the script to hopefully make our build as close to using the true craft recipes as possible.  Then maybe I can get whatever changes are actually needed to get accepted into craft so we can get binary factory builds.  So you might get a failure just because I am trying some changes to the build right now.  But now there are just a few recipes that are different than craft's recipes so if the build fails on any of those recipes I didn't change, we can get craft people to address it I think.

(In reply to Castor Fu from comment #2)
> It doesn't actually need to involve a remote ekos.  If I select the default
> simulators ekos profile and launch it, kstars crashes.

Here's a better stack trace after running on a copy I built:
VM Regions Near 0:
--> 
    __TEXT                      108e45000-109c29000    [ 13.9M] r-x/r-x SM=COW  /Users/*/kstars.app/Contents/MacOS/kstars

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   org.qt-project.QtCore         	0x000000010cff8e0a QObject::setProperty(char const*, QVariant const&) + 26 (qobject.cpp:4031)
1                                 	0x00000001091b7176 Ekos::Mount::Mount() + 6598 (mount.cpp:209)
2                                 	0x0000000108fadd08 Ekos::Manager::initMount() + 72 (manager.cpp:2435)
3                                 	0x0000000108fa567e Ekos::Manager::setTelescope(ISD::GDInterface*) + 414 (manager.cpp:1443)
4   org.qt-project.QtCore         	0x000000010cff89c4 QtPrivate::QSlotObjectBase::call(QObject*, void**) + 21 (qobjectdefs_impl.h:398) [inlined]
5   org.qt-project.QtCore         	0x000000010cff89c4 void doActivate<false>(QObject*, int, void**) + 1124 (qobject.cpp:3886)
6                                 	0x0000000108ea0bcf INDIListener::newTelescope(ISD::GDInterface*) + 63 (moc_indilistener.cpp:413)
7                                 	0x0000000108f23845 INDIListener::registerProperty(INDI::Property) + 789 (indilistener.cpp:260)
8                                 	0x0000000108f25913 QtPrivate::FunctorCall<QtPrivate::IndexesList<0>, QtPrivate::List<INDI::Property>, void, void (INDIListener::*)(INDI::Property)>::call(void (INDIListener::*)(INDI::Property), INDIListener*, void**) + 55 (qobjectdefs_impl.h:152) [inlined]
9                                 	0x0000000108f25913 void QtPrivate::FunctionPointer<void (INDIListener::*)(INDI::Property)>::call<QtPrivate::List<INDI::Property>, void>(void (INDIListener::*)(INDI::Property), INDIListener*, void**) + 55 (qobjectdefs_impl.h:185) [inlined]
10                                	0x0000000108f25913 QtPrivate::QSlotObject<void (INDIListener::*)(INDI::Property), QtPrivate::List<INDI::Property>, void>::impl(int, QtPrivate::QSlotObjectBase*, QObject*, void**, bool*) + 179 (qobjectdefs_impl.h:418)
11  org.qt-project.QtCore         	0x000000010cff0d39 QObject::event(QEvent*) + 745 (qobject.cpp:1314)
12  org.qt-project.QtWidgets      	0x000000010c1c6586 QApplicationPrivate::notify_helper(QObject*, QEvent*) + 262 (qapplication.cpp:3632)
13  org.qt-project.QtWidgets      	0x000000010c1c7940 QApplication::notify(QObject*, QEvent*) + 480
14  org.qt-project.QtCore         	0x000000010cfc8117 QCoreApplication::notifyInternal2(QObject*, QEvent*) + 167 (qcoreapplication.cpp:1063)
15  org.qt-project.QtCore         	0x000000010cfc9228 QCoreApplication::sendEvent(QObject*, QEvent*) + 21 (qcoreapplication.cpp:1458) [inlined]
16  org.qt-project.QtCore         	0x000000010cfc9228 QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) + 792 (qcoreapplication.cpp:1817)
17  libqcocoa.dylib               	0x000000010e6296f8 QCocoaEventDispatcherPrivate::processPostedEvents() + 312 (qcocoaeventdispatcher.mm:904)
18  libqcocoa.dylib               	0x000000010e629e18 QCocoaEventDispatcherPrivate::postedEventsSourceCallback(void*) + 40 (qcocoaeventdispatcher.mm:927)
19  com.apple.CoreFoundation      	0x00007fff20a6137c __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
20  com.apple.CoreFoundation      	0x00007fff20a612e4 __CFRunLoopDoSource0 + 180
21  com.apple.CoreFoundation      	0x00007fff20a610c6 __CFRunLoopDoSources0 + 340
22  com.apple.CoreFoundation      	0x00007fff20a5fa8c __CFRunLoopRun + 893

Hmm, I've tried adding some debug statements, and I can't seem to get it to crash when running it within the VM, but after turning on logging in the mount code, I see the following on a crash (the XX lines are ones added by me).  It looks like maybe there's a race installing some code that mountbox.qml is using?    On successful runs I don't see the warnings about modules not being installed.


[2022-01-01T21:05:28.425 PST INFO ][           org.kde.kstars.ekos] - "Telescope Simulator is online."
[2022-01-01T21:05:28.455 PST DEBG ][     org.kde.kstars.ekos.mount] - XXA  QObject(0x0)
[2022-01-01T21:05:28.619 PST WARN ][                       default] - qrc:/qml/mount/mountbox.qml:4:1: module "QtQuick.Controls.Styles" is not installed
     import QtQuick.Controls.Styles 1.4
     ^
[2022-01-01T21:05:28.620 PST WARN ][                       default] - qrc:/qml/mount/mountbox.qml:3:1: module "QtQuick.Layouts" is not installed
     import QtQuick.Layouts 1.1
     ^
[2022-01-01T21:05:28.620 PST WARN ][                       default] - qrc:/qml/mount/mountbox.qml:2:1: module "QtQuick.Controls" is not installed
     import QtQuick.Controls 1.4
     ^
[2022-01-01T21:05:28.620 PST WARN ][                       default] - qrc:/qml/mount/mountbox.qml:4:1: module "QtQuick.Controls.Styles" is not installed
     import QtQuick.Controls.Styles 1.4
     ^
[2022-01-01T21:05:28.621 PST WARN ][                       default] - qrc:/qml/mount/mountbox.qml:3:1: module "QtQuick.Layouts" is not installed
     import QtQuick.Layouts 1.1
     ^
[2022-01-01T21:05:28.621 PST WARN ][                       default] - qrc:/qml/mount/mountbox.qml:2:1: module "QtQuick.Controls" is not installed
     import QtQuick.Controls 1.4
     ^
[2022-01-01T21:05:28.621 PST WARN ][                       default] - qrc:/qml/mount/mountbox.qml:4:1: module "QtQuick.Controls.Styles" is not installed
     import QtQuick.Controls.Styles 1.4
     ^
[2022-01-01T21:05:28.621 PST WARN ][                       default] - qrc:/qml/mount/mountbox.qml:3:1: module "QtQuick.Layouts" is not installed
     import QtQuick.Layouts 1.1
     ^
[2022-01-01T21:05:28.621 PST WARN ][                       default] - qrc:/qml/mount/mountbox.qml:2:1: module "QtQuick.Controls" is not installed
     import QtQuick.Controls 1.4
     ^
[2022-01-01T21:05:28.624 PST DEBG ][     org.kde.kstars.ekos.mount] - XXY   QQuickItem(0)  m_BaseObj QQuickItem(0)
[2022-01-01T21:05:28.624 PST DEBG ][     org.kde.kstars.ekos.mount] - XXX m_leftRightCheck null

Sorry, the whole mount error was a red herring because of not copying enough of the environment over.  

This looks like a better trace:
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x400002d80)
  * frame #0: 0x00007fff209ac552 libsystem_platform.dylib`_platform_strlen + 18
    frame #1: 0x0000000100019d13 kstars`QDebug::operator<<(char const*) [inlined] QString::fromUtf8(str="", size=-1) at qstring.h:706:63 [opt]
    frame #2: 0x0000000100019d06 kstars`QDebug::operator<<(char const*)(this=0x00007ffeefbfd458, t="") at qdebug.h:159:62 [opt]
    frame #3: 0x00000001000de2cc kstars`INDIListener::registerProperty(this=0x000000013893e030, prop=Property @ 0x00007ffeefbfd4a0) at indilistener.cpp:241:33 [opt]
    frame #4: 0x00000001000e0613 kstars`QtPrivate::QSlotObject<void (INDIListener::*)(INDI::Property), QtPrivate::List<INDI::Property>, void>::impl(int, QtPrivate::QSlotObjectBase*, QObject*, void**, bool*) [inlined] QtPrivate::FunctorCall<QtPrivate::IndexesList<0>, QtPrivate::List<INDI::Property>, void, void (INDIListener::*)(INDI::Property)>::call(f=<unavailable>, o=<unavailable>, arg=<unavailable>)(INDI::Property), INDIListener*, void**) at qobjectdefs_impl.h:152:13 [opt]
    frame #5: 0x00000001000e05dc kstars`QtPrivate::QSlotObject<void (INDIListener::*)(INDI::Property), QtPrivate::List<INDI::Property>, void>::impl(int, QtPrivate::QSlotObjectBase*, QObject*, void**, bool*) [inlined] void QtPrivate::FunctionPointer<void (INDIListener::*)(INDI::Property)>::call<QtPrivate::List<INDI::Property>, void>(f=<unavailable>, o=<unavailable>, arg=<unavailable>)(INDI::Property), INDIListener*, void**) at qobjectdefs_impl.h:185:13 [opt]
    frame #6: 0x00000001000e05dc kstars`QtPrivate::QSlotObject<void (INDIListener::*)(INDI::Property), QtPrivate::List<INDI::Property>, void>::impl(which=<unavailable>, this_=<unavailable>, r=<unavailable>, a=<unavailable>, ret=<unavailable>) at qobjectdefs_impl.h:418:17 [opt]
    frame #7: 0x0000000104180d39 QtCore`QObject::event(this=0x000000013893e030, e=0x000000012c2996f0) at qobject.cpp:1314:18 [opt]
    frame #8: 0x0000000103356586 QtWidgets`QApplicationPrivate::notify_helper(this=<unavailable>, rece

It seems like there may be a race in ClientManager::newProperty(INDI::Property *pprop)
where prop's initialization hasn't flushed through before the thread in INDIListener::registerProperty manages to read it?

(In reply to Castor Fu from comment #9)
> It seems like there may be a race in
> ClientManager::newProperty(INDI::Property *pprop)
> where prop's initialization hasn't flushed through before the thread in
> INDIListener::registerProperty manages to read it?

More likely something is corrupting memory?  If we try printing out the property we can see something which looks like memory corruption.

(lldb) p *prop.d_ptr.__ptr_
(std::shared_ptr<INDI::PropertyPrivate>::element_type) $36 = {
  property = 0xb00000001102892f
            ^^^^^^ something is garbled here... ≈
  baseDevice = 0x000000000000001c
  type = INDI_NUMBER
  registered = false
  dynamic = false
  self = {
    d_ptr = nullptr {
      __ptr_ = nullptr
    }
  }
}

Sorry I haven't been able to help you much with this so far.  I have been super busy trying to make all the changes in my script and fix any issues that creates that I haven't been able to look at this yet.  Also, I have a bunch of grading that I need to do.  If it helps, one way that can really help to investigate the cause is to open it up in QT Creator or in Xcode and run the program with debugging turned on, then you get more information about what is going on when the crash occurs.  I usually use my other repo for that, KStars-INDI-Mac-Dev, rather than kstars-on-osx-craft, because I like to keep the one repo for distribution and the other for development.  The KStars-INDI-Mac-Dev can be a little tricky however, plus I will probably have to redo some of it after I get done with what I am doing to kstars-on-osx-craft.  But, I do have a QT Creator project all set up right now with the latest KStars and INDI code using the KStars-INDI-Mac-Dev if I can do a debugging run or two to help you.  Let me know what you want me to try.

Also, It does look like you might be on to something.  I don't have time right now to try to solve it, but I can run some tests.  Also If it helps, I remember I was tracking down an intermittent bug a couple of months ago, I think it was in INDI Guide Simulator that might be similar to what you are checking out now, if I remember right, for some reason when INDI was starting up and properties were coming in, INDIListener received a property that either basically null or had inaccessible or bad data in it, which caused KStars to crash because it tried to access the bad data.  I might have some of the screenshots of the debugging runs I did if it helps.

Since it looks more like memory corruption I'm going to look into using kstars with some sanitizing checker like AddressSanitizer.  Do you know if that's normally done/supported  That seems like the best way to deal with this without familiarity with the code, though it will mean delving into cmake, etc.  Maybe it'd be easier to try tracking it down under linux.  I haven't done any real mac development so it's all relatively slow for me. (And my laptop is going into the shop tomorrow). So I'll keep plodding along.

I don't know anything about Address Sanitizer, but I have run analyses using tools in Xcode before.  The challenging part about this bug will be that it is not clear if it is a bug in KStars, or a bug in INDI, or if it's some issue between them.  As I said the test I had run before for a similar problem was to run it in QTCreator in the debugger and put in some breakpoints to try to see the value of some variables prior to the crash to try to see where and why the crash happens.

The issue would be easier to track down on Linux if it exists on that platform as well because MacOS debugging cannot debug child processes.  But also it is easier because the libraries are all natively installed on Linux, whereas on MacOS you have the added challenge of the needed libraries being installed in the App Bundle.

I may have an update on this. I mentioned it in the forum where I was posting the latest KStars release and the dmg I was building for MacOS.  One person said it wasn't MacOS only and that he had seen it on linux.  Another person Jean-Claude posted that he saw the issue until he replaced his kstarsrc file and now it never crashes on startup.   So I tried his experiment and I found that he might be right.  I did a comparison between the settings in the new and old kstarsrc and I found one thing that stood out to me,  I had CCD logging turned on in the one that crashes but not in the one that doesn't.  I also just tried an experiment.  I started kstars and turned off that setting and the indiserver started repeatedly with no issues, but I turned on that setting and it crashed like 50% of the time.    Can you check if this is also true for you?

https://indilib.org/forum/ekos/11081-mac-kstars-3-5-7-beta-new-build-please-test.html?start=48#79662

If this is true, it would definitely explain why I was having the issue a few months ago when nobody else seemed to be having it, and also why we have gotten so few crash reports, when it was happening to me fairly often.

(In reply to Rob from comment #14)
> I may have an update on this. I mentioned it in the forum where I was
> posting the latest KStars release and the dmg I was building for MacOS.  One
> person said it wasn't MacOS only and that he had seen it on linux.  Another
> person Jean-Claude posted that he saw the issue until he replaced his
> kstarsrc file and now it never crashes on startup.   So I tried his
> experiment and I found that he might be right.  I did a comparison between
> the settings in the new and old kstarsrc and I found one thing that stood
> out to me,  I had CCD logging turned on in the one that crashes but not in
> the one that doesn't.  I also just tried an experiment.  I started kstars
> and turned off that setting and the indiserver started repeatedly with no
> issues, but I turned on that setting and it crashed like 50% of the time.   
> Can you check if this is also true for you?
> 
> https://indilib.org/forum/ekos/11081-mac-kstars-3-5-7-beta-new-build-please-
> test.html?start=48#79662
> 
> If this is true, it would definitely explain why I was having the issue a
> few months ago when nobody else seemed to be having it, and also why we have
> gotten so few crash reports, when it was happening to me fairly often.

Wow, thanks!
I didn't have CCD logging turned on, but I did try disabling logging and then things worked.  And as I've tried re-enabling logging I haven't reproduced the failure.  Very strange.  I'll go ahead and load the beta image that you've mentioned.

If I don't see anymore repeats I guess I could resolve this as not reproducible.  We'll see...

Ok it may be something else with logging then.  I think it certainly bears further investigation, but this is a very important clue.  When I investigated this before, it was failing on registering properties as they came in.  I almost certainly had logging turned on then since I was trying to get all the info I could.  But what registering the properties coming in from the indiserver has to do with logging, I am not sure right now, but maybe it is connected somehow.