I have an idea what going on. Can you get a backtrace with debug symbols and would you be able to compile a KWin with a patch?

Adding my idea in case I forget again: the mmap of the DrmBuffer fails for the cursor buffer. We need to catch this situation properly.

(In reply to Martin Gräßlin from comment #1)
> I have an idea what going on. Can you get a backtrace with debug symbols and
> would you be able to compile a KWin with a patch?
Sure!

Backtrace with debug symbols:

#0  0x00007fffee75471a in QImage::fill(QColor const&) () from /usr/lib64/libQt5Gui.so.5                                         
#1  0x00007fffee7549ec in QImage::fill(Qt::GlobalColor) () from /usr/lib64/libQt5Gui.so.5                                       
#2  0x00007fffdcccd021 in KWin::DrmBackend::initCursor (this=this@entry=0x66dd60) at /usr/src/debug/kwin-5.6.90git~20160623T210703~03f0dc5/plugins/platforms/drm/drm_backend.cpp:499                                                                            
#3  0x00007fffdccce52f in KWin::DrmBackend::openDrm (this=0x66dd60) at /usr/src/debug/kwin-5.6.90git~20160623T210703~03f0dc5/plugins/platforms/drm/drm_backend.cpp:267                                                                                          
#4  0x00007ffff5e7b44e in QMetaObject::activate(QObject*, int, int, void**) () from /usr/lib64/libQt5Core.so.5                  
#5  0x00007ffff7b29652 in KWin::LogindIntegration::hasSessionControlChanged (this=<optimized out>, _t1=<optimized out>, _t1@entry=true) at /usr/src/debug/kwin-5.6.90git~20160623T210703~03f0dc5/build/moc_logind.cpp:187                                       
#6  0x00007ffff7abc894 in KWin::LogindIntegration::<lambda(QDBusPendingCallWatcher*)>::operator() (self=<optimized out>, __closure=0x6dd050) at /usr/src/debug/kwin-5.6.90git~20160623T210703~03f0dc5/logind.cpp:281                                            
#7  QtPrivate::FunctorCall<QtPrivate::IndexesList<0>, QtPrivate::List<QDBusPendingCallWatcher*>, void, KWin::LogindIntegration::takeControl()::<lambda(QDBusPendingCallWatcher*)> >::call (arg=<optimized out>, f=...) at /usr/include/qt5/QtCore/qobjectdefs_impl.h:501                                                                                                                        
#8  QtPrivate::Functor<KWin::LogindIntegration::takeControl()::<lambda(QDBusPendingCallWatcher*)>, 1>::call<QtPrivate::List<QDBusPendingCallWatcher*>, void> (arg=<optimized out>, f=...) at /usr/include/qt5/QtCore/qobjectdefs_impl.h:558                     
#9  QtPrivate::QFunctorSlotObject<KWin::LogindIntegration::takeControl()::<lambda(QDBusPendingCallWatcher*)>, 1, QtPrivate::List<QDBusPendingCallWatcher*>,void>::impl(int, QtPrivate::QSlotObjectBase *, QObject *, void **, bool *) (which=<optimized out>, this_=0x6dd040, r=<optimized out>, a=<optimized out>, ret=<optimized out>) at /usr/include/qt5/QtCore/qobject_impl.h:198         
#10 0x00007ffff5e7b44e in QMetaObject::activate(QObject*, int, int, void**) () from /usr/lib64/libQt5Core.so.5                  
#11 0x00007fffeefc35ff in QDBusPendingCallWatcher::finished(QDBusPendingCallWatcher*) () from /usr/lib64/libQt5DBus.so.5        
#12 0x00007fffeefc36f8 in ?? () from /usr/lib64/libQt5DBus.so.5                                                                 
#13 0x00007ffff5e7bf99 in QObject::event(QEvent*) () from /usr/lib64/libQt5Core.so.5                                            
#14 0x00007ffff63fd9ac in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib64/libQt5Widgets.so.5           
#15 0x00007ffff6405151 in QApplication::notify(QObject*, QEvent*) () from /usr/lib64/libQt5Widgets.so.5                         
#16 0x00007ffff5e4f018 in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /usr/lib64/libQt5Core.so.5               
#17 0x00007ffff5e517f0 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib64/libQt5Core.so.5                                                                                                                             
#18 0x00007ffff5ea086a in QEventDispatcherUNIX::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib64/libQt5Core.so.5                                                                                                                         
#19 0x00007fffdee3196d in QUnixEventDispatcherQPA::processEvents (this=<optimized out>, flags=...) at eventdispatchers/qunixeventdispatcher.cpp:68                                                                                                              
#20 0x00007ffff5e4cfca in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib64/libQt5Core.so.5            
#21 0x00007ffff5e558bc in QCoreApplication::exec() () from /usr/lib64/libQt5Core.so.5                                           
#22 0x0000000000408cb5 in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/kwin-5.6.90git~20160623T210703~03f0dc5/main_wayland.cpp:724

> Adding my idea in case I forget again: the mmap of the DrmBuffer fails for the
cursor buffer. We need to catch this situation properly.

Looks like you're right:

(gdb) print *m_cursor[0]                                                                                                        |
$2 = {m_backend = 0x66dd60, m_surface = 0x0, m_bo = 0x0, m_size = {wd = 64, ht = 64}, m_handle = 2, m_bufferId = 0, m_stride = 2|
56, m_bufferSize = 16384, m_memory = 0x0, m_image = 0x0}

thanks for checking my theory in the debugger.

Now I'm wondering why did the map fail and what's the best thing to handle the fail. We could just not update the cursor, but that's not really a solution. Or we could terminate, because we can expect that it won't work.

(In reply to Martin Gräßlin from comment #4)
> thanks for checking my theory in the debugger.
> 
> Now I'm wondering why did the map fail and what's the best thing to handle
> the fail. We could just not update the cursor, but that's not really a
> solution. Or we could terminate, because we can expect that it won't work.

Some more information: drmModeAddFB fails because the DRM ioctl returns -1 with errno set to 22 (EINVAL). A guess: Maybe 32 bpp or 24 bit depth aren't supported?

if ((ret = DRM_IOCTL(fd, DRM_IOCTL_MODE_ADDFB, &f)))
(gdb) print f
$9 = {fb_id = 0, width = 64, height = 64, pitch = 256, bpp = 32, depth = 24, handle = 2}

> A guess: Maybe 32 bpp or 24 bit depth aren't supported?

quite likely. In that case it doesn't make sense to try an RGB only buffer for the cursor. That would be not usable either. So the way forward seems to me to catch the condition and enable software cursor rendering. Which probably will trigger the same problem as the framebuffer backend.

I think I need to setup a qemu instance tomorrow.

Patch which should resolve the problem: https://phabricator.kde.org/D2026

and for proper rendering of the now used software cursor in the QPainter backend we have:
* https://phabricator.kde.org/D2027
* https://phabricator.kde.org/D2028

Git commit 7ea84cd346424161affbfddf653b16b6f90442f1 by Martin Gräßlin.
Committed on 29/06/2016 at 06:49.
Pushed by graesslin into branch 'Plasma/5.7'.

[platforms/drm] If mapping a DrmBuffer for a cursor fails, fallback to software cursor

Summary:
So far the drm platform did not verify whether creating and mapping a
DrmBuffer for a cursor works. This could result in a crash in the worst
case.

This change verfies whether mapping the two cursor buffers works, if
not software cursor is enabled. The code is adjusted to ensure that
none of the cursor buffers is accessed in case software cursor are
enabled.

Please note that right now the drm platform's rendering does not
support software cursors. Thus currently this change results in no
cursor at all. This will be addressed by following patches.
FIXED-IN: 5.7

Test Plan:
Verfied that it properly falls back to software cursor,
but could not verify that the crash is actually fixed.

Reviewers: #kwin, #plasma_on_wayland

Subscribers: plasma-devel, kwin

Tags: #plasma_on_wayland, #kwin

Differential Revision: https://phabricator.kde.org/D2026

M  +28   -14   plugins/platforms/drm/drm_backend.cpp

http://commits.kde.org/kwin/7ea84cd346424161affbfddf653b16b6f90442f1