Bug 457529

https://invent.kde.org/frameworks/kcoreaddons/-/merge_requests/245 maybe?

(In reply to Harald Sitter from comment #1)
> https://invent.kde.org/frameworks/kcoreaddons/-/merge_requests/245 maybe?

No, there are no dirs involved here. This is fixed in dolphin 22.08 by 57909ebe66fd4aa83d1fbaf2c746d408451e80cb, but git master is still affected.

(In reply to Antonio Rojas from comment #2)
> (In reply to Harald Sitter from comment #1)
> > https://invent.kde.org/frameworks/kcoreaddons/-/merge_requests/245 maybe?
> 
> No, there are no dirs involved here. This is fixed in dolphin 22.08 by
> 57909ebe66fd4aa83d1fbaf2c746d408451e80cb, but git master is still affected.

And this is still an issue in 22.12 beta, where the commit wasn't reverted

Can't reproduce the problem. What exact call is it blocking on?

Here is the full backtrace. You need to drag a significant number of files to reproduce it (about 25)

#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555564529a0) at futex-internal.c:57
#1  __futex_abstimed_wait_common
    (futex_word=futex_word@entry=0x5555564529a0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007ffff549c51f in __GI___futex_abstimed_wait_cancelable64
    (futex_word=futex_word@entry=0x5555564529a0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0)
    at futex-internal.c:139
#3  0x00007ffff549ecd0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x555556452950, cond=0x555556452978) at pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x555556452978, mutex=0x555556452950) at pthread_cond_wait.c:618
#5  0x00007ffff5aeb784 in QWaitConditionPrivate::wait(QDeadlineTimer) (deadline=..., this=0x555556452950) at thread/qwaitcondition_unix.cpp:146
#6  QWaitCondition::wait(QMutex*, QDeadlineTimer) (this=this@entry=0x55555658ade0, mutex=mutex@entry=0x55555658add8, deadline=...)
    at thread/qwaitcondition_unix.cpp:225
#7  0x00007ffff708dd66 in QDBusPendingCallPrivate::waitForFinished() (this=0x55555658ada0)
    at /usr/src/debug/qt5-base/qtbase/src/dbus/qdbuspendingcall.cpp:240
#8  0x00007ffff71b4fa1 in KUrlMimeData::exportUrlsToPortal(QMimeData*) () at /usr/lib/libKF5CoreAddons.so.5
#9  0x00007ffff7ed8b3f in KItemListController::startDragging() (this=this@entry=0x555555ea4eb0)
    at /usr/src/debug/dolphin/dolphin-22.11.80/src/kitemviews/kitemlistcontroller.cpp:1341
#10 0x00007ffff7edace3 in KItemListController::mouseMoveEvent(QGraphicsSceneMouseEvent*, QTransform const&)
    (transform=..., event=0x7fffffffd460, this=0x555555ea4eb0) at /usr/src/debug/dolphin/dolphin-22.11.80/src/kitemviews/kitemlistcontroller.cpp:608
#11 KItemListController::mouseMoveEvent(QGraphicsSceneMouseEvent*, QTransform const&) (this=0x555555ea4eb0, event=0x7fffffffd460, transform=...)
    at /usr/src/debug/dolphin/dolphin-22.11.80/src/kitemviews/kitemlistcontroller.cpp:578
#12 0x00007ffff7ede64b in KItemListController::processEvent(QEvent*, QTransform const&)
    (this=0x555555ea4eb0, event=0x7fffffffd460, transform=<optimized out>)
    at /usr/src/debug/dolphin/dolphin-22.11.80/src/kitemviews/kitemlistcontroller.cpp:1184
#13 0x00007ffff7ee3402 in KItemListView::event(QEvent*) (this=0x555555ec9850, event=0x7fffffffd460)
--Type <RET> for more, q to quit, c to continue without paging--
    at /usr/src/debug/dolphin/dolphin-22.11.80/src/kitemviews/kitemlistview.cpp:990
#14 0x00007ffff6978b1c in QApplicationPrivate::notify_helper(QObject*, QEvent*) (this=<optimized out>, receiver=0x555555ec9850, e=0x7fffffffd460)
    at kernel/qapplication.cpp:3637
#15 0x00007ffff5c8cf98 in QCoreApplication::notifyInternal2(QObject*, QEvent*) (receiver=0x555555ec9850, event=0x7fffffffd460)
    at kernel/qcoreapplication.cpp:1064
#16 0x00007ffff6c93ed0 in qt_sendSpontaneousEvent(QObject*, QEvent*) (event=0x7fffffffd460, receiver=<optimized out>) at kernel/qapplication.cpp:4061
#17 QGraphicsScenePrivate::sendEvent(QGraphicsItem*, QEvent*) (this=<optimized out>, item=<optimized out>, event=0x7fffffffd460)
    at graphicsview/qgraphicsscene.cpp:1254
#18 0x00007ffff6c9649a in QGraphicsScenePrivate::sendMouseEvent(QGraphicsSceneMouseEvent*) (this=0x5555558cdcd0, mouseEvent=0x7fffffffd460)
    at graphicsview/qgraphicsscene.cpp:1335
#19 0x00007ffff6ca10da in QGraphicsScene::mouseMoveEvent(QGraphicsSceneMouseEvent*) (this=<optimized out>, mouseEvent=0x7fffffffd460)
    at graphicsview/qgraphicsscene.cpp:4097
#20 0x00007ffff6c9f50f in QGraphicsScene::event(QEvent*) (this=0x555555edfad0, event=<optimized out>) at graphicsview/qgraphicsscene.cpp:3429
#21 0x00007ffff6978b1c in QApplicationPrivate::notify_helper(QObject*, QEvent*) (this=<optimized out>, receiver=0x555555edfad0, e=0x7fffffffd460)
    at kernel/qapplication.cpp:3637
#22 0x00007ffff5c8cf98 in QCoreApplication::notifyInternal2(QObject*, QEvent*) (receiver=0x555555edfad0, event=0x7fffffffd460)
    at kernel/qcoreapplication.cpp:1064
#23 0x00007ffff6cbe318 in qt_sendSpontaneousEvent(QObject*, QEvent*) (event=0x7fffffffd460, receiver=<optimized out>) at kernel/qapplication.cpp:4061
#24 QGraphicsViewPrivate::mouseMoveEventHandler(QMouseEvent*) (this=0x555555edffd0, event=0x7fffffffda30) at graphicsview/qgraphicsview.cpp:672
#25 0x00007ffff69af6bf in QWidget::event(QEvent*) (this=0x555555edfa60, event=0x7fffffffda30) at kernel/qwidget.cpp:8661
#26 0x00007ffff6a5d823 in QFrame::event(QEvent*) (this=0x555555edfa60, e=0x7fffffffda30) at widgets/qframe.cpp:550
#27 0x00007ffff5c8cc02 in QCoreApplicationPrivate::sendThroughObjectEventFilters(QObject*, QEvent*)
    (receiver=receiver@entry=0x555555ef77d0, event=event@entry=0x7fffffffda30) at kernel/qcoreapplication.cpp:1190
#28 0x00007ffff6978b0c in QApplicationPrivate::notify_helper(QObject*, QEvent*)
    (this=this@entry=0x55555569a420, receiver=receiver@entry=0x555555ef77d0, e=e@entry=0x7fffffffda30) at kernel/qapplication.cpp:3631
--Type <RET> for more, q to quit, c to continue without paging--
#29 0x00007ffff697e339 in QApplication::notify(QObject*, QEvent*) (this=<optimized out>, receiver=<optimized out>, e=0x7fffffffda30)
    at kernel/qapplication.cpp:3081
#30 0x00007ffff5c8cf98 in QCoreApplication::notifyInternal2(QObject*, QEvent*) (receiver=0x555555ef77d0, event=0x7fffffffda30)
    at kernel/qcoreapplication.cpp:1064
#31 0x00007ffff697c337 in QApplicationPrivate::sendMouseEvent(QWidget*, QMouseEvent*, QWidget*, QWidget*, QWidget**, QPointer<QWidget>&, bool, bool)
    (receiver=0x555555ef77d0, event=0x7fffffffda30, alienWidget=<optimized out>, nativeWidget=0x5555556d99a0, buttonDown=<optimized out>, lastMouseReceiver=..., spontaneous=true, onlyDispatchEnterLeave=false) at kernel/qapplication.cpp:2619
#32 0x00007ffff69cd3b5 in QWidgetWindow::handleMouseEvent(QMouseEvent*) (this=0x555555be75d0, event=0x7fffffffdd90) at kernel/qwidgetwindow.cpp:683
#33 0x00007ffff69cf15e in QWidgetWindow::event(QEvent*) (this=0x555555be75d0, event=0x7fffffffdd90) at kernel/qwidgetwindow.cpp:300
#34 0x00007ffff6978b1c in QApplicationPrivate::notify_helper(QObject*, QEvent*) (this=<optimized out>, receiver=0x555555be75d0, e=0x7fffffffdd90)
    at kernel/qapplication.cpp:3637
#35 0x00007ffff5c8cf98 in QCoreApplication::notifyInternal2(QObject*, QEvent*) (receiver=0x555555be75d0, event=0x7fffffffdd90)
    at kernel/qcoreapplication.cpp:1064
#36 0x00007ffff613f16c in QGuiApplicationPrivate::processMouseEvent(QWindowSystemInterfacePrivate::MouseEvent*) (e=0x5555577601d0)
    at kernel/qguiapplication.cpp:2285
#37 0x00007ffff6128985 in QWindowSystemInterface::sendWindowSystemEvents(QFlags<QEventLoop::ProcessEventsFlag>) (flags=...)
    at kernel/qwindowsysteminterface.cpp:1169
#38 0x00007ffff355e8e5 in userEventSourceDispatch(_GSource*, int (*)(void*), void*) () at /usr/lib/libQt5WaylandClient.so.5
#39 0x00007ffff3e6d87b in g_main_dispatch (context=0x7fffe8005010) at ../glib/glib/gmain.c:3444
#40 g_main_context_dispatch (context=0x7fffe8005010) at ../glib/glib/gmain.c:4162
#41 0x00007ffff3ec4279 in g_main_context_iterate.constprop.0 (context=0x7fffe8005010, block=1, dispatch=1, self=<optimized out>)
    at ../glib/glib/gmain.c:4238
#42 0x00007ffff3e6c132 in g_main_context_iteration (context=0x7fffe8005010, may_block=1) at ../glib/glib/gmain.c:4303
#43 0x00007ffff5cd7c4c in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) (this=0x5555556da580, flags=...)
    at kernel/qeventdispatcher_glib.cpp:423
--Type <RET> for more, q to quit, c to continue without paging--
#44 0x00007ffff5c8573c in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) (this=0x7fffffffe130, flags=...)
    at ../../include/QtCore/../../src/corelib/global/qflags.h:69
#45 0x00007ffff5c90269 in QCoreApplication::exec() () at ../../include/QtCore/../../src/corelib/global/qflags.h:121
#46 0x00007ffff613a112 in QGuiApplication::exec() () at kernel/qguiapplication.cpp:1870
#47 0x00007ffff6976f2a in QApplication::exec() () at kernel/qapplication.cpp:2829
#48 0x000055555559cf53 in main(int, char**) (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/dolphin/dolphin-22.11.80/src/main.cpp:258

On the console output I see a bunch of "QDragManager::drag in possibly invalid state" errors.

Could it be that your xdg-desktop-portal(-kde) or kio-fuse is somehow malfunctioning/broken? This trace looks a whole lot like the portal isn't responding which either means kio-fuse is being slow or the portal is.

(In reply to Harald Sitter from comment #7)
> Could it be that your xdg-desktop-portal(-kde) or kio-fuse is somehow
> malfunctioning/broken? This trace looks a whole lot like the portal isn't
> responding which either means kio-fuse is being slow or the portal is.

I haven't noticed any other issue with either of them. Everything else seems to work fine.

Same thing with sftp: eg. open sftp://deino.kde.org/srv/archives/ftp/stable/plasma/5.26.3/ on a dolphin split view vs. a local dir, select all items and try to drag them to the local pane.

Still can't reproduce :(

This is very puzzling. How can I reproduce your setup?

So the issue seems to depend on the number of selected files. Usually when you drag files, the thumbnails are drawn next to the cursor while moving. When dragging them from a remote location, the thumbnails take a while to show up (depending on the number of dragged files, about 20 seconds for ~100 files). If you release the button before the thumbnails are shown, the drag is cancelled. If you wait long enough, they eventually appear and then the copy/move operation starts. I guess you just need to select more files to make it noticeable.

I'm trying with 1000 files and at most it locks up for a second or two. Indeed I've added timers to the code and the only thing blocking for me is the open() calls for some milliseconds (which is to be expected because linux :()

We really need a way to reproduce this in a clean environment I think

dolphin-22.11.90 freezes here on Gentoo as soon as I only select e.g. all KDE Gear 22.12.0 tarballs on tinami.

Dolphin 22.11.90 freezes on fish/fstp on KaOS too, workaround used was reverting https://invent.kde.org/system/dolphin/-/commit/57909ebe66fd4aa83d1fbaf2c746d408451e80cb again.
With that reverted, issues are gone.

I guess we should simply revert for now. Freezes on SFTP are probably a bigger issue than not being able to drop files in Flatpak Dolphin. Let me know if that might be a bad judgement on my part.

A possibly relevant merge request was started @ https://invent.kde.org/system/dolphin/-/merge_requests/479

Please check out the merge request above and let me know if this seems like the right thing to do. I am again only reverting in the 22.12 branch currently because this is the most pressing matter for now.

I'd appreciate if we could identify a way to reliably reproduce this on a pristine setup of some sort so I can actually figure out why it gets hung up.

Created attachment 154321 [details]
test patchy

I've had a genious idea! someone please try the attached patch. though I am rather thinking it'll only improve the situation, not fix it :(

Thanks, will do on the affected system of mine - right now working on the laptop, with same package versions, that does not reproduce this problem, so it is not straightforward unfortunately.

(In reply to Andreas Sturmlechner from comment #20)
> Thanks, will do on the affected system of mine - right now working on the
> laptop, with same package versions, that does not reproduce this problem, so
> it is not straightforward unfortunately.
Correction, should have looked at runtime version while updates were rattling through - reproduced as soon as restarting dolphin as 22.12.0.

(In reply to Andreas Sturmlechner from comment #13)
> dolphin-22.11.90 freezes here on Gentoo as soon as I only select e.g. all
> KDE Gear 22.12.0 tarballs on tinami.
To clarify here, just select is not enough, freeze occurs after Ctrl+C for copying.

By - seemingly - just upgrading to 5.101.0 this seems to have improved in not freezing endlessly anymore but just for several seconds.

(In reply to Harald Sitter from comment #19)
> Created attachment 154321 [details]
> test patchy
> 
> I've had a genious idea! someone please try the attached patch. though I am
> rather thinking it'll only improve the situation, not fix it :(
This unfortunately did not make a difference on top of that though.

Not seeing any improvement myself with 5.101

With how dolphin is currently written there's always going to be N seconds freeze. Drag creation is a sync operation so we need to synchronously fuse the urls into the file system with kio-fuse. That does take time. Perhaps too much time, but ultimately it will always need some time. I know this is crappy but I don't see a good way out of this. We could stop fusing remote urls, that would remove most of the freezyness (open() on local files should be plenty fast, although, still synchronous mind you) perhaps that is the least price to pay here.

    const auto urls = KUrlMimeData::urlsFromMimeData(data);
    const auto allLocal = std::all_of(urls.cbegin(), urls.cend(), [](const auto &url) {
        return KProtocolInfo::protocolClass(url.scheme()) == QStringLiteral(":local");
    });
    if (allLocal) {
        KUrlMimeData::exportUrlsToPortal(data);
    }

Would you please elaborate what was different before? Your last comment makes it sound like Dolphin is coded wrong while this bug seemingly only appeared recently.

To which file does your code snippet belong?

(In reply to Felix Ernst from comment #25)
> Would you please elaborate what was different before? Your last comment
> makes it sound like Dolphin is coded wrong while this bug seemingly only
> appeared recently.

It's not lining up with our requirements anymore. QMimeData creation for drag events is synchronous, we'd need it to be asynchronous because fusing URLs for non-KDE apps does take N amount of time (N being at least the amount of time it takes to make M dbus calls). 
 
> To which file does your code snippet belong?

src/kitemviews/kitemlistcontroller.cpp, though I mean, it applies to every invocation of exportUrlsToPortal

(In reply to Harald Sitter from comment #26)
> It's not lining up with our requirements anymore. QMimeData creation for
> drag events is synchronous, we'd need it to be asynchronous because fusing
> URLs for non-KDE apps does take [time]. 

I familiarised myself with the issue a bit now and have an idea. Let me know if I am missing something.

How about we duplicate the `QMimeData` object, run `exportUrlsToPortal()` in a separate thread on that copy of the data while the drag operation with the original `QMimeData` is already happening, and then as soon as `exportUrlsToPortal()` is complete, we switch the `QMimeData` of the on-going drag-operation with the enriched `QMimeData` from the `exportUrlsToPortal()` call by using `QDrag::setMimeData()`?

If we do this, the only remaining problem is when quickly drag-hovering over a sandboxed application while `exportUrlsToPortal` hasn't completed yet. To avoid any confusion with this we should show a loading cursor as long as `exportUrlsToPortal()` is still ongoing.

*** Bug 462851 has been marked as a duplicate of this bug. ***

It's an amazing idea. Needs testing if switching out the mimedata actually works though, I am not super confident that it does

Git commit c8aed8ac81d9f7f3dc93a7570037041228a98bf4 by Felix Ernst.
Committed on 14/12/2022 at 12:33.
Pushed by felixernst into branch 'release/22.12'.

Revert "portalize drag urls"

This reverts commit 8d7e600f63a1961294dfe2c278a710b4ce0716e9.

While this revert unfortunately removes Dolphin's ability to copy to
sandboxed applications, the bugs being temporarily fixed by this seem
more important. See the bugs mentioned below for details.

Especially copy-pasting needs to work flawlessly for an application
like Dolphin. After the revert this will either work correctly or – in
the case of sandboxed applications – not at all.
Related: bug 462928

M  +1    -3    src/kitemviews/kitemlistcontroller.cpp
M  +0    -1    src/panels/folders/treeviewcontextmenu.cpp
M  +1    -4    src/views/dolphinview.cpp

https://invent.kde.org/system/dolphin/commit/c8aed8ac81d9f7f3dc93a7570037041228a98bf4

(In reply to Felix Ernst from comment #27)
> (In reply to Harald Sitter from comment #26)
> > It's not lining up with our requirements anymore. QMimeData creation for
> > drag events is synchronous, we'd need it to be asynchronous because fusing
> > URLs for non-KDE apps does take [time]. 
> 
> I familiarised myself with the issue a bit now and have an idea. Let me know
> if I am missing something.
> 
> How about we duplicate the `QMimeData` object, run `exportUrlsToPortal()` in
> a separate thread on that copy of the data while the drag operation with the
> original `QMimeData` is already happening, and then as soon as
> `exportUrlsToPortal()` is complete, we switch the `QMimeData` of the
> on-going drag-operation with the enriched `QMimeData` from the
> `exportUrlsToPortal()` call by using `QDrag::setMimeData()`?
> 
> If we do this, the only remaining problem is when quickly drag-hovering over
> a sandboxed application while `exportUrlsToPortal` hasn't completed yet. To
> avoid any confusion with this we should show a loading cursor as long as
> `exportUrlsToPortal()` is still ongoing.

I've been talking about this with David Edmundson a while ago and IIRC this isn't going to be possible - right now. Currently the mimedata business must be a blocking operation, doing it any other way requires somewhat invasive changes to Qt's QPAs where at least on wayland we can make things more async. The core problem is that our mimedata **must** be finished by the time the drop/paste happens, simply running it in a thread wouldn't help one way or another because the thread might take longer than the user and then we have a race condition between user dropping and mimedata being complete.
What we would need to happen is Qt supporting "temporary" mimetypes, where we set the mimetype "application/vnd.portal.filetransfer" with some temporary dummy data (so the potential receiver can see this mimetype as available) and when the actual drop/paste happens Qt  goes "right, this mimedata is now being consumed, complete it **right now**" which would then force the async qmimedata filling to become sync.

e.g.

- drag initiates
- dolphin fills "application/vnd.portal.filetransfer" with dummy data
- dolphin starts thread to actually build the real filetransfer data
- user moves the cursor to gwenview
- gwenview looks at mimedata - finds "application/vnd.portal.filetransfer" and knows how to handle it
- gwenview can receive the drop
- user drops
- dolphin receives Qt signal polishMimeData or something
- dolphin waits for thread if it hasn't finished yet
- dolphin fills real "application/vnd.portal.filetransfer" into mimedata
- gwenview receives real data

Notably this still means things are blocking, eventually, so IMO the first step is to actually improve the performance of kio-fuse so exportUrlsToPortal finishes in a more reasonable time frame than 10s. As a stop-gap measure I would still propose only exporting local urls https://bugs.kde.org/show_bug.cgi?id=457529#c24

Also see https://github.com/flatpak/xdg-desktop-portal/issues/961

A possibly relevant merge request was started @ https://invent.kde.org/frameworks/kcoreaddons/-/merge_requests/295

Git commit 9aea173ca81baadd624d4479771de9f20a3dcb73 by Harald Sitter.
Committed on 07/02/2023 at 05:41.
Pushed by sitter into branch 'master'.

exportUrlsToPortal: stop fusing remote urls

this proofs insanely troublesome. because of the way the filetransfer
portal works we need to open() paths before sending them to the portal,
this however means open() on our fuse-redirected paths which behind the
scenes is actually a remote stat() to determine the file type. this is
insanely expensive and just will not ever work for more than a couple of
files.

for the time being we'll disable fuse redirection unless explicitly
opted into via KCOREADDONS_FUSE_REDIRECT so it can still be tested.

M  +33   -14   src/lib/io/kurlmimedata.cpp

https://invent.kde.org/frameworks/kcoreaddons/commit/9aea173ca81baadd624d4479771de9f20a3dcb73

Git commit 9a38add4f965f3f6ac7a1dca506e064b143c7d89 by Harald Sitter.
Committed on 10/02/2023 at 13:54.
Pushed by sitter into branch 'kf5'.

exportUrlsToPortal: stop fusing remote urls

this proofs insanely troublesome. because of the way the filetransfer
portal works we need to open() paths before sending them to the portal,
this however means open() on our fuse-redirected paths which behind the
scenes is actually a remote stat() to determine the file type. this is
insanely expensive and just will not ever work for more than a couple of
files.

for the time being we'll disable fuse redirection unless explicitly
opted into via KCOREADDONS_FUSE_REDIRECT so it can still be tested.
(cherry picked from commit 9aea173ca81baadd624d4479771de9f20a3dcb73)

M  +33   -14   src/lib/io/kurlmimedata.cpp

https://invent.kde.org/frameworks/kcoreaddons/commit/9a38add4f965f3f6ac7a1dca506e064b143c7d89