For the record this seems to be somehow related to the Server Service DBus API. I didn't downgrade but just disabling this in systemsettings solved the issue for me.

I can confirm this started occurring after updating to to 5.97.0-1.1 (Opensuse Tumbleweed rolling release, August 22nd, 2022).
However, enabling or disabling Secret Service DBus API and rebooting does not solve the issue in my case.
Current workaround is to downgrade back to 5.96 version.

Disabling Secret Service DBus API doesn't help for me either.

May be related to https://bugs.kde.org/show_bug.cgi?id=456872

Well, the problem resolved itself after an Arch update. The only two KDE packages that were upgraded were:

[2022-08-26T03:11:00+0200] [ALPM] upgraded kwallet (5.97.0-2 -> 5.97.0-3)
[2022-08-26T03:11:00+0200] [ALPM] upgraded kaccounts-integration (22.08.0-1 -> 22.08.0-2)

It looks like there were no upstream updates, just a rebuilt on Arch's side. So I'm setting this report as RESOLVED - DOWNSTREAM.

*** Bug 458309 has been marked as a duplicate of this bug. ***

Good news! But the problem still exists here on kde neon.

(In reply to pipapo from comment #7)
> Good news! But the problem still exists here on kde neon.

The Arch patch in kwallet 5.97.0-3 is:

https://invent.kde.org/frameworks/kwallet/-/merge_requests/33.patch

The patch is from Nicolas Fella who replied earlier, so maybe he can help. The kaccounts-integration change is just a change to the description of the package.

(In reply to nic.christin@gmail.com from comment #8)
> The Arch patch in kwallet 5.97.0-3 is:
> https://invent.kde.org/frameworks/kwallet/-/merge_requests/33.patch

In that case, this is just another symptom of https://bugs.kde.org/show_bug.cgi?id=458069

*** This bug has been marked as a duplicate of bug 458069 ***

(In reply to michaelk83 from comment #9)
> (In reply to nic.christin@gmail.com from comment #8)
> > The Arch patch in kwallet 5.97.0-3 is:
> > https://invent.kde.org/frameworks/kwallet/-/merge_requests/33.patch
> 
> In that case, this is just another symptom of
> https://bugs.kde.org/show_bug.cgi?id=458069
> 
> *** This bug has been marked as a duplicate of bug 458069 ***

Not really, disabling org.freedesktop.secrets in kwallet is a workaround at best.

This suggests that kwallet doesn't start properly if another secrets implementation is already running, but I can't reproduce that. If gnome-keyring or keepassxc is already running kwallet starts fine, it just doesn't register the secrets interface

> The dialog asking for you wallet's password appears after a long delay of about 50 seconds.

50 seconds would be twice the default method call timeout in (Qt)DBus (25 seconds)

When I terminate kwallet, then open Chromium it immediately asks for unlocking the wallet. When ignoring the prompt after 25 seconds Chromium starts appearing

Creating a new user account and configure a new wallet with gnugpg leads to the same problem - The wallet does not open automaticly and after a minute or so a dialog is asking for the passphrase to the key.
Interestingly element, the matrix client, lost the keys for the chats. This was the case in two different user accounts. I had to logout, login again and verify the chats.
I tried to go back to previous version 5.96 with some packages without success.

I have a theory what's going wrong:

- Some application accesses kwallet
- kwallet starts and unlocks the GPG wallet
- During that kwallet invokes gpg
- gpg calls org.freedesktop.secrets to get the key's passprase
- ???
- problems

That means when creating a traditional, non-GPG wallet the problem should not appear, which also explains why I wasn't affected by the problem

(In reply to Nicolas Fella from comment #13)
> I have a theory what's going wrong:
> 
> - Some application accesses kwallet
> - kwallet starts and unlocks the GPG wallet
> - During that kwallet invokes gpg
> - gpg calls org.freedesktop.secrets to get the key's passprase
> - ???
> - problems
> 
> That means when creating a traditional, non-GPG wallet the problem should
> not appear, which also explains why I wasn't affected by the problem

I can confirm this. Creating a new wallet with blowfish and configuring as standard wallet produce the expected results. Unfortunately the gnugpg-signing is the recomanded procedure for kwallet.

kwalletd5 freezes with the following backtrace, which is consistent with my theory

#0  0x00007fe3d76d988f in __GI___poll (fds=fds@entry=0x14e1910, nfds=nfds@entry=3, timeout=timeout@entry=1000) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007fe3d943b88e in _gpgme_io_select_poll (nonblock=0, nfds=10, fds=0x14aaf00) at /home/nico/kde/src/gpgme/src/posix-io.c:761
#2  _gpgme_io_select (fds=0x14aaf00, nfds=10, nonblock=nonblock@entry=0) at /home/nico/kde/src/gpgme/src/posix-io.c:928
#3  0x00007fe3d941522e in _gpgme_wait_on_condition (ctx=ctx@entry=0x14e4e10, cond=cond@entry=0x0, op_err_p=op_err_p@entry=0x0) at /home/nico/kde/src/gpgme/src/wait-private.c:87
#4  0x00007fe3d9415409 in _gpgme_wait_one (ctx=ctx@entry=0x14e4e10) at /home/nico/kde/src/gpgme/src/wait-private.c:170
#5  0x00007fe3d94184e3 in gpgme_op_decrypt_ext (ctx=0x14e4e10, flags=<optimized out>, cipher=<optimized out>, plain=0x1508d30) at /home/nico/kde/src/gpgme/src/decrypt-verify.c:180
#6  0x00007fe3d94846c2 in GpgME::Context::decrypt (this=0x14dcd60, cipherText=..., plainText=..., flags=<optimized out>) at /home/nico/kde/src/gpgme/lang/cpp/src/context.cpp:1072
#7  0x00007fe3d948473c in GpgME::Context::decrypt (this=this@entry=0x14dcd60, cipherText=..., plainText=...) at /home/nico/kde/src/gpgme/lang/cpp/src/context.cpp:1078
#8  0x00007fe3d98f8ada in KWallet::GpgPersistHandler::read (this=<optimized out>, wb=0x7fe3c0010c50, sf=..., w=0) at /home/nico/kde/src/kwallet/src/runtime/kwalletd/backend/backendpersisthandler.cpp:625
#9  0x00007fe3d98f1aec in KWallet::Backend::openInternal (this=this@entry=0x7fe3c0010c50, w=w@entry=0) at /home/nico/kde/src/kwallet/src/runtime/kwalletd/backend/kwalletbackend.cc:381
#10 0x00007fe3d98f3ef8 in KWallet::Backend::open (this=this@entry=0x7fe3c0010c50, password=..., w=w@entry=0) at /home/nico/kde/src/kwallet/src/runtime/kwalletd/backend/kwalletbackend.cc:299
#11 0x000000000042cd90 in KWalletD::internalOpen (this=0x7ffc44c41630, appid=..., wallet=..., isPath=<optimized out>, w=0, modal=true, service=...)
    at /home/nico/kde/src/kwallet/src/runtime/kwalletd/kwalletd.cpp:606
#12 0x000000000042f45d in KWalletD::processTransactions (this=0x7ffc44c41630) at /home/nico/kde/src/kwallet/src/runtime/kwalletd/kwalletd.cpp:268
#13 0x00007fe3d7de63e0 in QObject::event (this=0x7ffc44c41630, e=0x7fe3b8001d80) at kernel/qobject.cpp:1314
#14 0x00007fe3d8b4337e in QApplicationPrivate::notify_helper (this=<optimized out>, receiver=0x7ffc44c41630, e=0x7fe3b8001d80) at kernel/qapplication.cpp:3637
#15 0x00007fe3d7dbc458 in QCoreApplication::notifyInternal2 (receiver=0x7ffc44c41630, event=0x7fe3b8001d80) at kernel/qcoreapplication.cpp:1064
#16 0x00007fe3d7dbf131 in QCoreApplicationPrivate::sendPostedEvents (receiver=0x0, event_type=0, data=0x1409ba0) at kernel/qcoreapplication.cpp:1821
#17 0x00007fe3d7e10d83 in postEventSourceDispatch (s=0x1441290) at kernel/qeventdispatcher_glib.cpp:277
#18 0x00007fe3d5046faf in g_main_dispatch (context=0x7fe3c0005010) at ../glib/gmain.c:3417
#19 g_main_context_dispatch (context=0x7fe3c0005010) at ../glib/gmain.c:4135
#20 0x00007fe3d509c2c8 in g_main_context_iterate.constprop.0 (context=context@entry=0x7fe3c0005010, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4211
#21 0x00007fe3d5044940 in g_main_context_iteration (context=0x7fe3c0005010, may_block=1) at ../glib/gmain.c:4276
#22 0x00007fe3d7e104a6 in QEventDispatcherGlib::processEvents (this=0x1446410, flags=...) at kernel/qeventdispatcher_glib.cpp:423
#23 0x00007fe3d7dbb03b in QEventLoop::exec (this=this@entry=0x7ffc44c41500, flags=..., flags@entry=...) at ../../include/QtCore/../../src/corelib/global/qflags.h:69
#24 0x00007fe3d7dc2b42 in QCoreApplication::exec () at ../../include/QtCore/../../src/corelib/global/qflags.h:121
#25 0x00007fe3d8436a0c in QGuiApplication::exec () at kernel/qguiapplication.cpp:1867
#26 0x00007fe3d8b432f5 in QApplication::exec () at kernel/qapplication.cpp:2829
#27 0x000000000041ac1c in main (argc=<optimized out>, argv=<optimized out>) at /home/nico/kde/src/kwallet/src/runtime/kwalletd/main.cpp:221

A possible solution might be to temporarily disable the Secret Service API while unlocking the wallet.

(In reply to michaelk83 from comment #16)
> A possible solution might be to temporarily disable the Secret Service API while unlocking the wallet.
However, this can cause all sorts of race condition bugs, so a better solution would be to figure out which requests are being sent to the Secret Service API, and short-circuit it to return some sensible response while the wallet is being unlocked. As I understand, this is only needed while waiting for GPG.

(In reply to michaelk83 from comment #17)
> (In reply to michaelk83 from comment #16)
> > A possible solution might be to temporarily disable the Secret Service API while unlocking the wallet.
> However, this can cause all sorts of race condition bugs, so a better
> solution would be to figure out which requests are being sent to the Secret
> Service API, and short-circuit it to return some sensible response while the
> wallet is being unlocked. As I understand, this is only needed while waiting
> for GPG.

A related question is where does GPG take its passphrase from when Secret Service is not running? Can KWallet force GPG to still use that source when KWallet provides Secret Service?

(Btw, this situation was predicted in Bug 313216 comment 32.)

So it seems the issue is now understood, but just to confirm:

- I am indeed using a gpg wallet
- https://invent.kde.org/frameworks/kwallet/-/merge_requests/33.patch only fixes the issue if "Secret Service DBus API" is disabled in KDE's System Settings

(In reply to michaelk83 from comment #18)
> A related question is where does GPG take its passphrase from when Secret Service is not running?
To answer my own question after re-reading the OP, the alternative passphrase source is simply a user prompt (and I guess gpg-agent after that). According to https://unix.stackexchange.com/a/188813/455274 :

> When a GPG process needs the key, it contacts the running gpg-agent program through a socket and  
> requests the key. If the agent process has the key, it provides it to gpg. If it doesn't, it attempts  
> to load the encrypted key from your keyring, and prompts you for the key's passphrase. Once the agent  
> has obtained the decrypted key, it passes it to the gpg process. ...  
> The main point of using a key agent is so that you don't have to type your passphrase every single  
> time you use your key. The agent keeps the key in memory from one time to the next.

The problem step is "attempts to load the encrypted key from your keyring".

@nic.christin, could you run `dbus-monitor "destination=org.freedesktop.secrets" "sender=org.freedesktop.secrets"` before launching the app that tries to access KWallet, and until the passphrase prompt shows up? Mask out any byte arrays before posting that here, as they may contain secret data. (Hmm.. we might need Bug 458341 fixed first, not sure. Anyway, no harm in trying.)

And two other experiments:
1. After providing the passphrase, if you close the wallet, and then launch that app again, do you get the delay again? In theory, the key should already be in gpg-agent, so the problem step should be skipped.
2. If you load the key/passphrase into gpg-agent before accessing KWallet for the 1st time, do you still get the delay? (For example, try using the same key to encrypt/decrypt some other file.)

(In reply to toastiwall from comment #2)
> I can confirm this started occurring after updating to to 5.97.0-1.1
> (Opensuse Tumbleweed rolling release, August 22nd, 2022).
> However, enabling or disabling Secret Service DBus API and rebooting does
> not solve the issue in my case.
> Current workaround is to downgrade back to 5.96 version.

JFYI: Given openSUSE Tumbleweed 20220826, disabling Server Service DBus API solves the issue for me after logout/login.

(In reply to pipapo from comment #7)
> Good news! But the problem still exists here on kde neon.

So the patch only applies to Archlinux? I am also on kde neon and this bug after the upgrade breaks my daily workflow spectacularly. I struggled to even log in to this website to write this.

Wouldn't every distro that uses KDE be affected? Is it possible to apply the patch so that all distros are fixed?

(In reply to toastiwall from comment #2)
> I can confirm this started occurring after updating to to 5.97.0-1.1
> (Opensuse Tumbleweed rolling release, August 22nd, 2022).
> However, enabling or disabling Secret Service DBus API and rebooting does
> not solve the issue in my case.
> Current workaround is to downgrade back to 5.96 version.

Does anyone know which version of KDE neon I would need to downgrade to to get this working again? I upgraded a few times and one of them broke this, not sure which one. I am currently on Wallet Manager Version 22.08.0. KDE neon version 5.25.4. KDE framework 5.87.0 Qt 5.15.5

(In reply to ioo+kde from comment #23)
> Does anyone know which version of KDE neon I would need to downgrade to to get this working again?
You'll need to specifically downgrade the KWallet packages to any version before 5.97.0. No need to downgrade everything else.

(In reply to ioo+kde from comment #22)
> Wouldn't every distro that uses KDE be affected?
In theory, yes.

> Is it possible to apply the patch so that all distros are fixed?
For now, you can apply the patch manually, if you know how. It's a very simple patch. See link in comment 8. But as noted, it's only a workaround (or rather, that patch fixes one of the workarounds).

(In reply to michaelk83 from comment #20)
> @nic.christin, could you run `dbus-monitor
> "destination=org.freedesktop.secrets" "sender=org.freedesktop.secrets"`
> before launching the app that tries to access KWallet, and until the
> passphrase prompt shows up? Mask out any byte arrays before posting that
> here, as they may contain secret data. (Hmm.. we might need Bug 458341 fixed
> first, not sure. Anyway, no harm in trying.)

I'm not sure what app tries to access the wallet first, since it happens automatically when I log into KDE. But if I run dbus-monitor on a console before I log into KDE, I get the following output before the prompt for the wallet password appears:

signal time=1661782312.393588 sender=org.freedesktop.DBus -> destination=:1.364 serial=2 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=NameAcquired
   string ":1.364"
signal time=1661782312.393609 sender=org.freedesktop.DBus -> destination=:1.364 serial=4 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=NameLost
   string ":1.364"
method return time=1661782321.165595 sender=org.freedesktop.DBus -> destination=:1.371 serial=5 reply_serial=12
   uint32 1
method call time=1661782321.269761 sender=:1.378 -> destination=:1.371 serial=7 path=/org/freedesktop/secrets; interface=org.freedesktop.DBus.Properties; member=GetAll
   string "org.freedesktop.Secret.Service"
method call time=1661782346.297437 sender=:1.378 -> destination=:1.371 serial=8 path=/org/freedesktop/secrets; interface=org.freedesktop.Secret.Service; member=OpenSession
   string "dh-ietf1024-sha256-aes128-cbc-pkcs7"
   variant       array of bytes [
         <redacted>
      ]

> And two other experiments:
> 1. After providing the passphrase, if you close the wallet, and then launch
> that app again, do you get the delay again? In theory, the key should
> already be in gpg-agent, so the problem step should be skipped.

No, I don't get any delay in that case. In fact I don't get a prompt at all, and secrets can be read correctly.

> 2. If you load the key/passphrase into gpg-agent before accessing KWallet
> for the 1st time, do you still get the delay? (For example, try using the
> same key to encrypt/decrypt some other file.)

No, I do not get the delay in this case. I don't get any prompt and the secrets can be read correctly. What I did for the test is run

echo Hello | GPG_TTY=/dev/tty2 gpg --sign --armor

from the console (which asked me for my GPG password) before logging into KDE.

(In reply to nic.christin@gmail.com from comment #25)
> (In reply to michaelk83 from comment #20)
> > And two other experiments:
> > 1. After providing the passphrase, if you close the wallet, and then launch
> > that app again, do you get the delay again?
> No, I don't get any delay in that case. In fact I don't get a prompt at all,
> and secrets can be read correctly.
> 
> > 2. If you load the key/passphrase into gpg-agent before accessing KWallet
> > for the 1st time, do you still get the delay? 
> No, I do not get the delay in this case. I don't get any prompt and the secrets
> can be read correctly.

Thanks! This confirms that the problem step is when gpg-agent tries to load the key/passphrase for the first time. Once it's loaded and cached in gpg-agent, further use bypasses Secret Service and works as expected.

> > could you run `dbus-monitor "destination=org.freedesktop.secrets"
> > "sender=org.freedesktop.secrets"` before launching the app that tries to
> > access KWallet, and until the passphrase prompt shows up?
> 
> ...
> if I run dbus-monitor on a console before I log into KDE, I get the
> following output before the prompt for the wallet password appears:
> 
> ...
> method call time=1661782321.269761 sender=:1.378 -> destination=:1.371
> serial=7 path=/org/freedesktop/secrets;
> interface=org.freedesktop.DBus.Properties; member=GetAll
>    string "org.freedesktop.Secret.Service"
> method call time=1661782346.297437 sender=:1.378 -> destination=:1.371
> serial=8 path=/org/freedesktop/secrets;
> interface=org.freedesktop.Secret.Service; member=OpenSession
>    string "dh-ietf1024-sha256-aes128-cbc-pkcs7"
>    variant       array of bytes [
>          <redacted>
>       ]
> 

Presumably, this is gpg-agent trying to open a session with Secret Service, to retrieve the key passphrase. If this is the only output, then it seems kwalletd is stuck on the `OpenSession` call until it eventually times out. Then gpg-agent gives up and shows the prompt.

So short-circuiting `OpenSession` while waiting on GPG should fix this, IMO. It should return something like `InvalidMethodCall`, or simply `NotSupported`. Also, the GPG unlock call needs to be asynchronous, so it won't block the Secret Service API handler. The two lines of interest are:
https://invent.kde.org/frameworks/kwallet/-/blob/master/src/runtime/kwalletd/kwalletfreedesktopservice.cpp#L261 and
https://invent.kde.org/frameworks/kwallet/-/blob/master/src/runtime/kwalletd/backend/backendpersisthandler.cpp#L625 (or one of its callers, see the backtrace in comment 15).

I don't have an environment set up to make a patch, but hopefully @Nicolas Fella or someone else can pick this up from here.

Btw, there's another small bug in `kwalletfreedesktopservice.cpp` line 264 - according to the specs, it should return `org.freedesktop.DBus.Error.NotSupported`, instead of `InvalidArgs`: https://specifications.freedesktop.org/secret-service/latest/ch07.html#idm46274163163312
> If a service does not support a specific set of algorithms, a org.freedesktop.DBus.Error.NotSupported
> error is returned, and the client is free to try another set of algorithms.

(In reply to michaelk83 from comment #26)
> So short-circuiting `OpenSession` while waiting on GPG should fix this, IMO.
> It should return something like `InvalidMethodCall`, or simply `NotSupported`.
On further thought, this can still cause problems if two or more apps try to unlock the wallet at the same time. The 1st one would short-circuit `OpenSession` while unlocking the wallet, and then the 2nd may get the short-circuit response instead of the normal response.

A more robust approach is to check if the key/passphrase are already loaded into gpg-agent, and if not, display a prompt and pre-load the passphrase before trying to unlock. See https://linux.die.net/man/1/gpg-preset-passphrase .

(In reply to michaelk83 from comment #27)
> A more robust approach is to check if the key/passphrase are already loaded into gpg-agent,
> and if not, display a prompt and pre-load the passphrase before trying to unlock.
This is also not ideal, since it will force manual entry of the passphrase. This would break the workflow for users that prefer to use a smart-card, for example.

The fundamental problem is that KWallet relies on an external tool (GPG) for its encryption, instead of a library. If that tool tries to load the passphrase from KWallet, we get this bug.

But I found that one should be able to control this for GPG using it's `pinentry` setting - https://wiki.archlinux.org/title/GnuPG#pinentry . In particular:
> `/usr/bin/pinentry-gtk-2` and `/usr/bin/pinentry-gnome3` support the DBus Secret Service API,
> which allows for remembering passwords via a compliant manager such as GNOME Keyring or KeePassXC.
The same problem should occur with the old KWallet API if `pinentry-kwallet` is used.

So using a different pinentry program should fix this bug without changing any code, and without the problems of the other solutions I've proposed above. @nic.christin, could you try entering the following line in your `gpg-agent.conf`:
> pinentry-program /usr/bin/pinentry-qt
You may need to install the `pinentry-qt` package.

(In reply to michaelk83 from comment #28)
> @nic.christin, could you try entering the following line in your
> `gpg-agent.conf`:
> > pinentry-program /usr/bin/pinentry-qt
> You may need to install the `pinentry-qt` package.

Unfortunately I'm already using pinentry-qt.

I see gpg has a "--pinentry-mode loopback" option, that redirects the password prompt to the caller. Could this be used to have gpg request the password from kwallet instead of calling back into org.freedesktop.secrets? I guess it would require implementing the pinentry Assuan protocol into kwallet (http://info2html.sourceforge.net/cgi-bin/info2html-demo/info2html?(pinentry)Protocol). I did try replacing the gpg binary with a wrapper script that sets "--pinentry-mode loopback", but of course that doesn't since kwallet doesn't handle the protocol.

(In reply to nic.christin@gmail.com from comment #29)
> Unfortunately I'm already using pinentry-qt.
Hmm.. that is unfortunate.

> I see gpg has a "--pinentry-mode loopback" option, that redirects the
> password prompt to the caller. Could this be used to have gpg request the
> password from kwallet instead of calling back into org.freedesktop.secrets?
> I guess it would require implementing the pinentry Assuan protocol into kwallet
If someone's going to implement the Assuan protocol, they may as well make a new pinentry tool that *doesn't* try to access KWallet (or Secret Service) while it's unlocking. But the loopback method is similar to my suggestion to have KWallet prompt for the passphrase before trying to unlock GPG.

If you're willing to mess with this some more, I'd like to confirm which process is making the `OpenSession` call - is it indeed pinentry-qt, or not? Rather than trying to do this at login, you can close KWallet and clear the key from gpg-agent before launching whatever tries to read secrets. But I'm not sure how to translate the DBus sender ID to the process ID... I've found these two suggestions, not sure how useful:
https://stackoverflow.com/a/69318327/15047974
https://unix.stackexchange.com/a/498093/455274

For clearing gpg-agent:
https://askubuntu.com/questions/349238/how-can-i-clear-my-cached-gpg-password

I can confirm that the process calling OpenSession is indeed pinentry-qt:

method call time=1661860281.711391 sender=:1.137 -> destination=:1.6 serial=8 path=/org/freedesktop/secrets; interface=org.freedesktop.Secret.Service; member=OpenSession

$ busctl --no-pager --unique --user | grep ^:1.137
:1.137 15321 pinentry-qt     nicolas :1.137     user@1000.service -       -

(In reply to nic.christin@gmail.com from comment #31)
> I can confirm that the process calling OpenSession is indeed pinentry-qt:
Thanks. Not surprising, but does make a full fix more difficult...

@nic.christin, one more thing to try: https://man.archlinux.org/man/gpg-agent.1#no-allow-external-cache
> --no-allow-external-cache
>    Tell Pinentry not to enable features which use an external cache for passphrases.
So either start gpg-agent with `--no-allow-external-cache` flag, or add `no-allow-external-cache` option in `gpg-agent.conf`.

(In reply to michaelk83 from comment #33)
> @nic.christin, one more thing to try:
> https://man.archlinux.org/man/gpg-agent.1#no-allow-external-cache
> > --no-allow-external-cache
> >    Tell Pinentry not to enable features which use an external cache for passphrases.
> So either start gpg-agent with `--no-allow-external-cache` flag, or add
> `no-allow-external-cache` option in `gpg-agent.conf`.

Yes, that fixes the problem and there are no ill-effects that I can see. pinentry-qt doesn't query dbus, the password prompt is displayed immediately, and gpg still caches the passphrase.

Perfect, thanks!

Great! Then I guess this can be marked as resolved.

Reopening because the fix should ideally be automated by KWallet when setting up a GPG wallet, or at least properly documented.
To summarize:

This affects GPG wallets with Secret Service enabled. By default, GPG tries to retrieve the key passphrase from Secret Service. If Secret Service is provided by KWallet, this creates a dependency cycle, which prevents the wallet from unlocking. There are three ways to break the cycle:

- Option A (recommended): To prevent GPG from trying to retrieve the passphrase from Secret Service, add the `no-allow-external-cache` option to `~/.gnupg/gpg-agent.conf`. You can either edit the file manually, or run: `echo no-allow-external-cache:0:1 | gpgconf --change-options gpg-agent`. GPG will prompt the user directly when the key passphrase is needed for the first time.
- Option B: Disable the Secret Service API support in KWallet settings. You may use a different Secret Service provider if desired.
- Option C: Do not use a GPG wallet (downgrade to a traditional blowfish wallet).

Those would be separate issues/requests. Please file new bug reports to track them. Thanks!

(In reply to Nate Graham from comment #37)
> Those would be separate issues/requests. Please file new bug reports to
> track them. Thanks!

Not really, we just needs to find one approach that works and stick with it

(In reply to Nicolas Fella from comment #38)
> we just needs to find one approach that works and stick with it
Option A (`no-allow-external-cache`) is the correct solution IMO, if you want to use a GPG wallet with Secret Service enabled. It breaks the circular dependency without affecting other Secret Service clients or potentially introducing race condition bugs, and using a feature of GnuPG that was intended specifically to address similar situations - https://man.archlinux.org/man/gpg-agent.1#no-allow-external-cache :
> Some desktop environments prefer to unlock all credentials with one master password and
> may have installed a Pinentry which employs an additional external cache to implement
> such a policy. By using this option the Pinentry is advised not to make use of such a
> cache and instead always ask the user for the requested passphrase.

Options B and C (disable Sercret Service or not use a GPG wallet) are alternatives that a user may choose depending on what they want to achieve.

But the `no-allow-external-cache` setting is pretty esoteric. Users trying to set up a GPG wallet with Secret Service enabled would not usually know about it, and will run into this issue. So it either needs to be set by KWallet when needed (by calling `gpgconf`), or documented clearly enough for users to not miss it.

(In reply to michaelk83 from comment #39)
> Option A (`no-allow-external-cache`) is the correct solution IMO
@Nicolas Fella, the patch would be something like the following (pseudo-code, in backendpersisthandler.cpp line 625)
https://invent.kde.org/frameworks/kwallet/-/blob/master/src/runtime/kwalletd/backend/backendpersisthandler.cpp#L625
> + original = run("/bin/sh -c 'gpgconf --list-options gpg-agent | grep no-allow-external-cache | cut -d: -f1-3'")->getstdout();
> + if (!original) {
> +     original = "no-allow-external-cache:16:";
> + }
> + run("/bin/sh -c 'echo no-allow-external-cache:8:1 | gpgconf --change-options gpg-agent'");
>   GpgME::DecryptionResult res = ctx->decrypt(encryptedData, decryptedData);
> + run("/bin/sh -c 'echo " + original + " | gpgconf --change-options gpg-agent'");

As I mentioned earlier, I don't have the environment set up to make a proper patch and test it.

(In reply to nic.christin@gmail.com from comment #34)
> (In reply to michaelk83 from comment #33)
> > @nic.christin, one more thing to try:
> > https://man.archlinux.org/man/gpg-agent.1#no-allow-external-cache
> > > --no-allow-external-cache
> > >    Tell Pinentry not to enable features which use an external cache for passphrases.
> > So either start gpg-agent with `--no-allow-external-cache` flag, or add
> > `no-allow-external-cache` option in `gpg-agent.conf`.
> 
> Yes, that fixes the problem and there are no ill-effects that I can see.
> pinentry-qt doesn't query dbus, the password prompt is displayed
> immediately, and gpg still caches the passphrase.
> 
> Perfect, thanks!

Will this solution allow users to again use a third party secret service agent along with kwallet again?

(In reply to ioo+kde from comment #41)
> Will this solution allow users to again use a third party secret service agent along with kwallet again?
You don't need this patch to run a 3rd-party Secret Service provider. This only applies to when Secret Service is provided by KWallet, and only if combined with a GPG wallet.

To run a 3rd-party Secret Service provider, you need the patch from Bug 458069, and then disable the KWallet Secret Service integration in the KWallet settings. Alternatively, downgrade KWallet to a version earlier than 5.97.0.

(In reply to michaelk83 from comment #36)
> Reopening because the fix should ideally be automated by KWallet when
> setting up a GPG wallet, or at least properly documented.
> To summarize:
> 
> This affects GPG wallets with Secret Service enabled. By default, GPG tries
> to retrieve the key passphrase from Secret Service. If Secret Service is
> provided by KWallet, this creates a dependency cycle, which prevents the
> wallet from unlocking. There are three ways to break the cycle:
> 
> - Option A (recommended): To prevent GPG from trying to retrieve the
> passphrase from Secret Service, add the `no-allow-external-cache` option to
> `~/.gnupg/gpg-agent.conf`. You can either edit the file manually, or run:
> `echo no-allow-external-cache:0:1 | gpgconf --change-options gpg-agent`. GPG
> will prompt the user directly when the key passphrase is needed for the
> first time.
> - Option B: Disable the Secret Service API support in KWallet settings. You
> may use a different Secret Service provider if desired.
> - Option C: Do not use a GPG wallet (downgrade to a traditional blowfish
> wallet).

Recommendation A works fine for me - thanks for the "solution". However, I am really puzzled by the fact that the regression definitely introduced by the upgrade to KDE's KWallet 5.97.0 for those of us who are using GPG wallet together with Secret Service API support in KWallet is simply called "resolved upstream". So, who is going to inform all the users hit by this bug and using different distros to change their ~/.gnupg/gpg-agent.conf accordingly? Sorry, but the bug is not solved at all!

(In reply to Frank Kruger from comment #43)
> Recommendation A works fine for me - thanks for the "solution". However, I
> am really puzzled by the fact that the regression definitely introduced by
> the upgrade to KDE's KWallet 5.97.0 for those of us who are using GPG wallet
> together with Secret Service API support in KWallet is simply called
> "resolved upstream". So, who is going to inform all the users hit by this
> bug and using different distros to change their ~/.gnupg/gpg-agent.conf
> accordingly? Sorry, but the bug is not solved at all!

Right, marking it "resolved" was my mistake. That's why I reopened it.

(In reply to michaelk83 from comment #44)
> (In reply to Frank Kruger from comment #43)
> > Recommendation A works fine for me - thanks for the "solution". However, I
> > am really puzzled by the fact that the regression definitely introduced by
> > the upgrade to KDE's KWallet 5.97.0 for those of us who are using GPG wallet
> > together with Secret Service API support in KWallet is simply called
> > "resolved upstream". So, who is going to inform all the users hit by this
> > bug and using different distros to change their ~/.gnupg/gpg-agent.conf
> > accordingly? Sorry, but the bug is not solved at all!
> 
> Right, marking it "resolved" was my mistake. That's why I reopened it.

Appreciated, but your solution seems to be restricted to a fix when setting up a GPG wallet. What about current users?

(In reply to Frank Kruger from comment #45)
> Appreciated, but your solution seems to be restricted to a fix when setting
> up a GPG wallet. What about current users?
The proposed fix should work the same whether the wallet is new or existed previously. It does not affect the wallet data in any way.

The patch I proposed in comment 40 (once translated from pseudo- to actual code - still TODO) would take effect when unlocking the GPG wallet, so it should work with existing wallets as well. It's the same fix, just automated, and only applied while unlocking the wallet (instead of a permanent configuration change).

Btw, Secret Service integration did not exist at all in KWallet prior to 5.97.0. So there are three types of KWallet users:
- Those using a blowfish wallet. They are not affected by this bug.
- Those who were using a GPG wallet with a 3rd-party Secret Service provider. In their case, GPG was likely taking the key passphrase from the 3rd party Secret Service provider. These need the patch from Bug 458069, and to disable the new integration in KWallet, so that the 3rd party provider can function as before.
- Those who were using a GPG wallet *without* a Secret Service provider (likely the majority). In their case, GPG was *not* taking the key passphrase from Secret Service, since Secret Service was not available. Adjusting `gpg-agent.conf` as suggested above (comment 36 Option A) restores the previous GPG behavior for those users.

(In reply to michaelk83 from comment #46)
> (In reply to Frank Kruger from comment #45)
> > Appreciated, but your solution seems to be restricted to a fix when setting
> > up a GPG wallet. What about current users?
> The proposed fix should work the same whether the wallet is new or existed
> previously. It does not affect the wallet data in any way.
> 
> The patch I proposed in comment 40 (once translated from pseudo- to actual
> code - still TODO) would take effect when unlocking the GPG wallet, so it
> should work with existing wallets as well. It's the same fix, just
> automated, and only applied while unlocking the wallet (instead of a
> permanent configuration change).
> 
Thanks for the clarification. I apprecitate it.

Here on kde neon option A is working in the sense that a dialog is asking for the passphrase immediately. But the key is not used automatically with the login as before.
Option B has no observable effect here.
Solution C is working. But to make it work for existing configuration you have to export/import the wallet and set it as standard.
Interestingly when applying changes to the wallet service the matrix-client element is complaining that it is unable to decrypt. You have to logout and login again and confirm the session via QR-code.
After learning some apt I managed to go back to 5.96.0 and everything is back to normal. Going back with the package manager synaptic will remove nearly all kde stuff because of unresolved dependencies.
At the moment kwallet version 5.97.0 does not work as expected.

(In reply to pipapo from comment #48)
> Here on kde neon option A is working in the sense that a dialog is asking
> for the passphrase immediately. But the key is not used automatically with
> the login as before.
@pipapo, can you explain what you mean by "key is not used automatically with the login"?
Do you have kwallet PAM or gnome-keyring installed?

> Option B has no observable effect here.
Option B is currently broken without the patch from Bug 458069. That patch is not out yet in Neon, so you'd have to compile from source with that patch applied (or wait until the next KWallet version is released).

> Solution C is working. But to make it work for existing configuration you
> have to export/import the wallet and set it as standard.
Yes, because you're essentially creating a new wallet.

> Interestingly when applying changes to the wallet service the matrix-client
> element is complaining that it is unable to decrypt. You have to logout and
> login again and confirm the session via QR-code.
Not sure what you mean by "applying changes to the wallet service", but the situation is that this bug is causing the wallet to fail to unlock, so secrets are inaccessible. Without the fixes, you'd need to enter the passphrase when prompted, and then unlock again. And after applying the fixes, if the client app doesn't try again to unlock and retrieve its secrets, they will still remain locked. So you need to restart your client apps after applying one of the fixes.

(In reply to michaelk83 from comment #49)
> (In reply to pipapo from comment #48)
> > Here on kde neon option A is working in the sense that a dialog is asking
> > for the passphrase immediately. But the key is not used automatically with
> > the login as before.
> @pipapo, can you explain what you mean by "key is not used automatically
> with the login"?
> Do you have kwallet PAM or gnome-keyring installed?

Since @pipapo has not replied, what I'm guessing was happening there is they had Gnome keyring installed. Gnome keyring was being automatically unlocked on login, and then GPG would take its passphrase from there.

Gnome keyring is also a Secret Service provider, so to restore this behavior, they need to disable the Secret Service API support in KWallet settings (comment 36 option B). The setting was broken in 5.97.0 (Bug 458069), but should be fixed now in 5.98.0 (released today).

Also, if they've set `no-allow-external-cache` in `gpg-agent.conf` (comment 36 option A), that should be removed in their case, so that pinentry would again try to read the passphrase from Secret Service, meaning in their case from Gnome keyring.

Sorry for not replying. I was very busy. For that reason I stayed at version 0.96.
But now, with the upgrade to 5.98 everything is working fine again:
KDE neon 5.25
KDE-Plasma-Version: 5.25.5
KDE-Frameworks-Version: 5.98.0

Many thanks to all involved developers!

Great news!

Nothing has happend to properly fix this, the only thing that changed in 5.98 is that disabling org.freedesktop.secrets was fixed, which can be used to work around this

I have updated my kwalletd5 package from 5.96 to 5.98 now, and the password entry dialog still takes its sweet time to show up.

On 5.96 it shows up right after login.

(In reply to kde from comment #54)
> I have updated my kwalletd5 package from 5.96 to 5.98 now, and the password
> entry dialog still takes its sweet time to show up.
> 
> On 5.96 it shows up right after login.

It's broken since 5.97. Use one of the workarounds from comment 36 for now, depending on your use case. If you want to use a 3rd party Secret Service provider such as Gnome keyring or KeePassXC, you'll need option B, otherwise most likely option A.

I am running KDE Frameworks 5.99.0 and I think the problem I am having could be related. To prevent being asked for my ssh passphrase, I have an autostart .desktop file that launches /usr/bin/ssh-add -k PATH_TO_SSH_KEY. However, I could only make it work by launching after a delay rather than immediately. Similarly I have set up Skype to launch after a delay, otherwise it closes after starting. I can see Skype credentials in KWalletManager.

I disagree that no-allow-external-cache is a proper solution. It's a workaround, but it prevents one from using GPG/pinentry for other tasks and saving the credential cache in the KWallet-provided Secret Service. It's probably the least-intrusive option because gpg-agent does caching on its own anyway, but it's not a solution.

The solution would be to make kwalletd be able to answer this query coming from pinentry. To do that, make the kwalletd code asking for the unlocking of GPG wallet not block - if there's an asynchronous API for that, you can use it, otherwise move the entire workload to a separate thread. QtDBus is already threaded, so the query was received by the the process; however, the thread owning the QObject that would've answered the call was blocked and didn't answer.

You need to make sure this doesn't loop again: make sure this Secret Service call isn't going to trigger another call back to pinentry and thus loop again.

This has the added benefit that kwalletd isn't frozen while pinentry is waiting for the user to type the password. I've seen before that applications waiting for kwalletd also freeze if kwalletd won't answer, though this particular change may not affect the applications that are freezing, if kwalletd can't answer the call in the first place. This benefit comes with its drawback though: kwalletd needs to know that it's already waiting for the wallet to open and queue the reply going back to the application.

(In reply to Thiago Macieira from comment #57)
@Thiago, the issue is bigger than just synchronous vs asynchronous. The issue is that if you use a GPG wallet, with Secret Service provided by KWallet, and while allowing pinentry to read the key passphrase from Secret Service, what ends up happening is that KWallet asks for the key passphrase from itself instead of the user. It's like trying to unlock a safe with the key that's locked inside that same safe. Not going to work. Even if you make this asynchronous, you'll just end up with an infinite recursion.

`no-allow-external-cache` is one way to break that cycle, but you're correct that it's just a workaround. As are all three of the options that I proposed in comment 36.

In comment 40 I've proposed a simple pseudo-code patch that would automate this particular workaround and restrict it to only when a wallet is being unlocked. This has two advantages:
1. It would be handled entirely and automatically by KWallet under the hood, so users won't need to mess with it or be aware of it.
2. Once the wallet is unlocked, it would still allow other key passphrases to be stored and retrieved from KWallet via Secret Service.

It's not a perfect solution either, for example there could still be some timing conditions where things don't work properly, but it's a fairly simple patch. A proper solution would be for KWallet to handle the pinentry request, as you suggest, but it's more complicated to implement. It would require KWallet to implement the Assuan protocol to display a prompt to the user on its own. This was discussed briefly in comment 29 and 30.

> @Thiago, the issue is bigger than just synchronous vs asynchronous. The
> issue is that if you use a GPG wallet, with Secret Service provided by
> KWallet, and while allowing pinentry to read the key passphrase from Secret
> Service, what ends up happening is that KWallet asks for the key passphrase
> from itself instead of the user. It's like trying to unlock a safe with the
> key that's locked inside that same safe. Not going to work. Even if you
> make this asynchronous, you'll just end up with an infinite recursion.

I understand that we ended up with KWallet asking itself for the password. But the fact is that if the query was asynchronous, then pinentry would have got its answer instead of timing out, and then would have prompted the use for the password. Or maybe not, maybe KWallet is storing the cached answer in memory and would have provided it to pinentry. But do note I talked about KWallet ensuring it doesn't recurse infinitely, which is why we'd need to figure out if where this particular password could be saved if it is provided to KWallet; refusing to store it is a way to break the chain.

I'm not saying it's easy to implement this.

Your idea from comment 40 -- to tell gpg-agent that we want a password with no-external-cache -- is a solution too. Probably the Right Solution (with capital R and S).

But not the way you described it. Modifying ~/.gnupg/gpg-agent.conf is not acceptable, because it's not atomic. Other passwords may be getting requested at the same time as KWallet is trying to open. In fact, if we are trying to open the wallet now because something wants a stored password, then it stands to reason another program could be trying to do the same. Moreover, because we're waiting for user interaction, the time during which the gpg-agent.conf file is modified is measured in human time.

Therefore, this solution requires that we inform gpg-agent that we want a no-external-auth-cache answer for THIS query only and that it inform the pinentry tool that it shouldn't query the external auth cache. That requires those two tools to be updated and their updates deployed; plus probably libgpgme too. It's probably the right thing to do, so we should interact with upstream to get them to implement this.

But if there is a KWallet-only solution, we should investigate it.

(In reply to Thiago Macieira from comment #59)
> But the fact is that if the query was asynchronous, then pinentry would have
> got its answer instead of timing out, and then would have prompted the user
> for the password.

Pinentry is asking for the passphrase through the same Secret Service API as any other client (see comment 31). KWallet has no way to tell it apart from any other client, so can't handle it differently. As far as KWallet can tell at this point, the passphrase that pinentry wants is inside the same wallet (but it's not).

Currently pinentry's request blocks in the `OpenSession` call because KWallet is still waiting for GPG to unlock the wallet (for the original request of some other client app). If this was asynchronous, what would happen is KWallet would try to unlock the wallet a 2nd time to look for the passphrase there, which would invoke GPG and pinentry a 2nd time, which would ask KWallet again, and so on and on and on.

> But do note I talked about KWallet ensuring it doesn't recurse infinitely,
> which is why we'd need to figure out if where this particular password
> could be saved if it is provided to KWallet; refusing to store it is a way to
> break the chain.

This is the passphrase for KWallet itself (or rather for the GPG key that encrypts the wallet). It's not stored anywhere. The cycle is when trying to retrieve it, not store it. But as noted above, as long as pinentry makes the request via the Secret Service API, KWallet has no way to know that it's its own password that is being requested here.

> Your idea from comment 40 -- to tell gpg-agent that we want a password with
> no-external-cache -- is a solution too. Probably the Right Solution (with
> capital R and S).
> 
> But not the way you described it. Modifying ~/.gnupg/gpg-agent.conf is not
> acceptable, because it's not atomic.

Yes, as I said, there could be timing issues and maybe other problems. That patch is still just an automated and time-limited workaround.
I'm not aware of any other way to tell pinentry to not use the external cache, other than maybe by implementing the Assuan protocol.

(In reply to michaelk83 from comment #60)
> Pinentry is asking for the passphrase through the same Secret Service API as
> any other client (see comment 31). KWallet has no way to tell it apart from
> any other client, so can't handle it differently. As far as KWallet can tell
> at this point, the passphrase that pinentry wants is inside the same wallet
> (but it's not).

That is understood. I did say that my idea would only work if there were a way to break the loop. If there isn't, then it doesn't help.

> Currently pinentry's request blocks in the `OpenSession` call because
> KWallet is still waiting for GPG to unlock the wallet (for the original
> request of some other client app). If this was asynchronous, what would
> happen is KWallet would try to unlock the wallet a 2nd time to look for the
> passphrase there, which would invoke GPG and pinentry a 2nd time, which
> would ask KWallet again, and so on and on and on.

Only if it got coded this poorly. A proper implementation would realise that the request for GPG to open the wallet is still pending and queue the request to be answered when the wallet got opened. So the loop breaks, but doesn't solve the problem.

> > But not the way you described it. Modifying ~/.gnupg/gpg-agent.conf is not
> > acceptable, because it's not atomic.
> 
> Yes, as I said, there could be timing issues and maybe other problems. That
> patch is still just an automated and time-limited workaround.
> I'm not aware of any other way to tell pinentry to not use the external
> cache, other than maybe by implementing the Assuan protocol.

It also wouldn't work if I had my ~/.gnupg directory protected against unwanted reads and writes.

I really think we need to talk to the gpg-agent/pinentry folks.

(In reply to Thiago Macieira from comment #61)
> A proper implementation would realise that
> the request for GPG to open the wallet is still pending and queue the
> request to be answered when the wallet got opened.

So the request from pinentry gets queued and waits for the wallet to unlock, which is waiting on pinentry to return the passphrase. It just brings us back to the synchronous deadlock we have now.
(In fact, the queuing behavior may very well be how it's actually implemented currently. Though I can't say for sure.)

> It also wouldn't work if I had my ~/.gnupg directory protected against
> unwanted reads and writes.

The proposed patch doesn't attempt to write to ~/.gnupg directly. It makes calls to gpgconf.

(In reply to michaelk83 from comment #62)
> > It also wouldn't work if I had my ~/.gnupg directory protected against
> > unwanted reads and writes.
> 
> The proposed patch doesn't attempt to write to ~/.gnupg directly. It makes
> calls to gpgconf.

Does it support setting ephemeral settings not saved to disk? I didn't see that in the documentation.

Otherwise, you're just asking to write to disk, only indirectly. It's like replacing the KConfig classes with a subprocess call to kwriteconfig5 and say.

(In reply to Thiago Macieira from comment #63)
> Does it support setting ephemeral settings not saved to disk? I didn't see
> that in the documentation.
> 
> Otherwise, you're just asking to write to disk, only indirectly.

I don't know, but if you configure your system such that gpgconf can't do its job, then you have a broken configuration, IMO.
If you really want ephemeral settings without writing to disk, you can mount them with overlayfs. But that's really an edge case.

> I don't know, but if you configure your system such that gpgconf can't do
> its job, then you have a broken configuration, IMO.
> If you really want ephemeral settings without writing to disk, you can mount
> them with overlayfs. But that's really an edge case.

gpgconf can do its job when *I* want to run it. It's a tool for me, not for other applications. KWallet shouldn't change my GPG settings behind my back.

In any case, this doesn't prevent a race condition anyway. How about we try to fix it the right way first?

(In reply to Thiago Macieira from comment #65)
> In any case, this doesn't prevent a race condition anyway. How about we try
> to fix it the right way first?

I think handling the Assuan protocol is our best bet for a proper fix, currently.

Hello.

Is there any progress?

I have kwallet 5.111 and the issue is still there.