Bug 402780

Summary: Akonadi doesn't work with Exchange again
Product: [Frameworks and Libraries] Akonadi Reporter: Tom Kijas <t.kijas>
Component: EWS ResourceAssignee: kdepim bugs <kdepim-bugs>
Status: CONFIRMED ---    
Severity: critical CC: bart, brendon, dorianvogel, hujq, jaap.geurts, jlp, jscott, kde, krissn, kurse, Martin, mcadoo.chen, nerumo, t.kijas, zawertun
Priority: NOR    
Version: GIT (master)   
Target Milestone: ---   
Platform: Neon   
OS: Linux   
Latest Commit: Version Fixed In:
Attachments: [mva] request.xml
[mva] response.xml

Description Tom Kijas 2019-01-02 08:12:51 UTC
SUMMARY
KMail (korganizer etc.) exchange connection is not possible anymore on several PCs with several accounts.

STEPS TO REPRODUCE
1. Install KMail, korganizer etc.
2. Try to use Server microsoft exchange (EWS)
3. Auth failed:  

a)
kf5.kio.core: Invalid URL: QUrl("")
org.kde.pim.ews.client: Failed to process EWS request: Přístup k omezenému portu je metodě POST zakázán.
ERROR "Failed to process EWS request: Přístup k omezenému portu je metodě POST zakázán."

b)
org.kde.pim.ews.client: Failed to process EWS request - HTTP code 404

org.kde.pim.ews.client: Failed to process EWS request: Nelze se spojit s hostitelem autodiscover.pravniciostrava.cz: Síť je nedosažitelná.

org.kde.pim.ews.client: Failed to process EWS request - HTTP code 404
kf5.kio.core: mimeType() not emitted when sending first data!; job URL = QUrl("https://autodiscover-s.outlook.com/autodiscover/autodiscover.xml") data size = 0

SOFTWARE/OS VERSIONS
Windows: 
MacOS: 
Linux/KDE Plasma: KDE NEON latest user edition, updated
(available in About System)
KDE Plasma Version: 5.14.4
KDE Frameworks Version: 5.53
Qt Version: 5.11.2

ADDITIONAL INFORMATION
Because of that, our company, that supports KDE, has no connection to its emails using KMail, that's just bad. :/
Comment 1 dvogel 2019-01-11 08:52:53 UTC
Hey, I was having the same issue on my laptop since a few days (Antergos):
KDE Plasma Version: 5.14.5
Qt Version: 5.12.0
KDE Frameworks Version: 5.53.0

However is was still working on my home computer (vanilla arch, same versions).
So really confusing.
However I just solved it thanks to https://github.com/KrissN/akonadi-ews/issues/48

I set the password with:
`qdbus org.freedesktop.Akonadi.Resource.akonadi_ews_resource_6 /Settings org.kde.Akonadi.Ews.Wallet.setPassword "my_app_password"`

(you can check that the command changed the password successfully by opening the settings of the account in kmail wit the number of dots in the masked password).
And it started working again. My password has '$', the rest are just upper and lower case alphanumeric chars so the dollar was probably the issue.

Hope that helps.
Comment 2 Tom Kijas 2019-01-16 12:29:55 UTC
Exactly, it still works on already set-up computers. Maybe someone who cares about KMail can start to deal with bug reports? Just an idea...
Comment 3 Bart 2019-02-01 23:08:43 UTC
Same problem over here, 
KDE Plasma Version: 5.14.90
KDE Frameworks Version: 5.54.0
Qt Version: 5.12.0

it's still working on older version:
Comment 4 Tom Kijas 2019-02-08 08:19:24 UTC
The speed of repair of KMail is insane. I switched to google mail.
And you know what? KMail after period of time REFUSES to send emails using google SMTP. That's just crazy.

KMail in this health is piece of unusable software, mainly in business and noone in KDE really cares.
Comment 5 Martin Steigerwald 2019-02-08 10:20:57 UTC
Dear Tom, while I certainly get your frustration and also share it to quite some extent, I actually know that KDEPIM developers care *a lot* about KDEPIM. But there are only a few of them regularly working on KDEPIM. About the developer of EWS resource I know that he started developing it as his employer migrated to Office 365. That means: The developer has a day job. The money is coming from that day job. Your bug report is about one month old. That is not all that old for a component of Akonadi that someone develops in his spare time.

It is a free software project, so in case you like to change the situation you are free to help. So please keep it constructive here and vent your frustration on a different place (like the kdepim-users mailing list if you must, but even there, I suggest you avoid blaming others). This is a bug tracker, no discussion forum. The longer a bug report gets, the less likely it is for someone to be able to dig out the relevant information quickly.
Comment 6 Tom Kijas 2019-02-11 08:57:58 UTC
OK, thats relevant. The point is, I switched to 365 from GoogleApps, where my original bug report is 3 years old (sending sometimes silenty dies, leaving emails not sended). But yeah u are right it's not something I should express here. 
I must admit I have no abilities in coding, I am lawyer, but I can financially support creator(s), if is it possible somehow?
Comment 7 Tom Kijas 2019-04-24 10:31:00 UTC
So this is power of free software, both Google apps and Exchange doesn't work properly and noone cares. :) I will really continue to support that. Lol
Comment 8 Krzysztof Nowicki 2019-04-24 17:51:12 UTC
Hi Tom,

Looking at your original report it is unclear to me at which step the set-up fails. I can see some autodiscover attempts, which have failed. I have seen this sometimes and it appears that it is related to the network set-up at the given location.

Have you tried to enter the EWS URL manually?
Do you know if your company is using on-premise Exchange or Office365?

If Office365 is used, you can try to manually enter the following as the EWS URL:

http://outlook.office365.com/EWS/Exchange.asmx
Comment 9 Krzysztof Nowicki 2019-04-24 17:56:23 UTC
Correction:

https://outlook.office365.com/EWS/Exchange.asmx
Comment 10 Kurse 2019-06-07 14:11:45 UTC
I can confirm this still does not work, even when manually putting the autodiscover URL in. When attempting autodiscover, its always a 401 error.

Also, OAuth2 is greyed out, and there does not seem to be a way to activate it.

Fedora 30
Plasma 5.15.5
QT 5.12.1
Comment 11 Vadim A. Misbakh-Soloviov (mva) 2019-06-08 00:50:05 UTC
Moreover:
1) When using set-up-earlier account, akonadi-ews throws this in console:
```
Failed to validate the query against the schema: The content of the "FolderIds" element in the namespace "http://schemas.microsoft.com/exchange/services/2006/messages" is incomplete. The list of expected items: "FolderId, DistinguishedFolderId" in the namespace "http://schemas.microsoft.com/exchange/services/2006/types".
```
(translated using google-translate, if any, since corporate server tells that not in english)


2) When creating new EWS-account:
a) it looks like it doesn't store the password neither it kwallet nor somewhere in configs
b) it always fails with getting 401 from the server. Even on initial check on "Try connect"...
Comment 12 Krzysztof Nowicki 2019-06-10 08:19:27 UTC
(In reply to Kurse from comment #10)
> I can confirm this still does not work, even when manually putting the
> autodiscover URL in. When attempting autodiscover, its always a 401 error.
> 
> Also, OAuth2 is greyed out, and there does not seem to be a way to activate
> it.

In the older versions of Akonadi (<= 18.12) OAuth2 will be greyed out if the package was built without support for Qt NetworkAuth. Starting with 19.04 Qt NetworkAuth is a hard requirement and OAuth2 will always be enabled.

The 401 error may be there due to lack of OAuth2 support. It all depends on the authentication configuration of the Exchange server. In the organization I work for Office 365 is configured so that basic authentication is not supported and it is necessary to use OAuth2.
Comment 13 Vadim A. Misbakh-Soloviov (mva) 2019-06-11 18:37:42 UTC
Btw, I've another laptop with kdepim-runtime-18.12.0 and EWS (I mean, my Account, that I set up long time ago, when EWS resource was out of the kdepim) works fine there (doesn't throw FolderIds-related error I mentioned above).

Although, newly-created account gets 401 there too.
And also it doesn't savethe password to kwallet...  (and neither even trying to load it from there, if I create it manually)
Comment 14 Vadim A. Misbakh-Soloviov (mva) 2019-06-11 18:40:37 UTC
btw, additional info:
1) there is a password-only auth server (no 365, no autodiscovery, only OWA).
2) I've dumped the request and the answer, and I'll attach them next to this comment
Comment 15 Vadim A. Misbakh-Soloviov (mva) 2019-06-11 18:57:38 UTC
Created attachment 120793 [details]
[mva] request.xml
Comment 16 Vadim A. Misbakh-Soloviov (mva) 2019-06-11 18:58:18 UTC
Created attachment 120794 [details]
[mva] response.xml
Comment 17 Vadim A. Misbakh-Soloviov (mva) 2019-06-11 19:09:05 UTC
Well... I've used a qdbus hack from Comment1 on newly-created EWS account, and it looks like working fine even here, on 19.04.1.

Although, I still not sure, why the old one gets errors about XML verification :-/
Comment 18 mcadoo.chen 2019-10-01 06:46:17 UTC
Is this issue solved? what is the mitigation for the moment? I am new KDE user and I got the same problem.
Comment 19 Krzysztof Nowicki 2019-10-31 09:55:04 UTC
(In reply to Vadim A. Misbakh-Soloviov (mva) from comment #15)
> Created attachment 120793 [details]
> [mva] request.xml

OK, that request is definitely wrong and no wonder it fails.

It looks like the request for retrieving the IDs of subscribed folders (folders which are monitored for new messages). By default this list includes:

 * Inbox
 * Calendar
 * Tasks
 * Contacts

Folders may be added or removed from this list in the EWS resource configuration window.

For some odd reason it looks like in your case the list is empty. By itself this should be a valid configuration, as the user is free to stop automatically monitoring for new mails and check for them manually. Unfortunately the current EWS resource implementation is not ready for it and doesn't expect this to occur and stupidly tries to request IDs of an empty folder list, to which the server rightfully replies with an error.

As a workaround could you go to the configuration window, click the Subscriptions tab and select at least one folder (Inbox preferably).
Comment 20 Igor Poboiko 2020-03-20 16:10:46 UTC
Git commit 7afd99abbfa141f6e6dfbe69b01827af8f16ba27 by Igor Poboiko.
Committed on 20/03/2020 at 16:10.
Pushed by poboiko into branch 'master'.

[resources/ews] Save password to wallet

Summary:
Seems like the password entered via the UI actually never gets saved anywhere.
Just do it explicitly.
Related: bug 393002, bug 390798, bug 414789

Test Plan:
1) Try to setup EWS account using autodiscovery, using Username/Password auth
2) Set Username, Password, hit "Try connect" -> it works fine
3) Hit "OK" -> observe "Authentication failure" resource error
4) Browse `akonadi-ews` via `KWalletManager` -> it's empty
5) Apply the patch, repeat 1-4 - authentication succeeds, password entry inside wallet appears

Reviewers: dvratil, nowicki

Reviewed By: dvratil

Subscribers: kde-pim

Tags: #kde_pim

Differential Revision: https://phabricator.kde.org/D27813

M  +1    -0    resources/ews/ewsconfigdialog.cpp

https://commits.kde.org/kdepim-runtime/7afd99abbfa141f6e6dfbe69b01827af8f16ba27
Comment 21 Yaroslav Sidlovsky 2020-07-20 10:39:38 UTC
Can't find change from the patch in the kdepim-runtime-20.04.3 sources.
Is this intentional?
Comment 22 Brendon Higgins 2020-10-28 19:50:32 UTC
(In reply to Krzysztof Nowicki from comment #9)
> Correction:
> 
> https://outlook.office365.com/EWS/Exchange.asmx

Thanks, Krzysztof. Autodiscover fails for my organization, so I needed this. But it's not user-discoverable in the slightest. Would it make sense as being a default value when manual mode is selected?
Comment 23 Vadim A. Misbakh-Soloviov (mva) 2021-01-20 07:02:12 UTC
(In reply to Krzysztof Nowicki from comment #19)

> OK, that request is definitely wrong and no wonder it fails.

> Folders may be added or removed from this list in the EWS resource
> configuration window.

> As a workaround could you go to the configuration window, click the
> Subscriptions tab and select at least one folder (Inbox preferably).


Unfortunatelly, I can't: there is empty list and an error message appers on top, that says "Failes to get folders list" (it means "from server", I guess).
And there is no way to manually add custom (say, "Inbox") value to the list...

For now, I just disabled server-side subscription, but I dislike that way of "solving" the issue...