ok, this time clearing the akonoadi cache was not enough.

i had to run:
rm agent_config_akonadi_davgroupware_resource_0_changes.dat

to get the resource up again.

Same here. openSuSE 13.1,  KDE 4.13.0
I wanted to "copy" an old calender (several years, about 5000 entries) from outlook into KDE.
I exported from outlook into an *.ics file, then imported ("merge" into an owncloud calender) that ics file into Kontact.
Some entries (don't know how many) were merged and appeared at the owncloud server side.
Then suddenly the ownloud-DAV-resource stopped silently with "unexpected error 401"

I used akonadiconsole to find the offending calender entry, deleted it and restarted the DAV-resource. But it immediately stops at some other entry.

BTW: What does error 401 mean??

i guess the 401 refers to the http status code ( https://en.wikipedia.org/wiki/List_of_HTTP_status_codes )

Your trigger of this bug seems different to mine, since the server was not down. But maybe the server was unresponsive for a short time, because of the huge import.

I had a look at the conversation using wireshark. I remeber that the server resonded to a PUT request with something like "no authentication headers were found" IIRC.

In the meandtime, I delete/added the DAV resource to get my system running again.
But since this bug should be reproduceable, I could retry that import and make the resulting cache file (or any other files) available to developers for testing, if it helps.

Same here with Tine20 (SabreDAV).
You do a change that Tine20 does not want to take (for whatever reason) or you modify some event through another client and then in KOrganizer before sync.
Akonadi DAV resource simply does into broken state and never recovers, still trying to save or delete the element which will never succeed.

Sorry, but the purpose of the DAV resouce is to keep things in SYNC, not break when a conflict occurs.

This bug is also related to Bug 335833 which shows the same symptoms, but is triggered by deleting already deleted events, caused by Bug 335090

(In reply to comment #6)
> This bug is also related to Bug 335833 which shows the same symptoms, but is
> triggered by deleting already deleted events, caused by Bug 335090
sorry, i meant: caused by Bug 328734

I have the same issue with 3 KDE installations (2 users on 2 of the installations, so 5 configuration instances in total).
In all cases I tried to move a contact from one adressbook to another, afterwards I got the *412* error of owncloud. I fixed this by applying an update (sorry, I can't find the Bug-Number/Forum-Thread anymore :-( ).
Two owncloud installations were able to upload one or of the moved contacts, but afterwards all of my installation got stuck in the "401" problem.
Webserver-Log shows that there is always only one try for each upload (and than repeated after some time), so it seems that akonadi does try to provide authorization to the webserver.

*** This bug has been confirmed by popular vote. ***

^ Thanks :-)
Is Gregory still working on the resource?

I see similar behaviour with 4.13.1/Kubuntu 14.04 in conjunction with a Zimbra server. Every once in a while, the akonadi resource will enter the state described above and most of the time, I have to delete and re-add it to get it working again. From what I could figure out so far, this seems to happen most of the time when accepting an invitation, but that does not seem to be the only trigger.

Not sure why it reaches the preconition fail in the first place, however the correct resource bahaviou should be:

- in he short run:
Abandon PUT request and issue GET request to get updated item from server

-in the long run:
Ask the user whether he would like to write local changes and overwrite server -OR - whether he would like to reload from server and overwrite the outdated local item

and CONTINUE THEN instead of just hanging up ;-)

Is Gegory still doing the work on this resource or has it been abandoned?

Same problem here using Radicale 0.8 and KDE 4.13.3 / Akonadi 1.12.1

Hi, 

I confirm this bug using KDE 4.13.3 and owncloud 6.0.4 (which uses a SabreDAV backend).

This bugs makes the Dav agent totally useless, sadly...

I can confirm the bug. Authentication is not send to PUT request, when going online and thats the dead end and you cannot recover without deleting offline data.
  
I think that this bug https://bugs.kde.org/show_bug.cgi?id=304725 is the same bug (duplicate).

From my testing/observations and looking into the source code, there are a number of issues here and it's hard to separate them out.

General sync concept not working:
While the DAV protocol as such is correctly implemented, some actual sync issue resolution is completely lacking. If a "match" filter is not working and the server return a 412 "precondition failed", then the resource goes into offline/broken state by design. Which is totally wrong because such clashes can happen and do happen. Particularly with calendars etc accessed and updated by various users or devices.
The correct behavior would be one of the three options (to be configurable):
1) ask user for selection: reload latest or publish changes to server anyway
2) Server wins by default and reloads latest, discards user changes
3) Client wins by default and overwrites server

Is there anyone actively working on fixing this? If not, i would try to take a look at the code, because this bug is really annoying me and find it important

I am trapped in the this bug also. The DAV peer is horde 5.1.3.

Syncing of contacts stopped after a while and the error 401 appears. Now it is working againg after readding the source.

The calender syncing does not stop and stays at "100% synchronized" state. However, in the HTTP logs I see status code 401 also. Maybe this is related.

Conflict resolution: I think asking the whether the remote or the local change should win, in case of a conflict is a "non option". Idea: In case of a conflict the entry is doubled and tagged with "CONFLICT <TIMESTAMP>" in the name. This can happen without user interaction and no data is lost.

If someone wants to take a look in this, I am willing to do some sponsoring / funding if needed.

This patch corrects the behavior, please approve and release fixed version:

diff --git a/resources/dav/resource/settings.cpp b/resources/dav/resource/settings.cpp
index 931dab5..e9b0f93 100644
--- a/resources/dav/resource/settings.cpp
+++ b/resources/dav/resource/settings.cpp
@@ -291,6 +291,9 @@ Settings::UrlConfiguration * Settings::urlConfiguration( DavUtils::Protocol prot
 
 QString Settings::username( DavUtils::Protocol proto, const QString &url ) const
 {
+  if ( mUrls.isEmpty() )
+    buildUrlsList();
+
   QString key = url + QLatin1Char(',') + DavUtils::protocolName( proto );
 
   if ( mUrls.contains( key ) )
@@ -308,6 +311,9 @@ QString Settings::username( DavUtils::Protocol proto, const QString &url ) const
 
 QString Settings::password(DavUtils::Protocol proto, const QString& url)
 {
+  if ( mUrls.isEmpty() )
+    buildUrlsList();
+
   QString key = url + QLatin1Char(',') + DavUtils::protocolName( proto );
 
   if ( mUrls.contains( key ) )

(In reply to frynoh from comment #19)
> This patch corrects the behavior, please approve and release fixed version:
> 

Not quite sure what exactly this patch will solve, however it will not solve the main issue that there is absolutely no conflict resolution within the resource.
The code is quite clear on what happens: In case of ANY error, go to the retry function which does: emit broken, go offline. It does not exactly do what the name would imply... ;-)

I have started to work on the resource again, but it will take some time to resolve the matter, it requires some major rewrite.

The way the resource is designed is the following:

Say we delete an event from Calendar in KOrganizer:

1) Akonadi calls item-deleted function in resource
2) Resource grabs url and etag and send a delete request to the caldav server together with an if-match header and the stored etag.
3) If everything worked, resource deletes item from akonadi collection
Now comes the tricky bit:
If the above DOES NOT WORK (because the item was changed meanwhile before we sent that delete request), the resource gets a 412 error back from the server and calls its retry function and that does emit broken, offline.

So in order for this to work, we need to introduce conflict resolution handling.

Ie in the above, have three exclusive options: 1) ask the user for input, 2) delete anyway (we are the boss), 3) reload from server.

This currently does not work as the dav job is not given to the retry function, only the error message.

So I am working on some user configurable conflict resolution and properly handle the retry function.

In the above example of deleting an element:

1) Ask user for input and then do one of the below
2) Server wins: ignore delete request and start new job to retrieve updated item from DAV server - if that fails, either ignore silently or notify user
3) Client wins: Restart the job without the if-match header, delete the item. If that fails, delete item from collection only, and/or notify user

This all will need some proper handling of the various error codes, ie 401 and 501 probably should set the resource offline, 412 should update items, etc.
It will also need a check whether the server went away and the resource should then go offline too.

(In reply to Ingo Ratsdorf from comment #20)
> Not quite sure what exactly this patch will solve,

This solve the problem of the bug report, what Till Schafer say. When dav is back again then the calendar events are now created. Till forgot to add to restart the resource, else the bug will not reproduce.

> however it will not solve
> the main issue that there is absolutely no conflict resolution within the
> resource.

This is not the bug reported. Use another report.

This will fix the 401 indeed. 

Because the resource was offline, when it comes back online the first thing it tries to do is commit to the server the changes.

However, if my interpretation of the implementation of the resource is correct, the auth headers are established once at the beggining, when exploring for main connections and then carried on with the keep-alive 

This broke when the server had some kind of problem.

This fix WILL fix the 401, but only to leave the resource in the same state with a 412 error when, for example, the resource tries to remove a cal event already removed by a different client.

By the way, forgot to tell that the google calendar resource has in place some nice conflict resolution that could be reused (Check for conflicts and asks the use with both events side by side to allow the user to select conflict resolution)

I've been looking into this, but it may take some time to find a moment to properly fix it.

as far as i understand from your comments above, the fix will indeed fix the bug i reported, which is: 

- recommit a change after the server was unavailable at the fist commit for some reason and do not fall into broken state

However, there seem to be more than one trigger to cause the resource failing: 
1. Events are not removed from client if deleted on server: Bug 328734
2. Missing conflict resolution: The resource fails for conflicting modifications on client and server side. I have opened one more bug for this case: Bug 338570

Git commit 9e7377709da9f0aba758ea0d259a84b0c4858bc1 by Grégory Oestreicher.
Committed on 29/08/2014 at 15:23.
Pushed by goestreicher into branch 'KDE/4.14'.

Build URL list earlier

If the first task asked to the resource is not to fetch
all collections then the URL list is empty, resulting in
an authentication failure as no username/password will
be loaded. Thanks frynoh for a patch.

M  +3    -0    resources/dav/resource/settings.cpp

http://commits.kde.org/kdepim-runtime/9e7377709da9f0aba758ea0d259a84b0c4858bc1

Git commit c9a781f5b61813da80a31793bb6ce90b6f2e9046 by Grégory Oestreicher.
Committed on 31/08/2014 at 16:43.
Pushed by goestreicher into branch 'KDE/4.14'.

Don't unnecessarily go offline

Check if the error returned by the jobs while creating,
modifying or deleting items are transient or final
and act accordingly.
Related: bug 335833

M  +2    -1    resources/dav/common/davitemcreatejob.cpp
M  +2    -3    resources/dav/common/davitemcreatejob.h
M  +2    -1    resources/dav/common/davitemdeletejob.cpp
M  +2    -3    resources/dav/common/davitemdeletejob.h
M  +2    -1    resources/dav/common/davitemmodifyjob.cpp
M  +2    -3    resources/dav/common/davitemmodifyjob.h
A  +84   -0    resources/dav/common/davjobbase.cpp     [License: GPL (v2+)]
A  +78   -0    resources/dav/common/davjobbase.h     [License: GPL (v2+)]
M  +1    -0    resources/dav/resource/CMakeLists.txt
M  +20   -3    resources/dav/resource/davgroupwareresource.cpp

http://commits.kde.org/kdepim-runtime/c9a781f5b61813da80a31793bb6ce90b6f2e9046

thx a lot! … looking forward to see this in 4.14.1

Hi Grégory,

good to see you back. Thanks a lot for the work so far.
I had a look through the code and you seem not to handle error 412, "Precondition failed".
This is now separate bug #338570.

Hey Ingo,

(In reply to Ingo Ratsdorf from comment #29)
> I had a look through the code and you seem not to handle error 412,
> "Precondition failed".

Well actually 412 and 428 are considered unrecoverable errors, so an error message will be shown and the event not updated.

> This is now separate bug #338570.

Yup, I saw this bug and worked on the conflict resolution tonight. Now it's a bit tricky to get it working (at least I couldn't) because Akonadi only implement local/local conflicts, i.e. conflicts that happen when two items are updated locally by multiple applications. The local/remote conflicts are not implemented, though there's the idea in the code.

In the end I've tried to game the system by faking a local conflict but this did not work out because the resource is never notified of the choice made by the user and thus can't act accordingly (overwrite the remote item for example).

Another limiting point is that the conflict handler is declared in a private header that can't be used from the resources… I guess that the grunt of the work must be in Akonadi to implement the local/remote conflicts management. Actually I have a lot of work to bring the bugs back under control, so this is not a high priority, especially now that errors will be shown plainly, but it's clearly something that must be worked on.

Cheers,
Grégory

(In reply to Grégory Oestreicher from comment #27)
> Git commit c9a781f5b61813da80a31793bb6ce90b6f2e9046 by Grégory Oestreicher.
> Committed on 31/08/2014 at 16:43.

Great, thank you! I have applied the patch against kdepim-runtime-4.14.0 on my Gentoo box for testing.

Me too. 
Is there already a possibility to obtain the 4.14 with trusty?

@Wolf Drechsel

I tried it with ubuntu Backports, the error is the same on 4.14

@Wolf Drechsel

I tried it with ubuntu Backports, the error is the same on 4.14

I have the same errors with ownCloud 7.0.2. As client i user KDEpim 4.13.3 in Kubuntu 14.04.
Today i tried to reproduce the error again, with akonadiconsole running, but it seems to work now.

What attracted my attention, was that also modyfiyng events or contacts on the client caused the error:

Bei der Abfrage ist ein Problem aufgetreten. Der Eintrag wurde auf dem Server nicht verändert
Bei der Aktion „https://MYSERVER/owncloud/remote.php/carddav/addressbooks/USER/contacts/xxxxxxxx.vcf hochladen“ ist der unerwartete Fehler 401 aufgetreten. (401).

Only removing and adding the ressource again in Akonadi helps.  Other clients (CaldavSync on Android) are working well.

Can it be that there is some problem with the Akonadi's authentication at the server?

See comment #30.
For as long as Akonadi is not changed, there does not seem to be a fix for conflicts.
Some things were fixed, like authentication, but the conflict resolution seem to be hard and there's no fix yet. I had to abandon the use of Akonadi unfortunately as it was getting too hard and I have to rely on my calendar on a daily basis.

(In reply to Ingo Ratsdorf from comment #36)
> See comment #30.
> For as long as Akonadi is not changed, there does not seem to be a fix for
> conflicts.
> Some things were fixed, like authentication, but the conflict resolution
> seem to be hard and there's no fix yet. I had to abandon the use of Akonadi
> unfortunately as it was getting too hard and I have to rely on my calendar
> on a daily basis.

The missing Conflict resolution kills CalDAV with Akonadi and thus Kontact. Since I need to rely on my calendar on a daily basis as well, this throws me back to the buggy, laggy and really undesireable lightning plugin in Thunderbird.
it is reproduceable on Arch Linux as well.

I would highly appreciate if someone could give this a bit of love!

I totally agree with you, the lack of conflict management makes the whole dav resource unusable for serious work. Too bad because beside this, Kontact is a great piece of software! I'd really appreciate if someone could have a look into that. Unfortunatly I don't have the skills. I'd be happy to help with testing

I started to look into this a while back, but then real life® got in the way. It should be possible to take conflict resolution from google resource and port it to the Dav resource, but someone has to do it :/

(In reply to yves from comment #38)
Me too, just tell me what data (logs etc.) i should commit.

Is there any development here? Akonadi is totally unuseable this way.

Unfortunately this critical bug exists for years now, and nobody seems to be interested (skilled?) to fix it.
I like Kontact, but meanwhile I only use it to look up telephone numbers. What a waste. Still searching for a nice desktop companion for my OwnCloud server.

It is very unlikely that this will be fixed as the kde PIM team is now moving to KF5 and plans a release of "Akonadi Next" also called Akonadi 2, a new implemention of Akonadi. As stated here they don't have enough developers and will have to drop some PIM application as well as Akonadi resources. They did not mention the dav resource though...

 https://www.mail-archive.com/kde-devel@kde.org/msg06508.html

Oh! This sounds like good news to me. Sometimes it might be better to design things new when they come to age rather than putting a patch here and an extension there.
Good luck for the relaunch.

Akonadi 1 has never got usable over years (for me, but I seem not to be the only one). If I read the post linked above, I see again good ideas for Akonadi Next - but Akonadi 1 also was a good idea in theory, I suppose...

Given the obvious lack of manpower, I don't know if it wouldn't be better to develop reliable, no-nonsense and solid PIM apps for KDE instead of again putting all ressources into a lofty framework with many theoretic advantages, but lots of shortcomings for the practical use.

(In reply to Larx from comment #45)
> Akonadi 1 has never got usable over years (for me, but I seem not to be the
> only one). If I read the post linked above, I see again good ideas for
> Akonadi Next - but Akonadi 1 also was a good idea in theory, I suppose...
> 
> Given the obvious lack of manpower, I don't know if it wouldn't be better to
> develop reliable, no-nonsense and solid PIM apps for KDE instead of again
> putting all ressources into a lofty framework with many theoretic
> advantages, but lots of shortcomings for the practical use.

Maybe "not usable" is a bit overstating (and ungenerous), but from my experience I cannot say better than "painful". Larx's opinion therefore seems like a reasonable (or: sensible) line of reasoning to me. In spite of this, it could still be totally incorrect, if the aim of framework rewriting is precisely at targeting some fundamental and unresolved source of problems affecting the first version.

I define "usable" concerning PIM software as "I trust my valuable PIM data to this piece of software", and Akonadi never reached this goal: 
- I lost mails (yes, you read everywhere that Akonadi is only a kind of cache and therefore never can trash data, but this is exactly what it did several times, probably during copy/move actions), 
- I always had to double check any appointments I entered in KOrganizer, as many times it simply failed - silently - because of some obscure conflict, only solvable by re-creating the ressources, 
- it messed up my carefully organized contact groups with "akonadi=?tag" entries for some unexplainable reason
- and so on.

All that not only once in a while, but very, very often. I sticked very long to Akonadi, as I'd really like to have a KDE-native PIM suite, but one day it was simply too much: Everything works fine with Evolution and Thunderbird (both having their own flaws, to be sure), it works fine with quite a lot of different (and open source) DAV connectors on Android, it works fine when using the ownCloud web interface (or other Groupware-suites, btw). So there are quite a lot of open source PIM solutions out there which work reliable, only none for the KDE desktop.

Sorry for the rant... I'm just afraid that the same game starts anew with Akonadi Next, when seemingly more than half of the code lacks proper maintainership, but the goal is still the ultimate all-in-one PIM solution.

although the general situation may be not optimal: lets keep this a technical and problem focused bug please.

i also ended up having a broken akonadi resource w/o noticing for a week. so i added and edited contacts and events on my pc using kontact and on  my mobile phone. now i need to clean up the mess:

to resolve the conflicts in my address book i created a new akonadiresource (NEW) and renamed the old to BROKEN. i exported both address books to csv file using kaddressbook and did a diff (using kompare).
then i manually added the changes made to BROKEN.csv to the NEW resource (not optimal - eg when a phone number has been changed you need to remember which phone number is the correct one, the one you added to NEW using your mobile phone, or the one in BROKEN that might have been set using the PC.

for the events i exported both calendars to ical files in korganizer, but finding changes using diff is impossible, or at least very painful.
one can compare the calendars in korganizer week by week pretty well, but events (especially those a year or more in the future) can easily be overlooked.


the best solution would be to find a way to see which changes where made to the broken calendar (and address book) that have not been written to the server (the akonadi cache or changelog).

as this bug might not be fixed at all, we might come up with a decent workaround.

@Till Schäfer: what is the agent_config_akonadi_davgroupware_resource_0_changes.dat file you where writing about?

Due to another error, i had to reinstall Kubuntu. Now it seems to work (4.13.3).

@Harald Frießnegger: Maybe a workaround is removing agent_config_akonadi_davgroupware_resource_0_changes.dat when the error occurs again?

I also noticed that the problems sometimes seemed to be gone after a new, fresh installation. However, they always returned after a certain time.

To resume the current situation: 
- AFAIK The bug "Broken State" is resolved in 4.14.1. The bug "missing conflict resolution" is another one (see  Bug 338570)

@Harald: Is this bug still occurring in a version greater or equal than 4.14.1 ?

Otherwise i would like to close this bug.

@Harald: As a Layman in this area: The *changes.dat files are binary files which record not yet committed changes. You can find an exact specification of the format in kdepimlibs/akonadi/src/core/changerecorder_p.cpp.

Thanks for your feedback

@Sammy Bresel: removing  ~/.config/akonadi/agent_config_akonadi_davgroupware_resource_<N>_changes.dat might help, but i'll still need to export addressbooks and calendars first in order to check which changes would get lost.
so i don't really see the benefit of removing the changes file instead of adding a new resource.


@Till Schäfer: i'm running 4.14.2 and ran into the broken-state-problem :-(

thanks for the pointers to the changes format (here a link to the source file - just for the record: https://projects.kde.org/projects/kde/kdepimlibs/repository/revisions/master/entry/akonadi/src/core/changerecorder_p.cpp)

a script that can turn the contents of the changes.dat files into a human readable format would be nice for people having the same problem. however, writing such a script is out of my comfort zone unfortunately.

Git commit 44d38d363d6ff7b4f8c892becfb58a0339dbef0e by Grégory Oestreicher.
Committed on 18/06/2015 at 20:59.
Pushed by goestreicher into branch 'KDE/4.14'.

Add simple conflict handling

The conflict dialog will only be shown for local update / remote update
conflicts. For cases where one side deleted the item while it was updated
on the other, the deleting side loses the conflict automatically to prevent
data loss.
Related: bug 338570
FIXED-IN: 4.14.10

M  +30   -0    resources/dav/common/davitemdeletejob.cpp
M  +13   -0    resources/dav/common/davitemdeletejob.h
M  +5    -4    resources/dav/common/davitemfetchjob.cpp
M  +2    -3    resources/dav/common/davitemfetchjob.h
M  +32   -2    resources/dav/common/davitemmodifyjob.cpp
M  +13   -0    resources/dav/common/davitemmodifyjob.h
M  +5    -0    resources/dav/common/davjobbase.cpp
M  +5    -0    resources/dav/common/davjobbase.h
M  +168  -8    resources/dav/resource/davgroupwareresource.cpp
M  +8    -0    resources/dav/resource/davgroupwareresource.h

http://commits.kde.org/kdepim-runtime/44d38d363d6ff7b4f8c892becfb58a0339dbef0e

cool thx.

Looking forward to test this !! Thank you so much for the work ! this is great if it finally would work!

I had given up on that - thanks for (hopefully) fixing the bug!

still present (ubuntu 15.10 with backports, Akonadi 5.0.51).
CalDAV ressource: "Offline, Broken", "Invalid Address (URL): . (0)"