Bug 406917

Summary: Sync of webdav calendar stuck forever
Product: [Frameworks and Libraries] Akonadi Reporter: MartinG <gronslet>
Component: DAV ResourceAssignee: kdepim bugs <kdepim-bugs>
Status: CONFIRMED ---    
Severity: normal CC: ad1rie3, auxsvr, bugs.kde.org, csabi, kde.org, kfunk, rigo, torotil, ts+kde
Priority: NOR    
Version: unspecified   
Target Milestone: ---   
Platform: Other   
OS: Linux   
Latest Commit: Version Fixed In:

Description MartinG 2019-04-26 09:57:28 UTC
SUMMARY

When akonadi fails to read a calendar event from eg. a Nextcloud/Owncloud remote calendar (or any webdav resource I guess), it will enter a stale state making all other calendar interactions impossible

STEPS TO REPRODUCE
1. Setup a webdav remote calendar in korganizer
2. Be unlucky and end up with an event that for some reason is expected to exist, but it doesn't
3. Register state "Fetching collection (0 %)" or "Ready (0%)" forever. Not possible to get out of that state with "akonadictl restart" or even restart of computer.


OBSERVED RESULT

DAV resource becomes unusable.

EXPECTED RESULT

Akonadi should handle the bad state in some way, and move on, letting me use the calendar resource.

SOFTWARE/OS VERSIONS
linux 5.0.9-200.fc29.x86_64
kdelibs-common-4.14.38-9.fc29.x86_64
kdelibs-4.14.38-9.fc29.x86_64
korganizer-libs-18.12.2-1.fc29.x86_64
kontact-18.12.2-1.fc29.x86_64
kontact-libs-18.12.2-1.fc29.x86_64
kdelibs-webkit-4.14.38-9.fc29.x86_64
kf5-kdelibs4support-5.55.0-1.fc29.x86_64
kf5-kdelibs4support-libs-5.55.0-1.fc29.x86_64
kdelibs-ktexteditor-4.14.38-9.fc29.x86_64
korganizer-18.12.2-1.fc29.x86_64

Linux/KDE Plasma: 5.14.5
KDE Plasma Version: 5.14.5
KDE Frameworks Version: 5.55.0
Qt Version: 5.11.3

ADDITIONAL INFORMATION

I'm running Nextcloud 15.0.7, and have set up the calendar as a webdav resource. At some point, it stopped working, and in Kontact->Settings->Configure korganizer->General->Calendars, the resource is stuck at "Fetching collection (0 %)" or "Ready (0 %)" forever.

If I run "akonadictl restart" in the terminal, I get the following:

org.kde.pim.davresource: Error when uploading item: 420 "There was a problem with the request. The item was not modified on the server.\nAn unexpected error (412) occurred while attempting to upload https://USERNAME@MYURL/NEXTCLOUNDNAME/remote.php/dav/calendars/USERNAME/CALENDARNAME/1551289068.R271.ics. (412)."
parse error from icalcomponent_new_from_string. string= "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<d:error xmlns:d=\"DAV:\" xmlns:s=\"http://sabredav.org/ns\">\n  <s:exception>Sabre\\DAV\\Exception\\NotFound</s:exception>\n  <s:message>Calendar object not found</s:message>\n</d:error>\n"
org.kde.pim.davresource: DavGroupwareResource::onItemFetched: Failed to parse item data.  "https://USERNAME@MYURL/NEXTCLOUNDNAME/remote.php/dav/calendars/USERNAME/CALENDARNAME/1551289068.R271.ics"


Here is the corresponding error on the nextcloud server (15.0.7, nextcloud.log):

{"reqId":"BX7CkUh0Wd4i4Vl9crBA",
 "level":4,
 "time":"2019-04-26T09:49:21+00:00",
 "remoteAddr":"192.168.1.1",
 "user":"USERNAME",
 "app":"webdav",
 "method":"PUT",
 "url":"\/NEXTCLOUDNAME\/remote.php\/dav\/calendars\/USERNAME\/CALENDARNAME\/1551289068.R271.ics",
 "message":{"Exception":"Sabre\\DAV\\Exception\\BadRequest",
	    "Message":"Calendar object with uid already exists in this calendar collection.",
	    "Code":0,
	    "Trace":[{"file":"\/var\/www\/html\/NEXTCLOUDNAME\/3rdparty\/sabre\/dav\/lib\/CalDAV\/Calendar.php",
		      "line":201,
		      "function":"createCalendarObject",
		      "class":"OCA\\DAV\\CalDAV\\CalDavBackend",
		      "type":"->",
		      "args":["3",
			      "1551289068.R271.ics",
			      "BEGIN:VCALENDAR\r\nPRODID:-\/\/K Desktop Environment\/\/NONSGML libkcal 4.3\/\/EN\r\nVERSION:2.0\r\nX-KDE-ICAL-IMPLEMENTATION-VERSION:1.0\r\nBEGIN:VEVENT\r\nDTSTAMP:20190227T174129Z\r\nCREATED:20190227T173728Z\r\nUID:a151d2ad-e962-459f-a291-705071fe62ef\r\nSEQUENCE:2\r\nLAST-MODIFIED:20190227T174129Z\r\nDESCRIPTION:Test\\,
 with commas\r\nSUMMARY:Test comma\r\nLOCATION:Location\\,
 with comma\r\nDTSTART;VALUE=DATE:20190227\r\nDTEND;VALUE=DATE:20190228\r\nTRANSP:OPAQUE\r\nEND:VEVENT\r\nEND:VCALENDAR\r\n"]},
		     {"file":"\/var\/www\/html\/NEXTCLOUDNAME\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php",
		      "line":1096,
		      "function":"createFile",
		      "class":"Sabre\\CalDAV\\Calendar",
		      "type":"->",
		      "args":["1551289068.R271.ics",
			      "BEGIN:VCALENDAR\r\nPRODID:-\/\/K Desktop Environment\/\/NONSGML libkcal 4.3\/\/EN\r\nVERSION:2.0\r\nX-KDE-ICAL-IMPLEMENTATION-VERSION:1.0\r\nBEGIN:VEVENT\r\nDTSTAMP:20190227T174129Z\r\nCREATED:20190227T173728Z\r\nUID:a151d2ad-e962-459f-a291-705071fe62ef\r\nSEQUENCE:2\r\nLAST-MODIFIED:20190227T174129Z\r\nDESCRIPTION:Test\\,
 with commas\r\nSUMMARY:Test comma\r\nLOCATION:Location\\,
 with comma\r\nDTSTART;VALUE=DATE:20190227\r\nDTEND;VALUE=DATE:20190228\r\nTRANSP:OPAQUE\r\nEND:VEVENT\r\nEND:VCALENDAR\r\n"]},
		     {"file":"\/var\/www\/html\/NEXTCLOUDNAME\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php",
		      "line":525,
		      "function":"createFile",
		      "class":"Sabre\\DAV\\Server",
		      "type":"->",
		      "args":["calendars\/USERNAME\/CALENDARNAME\/1551289068.R271.ics",
			      "BEGIN:VCALENDAR\r\nPRODID:-\/\/K Desktop Environment\/\/NONSGML libkcal 4.3\/\/EN\r\nVERSION:2.0\r\nX-KDE-ICAL-IMPLEMENTATION-VERSION:1.0\r\nBEGIN:VEVENT\r\nDTSTAMP:20190227T174129Z\r\nCREATED:20190227T173728Z\r\nUID:a151d2ad-e962-459f-a291-705071fe62ef\r\nSEQUENCE:2\r\nLAST-MODIFIED:20190227T174129Z\r\nDESCRIPTION:Test\\,
 with commas\r\nSUMMARY:Test comma\r\nLOCATION:Location\\,
 with comma\r\nDTSTART;VALUE=DATE:20190227\r\nDTEND;VALUE=DATE:20190228\r\nTRANSP:OPAQUE\r\nEND:VEVENT\r\nEND:VCALENDAR\r\n",
			      null]},
		     {"function":"httpPut",
		      "class":"Sabre\\DAV\\CorePlugin",
		      "type":"->",
		      "args":[{"absoluteUrl":"https:\/\/MYURL\/NEXTCLOUDNAME\/remote.php\/dav\/calendars\/USERNAME\/CALENDARNAME\/1551289068.R271.ics",
			       "__class__":"Sabre\\HTTP\\Request"},
			      {"__class__":"Sabre\\HTTP\\Response"}]},
		     {"file":"\/var\/www\/html\/NEXTCLOUDNAME\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php",
		      "line":105,
		      "function":"call_user_func_array",
		      "args":[[{"__class__":"Sabre\\DAV\\CorePlugin"},
			       "httpPut"],
			      [{"absoluteUrl":"https:\/\/MYURL\/NEXTCLOUDNAME\/remote.php\/dav\/calendars\/USERNAME\/CALENDARNAME\/1551289068.R271.ics",
				"__class__":"Sabre\\HTTP\\Request"},
			       {"__class__":"Sabre\\HTTP\\Response"}]]},
		     {"file":"\/var\/www\/html\/NEXTCLOUDNAME\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php",
		      "line":479,
		      "function":"emit",
		      "class":"Sabre\\Event\\EventEmitter",
		      "type":"->",
		      "args":["method:PUT",
			      [{"absoluteUrl":"https:\/\/MYURL\/NEXTCLOUDNAME\/remote.php\/dav\/calendars\/USERNAME\/CALENDARNAME\/1551289068.R271.ics",
				"__class__":"Sabre\\HTTP\\Request"},
			       {"__class__":"Sabre\\HTTP\\Response"}]]},
		     {"file":"\/var\/www\/html\/NEXTCLOUDNAME\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php",
		      "line":254,
		      "function":"invokeMethod",
		      "class":"Sabre\\DAV\\Server",
		      "type":"->",
		      "args":[{"absoluteUrl":"https:\/\/MYURL\/NEXTCLOUDNAME\/remote.php\/dav\/calendars\/USERNAME\/CALENDARNAME\/1551289068.R271.ics",
			       "__class__":"Sabre\\HTTP\\Request"},
			      {"__class__":"Sabre\\HTTP\\Response"}]},
		     {"file":"\/var\/www\/html\/NEXTCLOUDNAME\/apps\/dav\/lib\/Server.php",
		      "line":301,
		      "function":"exec",
		      "class":"Sabre\\DAV\\Server",
		      "type":"->",
		      "args":[]},
		     {"file":"\/var\/www\/html\/NEXTCLOUDNAME\/apps\/dav\/appinfo\/v2\/remote.php",
		      "line":35,
		      "function":"exec",
		      "class":"OCA\\DAV\\Server",
		      "type":"->",
		      "args":[]},
		     {"file":"\/var\/www\/html\/NEXTCLOUDNAME\/remote.php",
		      "line":163,
		      "args":["\/var\/www\/html\/NEXTCLOUDNAME\/apps\/dav\/appinfo\/v2\/remote.php"],
		      "function":"require_once"}],
	    "File":"\/var\/www\/html\/NEXTCLOUDNAME\/apps\/dav\/lib\/CalDAV\/CalDavBackend.php",
	    "Line":1059,
	    "CustomMessage":"--"},
 "userAgent":"Mozilla\/5.0 (X11; Linux x86_64) AppleWebKit\/534.34 (KHTML,
 like Gecko) akonadi_davgroupware_resource_0\/5.10.2 Safari\/534.34",
 "version":"15.0.7.0"}


---

Writing this, I realize that the calendar event that is troublesome, is one I created "Testing with commas", that probably is related to another bug, where commas in location or some other field doesn't work in all apps. 

How do I get out of this stale state?

Related:
https://forum.kde.org/viewtopic.php?f=261&t=159812
Comment 1 Christophe Marin 2019-04-26 17:08:54 UTC
I tested yesterday with demo.owncloud.com, I didn't get any issue.

I also have no issue with my regular caldav/carddav server
Comment 2 heimlicher 2019-05-07 10:15:02 UTC
I can confirm the above issue.
Trying to sync to my baikal server fails with a similar error.

akonadictl restart output:

parse error from icalcomponent_new_from_string. string= "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<d:error xmlns:d=\"DAV:\" xmlns:s=\"http://sabredav.org/ns\">\n  <s:sabredav-version>3.1.3</s:sabredav-version>\n  <s:exception>Sabre\\DAV\\Exception\\NotFound</s:exception>\n  <s:message>Calendar object not found</s:message>\n</d:error>\n"

org.kde.pim.davresource: DavGroupwareResource::onItemFetched: Failed to parse item data.  "https://xxxxx/dav.php/calendars/c/2/1557222588.R271.ics"
Comment 3 heimlicher 2019-05-07 10:39:49 UTC
After hitting  “Clear Akonadi Cache” in akonadiconsole the error disappeared.
Comment 4 csabi 2019-07-04 12:53:05 UTC
Hi, I have the same issue, it tries to load an event which doesn't exist on the server, and crashes.

I don't have a "Clear akonadi cache" button in the akonadi console. Where is that?
Comment 5 heimlicher 2019-07-05 22:59:56 UTC
(In reply to csabi from comment #4)
> Hi, I have the same issue, it tries to load an event which doesn't exist on
> the server, and crashes.
> 
> I don't have a "Clear akonadi cache" button in the akonadi console. Where is
> that?

Here is a screenshot that will help you find it: https://ibb.co/0DgmwzS
Comment 6 csabi 2019-07-06 09:54:30 UTC
Thank you.
The clash clearing did unstuck the calendar, so it can be used again (but removed a few events which were not saved to the server).

This bug is particularly problematic because the calendar doesn't show any sign that it stopped synchronizing. So on the computer everything looks fine, the problem is detected when the calendar is checked on another device.

It would be good to get a notification when the calendar is stuck for some reason.
Comment 7 Rigo Wenning 2019-07-09 10:00:22 UTC
Confirmation from here. I cleared the cache and now I have no items and no sync anymore. Because clearing the cache did not do the trick anymore. I also had a regression on this here:
https://www.dvratil.cz/2017/01/kmail-multiple-merge-candidates-error-and-how-to-fix-it/
I identified that the error wasn't in caldav, but in personal contacts and this also stopped the caldav sync
Comment 8 Rigo Wenning 2019-07-09 10:24:10 UTC
To unstuck mine, I had to remove and re-create the resource. Clearing was not enough anymore as it seems akonadi remembered to fetch this non-existent item despite clearing the cache and restarting the server.
Comment 9 Rigo Wenning 2019-07-09 14:46:35 UTC
I found the culprit. It was an ICS from Air Lingus that wasn't parsable. I removed it from the calendar and now it works again. Looks like there is an algorithm missing in the davgroup that throws away stuff that it can't digest and notify the user. This should be coordinatable with the folks from nextcloud as most people using kontact typically also use owncloud or nextcloud (as google has certainly done the sanitization already)
Comment 10 Tim Schlotfeldt 2019-09-19 20:12:03 UTC
I can confirm this. In run often into this while copying contact data (eg vcf files) from one carddav ressource to another.
Comment 11 Rigo Wenning 2019-09-19 22:00:28 UTC
Yes, my calendar is back in the no-sync status and entries are only within akonadi and not synced to disk or nextcloud. I think this is a serious bug as the algorithm as no plan B if it can't parse an entry.
Comment 12 MartinG 2019-09-25 19:36:16 UTC
I checked mine today, and mine is also back in none-sync mode. The sad thing is, that I have put in a number of events in Korganizer, that are now not shown in my shared Nextcloud calendar. I guess I will have to reset the connection and lose all of those. I tend to import flight events into my calendar, and those are often not fully sanitized. I'd much rather be confronted with an error when this happens, with an option to ignore and move on, rather than to be left in "offline mode" without knowing it!

Please let me know if any more info is needed.
Comment 13 csabi 2019-10-07 12:19:13 UTC
Completely agreed, we an error message and ignoring the problematic event is needed, not silently crash the calendar.

I was able to recover using the akonadi console, copying the events which were not synced - no remote id - to a new calendar, then back to the original calendar.
And couple restarts. They did get a new remote id after that.

BEGIN:VCALENDAR
PRODID:-//K Desktop Environment//NONSGML libkcal 4.3//EN
VERSION:2.0
X-KDE-ICAL-IMPLEMENTATION-VERSION:1.0
BEGIN:VEVENT
DTSTAMP:20191002T075812Z
URL:https:
 //www.eventbrite.com/e/codecontrol-community-meetup-7-storytelling-drinks-
 tickets-74687007851
CREATED:20191002T075812Z
LAST-MODIFIED:20191002T075812Z
DESCRIPTION:For details\, click here: https:
 //www.eventbrite.com/e/codecontrol-community-meetup-7-storytelling-drinks-
 tickets-74687007851
SUMMARY:CodeControl Community Meetup #7  (Storytelling & Drinks)
LOCATION:KollwitzstraÃe 91 - KollwitzstraÃe 91 - 10435 Berlin - 
 Germany
DTSTART;TZID=UTC:20191010T163000
DTEND;TZID=UTC:20191010T190000
TRANSP:OPAQUE
END:VEVENT

END:VCALENDAR


Now I have all events working, saved in Nextcloud. So not sure it was a parsing problem.

But: I have 2 events with empty UID's, and they don't show up in KOrganizer. But they show up in Nextcloud and on my phone.
They were added through KOrganizer.
Comment 14 xorshid 2019-10-22 18:05:24 UTC
*** This bug has been confirmed by popular vote. ***
Comment 15 ad1rie3 2020-03-16 09:18:05 UTC
https://bugs.kde.org/show_bug.cgi?id=418910

oups i have created duplicate
Comment 16 Rigo Wenning 2020-03-16 10:00:18 UTC
(In reply to csabi from comment #13)
If you can see the event in other calendars (I can see them on nextcloud web and my Jolla Phone), but not in korganizer, I suspect, this has to do with another bug (I don't have the number anymore) If you have imported the event from an invitation letter and not "accepted" the invite in korganizer. The event is in the calendar, but will not be shown in korganizer. You need to accept it again in korganizer and it will show up.
Comment 17 tjbp 2020-04-02 10:50:53 UTC
Possibly now fixed - see comment: https://bugs.kde.org/show_bug.cgi?id=418910#c1
Comment 18 Roman Zimmermann 2024-02-06 17:24:05 UTC
Definitely not fixed. I’m still experiencing this regularly. My last attempts to make this work again lead to losing my entire calendar. :-/
Comment 19 Rigo Wenning 2024-02-07 08:42:04 UTC
Akonadi is unfit for synchronization. If you use nextcloud or any other remote calendar, you risk your data. This is known to me and wasn't fixed since 10 years now and that's why I unfortunately had to abandon KDE-PIM. Since then, no issue whatsoever with my remote calendar. Akonadi is just not usable for production. Too much play and candy, too little serious protocol work.