Bug 252082

Summary: CalDAV - Created events don't get synced to google calendar if this calendar isn't your primary one
Product: [Frameworks and Libraries] Akonadi Reporter: Johannes K. <joker777vs.666>
Component: DAV ResourceAssignee: Grégory Oestreicher <greg>
Status: RESOLVED FIXED    
Severity: normal CC: greg, martin.tlustos
Priority: NOR    
Version: 4.5   
Target Milestone: ---   
Platform: Compiled Sources   
OS: Linux   
Latest Commit: Version Fixed In:
Sentry Crash Report:
Attachments: not dropdown field --> it works
here I set my e-mail address in the system settings
and now you can see this dropdown menu

Description Johannes K. 2010-09-23 00:14:07 UTC
Version:           4.5 (using Devel) 
OS:                Linux

Everything is fine with syncing to my primary google calendar but creating new events with other google calendar just don't work at all. If I create a new event in Kontact it just get deleted on the next sync.
Modifying already existing events works though.

Reproducible: Always
Comment 1 Grégory Oestreicher 2010-10-13 07:25:40 UTC
Hi,

How have you configured your secondary calendars? I've just tested here and everything is working fine, both with primary and secondary calendars.

I created two calendars in the same resource, the primary with https://www.google.com/calendar/dav/[email]/events and the secondary with https://www.google.com/calendar/dav/[calendarid]/events.

Cheers,
Grégory
Comment 2 Johannes K. 2010-10-14 23:16:06 UTC
Hello Grégory

At first thank you very much for your answer and sorry for my late response. Yes after I updated my system to the current kubuntu 10.10 it works better but not always. For example when I'm logged in to google at the same time it quite often complains that it cannot authenticate to google though my username and passwort is definitely right. Than I have to click several times cancel until the popup doesn't show up anymore. It even happens sometimes when I sync to often. I also have to say that at the moment I've got 6 Calenders synchronising to google. This information could be propably important. The good thing is that my new created events doesn't get deleted anymore but they are not always get synced and it easily happens that after I tried several times to create an event google tells me "Sorry, Calendar is temporarily unavailable. Please try again in a few minutes. If the problem persists, visit the Help Centre" when I login to the web calendar.
Does this happens to you as well?

greetings

Johannes
Comment 3 Grégory Oestreicher 2010-10-15 09:42:45 UTC
Hi Johannes,

No problems for the "late" answer. You had to wait longer than I.

Your problem looks a lot like this one: https://bugs.kde.org/show_bug.cgi?id=254123 and this is the first time I see this. All my tests with Google never used more than three calendars.

In both cases there are many calendars configured, and I think this may trigger some kind of protection from Google as there is always a first connection made without user / password. This is speculation though, and I'll test the resource more thoroughly with this scenario.

Thanks a lot for the information, I'll update this bug with new findings.

Cheers,
Grégory
Comment 4 Grégory Oestreicher 2010-10-15 09:48:02 UTC
*** Bug 254123 has been marked as a duplicate of this bug. ***
Comment 5 Grégory Oestreicher 2010-10-16 09:52:39 UTC
Hi,

Despite having the resource run for 12 hours with 6 Google calendars I couldn't trigger the same behaviour.

Do any of you can test the same thing with 6 calendars that are not hosted on Google? A great playground is the Davical test server provided here: http://caldav-test.ioda.net/ (scroll down a bit to have the list of available calendars). If the same problem happens then I definitely did not test right, and am eager to have more informations about the steps you performed.

Thanks,
Grégory
Comment 6 Johannes K. 2010-10-21 21:24:23 UTC
hey Grégory
sorry again for mor late response...it's quite bussy here.
This morning I tried to use the caldav links of Davical after I added some calendar (inclcuding the archive) Kontact started to eat my RAM. I've got 2 GB Kontact processed 700 MB ... do you experience a similar behavior?

In terms of the strange authentification behaviour I have to say that I'm testing two different system. My pc which as an upgraded Kubuntu 10.10 and my Laptop which has a freshly installed Kubuntu 10.10. The Laptop behaves wonderful ;-) but the pc doesn't ...but I don't know why...anyway...both of them have this RAM problem...

By the way...how can I see if an event was added to a Davical calendar?

Thanks a lot

Johannes
Comment 7 Johannes K. 2010-10-21 22:14:54 UTC
another thing
since I created these calendars of Davical Akonadi is crashing at regular intervals.
Here is the crash report. I would like to make the crash report more precise but I don't know how to build kdepim with debug symbols. Could you explain me how to do this?


Application: Akonadi Resource (akonadi_davgroupware_resource), signal: Segmentation fault
[KCrash Handler]
#6  0x000000000041c539 in DavItemsListJob::start() ()
#7  0x000000000042e66c in DavGroupwareResource::retrieveItems(Akonadi::Collection const&) ()
#8  0x00007f905247c665 in Akonadi::ResourceBasePrivate::slotSynchronizeCollection (this=<value optimized out>, col=<value optimized out>) at ../../akonadi/resourcebase.cpp:482
#9  0x00007f905247dd4e in Akonadi::ResourceBase::qt_metacall (this=0x234f0c0, _c=QMetaObject::InvokeMetaMethod, _id=<value optimized out>, _a=0x7fff12595b10) at ./moc_resourcebase.cpp:121
#10 0x0000000000433565 in DavGroupwareResource::qt_metacall(QMetaObject::Call, int, void**) ()
#11 0x00007f9053f9fb27 in QMetaObject::activate (sender=0x22ef3e0, m=<value optimized out>, local_signal_index=<value optimized out>, argv=0x2351f10) at kernel/qobject.cpp:3280
#12 0x00007f905247e7d5 in Akonadi::ResourceScheduler::executeCollectionSync (this=0x0, _t1=<value optimized out>) at ./resourcescheduler_p.moc:131
#13 0x00007f9052480ecc in Akonadi::ResourceScheduler::executeNext (this=0x22ef3e0) at ../../akonadi/resourcescheduler.cpp:247
#14 0x00007f9052481d58 in Akonadi::ResourceScheduler::qt_metacall (this=0x22ef3e0, _c=QMetaObject::InvokeMetaMethod, _id=<value optimized out>, _a=0x227f360) at ./resourcescheduler_p.moc:113
#15 0x00007f9053f99bde in QObject::event (this=0x22ef3e0, e=0x0) at kernel/qobject.cpp:1219
#16 0x00007f9050259fdc in QApplicationPrivate::notify_helper (this=0x21ea950, receiver=0x22ef3e0, e=0x2324c40) at kernel/qapplication.cpp:4396
#17 0x00007f905025faed in QApplication::notify (this=0x7fff12596810, receiver=0x22ef3e0, e=0x2324c40) at kernel/qapplication.cpp:4277
#18 0x00007f9051536156 in KApplication::notify (this=0x7fff12596810, receiver=0x22ef3e0, event=0x2324c40) at ../../kdeui/kernel/kapplication.cpp:310
#19 0x00007f9053f87cdc in QCoreApplication::notifyInternal (this=0x7fff12596810, receiver=0x22ef3e0, event=0x2324c40) at kernel/qcoreapplication.cpp:732
#20 0x00007f9053f8ac22 in sendEvent (receiver=0x0, event_type=<value optimized out>, data=0x21d23e0) at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:215
#21 QCoreApplicationPrivate::sendPostedEvents (receiver=0x0, event_type=<value optimized out>, data=0x21d23e0) at kernel/qcoreapplication.cpp:1373
#22 0x00007f9053fb4653 in sendPostedEvents (s=0x21ee430) at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:220
#23 postEventSourceDispatch (s=0x21ee430) at kernel/qeventdispatcher_glib.cpp:277
#24 0x00007f904eab5342 in g_main_context_dispatch () from /lib/libglib-2.0.so.0
#25 0x00007f904eab92a8 in ?? () from /lib/libglib-2.0.so.0
#26 0x00007f904eab945c in g_main_context_iteration () from /lib/libglib-2.0.so.0
#27 0x00007f9053fb4193 in QEventDispatcherGlib::processEvents (this=0x21d1ad0, flags=<value optimized out>) at kernel/qeventdispatcher_glib.cpp:415
#28 0x00007f905030ca4e in QGuiEventDispatcherGlib::processEvents (this=0x0, flags=<value optimized out>) at kernel/qguieventdispatcher_glib.cpp:204
#29 0x00007f9053f86a02 in QEventLoop::processEvents (this=<value optimized out>, flags=) at kernel/qeventloop.cpp:149
#30 0x00007f9053f86dec in QEventLoop::exec (this=0x7fff125967a0, flags=) at kernel/qeventloop.cpp:201
#31 0x00007f9053f8aebb in QCoreApplication::exec () at kernel/qcoreapplication.cpp:1009
#32 0x00007f905247a0a2 in Akonadi::ResourceBase::init (r=0x234f0c0) at ../../akonadi/resourcebase.cpp:245
#33 0x000000000042d0d8 in main ()
Comment 8 Grégory Oestreicher 2010-10-22 11:32:19 UTC
Hi Johannes,

There are three things, so let's start with the easy one, the crash. This has been fixed in trunk. There was a problem if you changed the protocol in the resource configuration screen after the initial configuration. To fix this you can either delete the resource and configure a new one or manually edit the resource configuration file (named akonadi_davgroupware_resourceXrc in ~/.kde/share/apps/config) and change the protocol to CalDAV.

For the RAM usage I've never tested with so much events, so I guess it's time to do it. In this case you end up with ~10K items and this is a pretty big calendar. Do you know if it is Kontact, Korganizer or Akonadi that is eating up so much memory?

Regarding the authentication issues I must admit that I'm totally puzzled by what may be causing this. Looks like an unmigrated setting somewhere. Have you checked if there are invalid entries in KWallet? I'll have a look at this with a machine that is running KDE4.4 and update it to trunk.

Finally, what do you mean by "how can I see if an event was added to a Davical calendar?". There should be no differences between Davical and other CalDAV implementations.

Cheers,
Grégory
Comment 9 Johannes K. 2010-10-26 08:23:59 UTC
Hi Grégory

Thanks for information about the crash...that helped. About the RAM...after some testing I realized that it only happens when I add the archive calendar and Kontact is eating the RAM ...
This authentification issue doesn't actually happens anymore...thats good. On the other hand the problem which made me opening this bug report is occuring again. My newly created events of my secondary calendars of google just doesn't get synced but get deleted. Though the event creation with Ioda-Net calendars works just fine...

I meant with "Davical calendar" the calendar of Ioda-Net and I was wondering if there is a possibility to see the events of the test calendars somhow graphically online. I now just download the file for comparing.
Thanks a lot

Johannes
Comment 10 Grégory Oestreicher 2010-10-28 23:10:35 UTC
SVN commit 1190772 by goestreicher:

CCBUG: 252082

Implement section 5.3.4 of RFC4791 and do not assume that ETag is always
present in responses. This may be the cause of issues with Google Calendar.



 M  +32 -3     davgroupwareresource.cpp  
 M  +9 -0      davgroupwareresource.h  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1190772
Comment 11 Grégory Oestreicher 2010-10-28 23:23:38 UTC
Hi Johannes,

I didn't have time to test the RAM usage unfortunately. However what I had time
to do is looking at the way Google is speaking CalDAV and they have a behavior
I've never seen before, but that is RFC compliant. Note that I still wasn't
able to reproduce your issue BTW. So the code is now doing what Google may
expect it to do. This may fix the issue you are having with secondary
calendars.

If you can test with the latest trunk, this would be excellent.

As for a way to view Davical events online, for Ioda.net demo server, I don't
know any. You may want to try with another CalDAV enabled client, such as
Thunderbird + Lightning or Sunbird.

Cheers,
Grégory
Comment 12 Johannes K. 2010-10-29 09:16:03 UTC
Hey Grégory

Thanks a log!! I saw the commit already yesterday and started to compile the trunk version of kdepim kdepimlibs and akonadi...I'm excited :-)

A remark to the ram useage...it only happens when I used the archive calendar and this is a really big one...

greetings

Johannes
Comment 13 Johannes K. 2010-10-29 10:46:49 UTC
so...I compiled now everything. The good thing is that the created event doesn't get deleted anymore but there aren't synced. :-(

Right now I'm compiling kdepim on a different notebook to test it there too.
greetings

Johannes
Comment 14 Grégory Oestreicher 2010-10-29 14:56:02 UTC
Hi Johannes,

Looks like I'll have to build a setup similar to yours as I can't reproduce this issue.

Besides the self-compiled akonadi, kdepimlibs and kdepim (from trunk), which KDE SC release are you using, on which distribution, is it a fresh install, etc. The more details you can give me the better.

Thanks,
Grégory
Comment 15 Johannes K. 2010-10-29 15:03:26 UTC
Hello Grégory

thanks...for helping me...
I use a Kubuntu 10.10 with kde 4.5.2
I don't know which else information could be useful for you.
thanks

Johannes
Comment 16 Martin Tlustos 2010-10-29 15:23:57 UTC
Hi Johannes,
just a comment from my side - it could make a difference whether your system was updated from 10.04 or done as a fresh install and whether you used any non-standard repositories (even in 10.04) - this kind of rendered my system unusable, so I had to do a clean install of 10.10
Martin
Comment 17 Johannes K. 2010-10-29 15:42:00 UTC
thank martin for the hint but i compiled kdepim kdepimlibs and akonadi on my laptop too because this is a fresh install of kubuntu 10.10. The pc at home is an upgrade...
both have the same behaviour.
Comment 18 Johannes K. 2010-10-30 15:52:55 UTC
Hey Guys
I just installed Kubuntu 10.10 on a virtual machine and compiled kdepim & co and it just works. I really don't know why it is not working on my pc and notebook but I but on the virtual machine. I will try to investigate it...
Sorry for the trouble

Johannes
Comment 19 Grégory Oestreicher 2010-10-31 11:32:17 UTC
Hi Johannes,

I'm having troubles having everything working fine on KUbuntu by just recompiling akonadi, kdepimlibs and kdepim… Haven't you recompiled everything in fact?

Cheers,
Grégory
Comment 20 Johannes K. 2010-10-31 12:46:18 UTC
Hi Grégory

Hmm...I installed Kubuntu 10.10, upgraded everything, upgraded kde to kde 4.5.2, fetched akonadi from the git repo and compiled it, got kdepimlibs from svn trunk and copiled it and finally kdepim. I had to install several additional packages mainly *-dev packages to be able to compile them and I tried to satisfy also the optional packages, in fact I installed all optional packages except assuan2 for kdepim which I couldn't find in the repository sources...
How does the troubles look like?

greetings

Johannes
Comment 21 Johannes K. 2010-10-31 23:15:10 UTC
OK...I think I've found the problem...
I said that synchronizing of events does work on the virtual machine. I looked at the settings of these events which worked and realized that in the tab "Attendees" no "Meeting Organizer" is set...in fact there is noe dropdown field but when I close the event and just move the cursor to this event the information which shows up tells me that the Organizer is "the name of the Google Calendar" (in my case for example "Standard" or "Important"). In this Case everything works but when I set my e-mail address in "System Settings" > "Account Details" korganizer just uses this profile (the email adress is the same as the primary Google Calendar) as the Organizer for every new created event.
Now when I look at the details of an event there is a dropdown field next to "Meeting Organizer" which is set by default to my email address which I just set in the system settings.
In my virtual box this dropdown menu contains two entries "Johannes Köhler <joker777vs.666@gmail.com> and "Johannes Köhler <>". When I change the selected value to "Johannes Köhler <>" the event gets synchronized properly and and the information hint (which popups when I move the cursor to the created event) shows the right Calnedar Name as the "Organizer".
Strangely I don't have these two pupup entries in my kontact instances on my pc and netbook. There I only have "Johannes Köhler <joker777vs.666@gmail.com" which I can not change to anything else and which makes my events not get synchronized.

I hope I explained it quite understandable. I will try to upload some screenshots...

Greetings

Johannes
Comment 22 Johannes K. 2010-10-31 23:42:18 UTC
Created attachment 53027 [details]
not dropdown field --> it works
Comment 23 Johannes K. 2010-10-31 23:43:25 UTC
Created attachment 53028 [details]
here I set my e-mail address in the system settings
Comment 24 Johannes K. 2010-10-31 23:45:24 UTC
Created attachment 53029 [details]
and now you can see this dropdown menu
Comment 25 Johannes K. 2010-10-31 23:54:37 UTC
In my last commit I mentioned that on my pc and netbook I don't even have these two entries in the drop down menu but only "Johannes Köhler <joker777vs.666@gmail.com>". This is probably due to the fact that I added the same e-mail address to my identity in the settings section of Mail in Kontact. When I change this e-mail address I've go two entries in this dropdown menu...the second one is this changed e-mail address. Unfortunately I cannot leave this email address in my identity settings section empty to test if than the synchronisation works.

I hope that helps...

Johannes
Comment 26 Grégory Oestreicher 2010-11-01 09:54:51 UTC
Hi Johannes,

Thanks a lot for all this troubleshooting, this is impressive!

I've tested this and indeed this can be reproduced. From the new event dialog, when the Organizer is set, it is also added as an attendee. Removing the attendee seems to do the trick for Google, as it will then display the event in the web interface (note that the event exists on the Google side, it is just not displayed).

I'll see with KOrg devs if it is normal (from a standards point of view) that the organizer is also added as attendee. Yeah, this means that this bug is not in the resource (phew!), but either on Google's side or on KOrg.

Thanks a lot again, I'll keep this bug updated with new informations.

Cheers,
Grégory
Comment 27 Johannes K. 2010-11-01 13:25:04 UTC
Hi Grégory

It's really a pleasure and I'm happy to was able to trac this bug down to a level where it is reproducible and understandable.
For me it also works with Organizer listed as an attendee.
I think the main problem is that the value set for the organizer has influence on the synchronisation process of the event.
At the moment for google it has to have the value which represents the calendar otherwise it doesn't get synchronized. This makes sense if the calendar is a shared calendar from another person but in my case they are mainly secondary calendars created by myself and the organizer shouldn't be some kind of empty string but my personal identity with the email address of my primary google calendar.
What I just called "empty string" get changed anyway when I add an email address to my identity of my e-mail accounts and than I cannot set the Organizer to this "empty string" hence I cannot synchronize any new events anymore.
What do you think about this? If you want we can discuss this via icq/skype/jabber or I could even show you what I mean via teamviewer....

Johannes
Comment 28 Grégory Oestreicher 2010-11-03 23:03:10 UTC
Hi Johannes,

Sorry for the silence but I had little time to spend on my KDE activities. I've made a series of commits in kdepim that fix this issue for me. Can you test the latest trunk?

Cheers,
Grégory
Comment 29 Johannes K. 2010-11-09 09:13:48 UTC
Hello Grégory

I recompiled the sources (which took me some time) and now the organizer isn't listed as an attendee anymore...thats cool. But the issue that the organizer (or it's e-mail address which represents it) gets overwritten when one sets an e-mail address in the default identity of kmail does still exists.
Personally I actually like this because when I create an event for my secondary google calendar the organizer should be me and not the secondary google calendar but in this case the synchronisation doesn't work anymore. Somehow the organizer at least has to have the e-mail address which represents the calendar...
thanks a lot...

Johannes
Comment 30 Grégory Oestreicher 2010-11-09 09:33:22 UTC
Hi Johannes,

The fact that the organizer is overwritten is an issue with Google Calendar. It seems to think that the calendar ID must be the organizer for I don't know which reason, and it is not possible, when fetching an event to decide whether the organizer must be replaced by a configured identity or not. I've tried to work around it without breaking compatibility and creating a Google-specific resource. Now if one of your identities has the same email address configured then KOrg will be able to find it.

Back to the original bug, do you still have issues with viewing events from Google's web interface or is that working?

Cheers,
Grégory
Comment 31 Johannes K. 2010-11-11 09:22:16 UTC
Hello Grégory

In the dropdown list for the organizers I can now switch between "'my name' <my emailaddress>" and "'my name' <>" the first entry is the same value of my identity in the system settings and when I set the organizer to the second entry the synchronisation process works fine...the good thing is that the second value keeps be selectable eventhough I add an email adress to the Identity...thats great...
Unfortunatelly the selected entry by default is the first one hence I always have to change this when I want to create an event.

I don't undestand this sentence:
"Now if one of your identities has the same email address configured
then KOrg will be able to find it."
What do you mean with korg?

Nevertheless I really appreciate your work...thanks a lot!!

Johannes
Comment 32 Johannes K. 2010-11-13 11:55:56 UTC
Hello

In the beginning I still could set the organizer to "'your name' <>" but than suddenly I couldn't do this anymore again. There were only these values available which are set in system settings and the identity of kmail.
Than I found this configuration possibility in Calendar -> General where the radio button is set by "Use email settings from System Settings". When I unmark it and just set the "Full name" field but leave the "Email address" field empty I can again the organizer of an event with the value "'your name' <>" and this makes the synchronisations working again...

Johannes
Comment 33 Grégory Oestreicher 2010-11-19 12:57:36 UTC
Hi Johannes,

This kind of tweaking (creating an identity without email address) should not be necessary.

When testing however I had a bug with Google's web interface: I created an event from KOrganizer into a secondary calendar and refreshed the web view using the "Refresh" link, but the event was not displayed. I had to click once on the calendar name in the rightmost list to make the event appear. The event was correctly stored, but not displayed.

Cheers,
Grégory
Comment 34 Grégory Oestreicher 2011-01-15 21:57:49 UTC
Fixed a real bug, but a part of the issue is on Google. Closing.

Cheers,
Grégory