Bug 418811

Summary: Events stored in the database not displayed in KOrganizer
Product: [Applications] korganizer Reporter: Carlos <carloscg>
Component: generalAssignee: kdepim bugs <kdepim-bugs>
Status: RESOLVED FIXED    
Severity: major CC: agzgtrtvquysegfqtl, auxsvr, dag, dvratil, earendil, ek+kde, emil.ostwald, enboig, helmar, Lukas, rigo
Priority: NOR    
Version: 5.22.3   
Target Milestone: ---   
Platform: Arch Linux   
OS: Linux   
Latest Commit: Version Fixed In:
Attachments: Images showing the problem.
ICS files to reproduce KOrganizer bug #418811

Description Carlos 2020-03-13 11:37:26 UTC
Created attachment 126759 [details]
Images showing the problem.

SUMMARY
Sometimes, events that are stored and visible in the Akonadi database are not displayed in KOrganizer. For example, I might have an event stored in "Personal Calendar" that I can see in the "Browser" tab of Akonadi Console but cannot see in KOrganizer. 

I'm attaching an image of this: On the top of the image, the event is seen in the browser tab. On the bottom of the image, we see that the event is not shown on the 12 of March (2020).

I'm unfortunately unable to debug the problem which I assume might be related to how KOrganizer queries the databse for events.

STEPS TO REPRODUCE
I don't know how to reproduce it. If need be, I can share an example event that has the problem however.


SOFTWARE/OS VERSIONS
Linux/KDE Plasma: 5.18
KDE Frameworks Version: 5.67.0
Qt Version: 5.14.1
Comment 1 Rigo Wenning 2020-05-07 12:07:16 UTC
Confirming here. I also had this, when a calendar invite was recorded into the calendar by some other tool (nextcloud for me) and I did not use kontact to record the invite. But the invitation email was still present in the system. Now accepting the invite also in kontact made the entry appear in korganizer. 

But I now have an even stranger issue: There is a recurring event. The date of the event was changed, so I changed "this instance" from 14 May to 07 May. First, the event disappeared completely, but showed up in akonadiconsole correctly (07 May). As this is a nextcloud calendar, I used my browser in nextcloud to change the event. Now the event shows up correctly in the nextcloud browser and on my mobile phone and in akonadiconsole. But it did not show up in kontact/korganizer. 
But then (after some fsck & restart) BIG surprise, the entry shows up on 11 May and, when opened, says 07 May. This looks pretty strange IMHO. Why would an event show up on 11 May when the entry correctly says 07 May?
Comment 2 Carlos 2020-05-07 12:22:03 UTC
I can confirm that the events that don't show up were created with an external tool. In my case, events get added to my (Exchange) calendar which are then fetched with davmail using CalDav.
Comment 3 Rigo Wenning 2020-05-07 12:37:41 UTC
This is related to my earlier bug report: 

https://bugs.kde.org/show_bug.cgi?id=377453
Comment 4 Rigo Wenning 2020-10-15 14:23:30 UTC
The same bug stroke again. But this time I accepted the invite with kontact 5.15.1 in Tumbleweed 20201002. Now the meeting shows up in nextcloud and on my jolla phone, but not in korganizer. Very strange indeed. I haven't moved the invitations. But I received 2 invitations to two different email addresses. Even accepting both did not help. 
In Akonadiconsole I find one meeting with RemoteID and the other one without. I deleted the one without RemoteID in akonadiconsole, now I have a dirty item.
Comment 5 Lukas Spies 2020-10-19 17:41:28 UTC
Yup, same issue here. Some events are not shown in KOrganizer even though they are visibile in Nextcloud and on my Android calendar app. It didn't seem to matter from where I aceepted the event, some events that are not shown I accepted from Thunderbird and some from KMail. However, there are also some events that were created from invitation that *are* visible in KOrganizer.

SOFTWARE/OS VERSIONS
Linux/KDE Plasma: 5.20.0
KDE Frameworks Version: 5.75.0
Qt Version: 5.15.1
KMail version: 5.15.2 (20.08.2)
Comment 6 emil.ostwald 2021-11-17 20:05:26 UTC
Same issue here. For me it occours, if I'm creating an event on my smartphone with an attendee. Regardless which calendar app I use. Tested it with two different apps on 2 smartphones.

Here is the raw output  of an event in akonadiconsole, which is not shown in Korganizer:

BEGIN:VCALENDAR
PRODID:-//K Desktop Environment//NONSGML libkcal 4.3//EN
VERSION:2.0
X-KDE-ICAL-IMPLEMENTATION-VERSION:1.0
BEGIN:VEVENT
ORGANIZER:MAILTO:mail@example.com
DTSTAMP:20211117T185728Z
ATTENDEE;CN=Name Lastname;RSVP=TRUE;PARTSTAT=NEEDS-ACTION;
 ROLE=REQ-PARTICIPANT;CUTYPE=UNKNOWN;X-UID=94301368699264:mailto:
 me@there.com
ATTENDEE;CN=name@cloud me. com;RSVP=TRUE;PARTSTAT=ACCEPTED;
 ROLE=REQ-PARTICIPANT;CUTYPE=INDIVIDUAL;X-UID=94301368916912:mailto:
 name@cloud.me.com
CREATED:20211117T185728Z
UID:ff4cbc4d-2201-4e09-b4fd-e6d43e763c21
LAST-MODIFIED:20211117T185728Z
SUMMARY:Test X new 3
STATUS:CONFIRMED
DTSTART;TZID=Europe/Berlin:20211119T070000
DTEND;TZID=Europe/Berlin:20211119T080000
TRANSP:OPAQUE
END:VEVENT
BEGIN:VTIMEZONE
TZID:Europe/Berlin
BEGIN:DAYLIGHT
TZNAME:CEST
TZOFFSETFROM:+0000
TZOFFSETTO:+0200
DTSTART:19800406T010000
RDATE:19800406T010000
END:DAYLIGHT
BEGIN:STANDARD
TZNAME:CET
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
DTSTART:19971026T030000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10
END:STANDARD
BEGIN:STANDARD
TZNAME:CET
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
DTSTART:19800928T030000
RRULE:FREQ=YEARLY;UNTIL=19961027T030000;BYDAY=-1SU;BYMONTH=9
RDATE:19950924T030000
END:STANDARD
BEGIN:DAYLIGHT
TZNAME:CEST
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
DTSTART:19810329T020000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=3
END:DAYLIGHT
END:VTIMEZONE

END:VCALENDAR


And this is the output of an event without an attendee. This event is shown in Korganizer:

BEGIN:VCALENDAR
PRODID:-//K Desktop Environment//NONSGML libkcal 4.3//EN
VERSION:2.0
X-KDE-ICAL-IMPLEMENTATION-VERSION:1.0
BEGIN:VEVENT
DTSTAMP:20211117T195434Z
CREATED:20211117T195434Z
UID:7860a8de-decf-4bec-b298-7c565757df90
LAST-MODIFIED:20211117T195434Z
SUMMARY:Test X new 4
STATUS:CONFIRMED
DTSTART;TZID=Europe/Berlin:20211119T050000
DTEND;TZID=Europe/Berlin:20211119T060000
TRANSP:OPAQUE
END:VEVENT
BEGIN:VTIMEZONE
TZID:Europe/Berlin
BEGIN:DAYLIGHT
TZNAME:CEST
TZOFFSETFROM:+0000
TZOFFSETTO:+0200
DTSTART:19800406T010000
RDATE:19800406T010000
END:DAYLIGHT
BEGIN:STANDARD
TZNAME:CET
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
DTSTART:19971026T030000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10
END:STANDARD
BEGIN:STANDARD
TZNAME:CET
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
DTSTART:19800928T030000
RRULE:FREQ=YEARLY;UNTIL=19961027T030000;BYDAY=-1SU;BYMONTH=9
RDATE:19950924T030000
END:STANDARD
BEGIN:DAYLIGHT
TZNAME:CEST
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
DTSTART:19810329T020000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=3
END:DAYLIGHT
END:VTIMEZONE

END:VCALENDAR


An event created in Nextcloud and shown in Korganizer with one attendee looks like this:

BEGIN:VCALENDAR
PRODID:-//K Desktop Environment//NONSGML libkcal 4.3//EN
VERSION:2.0
X-KDE-ICAL-IMPLEMENTATION-VERSION:1.0
BEGIN:VEVENT
ORGANIZER;CN=Name:MAILTO:me@there.com
DTSTAMP:20211117T191318Z
ATTENDEE;CN=Name Lastname;RSVP=TRUE;PARTSTAT=NEEDS-ACTION;
 ROLE=REQ-PARTICIPANT;CUTYPE=INDIVIDUAL;X-UID=93951309397392:mailto:
 me@me.com
CREATED:20211117T191253Z
UID:7658b6c7-c4b0-4757-a537-09364ad00764
SEQUENCE:2
LAST-MODIFIED:20211117T191318Z
SUMMARY:Test NC neu
STATUS:CONFIRMED
DTSTART;TZID=Europe/Berlin:20211119T080000
DTEND;TZID=Europe/Berlin:20211119T083000
TRANSP:OPAQUE
END:VEVENT
BEGIN:VTIMEZONE
TZID:Europe/Berlin
BEGIN:DAYLIGHT
TZNAME:CEST
TZOFFSETFROM:+0000
TZOFFSETTO:+0200
DTSTART:19800406T010000
RDATE:19800406T010000
END:DAYLIGHT
BEGIN:STANDARD
TZNAME:CET
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
DTSTART:19971026T030000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10
END:STANDARD
BEGIN:STANDARD
TZNAME:CET
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
DTSTART:19800928T030000
RRULE:FREQ=YEARLY;UNTIL=19961027T030000;BYDAY=-1SU;BYMONTH=9
RDATE:19950924T030000
END:STANDARD
BEGIN:DAYLIGHT
TZNAME:CEST
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
DTSTART:19810329T020000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=3
END:DAYLIGHT
END:VTIMEZONE

END:VCALENDAR

Events are synchronized between both smartphones and Nextcloud with DAVx5. 

Maybe this helps for debuggung.
Comment 7 emil.ostwald 2021-11-17 21:18:00 UTC
And another example. Changed the DAVx5 account name to something without an '@'. The organizer email address is filled with this account name. Now there is only the attendee I added to the event and not the organizer again as second attendee:

BEGIN:VCALENDAR
PRODID:-//K Desktop Environment//NONSGML libkcal 4.3//EN
VERSION:2.0
X-KDE-ICAL-IMPLEMENTATION-VERSION:1.0
BEGIN:VEVENT
ORGANIZER:MAILTO:nameCloudServer
DTSTAMP:20211117T202723Z
ATTENDEE;RSVP=TRUE;PARTSTAT=NEEDS-ACTION;ROLE=REQ-PARTICIPANT;
 CUTYPE=INDIVIDUAL;X-UID=93951310843280:mailto:me@me.com
CREATED:20211117T202723Z
UID:1db96c2b-6df8-41b2-b607-8fac529553d3
LAST-MODIFIED:20211117T202723Z
SUMMARY:Test H new 4
STATUS:CONFIRMED
DTSTART;TZID=Europe/Berlin:20211119T100000
DTEND;TZID=Europe/Berlin:20211119T110000
TRANSP:OPAQUE
BEGIN:VALARM
DESCRIPTION:Test H new 4
ACTION:DISPLAY
TRIGGER:-P1D
X-KDE-KCALCORE-ENABLED:TRUE
END:VALARM
END:VEVENT
BEGIN:VTIMEZONE
TZID:Europe/Berlin
BEGIN:DAYLIGHT
TZNAME:CEST
TZOFFSETFROM:+0000
TZOFFSETTO:+0200
DTSTART:19800406T010000
RDATE:19800406T010000
END:DAYLIGHT
BEGIN:STANDARD
TZNAME:CET
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
DTSTART:19971026T030000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10
END:STANDARD
BEGIN:STANDARD
TZNAME:CET
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
DTSTART:19800928T030000
RRULE:FREQ=YEARLY;UNTIL=19961027T030000;BYDAY=-1SU;BYMONTH=9
RDATE:19950924T030000
END:STANDARD
BEGIN:DAYLIGHT
TZNAME:CEST
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
DTSTART:19810329T020000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=3
END:DAYLIGHT
END:VTIMEZONE

END:VCALENDAR
Comment 8 Helmar Gerloni 2022-03-21 07:16:51 UTC
Created attachment 147643 [details]
ICS files to reproduce KOrganizer bug #418811
Comment 9 Helmar Gerloni 2022-03-21 07:18:03 UTC
Same problem here (KDE on Debian Testing, korganizer 21.08.1, plasma 5.24.3, KDE Frameworks 5.90).

I can reproduce the error with the attached ICS files (replace me@here.net with your mail address configured in KDE).

- example_Error_MeAndSomeoneElse.ics:
ATTENDEE:mailto:me@here.net
ATTENDEE:mailto:someone@else.net

- example_Error_MeOnly.ics:
ATTENDEE:mailto:me@here.net

- example_NoError_MeOnlyAccepted.ics:
ATTENDEE;PARTSTAT=ACCEPTED:mailto:me@here.net

- example_NoError_SomeoneElse.ics:
ATTENDEE:mailto:someone@else.net

- example_NoError_NoAttendee.ics:
(no ATTENDEE entry)

Steps to reproduce:
1. Import example_Error_*.ics into an existing Nextcloud calendar in KOrganizer (by clicking on it in dolphin):
  - Shows up in Akonadiconsole and is synchronized to Nextcloud
  - Does not show up in KOrganizer
  - Shows up on Android (as translucent, not accepted yet)
  - Shows up in Nextcloud Web
2. Confirm event on Android:
  - Gets colored in Android (accepted)
  - Shows up in KOrganizer

=> Korganizer does not show events containing a invitation for the user itself that has not yet been accepted yet by the user (Android/Outlook show such events as translucent/brighter). As soon as the invitation is accepted by the user in another calendar app KOrganizer shows the event.
Comment 10 Tobias Klaus 2022-05-04 19:45:11 UTC
This still happens with kdepim-22.04.0 (gentoo/qt-5.15.3/plasma-5.24.4//kde-framework-5.93)

This is a major blocker for usage of kde-pim with ews, as most of my companies events are not displayed.

Can someone give me a hint, where one could look for the code that is responsible for filtering? As this bug can also be observed when using Kalendar, I suppose it is within akonadi itself?
Comment 11 enboig 2022-11-23 11:34:54 UTC
I confirm it still happens in 5.13.3
Comment 12 dag 2023-08-28 07:13:14 UTC
Still happens in 5.16.0. Is there anyone looking into this or should I try to find the time myself?
Comment 13 Rigo Wenning 2023-08-28 07:20:22 UTC
(In reply to dag from comment #12)
I've given up on akonadi and moved to thunderbird because of a very long history of not fixing critical bugs and changing shortcuts instead. While korganizer in KDE3 was rock solid, it has degraded to a point where it is not suitable for daily use anymore. It is not korganizer (frontend) it really is buggy akonadi that also eats tons of email. I really miss the UX though.
Comment 14 dag 2023-08-28 21:04:22 UTC
A bit more info on this:

1. Took the ics file for an event that did not show up
2. Edited the ics file removing all the ATTENDEE lines with continuation lines
3. Imported the file into korganizer and it did show up.

Importing the unedited file did show up in akonadiconsole, but not in korganizer.
Comment 15 dag 2023-08-28 21:21:14 UTC
And some more interesting info:

Merging the ATTENDEE continuation lines to one long line for each attendee also made the ics work correctly in korganizer.

This means that the bug is in the way korganizer handles a continuation line in the form:

This legal ICS line does not work:

ATTENDEE;CN=Some Name ;RSVP=TRUE;PARTSTAT=NEEDS-ACTION; 
 ROLE=OPT-PARTICIPANT;CUTYPE=INDIVIDUAL;X-UID=94249726501648:mailto:
 some.name@organisation.com

This line does work:

ATTENDEE;CN=Some Name ;RSVP=TRUE;PARTSTAT=NEEDS-ACTION;  ROLE=OPT-PARTICIPANT;CUTYPE=INDIVIDUAL;X-UID=94249726501648:mailto: some.name@organisation.com
Comment 16 Daniel Vrátil 2023-10-07 20:10:25 UTC
I'm trying to reproduce the issue locally on current git master, but it seems to work now.

If you can still reproduce this issue, could you please tell me what version of libical do you have installed that you can reproduce it with?
Comment 17 dag 2023-11-14 11:24:13 UTC
Hi. 

Using FC38 (korganizer 23.08.1 and libical 3.0.16) and Mageia 9 (korganizer 23.04.1 and libical 3.0.16).

They both behave the same way and both fail to show the entry.

Best Regards
Dag
Comment 18 dag 2023-11-14 11:53:11 UTC
Backported libical 3.0.17 to my Mageia also and still not working
Comment 19 dag 2023-11-14 12:11:49 UTC
And did the same on my FC38 where the newest libical available is also 3.0.17. No change.
Comment 20 dag 2023-11-14 12:28:37 UTC
And: Isn't akonadi-console using the same display routines as korganizer? Asking because akonadi-console shows the problematic entries just fine.
Comment 21 dag 2023-11-14 14:49:49 UTC
A bit more info by trial and error:

What works:
- Removing all continuation lines in the ATTENDEE: list
and/or
- Changing your own email address in the mailto: field of the attendee list to your primary email address or something bogus.

What doesn't work:
- Having an ATTENDEE entry with you name and a secondary working email address and continuation lines in it ???

Hope this helps
Comment 22 dag 2023-11-14 14:58:05 UTC
Sorry for the spam, but this might be important:

Just removing the continuation lines for your own entry is not enough to make things work. All the ATTENDEES continuation lines have to be merged.

So weird??
Comment 23 Daniel Vrátil 2023-11-25 20:57:19 UTC
Hmm, I must be missing something, because I'm seeing the events in the calendar. I installed Mageia 9 into a VM to make sure I have the same version as you, and followed those steps:

1) Copied the first event from comment #6 into kwrite
2) Changed one of the attendees to match my email configured in KOrganizer
3) Saved as event.ics
4) In KOrganizer, selected File -> Open -> event.ics -> "Merge into an existing calendar" -> selected the default "Personal Calendar"
5) Scrolled back to Nov 2021 in the month view, the event is shown

The code that shows events in Akonadi Console is different from KOrganizer - there's no calendaring logic, it just displays the list of database entries in the given folder and has some additional logic to extract like date, title etc. Unlike KOrganizer which is backed by a fairly complex KCalendarCore::MemoryCalendar which takes care of recurrence calculations and what not. Which is why Akonadi Console is useful for debugging :)
Comment 24 dag 2023-11-29 07:50:56 UTC
Try to change your email to another defined email account.

I have accounts dag @ newtech.fi and also an account dag.nygren @ consultor.fi.
Additionally the consultor account is currently offline due to many reasons.

An entry with the working email account is shown. An entry with the second offline account will not be shown.

Could the reason be the "offline" state?
Comment 25 dag 2023-11-29 08:12:40 UTC
Just tried this on a "no-show" event with the following result:

Also a fake email address for yourself will make the event invisible.

In other words:
Iff you are not on the ATTENDEE list with your own working email address the event will not be shown at all.
Comment 26 dag 2023-11-29 08:16:00 UTC
The funny thing is that merging the continuation lines into long lines will also make the event show :-)
Comment 27 dag 2023-11-29 08:23:03 UTC
And obviously is you are the organizer the event is also always shown
Comment 28 dag 2023-11-29 08:30:11 UTC
Possible explanation to why removing the line feeds in the continuation line makes things work:
The line becomes too long and the email address at the end cut off due to this.
Then korganizer has no idea about the identity of the ATTENDEE and can make no assumptions
Comment 29 dag 2023-12-15 08:56:04 UTC
Looking around at other bug reports like https://bugs.kde.org/show_bug.cgi?id=412722, https://bugs.kde.org/show_bug.cgi?id=418811 and this one it seems like korganizer is heavily filtering a lot of events due to different reasons and hiding them.
The wanted behaviour would be to show the event with a marker (color, text or other) that this event is "special" in some way.
The UI for a calender should NEVER EVER hide an event.

Now I cannot trust the korganizer display at all and will miss some events due to different strange conditions.

Where is this filtering done? Can somone point me to the source file so I can remove the filter?
Comment 30 dag 2024-01-17 09:16:45 UTC
Just noted another thing that might help in reproducing:

The problem item seems to be visible in "Agenda" mode - It is only left out from the Month view
Comment 31 Daniel Vrátil 2024-02-16 10:29:47 UTC
So, here's an answer on what's going on:

Some years ago, special "Search" -> "Open Invitations" and "Declined Invitations" calendars were introduced. Those calendar should contain all your open ("unanswered") and declined invitations from all calendars. To prevent such events appearing twice in KOrganizer, an additional filter was added to code that hides those events from regular calendars. The problem is that users often don't have those calendars enabled, and that the special "Search" collections don't work reliably at all and thus events might be missing.

I have accidentally fixed this in git master (to be released 24.02) by porting event views away from ETMCalendar, so this filter was also removed.

I have proposed to remove the special search collections completely (due to their unreliability as of now), always show unanswered invites  make it globally configurable in KOrganizer whether to show declined event.
Comment 32 Daniel Vrátil 2024-02-21 07:53:34 UTC
Open and declined invitations will appear in calendars again in 24.02 release, since the filter was removed recently (by accident, but that doesn't matter, now it's a feature :-)). KOrganizer master (to be released in 24.05) also has a checkbox in settings to globally show/hide declined invitations.

I'm closing this as fixed, since open and declined invitations are showing again, if there are still some other events that are not appearing in KOrganizer and are not open/declined invitations, please open a new issue.

Sorry this took so long to get fixed.