Bug 384309 - Adding a certain iCal resource makes Akonadi creates high CPU load
Summary: Adding a certain iCal resource makes Akonadi creates high CPU load
Status: RESOLVED FIXED
Alias: None
Product: Akonadi
Classification: Frameworks and Libraries
Component: ICal file resource (show other bugs)
Version: unspecified
Platform: Gentoo Packages Linux
: NOR normal
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-09-03 10:18 UTC by t.schmittlauch
Modified: 2024-02-28 21:42 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In: 24.02.1
Sentry Crash Report:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description t.schmittlauch 2017-09-03 10:18:55 UTC
Adding a certain Google Calendar iCal file to KOrganizer/ Akonadi makes Akonadi use 80-100% CPU for a very long time. I removed the iCal resource after ca. 15-20min and restarted Akonadi + KOrganizer to end this behaviour.

How to reproduce:
1. Open KOrganizer
2. Add a new calendar
3. Choose "ICal calendar file"
4. Enter the following URI as filename: https://calendar.google.com/calendar/ical/24l1i4pt8aosa0t34js2k7mrho@group.calendar.google.com/public/basic.ics
I also enabled the "read-only" option.
5. Add the calendar and observe CPU load.

This behaviour doesn't only happen directly after adding the calendar. After some tries I managed to add the calendar successfully without messing up my system, but after a reboot the 100% CPU load issue appeared again.

If I shall provide additional information or logfiles, just tell me how.

I'm using Akonadi 17.08.0 on 64bit Gentoo system with mariadb as a backend
Comment 1 Allen Winter 2017-09-04 17:27:49 UTC
confirmed but I really don't think this is an Akonadi problem.

either the libical low-level parsing lib or the KDE kcalcore abstraction library is at fault.
Comment 2 t.schmittlauch 2017-09-04 21:36:04 UTC
> either the libical low-level parsing lib or the KDE kcalcore abstraction library is at fault.

Akonadi was my first guess as I could only observe which process had this issue.
Can you forward this bug report to the relevant project teams or shall I do that?
Comment 3 t.schmittlauch 2019-01-11 21:36:37 UTC
I found yet another ical file that causes high CPU load, this time mostly in korganizer itself.
But when setting this resource to "offline" in Akonadi-Console, the problem vanishes in korganizer. So Akonadi (and maybe kcalcore) is definitely involved.

This is the problematic calendar URL: https://unichor-dresden.de/index.php?option=com_chor&task=default.openCalendar&id=2
Comment 4 auxsvr 2020-06-06 06:18:52 UTC
Also with https://calendar.google.com/calendar/ical/o4cqnocbu4ou1s5t556sec9hoo%40group.calendar.google.com/public/basic.ics. A similar bug a few years ago was that the akonadi resource would keep on refreshing for ever if it was marked as read only. Now it seems that after manually stopping refreshing, it resumes proper operation.
Comment 5 Daniel Vrátil 2024-02-21 10:45:20 UTC
The problem with the Google calendar is that every time it's fetched it's different, so the hashes don't match and we end up in an endless sync loop in the ical resource that fetches the ical, finds out that the hash differs from previous and triggers sync, which fetches the ical, finds out the hashes differ again and triggers another sync and so on.
Comment 6 Bug Janitor Service 2024-02-24 13:37:17 UTC
A possibly relevant merge request was started @ https://invent.kde.org/pim/kdepim-runtime/-/merge_requests/150
Comment 7 Daniel Vrátil 2024-02-28 21:24:37 UTC
Git commit a4e290c26d98278f729073b909cc963e3331c0a8 by Daniel Vrátil.
Committed on 28/02/2024 at 21:18.
Pushed by dvratil into branch 'release/24.02'.

Fix endless sync loop with some remote iCal sources

The sync in SingleFileResource implementations (iCal, vCard and MBox
resources) is implemented in a little strange way for remote files:

1) Akonadi (or someone) requests sync of the resource's collection
2) The resource starts a download of the new file
3) In the meantime, it syncs content of the current version of
   the file with Akonadi
4) When the file is downloaded, hash of the file is compared with
   previous hash and if they differ, goto 1

Some remote sources (e.g. Google Calendar) generate the iCal on-demand
and as such the file always contains a different DTSTAMP. Thus, whenever
the iCal resource completes the download, the hash will always differ
compared to the previous one, and so it will again re-schedule sync
over and over. While the data are actually available in Akonadi and
to the user, it causes a high CPU load as the iCal is constantly being
re-parsed, Akonadi database update, the Indexing agent is triggered
over and over again. And the bigger the calendar, the worse it is.

This fix changes the behavior by downloading the file, running the
parser (iCal/vCard/mbox/...) and after that performing a regular
sync, so basically:

1) Akonadi (or someone) requests sync of the resource's collection
2) The resource starts a download of the new file
3) When the file is downloaded, the resource syncs the content of the
   file with Akonadi.

I have opted for using QPromise+QFuture combo to perform the async
chaining of download->sync. It's the only solution that requires the
last amount of code refactoring and reorganization in order to properly
sequence the operations. This would be the perfect place for using
coroutines, but I don't want to introduce coroutines as part of a
regular bugfix. I might do an experimental port at some point later.
FIXED-IN: 24.02.1

M  +15   -6    resources/ical/shared/icalresourcebase.cpp
M  +42   -35   resources/mbox/mboxresource.cpp
M  +14   -6    resources/shared/singlefileresource/singlefileresource.h
M  +31   -28   resources/shared/singlefileresource/singlefileresourcebase.cpp
M  +8    -2    resources/shared/singlefileresource/singlefileresourcebase.h
M  +17   -13   resources/vcard/vcardresource.cpp

https://invent.kde.org/pim/kdepim-runtime/-/commit/a4e290c26d98278f729073b909cc963e3331c0a8
Comment 8 Daniel Vrátil 2024-02-28 21:42:24 UTC
Git commit d136c91ace0f6e52089004221c0aabb696bdee63 by Daniel Vrátil.
Committed on 28/02/2024 at 21:42.
Pushed by dvratil into branch 'master'.

Fix endless sync loop with some remote iCal sources

The sync in SingleFileResource implementations (iCal, vCard and MBox
resources) is implemented in a little strange way for remote files:

1) Akonadi (or someone) requests sync of the resource's collection
2) The resource starts a download of the new file
3) In the meantime, it syncs content of the current version of
   the file with Akonadi
4) When the file is downloaded, hash of the file is compared with
   previous hash and if they differ, goto 1

Some remote sources (e.g. Google Calendar) generate the iCal on-demand
and as such the file always contains a different DTSTAMP. Thus, whenever
the iCal resource completes the download, the hash will always differ
compared to the previous one, and so it will again re-schedule sync
over and over. While the data are actually available in Akonadi and
to the user, it causes a high CPU load as the iCal is constantly being
re-parsed, Akonadi database update, the Indexing agent is triggered
over and over again. And the bigger the calendar, the worse it is.

This fix changes the behavior by downloading the file, running the
parser (iCal/vCard/mbox/...) and after that performing a regular
sync, so basically:

1) Akonadi (or someone) requests sync of the resource's collection
2) The resource starts a download of the new file
3) When the file is downloaded, the resource syncs the content of the
   file with Akonadi.

I have opted for using QPromise+QFuture combo to perform the async
chaining of download->sync. It's the only solution that requires the
last amount of code refactoring and reorganization in order to properly
sequence the operations. This would be the perfect place for using
coroutines, but I don't want to introduce coroutines as part of a
regular bugfix. I might do an experimental port at some point later.
FIXED-IN: 24.02.1
(cherry picked from commit a4e290c26d98278f729073b909cc963e3331c0a8)

M  +15   -6    resources/ical/shared/icalresourcebase.cpp
M  +42   -35   resources/mbox/mboxresource.cpp
M  +14   -6    resources/shared/singlefileresource/singlefileresource.h
M  +31   -28   resources/shared/singlefileresource/singlefileresourcebase.cpp
M  +8    -2    resources/shared/singlefileresource/singlefileresourcebase.h
M  +17   -13   resources/vcard/vcardresource.cpp

https://invent.kde.org/pim/kdepim-runtime/-/commit/d136c91ace0f6e52089004221c0aabb696bdee63