Bug 454948 - Akonadi server will not start on fresh install; yields "unknown error"
Summary: Akonadi server will not start on fresh install; yields "unknown error"
Status: REOPENED
Alias: None
Product: Akonadi
Classification: Frameworks and Libraries
Component: server (show other bugs)
Version: 5.20.2
Platform: Arch Linux Linux
: NOR major
Target Milestone: ---
Assignee: kdepim bugs
URL:
Keywords:
: 455111 455329 455419 (view as bug list)
Depends on:
Blocks:
 
Reported: 2022-06-06 22:18 UTC by fake.name
Modified: 2023-07-09 18:37 UTC (History)
22 users (show)

See Also:
Latest Commit:
Version Fixed In: 22.04.3
Sentry Crash Report:


Attachments
output of running akonadictl start (1.13 KB, text/plain)
2022-06-06 22:18 UTC, fake.name
Details
Akonadi fresh install error mariadb (1.06 KB, text/plain)
2022-06-10 11:35 UTC, Schlaefer
Details

Note You need to log in before you can comment on or make changes to this bug.
Description fake.name 2022-06-06 22:18:26 UTC
Created attachment 149515 [details]
output of running akonadictl start

SUMMARY
This is a reinstall of KDE completed today; everything else works out of the box, save for the akonadi server, which does not start.

Running akonadictl start produces the attached output.







STEPS TO REPRODUCE
1. go to konsole; akonadictl start
OR
1. open kalendar or korganizer or kmail

OBSERVED RESULT


EXPECTED RESULT


SOFTWARE/OS VERSIONS
Linux/KDE Plasma:  Arch 251.2, running Zen kernel 5.18.1
KDE Plasma Version:  5.24.5
KDE Frameworks Version: 5.94.0
Qt Version: 5.15.4

ADDITIONAL INFORMATION
Comment 1 Andreas G. 2022-06-07 21:00:48 UTC
Same issue after updating Arch Linux today. Same program versions as you. Downgrading to the previous version of mariadb made Akonadi usable again. 

Packages that were downgraded:
mariadb-10.7.4-1
mariadb-clients-10.7.4-1
mariadb-libs-10.7.4-1
Comment 2 fake.name 2022-06-08 00:20:13 UTC
Alas, the reversion didn't do me any good. The problem seems to be in Mariadb, actually; starting mariadbd fails, and one gets the following:



systemctl status mariadb.service 
× mariadb.service - MariaDB 10.7.4 database server
     Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Tue 2022-06-07 20:09:31 EDT; 9min ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 4032 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 4033 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemct>
    Process: 4059 ExecStart=/usr/bin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
   Main PID: 4059 (code=exited, status=1/FAILURE)
     Status: "MariaDB server is down"
        CPU: 102ms

Jun 07 20:09:31 psyche mariadbd[4059]: 2022-06-07 20:09:31 0 [Note] InnoDB: Buffer pool(s) load completed at 220607 20:09:31
Jun 07 20:09:31 psyche mariadbd[4059]: 2022-06-07 20:09:31 0 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
Jun 07 20:09:31 psyche mariadbd[4059]: 2022-06-07 20:09:31 0 [Note] Server socket created on IP: '0.0.0.0'.
Jun 07 20:09:31 psyche mariadbd[4059]: 2022-06-07 20:09:31 0 [Note] Server socket created on IP: '::'.
Jun 07 20:09:31 psyche mariadbd[4059]: 2022-06-07 20:09:31 0 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.db' doesn't exist
Jun 07 20:09:31 psyche mariadbd[4059]: 2022-06-07 20:09:31 0 [ERROR] Aborting
Jun 07 20:09:31 psyche mariadbd[4059]: Warning: Memory not freed: 280
Jun 07 20:09:31 psyche systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Jun 07 20:09:31 psyche systemd[1]: mariadb.service: Failed with result 'exit-code'.
Jun 07 20:09:31 psyche systemd[1]: Failed to start MariaDB 10.7.4 database server.


So it's looking for things it isn't finding, and I cannot actually tell where it's looking...
Comment 3 fake.name 2022-06-09 19:10:14 UTC
Updating this:
a total removal -- i.e, pacman -Rcns -- plus a deletion of config files, followed by installation of the PRIOR mariadb version, followed by reinstallation of akonadi without letting mariadb update, fixes the problem. So the problem is in fact in the versioning. Thank you Andreas G.
Comment 4 Oleksandr Natalenko 2022-06-10 07:56:10 UTC
All, if you expect some help on this, post your `~/.local/share/akonadi/db_data/mysql.err`. It might be the case the DB needs upgrade, and to find out also post the output of this:

```
$ mariadb-upgrade --defaults-file=~/.local/share/akonadi/mysql.conf --socket=/run/user/$(id -u)/akonadi/mysql.socket --check-if-upgrade-is-needed
```
Comment 5 Antonio Rojas 2022-06-10 09:37:57 UTC
*** Bug 455111 has been marked as a duplicate of this bug. ***
Comment 6 harbind00 2022-06-10 09:59:22 UTC
Running 
```
touch /home/$HOME/.local/share/akonadi/db_data/ib_logfile0
```
fixed this for me.
Comment 7 Rober 2022-06-10 10:41:29 UTC
(In reply to harbind00 from comment #6)
> Running 
> ```
> touch /home/$HOME/.local/share/akonadi/db_data/ib_logfile0
> ```
> fixed this for me.

It doesn't work for me. mysql.err: 

2022-06-10 12:39:55 0 [Warning] option 'innodb-log-buffer-size': unsigned value 1048576 adjusted to 2097152
2022-06-10 12:39:55 0 [Note] InnoDB: Compressed tables use zlib 1.2.12
2022-06-10 12:39:55 0 [Note] InnoDB: Number of transaction pools: 1
2022-06-10 12:39:55 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-06-10 12:39:55 0 [Note] InnoDB: Using Linux native AIO
2022-06-10 12:39:55 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2022-06-10 12:39:55 0 [Note] InnoDB: Completed initialization of buffer pool
2022-06-10 12:39:55 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
2022-06-10 12:39:55 0 [ERROR] InnoDB: ib_logfile0 is empty, and LSN is unknown.
2022-06-10 12:39:55 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
2022-06-10 12:39:55 0 [Note] InnoDB: Starting shutdown...
2022-06-10 12:39:55 0 [ERROR] Plugin 'InnoDB' init function returned error.
2022-06-10 12:39:55 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2022-06-10 12:39:55 0 [ERROR] Unknown/unsupported storage engine: innodb
2022-06-10 12:39:55 0 [ERROR] Aborting
Comment 8 Oleksandr Natalenko 2022-06-10 11:32:01 UTC
(In reply to Rober from comment #7)
> 2022-06-10 12:39:55 0 [ERROR] InnoDB: ib_logfile0 is empty, and LSN is
> unknown.
> 2022-06-10 12:39:55 0 [ERROR] InnoDB: Plugin initialization aborted with
> error Data structure corruption

You may want to try starting mariadb with `innodb_force_recovery=X` specified in `~/.local/share/akonadi/mysql.conf`, where X would be a number from 1 to 6.
Comment 9 Schlaefer 2022-06-10 11:35:35 UTC
Created attachment 149595 [details]
Akonadi fresh install error mariadb

I'm seeing the same issue after installing mariadb 10.8.3 on Arch.

What worked here: 
- delete "~.local/share/akonadi/db_data" (make a backup first)
- run "akonadictl start"

That creates a new, working DB.
Comment 10 harbind00 2022-06-10 12:13:48 UTC
(In reply to harbind00 from comment #6)
> Running 
> ```
> touch /home/$HOME/.local/share/akonadi/db_data/ib_logfile0
> ```
> fixed this for me.

Oops, this should of course be `touch $HOME/.local/share/akonadi/db_data/ib_logfile0`
Comment 11 Emile de Weerd 2022-06-10 18:53:27 UTC
(In reply to harbind00 from comment #10)
> (In reply to harbind00 from comment #6)
> > Running 
> > ```
> > touch /home/$HOME/.local/share/akonadi/db_data/ib_logfile0
> > ```
> > fixed this for me.
> 
> Oops, this should of course be `touch
> $HOME/.local/share/akonadi/db_data/ib_logfile0`

Hi, I confirm the above fixed it for me too.
Comment 12 d3coder 2022-06-11 12:29:01 UTC
> touch $HOME/.local/share/akonadi/db_data/ib_logfile0

This fixed issue for me
Comment 13 Kishore Gopalakrishnan 2022-06-12 05:28:25 UTC
(In reply to harbind00 from comment #10)
> (In reply to harbind00 from comment #6)
> > Running 
> > ```
> > touch /home/$HOME/.local/share/akonadi/db_data/ib_logfile0
> > ```
> > fixed this for me.
> 
> Oops, this should of course be `touch
> $HOME/.local/share/akonadi/db_data/ib_logfile0`

Good find. This commit (included in 10.8.0): https://jira.mariadb.org/browse/MDEV-14425
says "Starting up without ib_logfile0 will no longer be supported; see also MDEV-27199."

Is akonadi somehow expected to create it as part of the startup process?
Comment 14 Kishore Gopalakrishnan 2022-06-12 05:44:52 UTC
(In reply to Oleksandr Natalenko from comment #4)
> All, if you expect some help on this, post your
> `~/.local/share/akonadi/db_data/mysql.err`. It might be the case the DB
> needs upgrade, and to find out also post the output of this:
> 
> ```
> $ mariadb-upgrade --defaults-file=~/.local/share/akonadi/mysql.conf
> --socket=/run/user/$(id -u)/akonadi/mysql.socket --check-if-upgrade-is-needed
> ```

The command you gave indeed says that an upgrade is required:

[kishore@kishorearchtestingVM ~]$  mariadb-upgrade --defaults-file=~/.local/share/akonadi/mysql.conf --socket=/run/user/$(id -u)/akonadi/mysql.socket --check-if-upgrade-is-needed
Major version upgrade detected from 10.7.4-MariaDB to 10.8.3-MariaDB. Check required!
Comment 15 Kishore Gopalakrishnan 2022-06-12 05:48:29 UTC
(In reply to Kishore Gopalakrishnan from comment #14)
> (In reply to Oleksandr Natalenko from comment #4)
> > All, if you expect some help on this, post your
> > `~/.local/share/akonadi/db_data/mysql.err`. It might be the case the DB
> > needs upgrade, and to find out also post the output of this:
> > 
> > ```
> > $ mariadb-upgrade --defaults-file=~/.local/share/akonadi/mysql.conf
> > --socket=/run/user/$(id -u)/akonadi/mysql.socket --check-if-upgrade-is-needed
> > ```
> 
> The command you gave indeed says that an upgrade is required:
> 
> [kishore@kishorearchtestingVM ~]$  mariadb-upgrade
> --defaults-file=~/.local/share/akonadi/mysql.conf --socket=/run/user/$(id
> -u)/akonadi/mysql.socket --check-if-upgrade-is-needed
> Major version upgrade detected from 10.7.4-MariaDB to 10.8.3-MariaDB. Check
> required!

Just running `mariadb-upgrade --defaults-file=~/.local/share/akonadi/mysql.conf --socket=/run/user/$(id -u)/akonadi/mysql.socket` and then rebooting fixed this, but it doesn't seem to be related to the reported issue (akonadi was able to start even with an outdated database as long as the ib_logfile0 file was present).
Comment 16 Martin Schnitkemper 2022-06-12 08:07:34 UTC
I have now run the update on two systems and in both cases akonadi no longer starts. I didn't get the "unknown error" error, but the file "ib_logfile0" was missing after the first login.

A "touch $HOME/.local/share/akonadi/db_data/ib_logfile0" as suggested here several times was not a solution because after the second logon I had 3200+ entries in mysql.err like
| [ERROR] InnoDB: Page [page id: space=0, page number=7] log sequence number 1447482201 is in the future! Current system log sequence number 1378510903.
| [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.

A "recovery" was not an option for me because it seems complicated to me, is only insufficiently described in the MariaDB knowledgebase and I am not sure if it solves the problem at all.

I had already completed the update to mariadb 10.8.3 last week and also updated the structures with "mariadb-upgrade --socket=/run/user/$(id -u)/akonadi/mysql.socket", it came therefore only the "KDE Gear 22.04.2" update with akonadi and its accompanying components.

Based on the experience of the first system, I copied now the "ib_logfile0" file to a safe place before updating the second system. After the file was missing again after the update and the first logon, the easiest solution for me was to copy the file back after the update and restart akonadi. It also survived a reboot.

My recommendation is therefore to back up the file before an update or restore it from a current backup. If it has already happened then it also helps to delete (or rename) the "db_data" folder and start new with a 
"mariadb-install-db --datadir=$HOME/.local/share/akonadi/db_data --basedir= /usr".

The question remains (and here I see the real problem) why akonadi or one of its components removes the "ib_logfile0" file on the first logon after the update.
Comment 17 Antonio Rojas 2022-06-15 13:56:21 UTC
*** Bug 455329 has been marked as a duplicate of this bug. ***
Comment 18 Maik Qualmann 2022-06-15 18:27:43 UTC
Git commit 8d16ea2e3fddbfa9a1615d506c342070502d1ce9 by Maik Qualmann.
Committed on 15/06/2022 at 18:26.
Pushed by mqualmann into branch 'master'.

do not remove the MySQL log files

M  +2    -2    core/libs/database/server/databaseserver.cpp

https://invent.kde.org/graphics/digikam/commit/8d16ea2e3fddbfa9a1615d506c342070502d1ce9
Comment 19 Antonio Rojas 2022-06-15 18:28:43 UTC
The file is being explicitely removed here:

https://invent.kde.org/pim/akonadi/-/blob/master/src/server/storage/dbconfigmysql.cpp#L395
Comment 20 Maik Qualmann 2022-06-15 18:34:42 UTC
Sorry for posting a commit message in a non-digiKam bug report.

Maik
Comment 21 Antonio Rojas 2022-06-16 15:14:07 UTC
*** Bug 455419 has been marked as a duplicate of this bug. ***
Comment 22 Bug Janitor Service 2022-06-17 16:32:29 UTC
A possibly relevant merge request was started @ https://invent.kde.org/pim/akonadi/-/merge_requests/105
Comment 23 KimHono 2022-06-19 10:27:36 UTC
In my case :
- "touch /home/$HOME/.local/share/akonadi/db_data/ib_logfile0" worked for several times but finally failed. Even while restoring a backup realized before the issue.
- "mariadb-upgrade --defaults-file=~/.local/share/akonadi/mysql.conf --socket=/run/user/$(id -u)/akonadi/mysql.socket --check-if-upgrade-is-needed" was clean but didn't resolve anything.
- "rm -r ~/.local/share/akonadi/db_data" followed by "akonadictl start" is finally the best solution. Thanks to IMAP protocol I didn't loose any email !
Comment 24 Sandro Knauß 2022-06-19 12:50:05 UTC
Git commit d92d0cdbad08de6318c0e0a88e358c3edd3bb702 by Sandro Knauß, on behalf of Antonio Rojas.
Committed on 19/06/2022 at 12:50.
Pushed by knauss into branch 'master'.

Don't delete mariadb log files

The ib_logfile0 file is mandatory in MariaDB 10.8 [1] so akonadi deleting it on upgrade breaks the database self-check and prevents akonadi from working

[1] https://jira.mariadb.org/browse/MDEV-14425
FIXED-IN: 22.04.3

M  +0    -6    src/server/storage/dbconfigmysql.cpp

https://invent.kde.org/pim/akonadi/commit/d92d0cdbad08de6318c0e0a88e358c3edd3bb702
Comment 25 Sandro Knauß 2022-06-19 12:54:45 UTC
Git commit aebb20a082d05b36458008fedff4397f022c3ffa by Sandro Knauß, on behalf of Antonio Rojas.
Committed on 19/06/2022 at 12:52.
Pushed by knauss into branch 'master'.

Don't delete mariadb log files

The ib_logfile0 file is mandatory in MariaDB 10.8 [1] so akonadi deleting it on upgrade breaks the database self-check and prevents akonadi from working

[1] https://jira.mariadb.org/browse/MDEV-14425
FIXED-IN: 22.04.3

M  +0    -6    src/server/storage/dbconfigmysql.cpp

https://invent.kde.org/pim/akonadi/commit/aebb20a082d05b36458008fedff4397f022c3ffa
Comment 26 benmordecai 2023-03-08 12:57:39 UTC
This problem is still present for me in 22.12.3.

Running `touch $HOME/.local/share/akonadi/db_data/ib_logfile0` fixed it but it should not need to be done.
Comment 27 Don Rudo 2023-07-09 18:37:20 UTC
(In reply to KimHono from comment #23)
> In my case :
> - "touch /home/$HOME/.local/share/akonadi/db_data/ib_logfile0" worked for
> several times but finally failed. Even while restoring a backup realized
> before the issue.
> - "mariadb-upgrade --defaults-file=~/.local/share/akonadi/mysql.conf
> --socket=/run/user/$(id -u)/akonadi/mysql.socket
> --check-if-upgrade-is-needed" was clean but didn't resolve anything.
> - "rm -r ~/.local/share/akonadi/db_data" followed by "akonadictl start" is
> finally the best solution. Thanks to IMAP protocol I didn't loose any email !

I confirm this was the best fix for me, with only one addition, I got into some issues beforehand because journalctl was only showing the warning about "mysqld" being deprecated in favor of "mariadbd" as a potential 'error', the full change was:

sed -i 's/mysqld/mysqldbd/g' ~/.config/akonadi/akonadiserverrc
rm -r ~/.local/share/akonadi/db_data
akonadictl start