Bug 475874 - Huge mysql.err files filled with [ERROR] Incorrect definition of table mysql.column_stats.
Summary: Huge mysql.err files filled with [ERROR] Incorrect definition of table mysql....
Status: RESOLVED FIXED
Alias: None
Product: digikam
Classification: Applications
Component: Database-Mysql (show other bugs)
Version: 8.2.0
Platform: Microsoft Windows Microsoft Windows
: NOR normal
Target Milestone: ---
Assignee: Digikam Developers
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-10-20 12:10 UTC by Steve Franks
Modified: 2023-11-14 09:52 UTC (History)
3 users (show)

See Also:
Latest Commit:
Version Fixed In: 8.2.0
Sentry Crash Report:


Attachments
MYSQL error file showing the error messages described above (498.21 KB, text/plain)
2023-10-20 12:10 UTC, Steve Franks
Details
Screenshot from 2023-11-12 10-26-49.png (527.55 KB, image/png)
2023-11-12 10:36 UTC, Steve Franks
Details
attachment-2961127-0.html (1.17 KB, text/html)
2023-11-12 10:44 UTC, Steve Franks
Details
attachment-3172951-0.html (1.17 KB, text/html)
2023-11-14 09:52 UTC, Steve Franks
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Steve Franks 2023-10-20 12:10:49 UTC
Created attachment 162449 [details]
MYSQL error file showing the error messages described above

SUMMARY
***
NOTE: If you are reporting a crash, please try to attach a backtrace with debug symbols.
See https://community.kde.org/Guidelines_and_HOWTOs/Debugging/How_to_create_useful_crash_reports
***


STEPS TO REPRODUCE
1. Run digikam
2.  Look in db_data directory at the size of mysql.err file
3. 

OBSERVED RESULT

mysql.err.old contains the following [pair of lines millions of times:
2023-10-20 12:48:32 14 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'hist_type' at position 9 to have type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB','JSON_HB'), found type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB').
2023-10-20 12:48:32 14 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'histogram' at position 10 to have type longblob, found type varbinary(255).

EXPECTED RESULT

No errors in mysql.err, only start and stop messages

SOFTWARE/OS VERSIONS
Windows:  11
macOS: 
Linux/KDE Plasma: 
(available in About System)
KDE Plasma Version:  
KDE Frameworks Version: 5.106.0
Qt Version:  5.15.8

ADDITIONAL INFORMATION
I have tried to modify the columns to the expected definitions. HeidiSQL confirms that the definitions are now correct, but the errors are still being reported.
Comment 1 caulier.gilles 2023-10-20 12:17:40 UTC
Which Mysql or MariaDb version did you use exactly ?

Gilles Caulier
Comment 2 Steve Franks 2023-10-20 12:28:17 UTC
MariaDB 11.0.2 and 11.2.1.
This does not seem to affect the working of digikam, but mysql.err.old has grown to 6.3gb, which is a waste of space.
Comment 3 Maik Qualmann 2023-10-20 14:33:08 UTC
After the log, use the internal MySQL server from digiKam. You have installed newer versions from the server without a "MySQL_upgrade" command for the database. We only do this in an absolute error if the server no longer starts. We will probably have to do it in the future at every start.

Maik
Comment 4 Maik Qualmann 2023-10-20 21:02:20 UTC
Git commit 020aa6b7c3f7f5de7fa40a146a3a9918ace6a2a0 by Maik Qualmann.
Committed on 20/10/2023 at 23:01.
Pushed by mqualmann into branch 'master'.

always perform a MySQL database upgrade
If the server version is not newer than the database,
no upgrade will be carried out automatically.
In the case of a newer server version, an upgrade is
carried out and a simple progress dialog is displayed.
FIXED-IN: 8.2.0

M  +1    -1    NEWS
M  +63   -68   core/libs/database/server/databaseserver.cpp
M  +2    -2    core/libs/database/server/databaseserver.h

https://invent.kde.org/graphics/digikam/-/commit/020aa6b7c3f7f5de7fa40a146a3a9918ace6a2a0
Comment 5 Maik Qualmann 2023-10-20 21:05:30 UTC
Note: I can't say whether an upgrade to the current server version is still possible for you because you have already edited the internal MySQL system table.

Maik
Comment 6 Maik Qualmann 2023-10-21 12:21:04 UTC
Git commit d5d923389605ffe66be61f73d10ed88d4bcfc161 by Maik Qualmann.
Committed on 21/10/2023 at 14:20.
Pushed by mqualmann into branch 'master'.

add the mysql upgrade command as required binary

M  +6    -3    core/libs/album/manager/albummanager_database.cpp
M  +2    -1    core/libs/database/CMakeLists.txt
M  +135  -120  core/libs/database/engine/dbengineparameters.cpp
M  +9    -2    core/libs/database/engine/dbengineparameters.h
M  +25   -17   core/libs/database/server/databaseserver.cpp
M  +9    -6    core/libs/database/utils/widgets/dbsettingswidget.cpp
M  +5    -3    core/libs/database/utils/widgets/dbsettingswidget_p.h
C  +3    -3    core/libs/database/utils/widgets/mysqlserverbinary.cpp [from: core/libs/database/utils/widgets/mysqlservbinary.cpp - 089% similarity]
C  +7    -7    core/libs/database/utils/widgets/mysqlserverbinary.h [from: core/libs/database/utils/widgets/mysqlservbinary.h - 068% similarity]
R  +8    -8    core/libs/database/utils/widgets/mysqlupgradebinary.cpp [from: core/libs/database/utils/widgets/mysqlservbinary.cpp - 059% similarity]
R  +10   -10   core/libs/database/utils/widgets/mysqlupgradebinary.h [from: core/libs/database/utils/widgets/mysqlservbinary.h - 050% similarity]
M  +36   -26   core/tests/database/databasemysqlinit_utest.cpp
M  +36   -26   core/tests/database/databaseswitch_utest.cpp

https://invent.kde.org/graphics/digikam/-/commit/d5d923389605ffe66be61f73d10ed88d4bcfc161
Comment 7 Maik Qualmann 2023-10-22 19:13:11 UTC
Git commit 51f425ecf2647e5f4a97eb275055a4f031904c4c by Maik Qualmann.
Committed on 22/10/2023 at 21:12.
Pushed by mqualmann into branch 'master'.

limit the file size to 100MB of the old MySQL error log file

M  +7    -1    core/libs/database/server/databaseserver.cpp

https://invent.kde.org/graphics/digikam/-/commit/51f425ecf2647e5f4a97eb275055a4f031904c4c
Comment 8 Steve Franks 2023-10-26 12:50:07 UTC
The mysql.err file is currently about 2mb, the mysql.err.old file grows after every digikam session.
Would it not be better to delete the old file and rename mysql.err to mysql.err.old at the beginning of each digikam session.
I can't think of a reason to keep for nearly 2 weeks (mysql.err.old contains messages from 14th October to today (26th October))
Comment 9 Maik Qualmann 2023-10-26 16:54:20 UTC
The problem is that when users have problems with MySQL, they start digiKam more often and then we only have the last log. The old error log file is now deleted after 100MB and starts again from the beginning. We can of course reduce 100MB, but I don't think it's too big at the moment.

Maik
Comment 10 Steve Franks 2023-11-12 10:36:07 UTC
Created attachment 163071 [details]
Screenshot from 2023-11-12 10-26-49.png

I downloaded 8.2.0 appimage for Linux. I have not updated Mariadb, because
I already have a recent version. When I start the appimage it says that it
can't find Mariadb. I found and set the correct folders, but then the
upgrade fails with an unknown error (see attached screenshot).
Regards
Steve

On Thu, 26 Oct 2023 at 17:54, Maik Qualmann <bugzilla_noreply@kde.org>
wrote:

> https://bugs.kde.org/show_bug.cgi?id=475874
>
> --- Comment #9 from Maik Qualmann <metzpinguin@gmail.com> ---
> The problem is that when users have problems with MySQL, they start digiKam
> more often and then we only have the last log. The old error log file is
> now
> deleted after 100MB and starts again from the beginning. We can of course
> reduce 100MB, but I don't think it's too big at the moment.
>
> Maik
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
> You reported the bug.
Comment 11 Steve Franks 2023-11-12 10:44:59 UTC
Created attachment 163072 [details]
attachment-2961127-0.html

I tried again with Mariadb 11.3 Alpha and it displayed the same error.
Regards
Steve

On Thu, 26 Oct 2023 at 17:54, Maik Qualmann <bugzilla_noreply@kde.org>
wrote:

> https://bugs.kde.org/show_bug.cgi?id=475874
>
> --- Comment #9 from Maik Qualmann <metzpinguin@gmail.com> ---
> The problem is that when users have problems with MySQL, they start digiKam
> more often and then we only have the last log. The old error log file is
> now
> deleted after 100MB and starts again from the beginning. We can of course
> reduce 100MB, but I don't think it's too big at the moment.
>
> Maik
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
> You reported the bug.
Comment 12 Maik Qualmann 2023-11-12 10:48:19 UTC
We now run "mysql_upgrade" at every start in order to adapt the database when the server software is updated. Since you have already tried to correct the system tables of the database manually, the "mysql_upgrade" will fail here because the database version does not match the changes. I think you need to start with a new database in this case.

Maik
Comment 13 Maik Qualmann 2023-11-12 10:58:33 UTC
While you changed the system tables manually, you certainly created a backup. You could use this now.

Maik
Comment 14 Steve Franks 2023-11-12 13:00:11 UTC
My Windows laptop is working fine with 8.2.
I didn’t modify the Linux data, simply downloaded the 8.2 appimage and ran it. It surprised me when it displayed the missing MySQL dialogue.
Comment 15 Maik Qualmann 2023-11-12 15:08:58 UTC
If the "mysql_upgrade" fails, the database may not be clean. We need to do this upgrade due to the many changes to the internal database structure with almost every new MariaDB release. Other projects that manage an internal database also do this (e.g. Akonadi).

Are there any messages in the MySQL error log in the database directory?

Maik
Comment 16 Maik Qualmann 2023-11-12 16:10:28 UTC
Another thing, the path to "mysql_upgrade" is in your home directory. The server uses the same instance as the path or the same MariaDB version?

Maik
Comment 17 Steve Franks 2023-11-14 09:52:19 UTC
Created attachment 163144 [details]
attachment-3172951-0.html

I think the problem is due to the digikan data directory being empty.
I have no idea where the data disappeared to, but will retrieve it from
backup and try again.
Sorry for troubling you
Regards
Steve

On Sun, 12 Nov 2023 at 16:10, Maik Qualmann <bugzilla_noreply@kde.org>
wrote:

> https://bugs.kde.org/show_bug.cgi?id=475874
>
> --- Comment #16 from Maik Qualmann <metzpinguin@gmail.com> ---
> Another thing, the path to "mysql_upgrade" is in your home directory. The
> server uses the same instance as the path or the same MariaDB version?
>
> Maik
>
> --
> You are receiving this mail because:
> You reported the bug.
> You are on the CC list for the bug.