Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bitnami/mysql] MySQL 8.4 won't start with helm charts #67103

Open
JWebDev opened this issue May 17, 2024 · 8 comments
Open

[bitnami/mysql] MySQL 8.4 won't start with helm charts #67103

JWebDev opened this issue May 17, 2024 · 8 comments
Assignees
@rafariossaa
Labels
mysql tech-issues The user has a technical issue about an application triage Triage is needed

Comments

@JWebDev
Copy link

JWebDev commented May 17, 2024 •
edited by carrodher

Name and Version

bitnami/mysql:8.4-debian-12

What architecture are you using?

amd64

What steps will reproduce the bug?

Hi.

I am trying to update my MySQL database to Stable 8.4.0 LTS, but with the current, newest Helm chart and minimal configuration, the image simply does not start the MySQL server. However, with image 8.3-debian-12, the server starts as expected.

P.S. I noticed that the default image is 8.0.37, and maybe this is not a bug and simply not jet implemented or tested.

helm upgrade --install mysql bitnami/mysql -n mysql -f values.yaml

My minimal values.yaml:

image:
  registry: docker.io
  repository: bitnami/mysql
  tag: 8.4-debian-12
#  tag: 8.3-debian-12

  debug: false
architecture: standalone

Thanks!

What is the expected behavior?

Normal MySQL Server start

What do you see instead?

Here is the start log

mysql 07:15:42.40 INFO  ==> 
mysql 07:15:42.40 INFO  ==> Welcome to the Bitnami mysql container
mysql 07:15:42.40 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
mysql 07:15:42.40 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
mysql 07:15:42.40 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
mysql 07:15:42.40 INFO  ==> 
mysql 07:15:42.40 INFO  ==> ** Starting MySQL setup **
mysql 07:15:42.41 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars
mysql 07:15:42.42 INFO  ==> Initializing mysql database
mysql 07:15:42.42 WARN  ==> The mysql configuration file '/opt/bitnami/mysql/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
mysql 07:15:42.42 INFO  ==> Using persisted data
mysql 07:15:42.44 INFO  ==> Running mysql_upgrade
mysql 07:15:42.44 INFO  ==> Starting mysql in background
2024-05-17T07:15:42.459846Z 0 [System] [MY-015015] [Server] MySQL Server - start.
2024-05-17T07:15:42.459929Z 0 [Warning] [MY-010139] [Server] Changed limits: max_open_files: 65535 (requested 323500)
2024-05-17T07:15:42.459931Z 0 [Warning] [MY-010142] [Server] Changed limits: table_open_cache: 412 (requested 4000)
2024-05-17T07:15:42.682210Z 0 [Warning] [MY-011068] [Server] The syntax 'skip_slave_start' is deprecated and will be removed in a future release. Please use skip_replica_start instead.
2024-05-17T07:15:42.682296Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld (mysqld 8.4.0) starting as process 48
2024-05-17T07:15:42.697427Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-05-17T07:15:42.908616Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
mysqld: Table 'mysql.plugin' doesn't exist
2024-05-17T07:15:43.044534Z 0 [ERROR] [MY-010735] [Server] Could not open the mysql.plugin table. Please perform the MySQL upgrade procedure.
2024-05-17T07:15:43.044728Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables
2024-05-17T07:15:43.044879Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables
2024-05-17T07:15:43.045004Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables
2024-05-17T07:15:43.045250Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables
2024-05-17T07:15:43.045376Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables
2024-05-17T07:15:43.045506Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables
2024-05-17T07:15:43.045628Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables
2024-05-17T07:15:43.173154Z 4 [System] [MY-013381] [Server] Server upgrade from '80400' to '80400' started.
2024-05-17T07:15:50.000930Z 4 [System] [MY-013381] [Server] Server upgrade from '80400' to '80400' completed.
2024-05-17T07:15:50.230301Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-05-17T07:15:50.230395Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-05-17T07:15:50.243430Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/opt/bitnami/mysql/tmp' in the path is accessible to all OS users. Consider choosing a different directory.
2024-05-17T07:15:50.245884Z 0 [ERROR] [MY-000067] [Server] unknown variable 'default_authentication_plugin=caching_sha2_password'.
2024-05-17T07:15:50.247034Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-05-17T07:15:51.768559Z 0 [System] [MY-010910] [Server] /opt/bitnami/mysql/bin/mysqld: Shutdown complete (mysqld 8.4.0)  Source distribution.
2024-05-17T07:15:51.768571Z 0 [System] [MY-015016] [Server] MySQL Server - end.
@JWebDev JWebDev added the tech-issues The user has a technical issue about an application label May 17, 2024
@github-actions github-actions bot added the triage Triage is needed label May 17, 2024
@carrodher
Copy link
Member

That's somehow expected. Please note Helm charts are built and tested with a specific application version. At this moment, MySQL 8.0, see https://github.com/bitnami/charts/blob/main/bitnami/mysql/Chart.yaml#L15. Substituting the version could cause degraded performance, broken chart features, or missing environment variables. Having said that, as you rightly said the current Helm chart works fine with MySQL 8.3, but it is not something tested on our side.

We're working on releasing the bitnami/mysql Helm chart based on MySQL 8.4 which will be the defacto version. We'll update this thread once it is released.

@JWebDev
Copy link
Author

JWebDev commented May 17, 2024

@carrodher Thanks for the answer. Yes, this is not a bug, as I thought.
I'll wait for the message.
Thanks!

@carrodher carrodher mentioned this issue May 20, 2024
Closed
@rafariossaa
Copy link
Contributor

Hi @JWebDev,
Could you give release 11.0.0 a try ?

@Divinii
Copy link

Divinii commented May 30, 2024

@rafariossaa I just tested an upgrade to 11.0.0 and 11.1.0 and the Liveness probe fails with the error Plugin 'mysql_native_password' is not loaded

10s (x3 over 30s)      Warning   Unhealthy                 Pod/mysql-0                 Liveness probe failed: mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqladmin: connect to server at 'localhost' failed
error: 'Plugin 'mysql_native_password' is not loaded'

In my application I am getting an similar error:

Error: Failed to make a database connection
Error: SQLSTATE[HY000] [1524] Plugin 'mysql_native_password' is not loaded

I have tried adding default_authentication_plugin=mysql_native_password to the mysql config which results in it crashing:

2024-05-30T13:15:42.232754181+02:00 2024-05-30T11:15:42.232676Z 0 [ERROR] [MY-000067] [Server] unknown variable 'default_authentication_plugin=mysql_native_password'.
2024-05-30T13:15:42.233292054+02:00 2024-05-30T11:15:42.233233Z 0 [ERROR] [MY-010119] [Server] Aborting

@carrodher carrodher assigned rafariossaa and unassigned carrodher Jun 3, 2024
@rafariossaa
Copy link
Contributor

Which chart version are you upgrading to 11.0.0 from ?

@Divinii
Copy link

Divinii commented Jun 7, 2024

I am upgrading from chart version mysql:10.3.0

@tommytob
Copy link

tommytob commented Jun 7, 2024 •
edited

@Divinii

I encountered the same problem, By default de mysql_native_password is not enabled anymore in mysql release 8.4 , So you have 2 options.

Options 1:
Rollback the upgrade and update the user passwords to a caching_sha2_password and upgrade again. I think this will work

Option 2:
What I have done

@Divinii
Copy link

Divinii commented Jun 7, 2024

@tommytob This is exactly what was wrong with my instances, I was not aware of that and went down the wrong rabbit hole. :) Thank you very much for taking the time and explaining that to me!

@javsalgar javsalgar changed the title MySQL 8.4 won't start with helm charts [bitnami/mysql] MySQL 8.4 won't start with helm charts Jun 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rafariossaa rafariossaa

Labels
mysql tech-issues The user has a technical issue about an application triage Triage is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[bitnami/mysql] Release 11.0.0 bitnami/charts
5 participants
@JWebDev @tommytob @rafariossaa @carrodher @Divinii