-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
When extracting hive data to mysql, the parallelism parameter setting is invalid. #6830
Comments
try to add |
Thank you, but this parameter has been added before,did not meet expectations |
try with this config, it will auto generate insert sql, i use this config, the write speed is good |
`env { source { transform { sink { url = "jdbc:mysql://xxx:3306/xxx?allowMultiQueries=true&useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true&useSSL=false" The version I am using is 2.3.1, and the configuration is as shown above, but the extraction speed is the same as before and has not improved. Is there something wrong with my configuration? |
Hello, now in the seatunnel-2.3.5 version, using the same configuration, the parallelism parameter will not take effect. |
Search before asking
What happened
The amount of data is 15 million rows. The speed of extracting hive data to mysql can only reach 13,000 records per second, and it takes nearly 30 minutes to complete the extraction.
But I set the seatunnel parallelism parameter to 10. After the data extraction task started, I used the show processlist command on the MySQL client and found that there was only one insert thread.
Then I tested extracting hive data to the console. The same amount of data took less than 1 minute.
SeaTunnel Version
2.3.1 and 2.3.5
SeaTunnel Config
Running Command
Error Exception
Zeta or Flink or Spark Version
Zeta
Java or Scala Version
1.8
Screenshots
No response
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: