Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BugFix] clean stale column stats periodically #45839

Merged
merged 14 commits into from
May 23, 2024
Merged

[BugFix] clean stale column stats periodically #45839

merged 14 commits into from
May 23, 2024

Conversation

packy92
Copy link
Contributor

@packy92 packy92 commented May 17, 2024 •
edited

Why I'm doing:

Truncate operation didn't clear the stale column statistics.

What I'm doing:

  • The record drop partition step in OlapTable.dropPartition() interface leads it will be processed in follower replay operation. So move it to the outer process to ensure it only be processed in leader.
  • add a periodic task to clear the stale column statistics. The strategy is:
    • do the clear task once every 12 hours.
    • After the last cleanup, if a table has successfully undergone a statistics collection, and the collection completion time is after the last cleanup time (or it is before the cleanup time in 30 minutes), then during the next cleanup process, the stale column statistics would be cleared.

Fixes #issue

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.3
    • 3.2
    • 3.1
    • 3.0
    • 2.5

@packy92
[BugFix] clean stale column stats periodically
37d939e 
Signed-off-by: packy92 <wangchao@starrocks.com>
@packy92 packy92 requested review from a team as code owners May 17, 2024 11:41
@packy92
add ut
cd94b12 
Signed-off-by: packy92 <wangchao@starrocks.com>
@packy92
add ut
a2ba9c0 
Signed-off-by: packy92 <wangchao@starrocks.com>
@packy92
fix ut
13e9271 
Signed-off-by: packy92 <wangchao@starrocks.com>
@packy92
add ut
a30684a 
Signed-off-by: packy92 <wangchao@starrocks.com>
@packy92
add ut
98e8bbc 
Signed-off-by: packy92 <wangchao@starrocks.com>
@packy92
fix comment
c288e0e 
Signed-off-by: packy92 <wangchao@starrocks.com>
@packy92
fix comment
285b643 
Signed-off-by: packy92 <wangchao@starrocks.com>
@packy92
fix comment
57a76ba 
Signed-off-by: packy92 <wangchao@starrocks.com>
@packy92
fix conflict
03a4160 
Signed-off-by: packy92 <wangchao@starrocks.com>
HangyuanLiu
HangyuanLiu previously approved these changes May 22, 2024
View reviewed changes
Seaven
Seaven previously approved these changes May 22, 2024
View reviewed changes
@packy92
fix comment
b2adc11 
Signed-off-by: packy92 <wangchao@starrocks.com>
@packy92 packy92 dismissed stale reviews from Seaven and HangyuanLiu via b2adc11 May 22, 2024 08:40
@packy92
fix comment
123e0d6 
Signed-off-by: packy92 <wangchao@starrocks.com>
@mergify Mergify
Copy link
Contributor

mergify bot commented May 23, 2024 •
edited

backport branch-3.3

✅ Backports have been created

@mergify Mergify
Copy link
Contributor

mergify bot commented May 23, 2024 •
edited

backport branch-3.2

✅ Backports have been created

@mergify Mergify
Copy link
Contributor

mergify bot commented May 23, 2024 •
edited

backport branch-3.1

✅ Backports have been created

@mergify Mergify
Copy link
Contributor

mergify bot commented May 23, 2024 •
edited

backport branch-3.0

✅ Backports have been created

@mergify Mergify
Copy link
Contributor

mergify bot commented May 23, 2024 •
edited

backport branch-2.5

✅ Backports have been created

mergify bot pushed a commit that referenced this pull request May 23, 2024
@packy92 @mergify
[BugFix] clean stale column stats periodically (#45839)
b398bc0 
Signed-off-by: packy92 <wangchao@starrocks.com>
(cherry picked from commit 1e4bd06)

# Conflicts:
#	fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java
@mergify mergify bot mentioned this pull request May 23, 2024
Merged
42 tasks
mergify bot pushed a commit that referenced this pull request May 23, 2024
@packy92 @mergify
[BugFix] clean stale column stats periodically (#45839)
edb7ab5 
Signed-off-by: packy92 <wangchao@starrocks.com>
(cherry picked from commit 1e4bd06)

# Conflicts:
#	fe/fe-core/src/main/java/com/starrocks/catalog/OlapTable.java
#	fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java
@mergify mergify bot mentioned this pull request May 23, 2024
Merged
24 tasks
mergify bot pushed a commit that referenced this pull request May 23, 2024
@packy92 @mergify
[BugFix] clean stale column stats periodically (#45839)
1063f01 
Signed-off-by: packy92 <wangchao@starrocks.com>
(cherry picked from commit 1e4bd06)

# Conflicts:
#	fe/fe-core/src/main/java/com/starrocks/catalog/OlapTable.java
#	fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java
#	fe/fe-core/src/main/java/com/starrocks/statistic/ExternalAnalyzeJob.java
#	fe/fe-core/src/main/java/com/starrocks/statistic/NativeAnalyzeJob.java
#	fe/fe-core/src/test/java/com/starrocks/statistic/StatisticsCollectJobTest.java
@mergify mergify bot mentioned this pull request May 23, 2024
Merged
42 tasks
mergify bot pushed a commit that referenced this pull request May 23, 2024
@packy92 @mergify
[BugFix] clean stale column stats periodically (#45839)
ba61f8b 
Signed-off-by: packy92 <wangchao@starrocks.com>
(cherry picked from commit 1e4bd06)

# Conflicts:
#	fe/fe-core/src/main/java/com/starrocks/catalog/OlapTable.java
#	fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java
#	fe/fe-core/src/main/java/com/starrocks/statistic/ExternalAnalyzeJob.java
#	fe/fe-core/src/main/java/com/starrocks/statistic/NativeAnalyzeJob.java
#	fe/fe-core/src/test/java/com/starrocks/statistic/StatisticsCollectJobTest.java
@mergify mergify bot mentioned this pull request May 23, 2024
Merged
42 tasks
mergify bot pushed a commit that referenced this pull request May 23, 2024
@packy92 @mergify
[BugFix] clean stale column stats periodically (#45839)
bc48608 
Signed-off-by: packy92 <wangchao@starrocks.com>
(cherry picked from commit 1e4bd06)

# Conflicts:
#	fe/fe-core/src/main/java/com/starrocks/catalog/OlapTable.java
#	fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java
#	fe/fe-core/src/main/java/com/starrocks/statistic/AnalyzeManager.java
#	fe/fe-core/src/main/java/com/starrocks/statistic/ExternalAnalyzeJob.java
#	fe/fe-core/src/main/java/com/starrocks/statistic/NativeAnalyzeJob.java
#	fe/fe-core/src/test/java/com/starrocks/statistic/StatisticsCollectJobTest.java
@mergify mergify bot mentioned this pull request May 23, 2024
Merged
42 tasks
wanpengfei-git pushed a commit that referenced this pull request May 24, 2024
@mergify @packy92
[BugFix] clean stale column stats periodically (backport #45839) (#46147
b1bce31 
)

Signed-off-by: packy92 <wangchao@starrocks.com>
Co-authored-by: packy92 <110370499+packy92@users.noreply.github.com>
Co-authored-by: packy92 <wangchao@starrocks.com>
wanpengfei-git pushed a commit that referenced this pull request May 24, 2024
@mergify @packy92
[BugFix] clean stale column stats periodically (backport #45839) (#46149
589b34b 
)

Signed-off-by: packy92 <wangchao@starrocks.com>
Co-authored-by: packy92 <110370499+packy92@users.noreply.github.com>
Co-authored-by: packy92 <wangchao@starrocks.com>
packy92 pushed a commit that referenced this pull request May 24, 2024
@mergify @packy92
[BugFix] clean stale column stats periodically (backport #45839) (#46149
409f2bb 
)

Signed-off-by: packy92 <wangchao@starrocks.com>
wanpengfei-git pushed a commit that referenced this pull request May 24, 2024
@mergify
[BugFix] clean stale column stats periodically (backport #45839) (#46150
47dc7ea 
)

Signed-off-by: packy92 <wangchao@starrocks.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
packy92 pushed a commit that referenced this pull request May 24, 2024
@mergify @packy92
[BugFix] clean stale column stats periodically (backport #45839) (#46150
0c53a8e 
)

Signed-off-by: packy92 <wangchao@starrocks.com>
packy92 added a commit that referenced this pull request May 24, 2024
@packy92
[BugFix] clean stale column stats periodically (backport #45839) (#46150
1a412b2 
)

Signed-off-by: packy92 <wangchao@starrocks.com>
wanpengfei-git pushed a commit that referenced this pull request May 24, 2024
@mergify @packy92
[BugFix] clean stale column stats periodically (backport #45839) (#46152
411c8b8 
)

Signed-off-by: packy92 <wangchao@starrocks.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: packy92 <wangchao@starrocks.com>
wanpengfei-git pushed a commit that referenced this pull request May 28, 2024
@mergify @packy92
[BugFix] clean stale column stats periodically (backport #45839) (#46148
d03aee6 
)

Signed-off-by: packy92 <wangchao@starrocks.com>
Co-authored-by: packy92 <110370499+packy92@users.noreply.github.com>
Co-authored-by: packy92 <wangchao@starrocks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@imay imay imay left review comments

@starrocks-cr starrocks-cr[bot] starrocks-cr left review comments

@Seaven Seaven Seaven approved these changes

@HangyuanLiu HangyuanLiu HangyuanLiu approved these changes

Assignees

@packy92 packy92

Labels
2.5-merged 3.0-merged 3.1-merged 3.2-merged 3.3-merged
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@packy92 @imay @Seaven @HangyuanLiu