Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix kubelet memory leak when device plugin is registered #124719

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

carlory
Copy link
Member

@carlory carlory commented May 7, 2024

What type of PR is this?

/kind bug

What this PR does / why we need it:

Two types of memory leaks:

  • (c *client) Connect() error creates a gRPC connection between device manager and device plugin. but if dail sccuessfully but do c.handler.PluginConnected(c.resource, c) fails, the caller will not close the connection. This is a memory leak. This PR fixes this by closing the connection if c.handler.PluginConnected(c.resource, c) fails.

  • 2 different socket file but shares the same plugin name which is populated from plugin via GetInfo interface. the old registerClient func will rewrite the old client with the new client. the old client will not be closed. This is a memory leak. This PR fixes this by adding a check. In this case, it return an error to the caller.

Which issue(s) this PR fixes:

Fixes #124716

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Fix kubelet memory leak when device plugin connection fails and register same plugin twice with different socket file. 

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels May 7, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: carlory
Once this PR has been reviewed and has the lgtm label, please assign dchen1107 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels May 7, 2024
Copy link
Contributor

@ffromani ffromani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/ok-to-test

we need to make sure we have test coverage for the flow modified here

Comment on lines 99 to 100
path1 := existing.(DevicePlugin).SocketPath()
path2 := c.(DevicePlugin).SocketPath()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we have a way to get the path avoiding the cast?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There‘s no other way except using client{} structure. The Interface way is more recommended in the Go world.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd try adding a SocketPath method to the Client interface and evaluate the impacts.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or, in general, carrying this information around somehow. I'd really like to explore options to avoid the cast.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd try adding a SocketPath method to the Client interface and evaluate the impacts.

I added it in this PR.

Comment on lines 75 to 85

err = c.handler.PluginConnected(c.resource, c)
if err != nil {
klog.ErrorS(err, "Failed to connect to device plugin", "resource", c.resource)
if err := conn.Close(); err != nil {
klog.V(2).ErrorS(err, "Failed to close grcp connection", "resource", c.resource)
}
return err
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder (and I'm not pushing for this solution, but let's give it a fair evaluation) if we can use a defer to catch the issue instead of moving PluginConnected above. Something like

func (c *client) Connect() (rerr error) {
    // ...

    c.mutex.Lock()
    c.grpc = conn
    c.client = client
    c.mutex.Unlock()
    defer func() {
        if rerr == nil {
            return
        }
        klog.ErrorS(err, "Failed to connect to device plugin", "resource", c.resource)
        if err := conn.Close(); err != nil {
            klog.V(2).ErrorS(err, "Failed to close grcp connection", "resource", c.resource)
           return
        }
        c.grpc = nil
    }()
    return c.handler.PluginConnected(c.resource, c)
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If so, we have to lock again to reset c.grpc and c.client in defer func

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right. On second look the order of operations should not be too critical here. I'll have a deeper look later on.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

line 80: Failed to close grcp connection
"grcp"?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed.

And @ffromani I follow your suggestion.

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label May 7, 2024
@ffromani
Copy link
Contributor

ffromani commented May 7, 2024

/triage accepted
/priority backlog

priority subjected to review like the linked issue

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. priority/backlog Higher priority than priority/awaiting-more-evidence. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels May 7, 2024
@carlory
Copy link
Member Author

carlory commented May 7, 2024

we need to make sure we have test coverage for the flow modified here.

I will do it later. Unfortunately, there is no existing test case in this package. I need some investigation how to create a test case for this problem.

@carlory carlory force-pushed the fix-124716 branch 5 times, most recently from 516071a to 2080bfb Compare May 7, 2024 10:55
c := s.getClient(name)
if c != nil {
if c.SocketPath() != socketPath {
return fmt.Errorf("the device plugin %s already registered with a different socket path %s", name, c.SocketPath())
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return fmt.Errorf("the device plugin %s already registered with a different socket path %s", name, c.SocketPath())
return fmt.Errorf("The device plugin %s already registered with a different socket path %s", name, c.SocketPath())

Fantastic work. Stick with one

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go-staticcheck: error strings should not be capitalized (ST1005)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got it, thx

@carlory
Copy link
Member Author

carlory commented May 8, 2024

/hold

Needs to handle re-registration case.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 8, 2024
@k8s-ci-robot
Copy link
Contributor

@carlory: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-kubernetes-unit d28810f link true /test pull-kubernetes-unit
pull-kubernetes-integration d28810f link true /test pull-kubernetes-integration

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@carlory carlory marked this pull request as draft May 8, 2024 10:31
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 8, 2024
@bart0sh bart0sh added this to Triage in SIG Node PR Triage May 8, 2024
@bart0sh bart0sh moved this from Triage to WIP in SIG Node PR Triage May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/kubelet cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/bug Categorizes issue or PR as related to a bug. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. priority/backlog Higher priority than priority/awaiting-more-evidence. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Development

Successfully merging this pull request may close these issues.

Kubelet memory leak when a plugin is registered twice
5 participants