Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FR: Customize kube api address #11397

Closed
bingkunyangvungle opened this issue Mar 12, 2024 · 3 comments · Fixed by #12115
Closed

FR: Customize kube api address #11397

bingkunyangvungle opened this issue Mar 12, 2024 · 3 comments · Fixed by #12115

Comments

@bingkunyangvungle
Copy link

What are you trying to do?

We're using a cloud provider and their cluster's ca certificate has a customized address and it's different (for example: https://kube.cloud.dev) from the default one: https://kubernetes.default.svc.

By default, the certificate looks like this:
`* Server certificate:

  • subject: CN=kube-apiserver
  • start date: Dec 29 07:59:06 2020 GMT
  • expire date: Mar 4 18:02:10 2025 GMT
  • issuer: CN=kubernetes`

In our case, it looks like this:
`* Server certificate:

  • subject: CN=*.kube.cloud.dev
  • start date: Jan 5 12:16:17 2024 GMT
  • expire date: Apr 4 12:16:16 2024 GMT
  • issuer: C=US; O=Let's Encrypt; CN=R3`

When I was trying to install tailscale operator, it has the following error:
failed to verify certificate: x509: certificate is valid for *.kube.cloud.dev, not kubernetes.default.svc

In this case, we would like the tailscale operator to support reading from the environment variables: KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT. While KUBERNETES_SERVICE_HOST can be kube.cloud.dev and KUBERNETES_SERVICE_PORT can be 443.

How should we solve this?

Just as I mentioned above, just enable the setting of these two environment variables: KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT and read the value and set it here: https://github.com/tailscale/tailscale/blob/main/kube/client.go#L31. In this way, the issue can be solved.

What is the impact of not solving this?

After enabling this feature, tailscale operator can enable the customized kube server address, other than https://kubernetes.default.svc. This can give the user more freedom.

Anything else?

No response

@irbekrm
Copy link
Contributor

irbekrm commented Mar 12, 2024

When I was trying to install tailscale operator, it has the following error:
failed to verify certificate: x509: certificate is valid for *.kube.cloud.dev, not kubernetes.default.svc

Do you see this in operator's logs? Or, in one of proxy's logs?

The operator itself uses controller-runtime client with a client-go mechanism for retrieving api-server address etc. It does not use the client that you linked. However, the proxies that we create do.

If possible, we should try to auto-detect the address.

@bingkunyangvungle
Copy link
Author

@irbekrm Thank you for the response! Yeah, I think there's only one container in the pod operator-58b8c85c97-5ztvb and the container's name is operator. So I think it should be the operator's logs.

@bingkunyangvungle
Copy link
Author

bingkunyangvungle commented Mar 18, 2024

I went through the Dockerfile for tailscale repo and found that it only builds the image for tailscale, but not the operator. Do you know where's the script/Dockerfile to build the operator? @irbekrm

Since I couldn't build and testing locally, here's one suggestion of change for detecting and updating the kube server address: (looks like I couldn't create the PR)
In the file: tailscale/cmd/k8s-operator/operator.go:

  1. Add this dependency: "k8s.io/client-go/tools/clientcmd"
  2. Add this function:
func loadConfigWithApiServer() (*rest.Config, error) {
	fmt.Printf("Entring loadConfigWithApiServer\n")
	loadingRules := clientcmd.NewDefaultClientConfigLoadingRules()
	if _, ok := os.LookupEnv("HOME"); !ok {
		u, err := user.Current()
		if err != nil {
			return nil, fmt.Errorf("could not get current user: %w", err)
		}
		loadingRules.Precedence = append(loadingRules.Precedence, filepath.Join(u.HomeDir, clientcmd.RecommendedHomeDir, clientcmd.RecommendedFileName))
	}	
	
	return config.loadConfigWithContext(fmt.Sprintf("https://%s:%s", os.Getenv("KUBERNETES_SERVICE_HOST"), os.Getenv("KUBERNETES_SERVICE_PORT_HTTPS")), loadingRules, "")
}
  1. Change this line in the main():
    restConfig := config.GetConfigOrDie()
    to this line:
    restConfig := loadConfigWithApiServer()

I refer to this function in config.go: link. In this way, we can dynamically update the kube service address easily. Please help to review if this one is feasible. Thank you! @irbekrm

ChandonPierre added a commit to ChandonPierre/tailscale that referenced this issue May 31, 2024
…erver URL via ENV

Updates tailscale#11397

Signed-off-by: Chandon Pierre <cpierre@coreweave.com>
ChandonPierre added a commit to ChandonPierre/tailscale that referenced this issue May 31, 2024
…erver URL via ENV

Updates tailscale#11397

Signed-off-by: Chandon Pierre <cpierre@coreweave.com>
irbekrm pushed a commit that referenced this issue May 31, 2024
…erver URL via ENV (#12115)

Updates #11397

Signed-off-by: Chandon Pierre <cpierre@coreweave.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants