Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"kubectl top nodes" reports "unknown" when executing multiple concurrent “kubectl exec” requests against a pod running on a Windows node #124700

Open
junjiezhang1997 opened this issue May 6, 2024 · 6 comments
Open

"kubectl top nodes" reports "unknown" when executing multiple concurrent “kubectl exec” requests against a pod running on a Windows node #124700

junjiezhang1997 opened this issue May 6, 2024 · 6 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/windows Categorizes an issue or PR as relevant to SIG Windows.

Comments

@junjiezhang1997
Copy link

What happened?

"kubectl top nodes" reports "unknown" when executing multiple concurrent “kubectl exec” requests against a pod

What did you expect to happen?

"kubectl top nodes" reports the status of nodes.

How can we reproduce it (as minimally and precisely as possible)?

  1. create an AKS Windows node akswin22000000.
  2. deploy a pod to this node with this yaml (kubectl apply -f hpc.yaml):
apiVersion: v1
kind: Pod
metadata:
  labels:
    pod: hpc
  name: hpc
spec:
  securityContext:
    windowsOptions:
      hostProcess: true
      runAsUserName: "NT AUTHORITY\\SYSTEM"
  hostNetwork: true
  containers:
    - name: test
      image: mcr.microsoft.com/windows/servercore:ltsc2022
      imagePullPolicy: IfNotPresent
      command:
        - powershell.exe
        - -Command
        - "Start-Sleep 2147483"
  nodeSelector:
    kubernetes.io/hostname: "akswin22000000"
  1. run the following script (test.py) and wait for several minutes.
import subprocess
import threading

def run_command(pod_name, command):
  subprocess.run(["kubectl", "exec", pod_name, "--", "powershell", "-command", command])

threads = []
for i in range(0, 50):
  thread = threading.Thread(target=run_command, args=("hpc", "& { Start-Sleep 0 }"))
  thread.start()
  threads.append(thread)

for thread in threads:
  thread.join()
  1. run kubectl top nodes and get the following result:
NAME                                CPU(cores)   CPU%        MEMORY(bytes)   MEMORY% 
aks-nodepool1-66451635-vmss000000   170m         8%          1877Mi          41%  
akswin22000000                      <unknown>    <unknown>   <unknown>       <unknown>

Anything else we need to know?

Restarting node can recover it.

But we still have the following questions:

  1. Why is “kubectl exec” causing problems? This looks like a bug: if the API is not designed to handle some load, we expect it to return an error, not hang.
  2. Why is the node not recovering its metrics, even after the “hanging” pod is restarted?

Kubernetes version

$ kubectl version
# paste output here

Client Version: version.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.1-eks-2f008fe", GitCommit:"abfec7d7e55d56346a5259c9379dea9f56ba2926", GitTreeState:"clean", BuildDate:"2023-04-14T20:43:13Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.3", GitCommit:"5214c5ddb5785ed9d8e47e79e67181e205555067", GitTreeState:"clean", BuildDate:"2024-04-12T23:22:53Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

AKS

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)
@junjiezhang1997 junjiezhang1997 added the kind/bug Categorizes issue or PR as related to a bug. label May 6, 2024
@k8s-ci-robot k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 6, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@brianpursley
Copy link
Member

I tried this w/ Linux nodes and did not experience the problem, so I think this problem is specific to Windows nodes.

/retitle "kubectl top nodes" reports "unknown" when executing multiple concurrent “kubectl exec” requests against a pod running on a Windows node

/sig windows

@junjiezhang1997 Can you run kubectl top nodes -v9 to get verbose log output? There might be something in there that could help identify what is causing this problem.

@k8s-ci-robot k8s-ci-robot changed the title "kubectl top nodes" reports "unknown" when executing multiple concurrent “kubectl exec” requests against a pod "kubectl top nodes" reports "unknown" when executing multiple concurrent “kubectl exec” requests against a pod running on a Windows node May 9, 2024
@k8s-ci-robot k8s-ci-robot added sig/windows Categorizes an issue or PR as relevant to SIG Windows. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels May 9, 2024
@junjiezhang1997
Copy link
Author

@brianpursley

I repro this issue and run kubectl top nodes -v9. The logs are as follows:

I0513 10:23:15.423644 3175933 loader.go:373] Config loaded from file:  /home/junjiezhang/.kube/config
I0513 10:23:15.425219 3175933 round_trippers.go:466] curl -v -XGET  -H "User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/abfec7d" -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "Authorization: Bearer <masked>" 'https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/api'
I0513 10:23:15.427724 3175933 round_trippers.go:495] HTTP Trace: DNS Lookup for test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io resolved to [{4.255.115.91 }]
I0513 10:23:15.653081 3175933 round_trippers.go:510] HTTP Trace: Dial to tcp:4.255.115.91:443 succeed
I0513 10:23:16.134266 3175933 round_trippers.go:553] GET https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/api 200 OK in 709 milliseconds
I0513 10:23:16.134306 3175933 round_trippers.go:570] HTTP Statistics: DNSLookup 2 ms Dial 225 ms TLSHandshake 252 ms ServerProcessing 228 ms Duration 709 ms
I0513 10:23:16.134319 3175933 round_trippers.go:577] Response Headers:
I0513 10:23:16.134334 3175933 round_trippers.go:580]     Cache-Control: public
I0513 10:23:16.134346 3175933 round_trippers.go:580]     Content-Type: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList
I0513 10:23:16.134357 3175933 round_trippers.go:580]     Etag: "7E2E4FBD8CDC884130EBFEF64757F6BC507A8EA6A2EA8F94E05F1B6F207B4E183C63D6D6EA5513FDDC10FC16233248F933DC787C288493D52745A3FAFD68CCD8"
I0513 10:23:16.134369 3175933 round_trippers.go:580]     Vary: Accept
I0513 10:23:16.134381 3175933 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: c63bbe1a-37b9-4027-a25b-c0af27c789f9
I0513 10:23:16.134392 3175933 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 52025469-7ffe-46a6-8b56-59ca942f158b
I0513 10:23:16.134400 3175933 round_trippers.go:580]     Date: Mon, 13 May 2024 10:23:16 GMT
I0513 10:23:16.134413 3175933 round_trippers.go:580]     Audit-Id: a6c10bfc-8ad3-4c62-af55-a3bf6bba25ba
I0513 10:23:16.134531 3175933 request.go:1188] Response Body: {"kind":"APIGroupDiscoveryList","apiVersion":"apidiscovery.k8s.io/v2beta1","metadata":{},"items":[{"metadata":{"creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"bindings","responseKind":{"group":"","version":"","kind":"Binding"},"scope":"Namespaced","singularResource":"binding","verbs":["create"]},{"resource":"componentstatuses","responseKind":{"group":"","version":"","kind":"ComponentStatus"},"scope":"Cluster","singularResource":"componentstatus","verbs":["get","list"],"shortNames":["cs"]},{"resource":"configmaps","responseKind":{"group":"","version":"","kind":"ConfigMap"},"scope":"Namespaced","singularResource":"configmap","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["cm"]},{"resource":"endpoints","responseKind":{"group":"","version":"","kind":"Endpoints"},"scope":"Namespaced","singularResource":"endpoints","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["ep"]},{"resource":"events","responseKind":{"group":"","version":"","kind":"Event"},"scope":"Namespaced","singularResource":"event","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["ev"]},{"resource":"limitranges","responseKind":{"group":"","version":"","kind":"LimitRange"},"scope":"Namespaced","singularResource":"limitrange","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["limits"]},{"resource":"namespaces","responseKind":{"group":"","version":"","kind":"Namespace"},"scope":"Cluster","singularResource":"namespace","verbs":["create","delete","get","list","patch","update","watch"],"shortNames":["ns"],"subresources":[{"subresource":"finalize","responseKind":{"group":"","version":"","kind":"Namespace"},"verbs":["update"]},{"subresource":"status","responseKind":{"group":"","version":"","kind":"Namespace"},"verbs":["get","patch","update"]}]},{"resource":"nodes","responseKind":{"group":"","version":"","kind":"Node"},"scope":"Cluster","singularResource":"node","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["no"],"subresources":[{"subresource":"proxy","responseKind":{"group":"","version":"","kind":"NodeProxyOptions"},"verbs":["create","delete","get","patch","update"]},{"subresource":"status","responseKind":{"group":"","version":"","kind":"Node"},"verbs":["get","patch","update"]}]},{"resource":"persistentvolumeclaims","responseKind":{"group":"","version":"","kind":"PersistentVolumeClaim"},"scope":"Namespaced","singularResource":"persistentvolumeclaim","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["pvc"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"PersistentVolumeClaim"},"verbs":["get","patch","update"]}]},{"resource":"persistentvolumes","responseKind":{"group":"","version":"","kind":"PersistentVolume"},"scope":"Cluster","singularResource":"persistentvolume","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["pv"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"PersistentVolume"},"verbs":["get","patch","update"]}]},{"resource":"pods","responseKind":{"group":"","version":"","kind":"Pod"},"scope":"Namespaced","singularResource":"pod","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["po"],"categories":["all"],"subresources":[{"subresource":"attach","responseKind":{"group":"","version":"","kind":"PodAttachOptions"},"verbs":["create","get"]},{"subresource":"binding","responseKind":{"group":"","version":"","kind":"Binding"},"verbs":["create"]},{"subresource":"ephemeralcontainers","responseKind":{"group":"","version":"","kind":"Pod"},"verbs":["get","patch","update"]},{"subresource":"eviction","responseKind":{"group":"policy","version":"v1","kind":"Eviction"},"verbs":["create"]},{"subresource":"exec","responseKind":{"group":"","version":"","kind":"PodExecOptions"},"verbs":["create","get"]},{"subresource":"log","responseKind":{"group":"","version":"","kind":"Pod"},"verbs":["get"]},{"subresource":"portforward","responseKind":{"group":"","version":"","kind":"PodPortForwardOptions"},"verbs":["create","get"]},{"subresource":"proxy","responseKind":{"group":"","version":"","kind":"PodProxyOptions"},"verbs":["create","delete","get","patch","update"]},{"subresource":"status","responseKind":{"group":"","version":"","kind":"Pod"},"verbs":["get","patch","update"]}]},{"resource":"podtemplates","responseKind":{"group":"","version":"","kind":"PodTemplate"},"scope":"Namespaced","singularResource":"podtemplate","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"resource":"replicationcontrollers","responseKind":{"group":"","version":"","kind":"ReplicationController"},"scope":"Namespaced","singularResource":"replicationcontroller","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["rc"],"categories":["all"],"subresources":[{"subresource":"scale","responseKind":{"group":"autoscaling","version":"v1","kind":"Scale"},"verbs":["get","patch","update"]},{"subresource":"status","responseKind":{"group":"","version":"","kind":"ReplicationController"},"verbs":["get","patch","update"]}]},{"resource":"resourcequotas","responseKind":{"group":"","version":"","kind":"ResourceQuota"},"scope":"Namespaced","singularResource":"resourcequota","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["quota"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"ResourceQuota"},"verbs":["get","patch","update"]}]},{"resource":"secrets","responseKind":{"group":"","version":"","kind":"Secret"},"scope":"Namespaced","singularResource":"secret","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"resource":"serviceaccounts","responseKind":{"group":"","version":"","kind":"ServiceAccount"},"scope":"Namespaced","singularResource":"serviceaccount","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["sa"],"subresources":[{"subresource":"token","responseKind":{"group":"authentication.k8s.io","version":"v1","kind":"TokenRequest"},"verbs":["create"]}]},{"resource":"services","responseKind":{"group":"","version":"","kind":"Service"},"scope":"Namespaced","singularResource":"service","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["svc"],"categories":["all"],"subresources":[{"subresource":"proxy","responseKind":{"group":"","version":"","kind":"ServiceProxyOptions"},"verbs":["create","delete","get","patch","update"]},{"subresource":"status","responseKind":{"group":"","version":"","kind":"Service"},"verbs":["get","patch","update"]}]}],"freshness":"Current"}]}]}
I0513 10:23:16.135217 3175933 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/abfec7d" -H "Authorization: Bearer <masked>" 'https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/apis'
I0513 10:23:16.363471 3175933 round_trippers.go:553] GET https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/apis 200 OK in 228 milliseconds
I0513 10:23:16.363505 3175933 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 228 ms Duration 228 ms
I0513 10:23:16.363524 3175933 round_trippers.go:577] Response Headers:
I0513 10:23:16.363543 3175933 round_trippers.go:580]     Cache-Control: public
I0513 10:23:16.363563 3175933 round_trippers.go:580]     Content-Type: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList
I0513 10:23:16.363581 3175933 round_trippers.go:580]     Etag: "F3CA980379F4458FDC5E712C487718213D325C01671AC580CFDF3CE5483D8E326404020681338D0251257E9D296F0F876D6CA648937AD3785D79557DE34A9BD2"
I0513 10:23:16.363603 3175933 round_trippers.go:580]     Vary: Accept
I0513 10:23:16.363621 3175933 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: c63bbe1a-37b9-4027-a25b-c0af27c789f9
I0513 10:23:16.363639 3175933 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 52025469-7ffe-46a6-8b56-59ca942f158b
I0513 10:23:16.363658 3175933 round_trippers.go:580]     Date: Mon, 13 May 2024 10:23:16 GMT
I0513 10:23:16.363676 3175933 round_trippers.go:580]     Audit-Id: e023d35b-d6c9-4e3d-b75b-65d4c3a6b421
I0513 10:23:16.588368 3175933 request.go:1188] Response Body: {"kind":"APIGroupDiscoveryList","apiVersion":"apidiscovery.k8s.io/v2beta1","metadata":{},"items":[{"metadata":{"name":"apiregistration.k8s.io","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"apiservices","responseKind":{"group":"","version":"","kind":"APIService"},"scope":"Cluster","singularResource":"apiservice","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"categories":["api-extensions"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"APIService"},"verbs":["get","patch","update"]}]}],"freshness":"Current"}]},{"metadata":{"name":"apps","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"controllerrevisions","responseKind":{"group":"","version":"","kind":"ControllerRevision"},"scope":"Namespaced","singularResource":"controllerrevision","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"resource":"daemonsets","responseKind":{"group":"","version":"","kind":"DaemonSet"},"scope":"Namespaced","singularResource":"daemonset","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["ds"],"categories":["all"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"DaemonSet"},"verbs":["get","patch","update"]}]},{"resource":"deployments","responseKind":{"group":"","version":"","kind":"Deployment"},"scope":"Namespaced","singularResource":"deployment","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["deploy"],"categories":["all"],"subresources":[{"subresource":"scale","responseKind":{"group":"autoscaling","version":"v1","kind":"Scale"},"verbs":["get","patch","update"]},{"subresource":"status","responseKind":{"group":"","version":"","kind":"Deployment"},"verbs":["get","patch","update"]}]},{"resource":"replicasets","responseKind":{"group":"","version":"","kind":"ReplicaSet"},"scope":"Namespaced","singularResource":"replicaset","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["rs"],"categories":["all"],"subresources":[{"subresource":"scale","responseKind":{"group":"autoscaling","version":"v1","kind":"Scale"},"verbs":["get","patch","update"]},{"subresource":"status","responseKind":{"group":"","version":"","kind":"ReplicaSet"},"verbs":["get","patch","update"]}]},{"resource":"statefulsets","responseKind":{"group":"","version":"","kind":"StatefulSet"},"scope":"Namespaced","singularResource":"statefulset","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["sts"],"categories":["all"],"subresources":[{"subresource":"scale","responseKind":{"group":"autoscaling","version":"v1","kind":"Scale"},"verbs":["get","patch","update"]},{"subresource":"status","responseKind":{"group":"","version":"","kind":"StatefulSet"},"verbs":["get","patch","update"]}]}],"freshness":"Current"}]},{"metadata":{"name":"events.k8s.io","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"events","responseKind":{"group":"","version":"","kind":"Event"},"scope":"Namespaced","singularResource":"event","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["ev"]}],"freshness":"Current"}]},{"metadata":{"name":"authentication.k8s.io","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"selfsubjectreviews","responseKind":{"group":"","version":"","kind":"SelfSubjectReview"},"scope":"Cluster","singularResource":"selfsubjectreview","verbs":["create"]},{"resource":"tokenreviews","responseKind":{"group":"","version":"","kind":"TokenReview"},"scope":"Cluster","singularResource":"tokenreview","verbs":["create"]}],"freshness":"Current"},{"version":"v1beta1","resources":[{"resource":"selfsubjectreviews","responseKind":{"group":"","version":"","kind":"SelfSubjectReview"},"scope":"Cluster","singularResource":"selfsubjectreview","verbs":["create"]}],"freshness":"Current"}]},{"metadata":{"name":"authorization.k8s.io","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"localsubjectaccessreviews","responseKind":{"group":"","version":"","kind":"LocalSubjectAccessReview"},"scope":"Namespaced","singularResource":"localsubjectaccessreview","verbs":["create"]},{"resource":"selfsubjectaccessreviews","responseKind":{"group":"","version":"","kind":"SelfSubjectAccessReview"},"scope":"Cluster","singularResource":"selfsubjectaccessreview","verbs":["create"]},{"resource":"selfsubjectrulesreviews","responseKind":{"group":"","version":"","kind":"SelfSubjectRulesReview"},"scope":"Cluster","singularResource":"selfsubjectrulesreview","verbs":["create"]},{"resource":"subjectaccessreviews","responseKind":{"group":"","version":"","kind":"SubjectAccessReview"},"scope":"Cluster","singularResource":"subjectaccessreview","verbs":["create"]}],"freshness":"Current"}]},{"metadata":{"name":"autoscaling","creationTimestamp":null},"versions":[{"version":"v2","resources":[{"resource":"horizontalpodautoscalers","responseKind":{"group":"","version":"","kind":"HorizontalPodAutoscaler"},"scope":"Namespaced","singularResource":"horizontalpodautoscaler","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["hpa"],"categories":["all"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"HorizontalPodAutoscaler"},"verbs":["get","patch","update"]}]}],"freshness":"Current"},{"version":"v1","resources":[{"resource":"horizontalpodautoscalers","responseKind":{"group":"","version":"","kind":"HorizontalPodAutoscaler"},"scope":"Namespaced","singularResource":"horizontalpodautoscaler","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["hpa"],"categories":["all"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"HorizontalPodAutoscaler"},"verbs":["get","patch","update"]}]}],"freshness":"Current"}]},{"metadata":{"name":"batch","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"cronjobs","responseKind":{"group":"","version":"","kind":"CronJob"},"scope":"Namespaced","singularResource":"cronjob","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["cj"],"categories":["all"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"CronJob"},"verbs":["get","patch","update"]}]},{"resource":"jobs","responseKind":{"group":"","version":"","kind":"Job"},"scope":"Namespaced","singularResource":"job","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"categories":["all"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"Job"},"verbs":["get","patch","update"]}]}],"freshness":"Current"}]},{"metadata":{"name":"certificates.k8s.io","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"certificatesigningrequests","responseKind":{"group":"","version":"","kind":"CertificateSigningRequest"},"scope":"Cluster","singularResource":"certificatesigningrequest","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["csr"],"subresources":[{"subresource":"approval","responseKind":{"group":"","version":"","kind":"CertificateSigningRequest"},"verbs":["get","patch","update"]},{"subresource":"status","responseKind":{"group":"","version":"","kind":"CertificateSigningRequest"},"verbs":["get","patch","update"]}]}],"freshness":"Current"}]},{"metadata":{"name":"networking.k8s.io","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"ingressclasses","responseKind":{"group":"","version":"","kind":"IngressClass"},"scope":"Cluster","singularResource":"ingressclass","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"resource":"ingresses","responseKind":{"group":"","version":"","kind":"Ingress"},"scope":"Namespaced","singularResource":"ingress","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["ing"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"Ingress"},"verbs":["get","patch","update"]}]},{"resource":"networkpolicies","responseKind":{"group":"","version":"","kind":"NetworkPolicy"},"scope":"Namespaced","singularResource":"networkpolicy","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["netpol"]}],"freshness":"Current"}]},{"metadata":{"name":"policy","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"poddisruptionbudgets","responseKind":{"group":"","version":"","kind":"PodDisruptionBudget"},"scope":"Namespaced","singularResource":"poddisruptionbudget","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["pdb"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"PodDisruptionBudget"},"verbs":["get","patch","update"]}]}],"freshness":"Current"}]},{"metadata":{"name":"rbac.authorization.k8s.io","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"clusterrolebindings","responseKind":{"group":"","version":"","kind":"ClusterRoleBinding"},"scope":"Cluster","singularResource":"clusterrolebinding","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"resource":"clusterroles","responseKind":{"group":"","version":"","kind":"ClusterRole"},"scope":"Cluster","singularResource":"clusterrole","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"resource":"rolebindings","responseKind":{"group":"","version":"","kind":"RoleBinding"},"scope":"Namespaced","singularResource":"rolebinding","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"resource":"roles","responseKind":{"group":"","version":"","kind":"Role"},"scope":"Namespaced","singularResource":"role","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]}],"freshness":"Current"}]},{"metadat [truncated 18294 chars]
I0513 10:23:16.589355 3175933 round_trippers.go:466] curl -v -XGET  -H "Accept: application/vnd.kubernetes.protobuf, */*" -H "User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/abfec7d" -H "Authorization: Bearer <masked>" 'https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/apis/metrics.k8s.io/v1beta1/nodes'
I0513 10:23:16.818811 3175933 round_trippers.go:553] GET https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/apis/metrics.k8s.io/v1beta1/nodes 200 OK in 229 milliseconds
I0513 10:23:16.818843 3175933 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 229 ms Duration 229 ms
I0513 10:23:16.818862 3175933 round_trippers.go:577] Response Headers:
I0513 10:23:16.818878 3175933 round_trippers.go:580]     Audit-Id: 0ad280de-994e-4617-be16-771d5d1f6a98
I0513 10:23:16.818899 3175933 round_trippers.go:580]     Audit-Id: 0ad280de-994e-4617-be16-771d5d1f6a98
I0513 10:23:16.818918 3175933 round_trippers.go:580]     Date: Mon, 13 May 2024 10:23:16 GMT
I0513 10:23:16.818935 3175933 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: c63bbe1a-37b9-4027-a25b-c0af27c789f9
I0513 10:23:16.818953 3175933 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 52025469-7ffe-46a6-8b56-59ca942f158b
I0513 10:23:16.818972 3175933 round_trippers.go:580]     Cache-Control: no-cache, private
I0513 10:23:16.818991 3175933 round_trippers.go:580]     Cache-Control: no-cache, private
I0513 10:23:16.819005 3175933 round_trippers.go:580]     Content-Type: application/vnd.kubernetes.protobuf
I0513 10:23:16.819035 3175933 round_trippers.go:580]     Content-Length: 1546
I0513 10:23:16.819120 3175933 request.go:1186] Response Body:
00000000  6b 38 73 00 0a 29 0a 16  6d 65 74 72 69 63 73 2e  |k8s..)..metrics.|
00000010  6b 38 73 2e 69 6f 2f 76  31 62 65 74 61 31 12 0f  |k8s.io/v1beta1..|
00000020  4e 6f 64 65 4d 65 74 72  69 63 73 4c 69 73 74 12  |NodeMetricsList.|
00000030  d4 0b 0a 06 0a 00 12 00  1a 00 12 c9 0b 0a 87 0b  |................|
00000040  0a 21 61 6b 73 2d 6e 6f  64 65 70 6f 6f 6c 31 2d  |.!aks-nodepool1-|
00000050  38 31 38 37 36 34 33 34  2d 76 6d 73 73 30 30 30  |81876434-vmss000|
00000060  30 30 30 12 00 1a 00 22  00 2a 00 32 00 38 00 42  |000....".*.2.8.B|
00000070  08 08 94 d4 87 b2 06 10  00 5a 16 0a 09 61 67 65  |.........Z...age|
00000080  6e 74 70 6f 6f 6c 12 09  6e 6f 64 65 70 6f 6f 6c  |ntpool..nodepool|
00000090  31 5a 20 0a 17 62 65 74  61 2e 6b 75 62 65 72 6e  |1Z ..beta.kubern|
000000a0  65 74 65 73 2e 69 6f 2f  61 72 63 68 12 05 61 6d  |etes.io/arch..am|
000000b0  64 36 34 5a 33 0a 20 62  65 74 61 2e 6b 75 62 65  |d64Z3. beta.kube|
000000c0  72 6e 65 74 65 73 2e 69  6f 2f 69 6e 73 74 61 6e  |rnetes.io/instan|
000000d0  63 65 2d 74 79 70 65 12  0f 53 74 61 6e 64 61 72  |ce-type..Standar|
000000e0  64 5f 44 53 32 5f 76 32  5a 1e 0a 15 62 65 74 61  |d_DS2_v2Z...beta|
000000f0  2e 6b 75 62 65 72 6e 65  74 65 73 2e 69 6f 2f 6f  |.kubernetes.io/o|
00000100  73 12 05 6c 69 6e 75 78  5a 32 0a 28 66 61 69 6c  |s..linuxZ2.(fail|
00000110  75 72 65 2d 64 6f 6d 61  69 6e 2e 62 65 74 61 2e  |ure-domain.beta.|
00000120  6b 75 62 65 72 6e 65 74  65 73 2e 69 6f 2f 72 65  |kubernetes.io/re|
00000130  67 69 6f 6e 12 06 65 61  73 74 75 73 5a 2b 0a 26  |gion..eastusZ+.&|
00000140  66 61 69 6c 75 72 65 2d  64 6f 6d 61 69 6e 2e 62  |failure-domain.b|
00000150  65 74 61 2e 6b 75 62 65  72 6e 65 74 65 73 2e 69  |eta.kubernetes.i|
00000160  6f 2f 7a 6f 6e 65 12 01  30 5a 2b 0a 1e 6b 75 62  |o/zone..0Z+..kub|
00000170  65 72 6e 65 74 65 73 2e  61 7a 75 72 65 2e 63 6f  |ernetes.azure.co|
00000180  6d 2f 61 67 65 6e 74 70  6f 6f 6c 12 09 6e 6f 64  |m/agentpool..nod|
00000190  65 70 6f 6f 6c 31 5a 47  0a 1c 6b 75 62 65 72 6e  |epool1ZG..kubern|
000001a0  65 74 65 73 2e 61 7a 75  72 65 2e 63 6f 6d 2f 63  |etes.azure.com/c|
000001b0  6c 75 73 74 65 72 12 27  4d 43 5f 6a 75 6e 6a 69  |luster.'MC_junji|
000001c0  65 7a 68 61 6e 67 2d 74  65 73 74 5f 74 65 73 74  |ezhang-test_test|
000001d0  2d 63 6c 75 73 74 65 72  5f 65 61 73 74 75 73 5a  |-cluster_eastusZ|
000001e0  5f 0a 37 6b 75 62 65 72  6e 65 74 65 73 2e 61 7a  |_.7kubernetes.az|
000001f0  75 72 65 2e 63 6f 6d 2f  63 6f 6e 73 6f 6c 69 64  |ure.com/consolid|
00000200  61 74 65 64 2d 61 64 64  69 74 69 6f 6e 61 6c 2d  |ated-additional-|
00000210  70 72 6f 70 65 72 74 69  65 73 12 24 35 61 61 65  |properties.$5aae|
00000220  61 35 35 36 2d 31 30 65  64 2d 31 31 65 66 2d 39  |a556-10ed-11ef-9|
00000230  34 66 31 2d 66 36 32 65  36 63 31 37 30 39 32 65  |4f1-f62e6c17092e|
00000240  5a 57 0a 2f 6b 75 62 65  72 6e 65 74 65 73 2e 61  |ZW./kubernetes.a|
00000250  7a 75 72 65 2e 63 6f 6d  2f 6b 75 62 65 6c 65 74  |zure.com/kubelet|
00000260  2d 69 64 65 6e 74 69 74  79 2d 63 6c 69 65 6e 74  |-identity-client|
00000270  2d 69 64 12 24 30 35 33  39 33 61 32 32 2d 65 34  |-id.$05393a22-e4|
00000280  63 65 2d 34 62 30 37 2d  62 66 65 30 2d 65 62 37  |ce-4b07-bfe0-eb7|
00000290  31 65 34 34 65 30 66 37  64 5a 23 0a 19 6b 75 62  |1e44e0f7dZ#..kub|
000002a0  65 72 6e 65 74 65 73 2e  61 7a 75 72 65 2e 63 6f  |ernetes.azure.co|
000002b0  6d 2f 6d 6f 64 65 12 06  73 79 73 74 65 6d 5a 53  |m/mode..systemZS|
000002c0  0a 27 6b 75 62 65 72 6e  65 74 65 73 2e 61 7a 75  |.'kubernetes.azu|
000002d0  72 65 2e 63 6f 6d 2f 6e  6f 64 65 2d 69 6d 61 67  |re.com/node-imag|
000002e0  65 2d 76 65 72 73 69 6f  6e 12 28 41 4b 53 55 62  |e-version.(AKSUb|
000002f0  75 6e 74 75 2d 32 32 30  34 67 65 6e 32 63 6f 6e  |untu-2204gen2con|
00000300  74 61 69 6e 65 72 64 2d  32 30 32 34 30 35 2e 30  |tainerd-202405.0|
00000310  33 2e 30 5a 3d 0a 22 6b  75 62 65 72 6e 65 74 65  |3.0Z=."kubernete|
00000320  73 2e 61 7a 75 72 65 2e  63 6f 6d 2f 6e 6f 64 65  |s.azure.com/node|
00000330  70 6f 6f 6c 2d 74 79 70  65 12 17 56 69 72 74 75  |pool-type..Virtu|
00000340  61 6c 4d 61 63 68 69 6e  65 53 63 61 6c 65 53 65  |alMachineScaleSe|
00000350  74 73 5a 25 0a 1b 6b 75  62 65 72 6e 65 74 65 73  |tsZ%..kubernetes|
00000360  2e 61 7a 75 72 65 2e 63  6f 6d 2f 6f 73 2d 73 6b  |.azure.com/os-sk|
00000370  75 12 06 55 62 75 6e 74  75 5a 22 0a 19 6b 75 62  |u..UbuntuZ"..kub|
00000380  65 72 6e 65 74 65 73 2e  61 7a 75 72 65 2e 63 6f  |ernetes.azure.co|
00000390  6d 2f 72 6f 6c 65 12 05  61 67 65 6e 74 5a 2e 0a  |m/role..agentZ..|
000003a0  23 6b 75 62 65 72 6e 65  74 65 73 2e 61 7a 75 72  |#kubernetes.azur|
000003b0  65 2e 63 6f 6d 2f 73 74  6f 72 61 67 65 70 72 6f  |e.com/storagepro|
000003c0  66 69 6c 65 12 07 6d 61  6e 61 67 65 64 5a 2f 0a  |file..managedZ/.|
000003d0  20 6b 75 62 65 72 6e 65  74 65 73 2e 61 7a 75 72  | kubernetes.azur|
000003e0  65 2e 63 6f 6d 2f 73 74  6f 72 61 67 65 74 69 65  |e.com/storagetie|
000003f0  72 12 0b 50 72 65 6d 69  75 6d 5f 4c 52 53 5a 1b  |r..Premium_LRSZ.|
00000400  0a 12 6b 75 62 65 72 6e  65 74 65 73 2e 69 6f 2f  |..kubernetes.io/|
00000410  61 72 63 68 12 05 61 6d  64 36 34 5a 3b 0a 16 6b  |arch..amd64Z;..k|
00000420  75 62 65 72 6e 65 74 65  73 2e 69 6f 2f 68 6f 73  |ubernetes.io/hos|
00000430  74 6e 61 6d 65 12 21 61  6b 73 2d 6e 6f 64 65 70  |tname.!aks-nodep|
00000440  6f 6f 6c 31 2d 38 31 38  37 36 34 33 34 2d 76 6d  |ool1-81876434-vm|
00000450  73 73 30 30 30 30 30 30  5a 19 0a 10 6b 75 62 65  |ss000000Z...kube|
00000460  72 6e 65 74 65 73 2e 69  6f 2f 6f 73 12 05 6c 69  |rnetes.io/os..li|
00000470  6e 75 78 5a 1b 0a 12 6b  75 62 65 72 6e 65 74 65  |nuxZ...kubernete|
00000480  73 2e 69 6f 2f 72 6f 6c  65 12 05 61 67 65 6e 74  |s.io/role..agent|
00000490  5a 21 0a 1d 6e 6f 64 65  2d 72 6f 6c 65 2e 6b 75  |Z!..node-role.ku|
000004a0  62 65 72 6e 65 74 65 73  2e 69 6f 2f 61 67 65 6e  |bernetes.io/agen|
000004b0  74 12 00 5a 33 0a 20 6e  6f 64 65 2e 6b 75 62 65  |t..Z3. node.kube|
000004c0  72 6e 65 74 65 73 2e 69  6f 2f 69 6e 73 74 61 6e  |rnetes.io/instan|
000004d0  63 65 2d 74 79 70 65 12  0f 53 74 61 6e 64 61 72  |ce-type..Standar|
000004e0  64 5f 44 53 32 5f 76 32  5a 19 0a 0e 73 74 6f 72  |d_DS2_v2Z...stor|
000004f0  61 67 65 70 72 6f 66 69  6c 65 12 07 6d 61 6e 61  |ageprofile..mana|
00000500  67 65 64 5a 1a 0a 0b 73  74 6f 72 61 67 65 74 69  |gedZ...storageti|
00000510  65 72 12 0b 50 72 65 6d  69 75 6d 5f 4c 52 53 5a  |er..Premium_LRSZ|
00000520  24 0a 20 74 6f 70 6f 6c  6f 67 79 2e 64 69 73 6b  |$. topology.disk|
00000530  2e 63 73 69 2e 61 7a 75  72 65 2e 63 6f 6d 2f 7a  |.csi.azure.com/z|
00000540  6f 6e 65 12 00 5a 33 0a  2e 74 6f 70 6f 6c 6f 67  |one..Z3..topolog|
00000550  79 2e 6b 75 62 65 72 6e  65 74 65 73 2e 61 7a 75  |y.kubernetes.azu|
00000560  72 65 2e 63 6f 6d 2f 73  75 62 2d 66 61 75 6c 74  |re.com/sub-fault|
00000570  2d 64 6f 6d 61 69 6e 12  01 30 5a 27 0a 1d 74 6f  |-domain..0Z'..to|
00000580  70 6f 6c 6f 67 79 2e 6b  75 62 65 72 6e 65 74 65  |pology.kubernete|
00000590  73 2e 69 6f 2f 72 65 67  69 6f 6e 12 06 65 61 73  |s.io/region..eas|
000005a0  74 75 73 5a 20 0a 1b 74  6f 70 6f 6c 6f 67 79 2e  |tusZ ..topology.|
000005b0  6b 75 62 65 72 6e 65 74  65 73 2e 69 6f 2f 7a 6f  |kubernetes.io/zo|
000005c0  6e 65 12 01 30 7a 00 12  08 08 d5 d3 87 b2 06 10  |ne..0z..........|
000005d0  00 1a 07 08 80 88 98 d3  e0 01 22 13 0a 03 63 70  |.........."...cp|
000005e0  75 12 0c 0a 0a 31 38 36  36 35 34 32 38 34 6e 22  |u....186654284n"|
000005f0  15 0a 06 6d 65 6d 6f 72  79 12 0b 0a 09 32 31 30  |...memory....210|
00000600  30 39 32 38 4b 69 1a 00  22 00                    |0928Ki..".|
I0513 10:23:16.819378 3175933 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/abfec7d" -H "Authorization: Bearer <masked>" 'https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/api/v1/nodes'
I0513 10:23:17.048296 3175933 round_trippers.go:553] GET https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/api/v1/nodes 200 OK in 228 milliseconds
I0513 10:23:17.048320 3175933 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 228 ms Duration 228 ms
I0513 10:23:17.048334 3175933 round_trippers.go:577] Response Headers:
I0513 10:23:17.048352 3175933 round_trippers.go:580]     Cache-Control: no-cache, private
I0513 10:23:17.048368 3175933 round_trippers.go:580]     Content-Type: application/json
I0513 10:23:17.048383 3175933 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: c63bbe1a-37b9-4027-a25b-c0af27c789f9
I0513 10:23:17.048400 3175933 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 52025469-7ffe-46a6-8b56-59ca942f158b
I0513 10:23:17.048416 3175933 round_trippers.go:580]     Date: Mon, 13 May 2024 10:23:16 GMT
I0513 10:23:17.048432 3175933 round_trippers.go:580]     Audit-Id: a78221cf-e726-4c11-aa62-66c21d97d1da
I0513 10:23:17.048642 3175933 request.go:1188] Response Body: {"kind":"NodeList","apiVersion":"v1","metadata":{"resourceVersion":"58839"},"items":[{"metadata":{"name":"aks-nodepool1-81876434-vmss000000","uid":"4c0e41c9-e79c-4f36-8765-25131f0f941f","resourceVersion":"57760","creationTimestamp":"2024-05-13T05:58:54Z","labels":{"agentpool":"nodepool1","beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"Standard_DS2_v2","beta.kubernetes.io/os":"linux","failure-domain.beta.kubernetes.io/region":"eastus","failure-domain.beta.kubernetes.io/zone":"0","kubernetes.azure.com/agentpool":"nodepool1","kubernetes.azure.com/cluster":"MC_junjiezhang-test_test-cluster_eastus","kubernetes.azure.com/consolidated-additional-properties":"5aaea556-10ed-11ef-94f1-f62e6c17092e","kubernetes.azure.com/kubelet-identity-client-id":"05393a22-e4ce-4b07-bfe0-eb71e44e0f7d","kubernetes.azure.com/mode":"system","kubernetes.azure.com/node-image-version":"AKSUbuntu-2204gen2containerd-202405.03.0","kubernetes.azure.com/nodepool-type":"VirtualMachineScaleSets","kubernetes.azure.com/os-sku":"Ubuntu","kubernetes.azure.com/role":"agent","kubernetes.azure.com/storageprofile":"managed","kubernetes.azure.com/storagetier":"Premium_LRS","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"aks-nodepool1-81876434-vmss000000","kubernetes.io/os":"linux","kubernetes.io/role":"agent","node-role.kubernetes.io/agent":"","node.kubernetes.io/instance-type":"Standard_DS2_v2","storageprofile":"managed","storagetier":"Premium_LRS","topology.disk.csi.azure.com/zone":"","topology.kubernetes.azure.com/sub-fault-domain":"0","topology.kubernetes.io/region":"eastus","topology.kubernetes.io/zone":"0"},"annotations":{"csi.volume.kubernetes.io/nodeid":"{\"disk.csi.azure.com\":\"aks-nodepool1-81876434-vmss000000\",\"file.csi.azure.com\":\"aks-nodepool1-81876434-vmss000000\"}","node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"},"managedFields":[{"manager":"ccp-initializer","operation":"Update","apiVersion":"v1","time":"2024-05-13T05:58:54Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-05-13T05:58:54Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.azure.com/agentpool":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/consolidated-additional-properties":{},"f:kubernetes.azure.com/kubelet-identity-client-id":{},"f:kubernetes.azure.com/mode":{},"f:kubernetes.azure.com/node-image-version":{},"f:kubernetes.azure.com/nodepool-type":{},"f:kubernetes.azure.com/os-sku":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.azure.com/storageprofile":{},"f:kubernetes.azure.com/storagetier":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:storageprofile":{},"f:storagetier":{}}}}},{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2024-05-13T05:58:55Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}}},{"manager":"cloud-node-manager","operation":"Update","apiVersion":"v1","time":"2024-05-13T05:59:10Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.azure.com/sub-fault-domain":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-05-13T10:18:11Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.disk.csi.azure.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}},"subresource":"status"},{"manager":"node-problem-detector","operation":"Update","apiVersion":"v1","time":"2024-05-13T10:18:26Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainerRuntimeProblem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FilesystemCorruptionProblem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KubeletProblem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"VMEventScheduled\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}},"subresource":"status"}]},"spec":{"providerID":"azure:///subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/mc_junjiezhang-test_test-cluster_eastus/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-81876434-vmss/virtualMachines/0"},"status":{"capacity":{"cpu":"2","ephemeral-storage":"129886128Ki","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"7097620Ki","pods":"30"},"allocatable":{"cpu":"1900m","ephemeral-storage":"119703055367","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"4652308Ki","pods":"30"},"conditions":[{"type":"FrequentDockerRestart","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:01:47Z","reason":"NoFrequentDockerRestart","message":"docker is functioning properly"},{"type":"FrequentContainerdRestart","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:01:47Z","reason":"NoFrequentContainerdRestart","message":"containerd is functioning properly"},{"type":"FrequentUnregisterNetDevice","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:01:47Z","reason":"NoFrequentUnregisterNetDevice","message":"node is functioning properly"},{"type":"KernelDeadlock","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:01:47Z","reason":"KernelHasNoDeadlock","message":"kernel has no deadlock"},{"type":"ReadonlyFilesystem","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:01:47Z","reason":"FilesystemIsNotReadOnly","message":"Filesystem is not read-only"},{"type":"FilesystemCorruptionProblem","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:01:47Z","reason":"FilesystemIsOK","message":"Filesystem is healthy"},{"type":"ContainerRuntimeProblem","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:01:47Z","reason":"ContainerRuntimeIsUp","message":"container runtime service is up"},{"type":"KubeletProblem","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:01:47Z","reason":"KubeletIsUp","message":"kubelet service is up"},{"type":"FrequentKubeletRestart","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:01:47Z","reason":"NoFrequentKubeletRestart","message":"kubelet is functioning properly"},{"type":"VMEventScheduled","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:02:52Z","reason":"NoVMEventScheduled","message":"VM has no scheduled event"},{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2024-05-13T10:18:11Z","lastTransitionTime":"2024-05-13T05:58:54Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2024-05-13T10:18:11Z","lastTransitionTime":"2024-05-13T05:58:54Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2024-05-13T10:18:11Z","lastTransitionTime":"2024-05-13T05:58:54Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2024-05-13T10:18:11Z","lastTransitionTime":"2024-05-13T05:58:54Z","reason":"KubeletReady","message":"kubelet is posting ready status. AppArmor enabled"}],"addresses":[{"type":"InternalIP","address":"10.224.0.4"},{"type":"Hostname","address":"aks-nodepool1-81876434-vmss000000"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"5a1df07f9c934f2c93edd6633c426d95","systemUUID":"36346b6d-bcdd-4540-91b3-26c5afadfdee","bootID":"fc0e2e97-3d64-40ad-bc24-a7bd9a208d18","kernelVersion":"5.15.0-1061-azure","osImage":"Ubuntu 22.04.4 LTS","containerRuntimeVersion":"containerd://1.7.15-1","kubeletVersion":"v1.28.9","kubeProxyVersion":"v1.28.9","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["mcr.microsoft.com/azuredefender/stable/ [truncated 19893 chars]
NAME                                CPU(cores)   CPU%        MEMORY(bytes)   MEMORY%     
aks-nodepool1-81876434-vmss000000   187m         9%          2051Mi          45%         
akswin22000000                      <unknown>    <unknown>   <unknown>       <unknown>

@brianpursley
Copy link
Member

Here's the code where <unknown> is printed.

kubernetes/staging/src/k8s.io/kubectl/pkg/metricsutil/metrics_printer.go

Line 185 in 71acf56

unknownMetricsStatus := "<unknown>"

I can't tell from the log output (because it gets truncated), but I'm guessing that /apis/metrics.k8s.io/v1beta1/nodes doesn't return metrics for akswin22000000.

You should be able to check for sure by running:

kubectl get --raw https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/apis/metrics.k8s.io/v1beta1/nodes

Typical output (formatted via jq) should look something like this:

{
  "kind": "NodeMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metadata": {
        "name": "k8s-master",
        "creationTimestamp": "2024-05-13T14:46:48Z",
        "labels": {
          "beta.kubernetes.io/arch": "amd64",
          "beta.kubernetes.io/os": "linux",
          "kubernetes.io/arch": "amd64",
          "kubernetes.io/hostname": "k8s-master",
          "kubernetes.io/os": "linux",
          "node-role.kubernetes.io/control-plane": "",
          "node.kubernetes.io/exclude-from-external-load-balancers": ""
        }
      },
      "timestamp": "2024-05-13T14:46:37Z",
      "window": "10.025s",
      "usage": {
        "cpu": "199102244n",
        "memory": "2754236Ki"
      }
    },
    {
      "metadata": {
        "name": "k8s-worker-1",
        "creationTimestamp": "2024-05-13T14:46:48Z",
        "labels": {
          "beta.kubernetes.io/arch": "amd64",
          "beta.kubernetes.io/os": "linux",
          "kubernetes.io/arch": "amd64",
          "kubernetes.io/hostname": "k8s-worker-1",
          "kubernetes.io/os": "linux"
        }
      },
      "timestamp": "2024-05-13T14:46:38Z",
      "window": "10.027s",
      "usage": {
        "cpu": "39892290n",
        "memory": "1849496Ki"
      }
    },
    {
      "metadata": {
        "name": "k8s-worker-2",
        "creationTimestamp": "2024-05-13T14:46:48Z",
        "labels": {
          "beta.kubernetes.io/arch": "amd64",
          "beta.kubernetes.io/os": "linux",
          "kubernetes.io/arch": "amd64",
          "kubernetes.io/hostname": "k8s-worker-2",
          "kubernetes.io/os": "linux"
        }
      },
      "timestamp": "2024-05-13T14:46:43Z",
      "window": "20.045s",
      "usage": {
        "cpu": "38712895n",
        "memory": "1770956Ki"
      }
    }
  ]
}

@junjiezhang1997
Copy link
Author

I tried kubectl get --raw https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/apis/metrics.k8s.io/v1beta1/nodes | jq and got the following results:

{
  "kind": "NodeMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metadata": {
        "name": "aks-nodepool1-81876434-vmss000000",
        "creationTimestamp": "2024-05-14T02:34:04Z",
        "labels": {
          "agentpool": "nodepool1",
          "beta.kubernetes.io/arch": "amd64",
          "beta.kubernetes.io/instance-type": "Standard_DS2_v2",
          "beta.kubernetes.io/os": "linux",
          "failure-domain.beta.kubernetes.io/region": "eastus",
          "failure-domain.beta.kubernetes.io/zone": "0",
          "kubernetes.azure.com/agentpool": "nodepool1",
          "kubernetes.azure.com/cluster": "MC_junjiezhang-test_test-cluster_eastus",
          "kubernetes.azure.com/consolidated-additional-properties": "5aaea556-10ed-11ef-94f1-f62e6c17092e",
          "kubernetes.azure.com/kubelet-identity-client-id": "05393a22-e4ce-4b07-bfe0-eb71e44e0f7d",
          "kubernetes.azure.com/mode": "system",
          "kubernetes.azure.com/node-image-version": "AKSUbuntu-2204gen2containerd-202405.03.0",
          "kubernetes.azure.com/nodepool-type": "VirtualMachineScaleSets",
          "kubernetes.azure.com/os-sku": "Ubuntu",
          "kubernetes.azure.com/role": "agent",
          "kubernetes.azure.com/storageprofile": "managed",
          "kubernetes.azure.com/storagetier": "Premium_LRS",
          "kubernetes.io/arch": "amd64",
          "kubernetes.io/hostname": "aks-nodepool1-81876434-vmss000000",
          "kubernetes.io/os": "linux",
          "kubernetes.io/role": "agent",
          "node-role.kubernetes.io/agent": "",
          "node.kubernetes.io/instance-type": "Standard_DS2_v2",
          "storageprofile": "managed",
          "storagetier": "Premium_LRS",
          "topology.disk.csi.azure.com/zone": "",
          "topology.kubernetes.azure.com/sub-fault-domain": "0",
          "topology.kubernetes.io/region": "eastus",
          "topology.kubernetes.io/zone": "0"
        }
      },
      "timestamp": "2024-05-14T02:32:58Z",
      "window": "1m0.239s",
      "usage": {
        "cpu": "177028171n",
        "memory": "2140828Ki"
      }
    }
  ]
}

I think you are right. It doesn't return the result for windows node.

@brianpursley
Copy link
Member

Possibly related:

Those issues are closed, but it could be a related problem, or they weren't entirely fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/windows Categorizes an issue or PR as relevant to SIG Windows.
Projects
[sig-windows] Issue Tracking
Status: No status
Milestone
No milestone
Development

No branches or pull requests
3 participants
@brianpursley @k8s-ci-robot @junjiezhang1997