multus kubeconfig file not populated correctly after node reboot

[geeky-akshay]

What happend: Apiserver IP address was empty in multus kubeconfig file

/etc/cni/net.d/multus.d/multus.kubeconfig (server: https://[])
`apiVersion: v1
kind: Config
clusters:
- name: local
  cluster:
    server: https://[]:
    certificate-authority-data: [snipped]==
users:
- name: multus
  user:
    token: "[snipped]=="
contexts:
- name: multus-context
  context:
    cluster: local
    user: multus
current-context: multus-context`

Pods
What you expected to happen: Pods were not scheduled and the below errors were appearing in /var/log/messages

May 20 14:13:20 ace-func1-3n1 kubelet[1683]: E0520 14:13:20.399208 1683 pod_workers.go:1298] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"d4fd3ede-c09e-4a3f-abd2-8eeb6ba0b3da\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"d02259dca386dfd70b0d6433452e99a880b57e849f79f74993ace14d90e55fb9\\\": plugin type=\\\"multus\\\" name=\\\"multus-cni-network\\\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: error loading config file \\\"/etc/cni/net.d/multus.d/multus.kubeconfig\\\": yaml: line 7: mapping values are not allowed in this context\"" pod="kube-system/coredns-58f4964b57-f8wkh" podUID="d4fd3ede-c09e-4a3f-abd2-8eeb6ba0b3da"

How to reproduce it (as minimally and precisely as possible): Reboot a node in multi-node Kubernetes cluster

Anything else we need to know?:

Environment:

• Multus version : ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2
• Kubernetes version (use kubectl version):
  Client Version: v1.29.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.1
• Primary CNI for Kubernetes cluster: Calico
• OS (e.g. from /etc/os-release):
  `
  NAME="Red Hat Enterprise Linux"
  VERSION="9.3 (Plow)"
  ID="rhel"
  ID_LIKE="fedora"
  VERSION_ID="9.3"
  PLATFORM_ID="platform:el9"
  PRETTY_NAME="Red Hat Enterprise Linux 9.3 (Plow)"
  ANSI_COLOR="0;31"
  LOGO="fedora-logo-icon"
  CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
  HOME_URL="https://www.redhat.com/"
  DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
  BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"
`

• File of '/etc/cni/net.d/'
• File of '/etc/cni/multus/net.d'
• NetworkAttachment info (use kubectl get net-attach-def -o yaml)
• Target pod yaml info (with annotation, use kubectl get pod <podname> -o yaml)
  `
  apiVersion: v1
  kind: Pod
  metadata:
  creationTimestamp: "2024-05-20T09:02:12Z"
  generateName: kube-multus-ds-
  labels:
  app: multus
  controller-revision-hash: 789c4467b8
  name: multus
  pod-template-generation: "1"
  tier: node
  name: kube-multus-ds-hxtkt
  namespace: kube-system
  ownerReferences:
  • apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: DaemonSet
    name: kube-multus-ds
    uid: cc6cd4f8-540a-4593-b099-4be99d0088bd
    resourceVersion: "2107922"
    uid: 308c615b-8284-4c79-91f5-9c3ec976e0e6
    spec:
    affinity:
    nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchFields:
    - key: metadata.name
    operator: In
    values:
    - ace-func1-3n1
    containers:
  • args:
    • --cni-version=0.3.1
    • --cni-conf-dir=/host/etc/cni/net.d
    • --multus-autoconfig-dir=/host/etc/cni/net.d
    • --multus-log-to-stderr=true
    • --multus-log-level=verbose
      command:
    • /thin_entrypoint
      image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2
      imagePullPolicy: IfNotPresent
      name: kube-multus
      resources:
      limits:
      cpu: 100m
      memory: 50Mi
      requests:
      cpu: 100m
      memory: 50Mi
      securityContext:
      privileged: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
    • mountPath: /host/etc/cni/net.d
      name: cni
    • mountPath: /host/opt/cni/bin
      name: cnibin
    • mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-5wxv9
      readOnly: true
      dnsPolicy: ClusterFirst
      enableServiceLinks: true
      hostNetwork: true
      initContainers:
  • args:
    • --type
    • thin
      command:
    • /install_multus
      image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2
      imagePullPolicy: IfNotPresent
      name: install-multus-binary
      resources:
      requests:
      cpu: 10m
      memory: 15Mi
      securityContext:
      privileged: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
    • mountPath: /host/opt/cni/bin
      mountPropagation: Bidirectional
      name: cnibin
    • mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-5wxv9
      readOnly: true
      nodeName: ace-func1-3n1
      preemptionPolicy: PreemptLowerPriority
      priority: 2000001000
      priorityClassName: system-node-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: multus
      serviceAccountName: multus
      terminationGracePeriodSeconds: 10
      tolerations:
  • effect: NoSchedule
    operator: Exists
  • effect: NoExecute
    operator: Exists
  • effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
  • effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
  • effect: NoSchedule
    key: node.kubernetes.io/disk-pressure
    operator: Exists
  • effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  • effect: NoSchedule
    key: node.kubernetes.io/pid-pressure
    operator: Exists
  • effect: NoSchedule
    key: node.kubernetes.io/unschedulable
    operator: Exists
  • effect: NoSchedule
    key: node.kubernetes.io/network-unavailable
    operator: Exists
    volumes:
  • hostPath:
    path: /etc/cni/net.d
    type: ""
    name: cni
  • hostPath:
    path: /opt/cni/bin
    type: ""
    name: cnibin
  • name: kube-api-access-5wxv9
    projected:
    defaultMode: 420
    sources:
    • serviceAccountToken:
      expirationSeconds: 3607
      path: token
    • configMap:
      items:
      • key: ca.crt
        path: ca.crt
        name: kube-root-ca.crt
    • downwardAPI:
      items:
      • fieldRef:
        apiVersion: v1
        fieldPath: metadata.namespace
        path: namespace
        status:
        conditions:
  • lastProbeTime: null
    lastTransitionTime: "2024-05-20T09:02:13Z"
    status: "True"
    type: PodReadyToStartContainers
  • lastProbeTime: null
    lastTransitionTime: "2024-05-20T09:02:14Z"
    status: "True"
    type: Initialized
  • lastProbeTime: null
    lastTransitionTime: "2024-05-20T09:02:15Z"
    status: "True"
    type: Ready
  • lastProbeTime: null
    lastTransitionTime: "2024-05-20T09:02:15Z"
    status: "True"
    type: ContainersReady
  • lastProbeTime: null
    lastTransitionTime: "2024-05-20T09:02:12Z"
    status: "True"
    type: PodScheduled
    containerStatuses:
  • containerID: containerd://fe1413443a789d09e25900c72cd3cfc2d33a6ccd5308bf75741284952ed58d18
    image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2
    imageID: sha256:2ce3492d2e6f598566f6ec26d08c39ca8c43df5beeefbb7ce38a1a4ef3809760
    lastState: {}
    name: kube-multus
    ready: true
    restartCount: 0
    started: true
    state:
    running:
    startedAt: "2024-05-20T09:02:14Z"
    hostIP: 10.52.214.11
    hostIPs:
  • ip: 10.52.214.11
    initContainerStatuses:
  • containerID: containerd://9e4195df464d4d98784b1d7521d5d71a775a34f334e0dc889e7f366171f8a0e3
    image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2
    imageID: sha256:2ce3492d2e6f598566f6ec26d08c39ca8c43df5beeefbb7ce38a1a4ef3809760
    lastState: {}
    name: install-multus-binary
    ready: true
    restartCount: 0
    started: false
    state:
    terminated:
    containerID: containerd://9e4195df464d4d98784b1d7521d5d71a775a34f334e0dc889e7f366171f8a0e3
    exitCode: 0
    finishedAt: "2024-05-20T09:02:13Z"
    reason: Completed
    startedAt: "2024-05-20T09:02:13Z"
    phase: Running
    podIP: 10.52.214.11
    podIPs:
  • ip: 10.52.214.11
    qosClass: Burstable
    startTime: "2024-05-20T09:02:12Z"
    `
• Other log outputs (if you use multus logging)

[dougbtv]

Do you have a reproducer for this? thanks!