multus daemonset init error: exec: "cp": executable file not found in $PATH: unknown when v4.0.2

[wenwutang1]

What happend:
deploy multus cni(v4.0.2) occur error: exec: "cp": executable file not found in $PATH: unknown, but change container image to "multus-cni:snapshot-thick" it working.
this multus-cni:v4.0.2 image hash:sha256:ed70b5729dcedf4fcfeefba25cd9d8f43fc7120ac4f768b42ba5de088ad2a509

daemonset config:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-multus-ds
  namespace: kube-system
  labels:
    tier: node
    app: multus
    name: multus
spec:
  selector:
    matchLabels:
      name: multus
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        tier: node
        app: multus
        name: multus
    spec:
      hostNetwork: true
      hostPID: true
      tolerations:
        - operator: Exists
          effect: NoSchedule
        - operator: Exists
          effect: NoExecute
      serviceAccountName: multus
      containers:
        - name: kube-multus
          image: multus-cni:v4.0.2
          command: [ "/usr/src/multus-cni/bin/multus-daemon" ]
          resources:
            requests:
              cpu: "100m"
              memory: "50Mi"
            limits:
              cpu: "100m"
              memory: "50Mi"
          securityContext:
            privileged: true
          volumeMounts:
            - name: cni
              mountPath: /host/etc/cni/net.d
            - name: cnibin
              mountPath: /opt/cni/bin
            - name: host-run
              mountPath: /host/run
            - name: host-var-lib-cni-multus
              mountPath: /var/lib/cni/multus
            - name: host-var-lib-kubelet
              mountPath: /var/lib/kubelet
            - name: host-run-k8s-cni-cncf-io
              mountPath: /run/k8s.cni.cncf.io
            - name: host-run-netns
              mountPath: /run/netns
              mountPropagation: HostToContainer
            - name: multus-daemon-config
              mountPath: /etc/cni/net.d/multus.d
              readOnly: true
            - name: hostroot
              mountPath: /hostroot
              mountPropagation: HostToContainer
          env:
            - name: MULTUS_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
      initContainers:
        - name: install-multus-binary
          image: multus-cni:v4.0.0
          command:
            - "cp"
            - "/usr/src/multus-cni/bin/multus-shim"
            - "/host/opt/cni/bin/multus-shim"
          resources:
            requests:
              cpu: "10m"
              memory: "15Mi"
          securityContext:
            privileged: true
          volumeMounts:
            - name: cnibin
              mountPath: /host/opt/cni/bin
              mountPropagation: Bidirectional
      terminationGracePeriodSeconds: 10
      volumes:
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: cnibin
          hostPath:
            path: /opt/cni/bin
        - name: hostroot
          hostPath:
            path: /
        - name: multus-daemon-config
          configMap:
            name: multus-daemon-config
            items:
            - key: daemon-config.json
              path: daemon-config.json
        - name: host-run
          hostPath:
            path: /run
        - name: host-var-lib-cni-multus
          hostPath:
            path: /var/lib/cni/multus
        - name: host-var-lib-kubelet
          hostPath:
            path: /var/lib/kubelet
        - name: host-run-k8s-cni-cncf-io
          hostPath:
            path: /run/k8s.cni.cncf.io
        - name: host-run-netns
          hostPath:
            path: /run/netns/

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Multus version
  image path and image ID (from 'docker images') k8snetworkplumbingwg/multus-cni:v4.0.2
• Kubernetes version (use kubectl version): v1.27.10
• Primary CNI for Kubernetes cluster: calico
• OS (e.g. from /etc/os-release):
• File of '/etc/cni/net.d/'
• File of '/etc/cni/multus/net.d'
• NetworkAttachment info (use kubectl get net-attach-def -o yaml)
• Target pod yaml info (with annotation, use kubectl get pod <podname> -o yaml)
• Other log outputs (if you use multus logging)

[dougbtv]

Thanks for the report...

Just a quick note you have a mistmatch with:

          image: multus-cni:v4.0.2

and in the init containers

          image: multus-cni:v4.0.0

This probably doesn't matter that the cp command isn't in the tagged versions, so, that's worth digging into. Thanks for now the snapshot does address that

[johren-hpe]

I ran into this same issue before I realized I was using the wrong image. I believe you want to use multus-cni:v4.0.2-thick since you are using the thick deployment. That image does have the cp command.