multus v4 thick mode breaks NetworkAttachmentDefinition with CNI config file

[leerin-ruby]

What happend:
I used multus + istio cni for app sidecar injection, while for the injection, pod reports error with "ERRORED: error configuring pod [nrd/nrd-v1-77c6dbbd48-586x5] networking: Multus: [nrd/nrd-v1-77c6dbbd48-586x5/4c1a126c-c889-472e-9e44-46193c8cfaa3]: error loading k8s delegates k8s args: TryLoadPodDelegates: error in getting k8s network for pod: GetNetworkDelegates: failed getting the delegate: GetCNIConfig: err in GetCNIConfigFromFile: no network available in the name istio-cni in cni dir /host/etc/cni/net.d"

Seems multus CNI directory where other CNI that is used by multus as net-attach-def without config was changed from "/etc/cni/multus/net.d" to "/host/etc/cni/net.d" in multus v4 thick mode.

What you expected to happen:
keep the directory as "/etc/cni/multus/net.d" the same as v3 and v4 thin mode

How to reproduce it (as minimally and precisely as possible):
install multus v4 thick
install istio with cni enabled
install a app with sidecar injection

Anything else we need to know?:
from the source code, seems the "confDir" in deamonset config is change to "/host/etc/cni/net.d" in v4.0.2 and was removed in master

Environment:
tanzu2.5

• Multus version
  image path and image ID (from 'docker images')
  Image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot-thick
  Image ID: ghcr.io/k8snetworkplumbingwg/multus-cni@sha256:e47cac8a15cc2f5f7e25a92bcd56e508888d2d7bed25b38aae5b0bb09fb84c1c

• Kubernetes version (use kubectl version): 1.28

• Primary CNI for Kubernetes cluster: calico

• OS (e.g. from /etc/os-release): tkg25-cni-md-0-gqrxm-rtxh5 6.1.62-6.ph5

• File of '/etc/cni/net.d/'
  00-multus.conf 10-calico.conflist calico-kubeconfig multus.d

• File of '/etc/cni/multus/net.d'
  istio-cni.conf ZZZ-istio-cni-kubeconfig

• NetworkAttachment info (use kubectl get net-attach-def -o yaml)
  `apiVersion: v1
  items:

• apiVersion: k8s.cni.cncf.io/v1
  kind: NetworkAttachmentDefinition
  metadata:
  creationTimestamp: "2024-03-06T08:40:55Z"
  generation: 1
  name: istio-cni
  namespace: default
  resourceVersion: "280413"
  uid: e41aa7e1-a5de-439e-932f-0e41b2268f25
  spec:
  config: ""
  kind: List
  metadata:
  resourceVersion: ""
  `

• Target pod yaml info (with annotation, use kubectl get pod <podname> -o yaml)

• Other log outputs (if you use multus logging)

[dougbtv]

I think that what you should do is modify the daemonset to add another mount point that points at /etc/cni/multus/net.d