Thick plugin: Add configuration options in documentation to configure Master CNI

[raghs-aws]

What happend:
Thick plugin doesn't set the master cni or the default cni, and keeps "multusConfigFile": "auto". This causes Multus to pick other cnis than primary-cni in some cases , if we have some other cnis installed like istio-cni.

What you expected to happen:

There should be an option to keep Master CNI defined to avoid multus accidently picking other cnis if there are more than 1 cnis in the path.

How to reproduce it (as minimally and precisely as possible):

install istio-cni with hostNetwork enabled

  # Configure ambient settings
  ambient:
    # If enabled, ambient redirection will be enabled
    enabled: true
    # Set ambient redirection mode: "iptables" or "ebpf"
    redirectMode: "iptables"

Anything else we need to know?:

we can override the behavior if add "multusMasterCNI" in the daemon-config. below is an option (in this case its vpc-cni)

    "multusMasterCNI": "10-aws.conflist"

i dont see documentation mentioning this for thick plugin. request is to update this in the Thick plugin documentation and/or configuration documentation, so that users are aware how to override the auto selection of primary cni.
Environment:

• Multus version : 4.0.2
  image path and image ID (from 'docker images')
• Kubernetes version (use kubectl version): 1.25
• Primary CNI for Kubernetes cluster: vpc-cni
• OS (e.g. from /etc/os-release): “Amazon Linux2” “centos rhel fedora”
• File of '/etc/cni/net.d/' :
  --rw-r--r-- 1 root root 906 Jan 25 17:24 10-aws.conflist
  -rw------- 1 root root 216 Jan 27 14:54 00-multus.conf
  drwxr-xr-x 2 root root 60 Mar 5 17:19 whereabouts.d
  -rw------- 1 root root 2947 Mar 5 17:52 ZZZ-istio-cni-kubeconfig
  -rw-r--r-- 1 root root 334 Mar 5 17:52 YYY-istio-cni.conf
• File of '/etc/cni/multus/net.d'
• NetworkAttachment info (use kubectl get net-attach-def -o yaml)
• Target pod yaml info (with annotation, use kubectl get pod <podname> -o yaml)
• Other log outputs (if you use multus logging)

[raghs-aws]

if needed, I can create a PR to update the documentation.

[dougbtv]

Thanks Raghs -- can you provide a documentation update PR and we can continue the discussion there? Thanks!

[raghs-aws]

Thanks @dougbtv . opened a PR : #1245

[raghs-aws]

@dougbtv Could you please review the above PR.

[abasitt]

@raghs-aws thank you for sharing about this, I was looking in to migration from thin to thick plugin and was wondering exactly about this master-cni configs in the thick plugin.
Is this understanding correct?.
thinplugin clusterNetwork= thinkplugin multusMasterCNI

what is equivalent to defaultnetworks in thickplugin configs ? and wondering if auto can be disabled ?

[abasitt]

Seems like there are are also manual options for masternetwork even in thick plugin, but still good to know about this flag to use if multusconfigs are set for auto. Anyway will play around with thick plugin to explore it.

[raghs-aws]

yes in old thin plugin it was multus-master-cni-file option. This flag multusMasterCNI with your primary CNI was the solution. we used and running it in our environment. This solved the race condition between istio and primary CNI, you can use auto and which is fine for other configs. i tried a few other configs like clusterNetwork etc, but somehow didnt help.
My PR for documentation update is pending in review.