Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multus NIC cannot connect to the external NW. #1255

Open
InoueReo opened this issue Apr 4, 2024 · 1 comment
Open

Multus NIC cannot connect to the external NW. #1255

InoueReo opened this issue Apr 4, 2024 · 1 comment
Labels
help wanted

Comments

@InoueReo
Copy link

InoueReo commented Apr 4, 2024

What happend:
I refer to https://github.com/Orange-OpenSource/towards5gs-helm , and I install helm. The free5GC helm install succeeds but cannot connect to the external NW. it appears that it is not tied to the Multus Master Interface. How to reslove this problems?

What you expected to happen:
If the Multus Master Interface is connected to the Multus Master Interface, it is recognized that XXX@ifYYYis displayed, but it appears to be displayed incorrectly.

bash output

$ ip a
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 4c:52:62:65:bb:fd brd ff:ff:ff:ff:ff:ff
    inet 172.31.16.37/24 brd 172.31.16.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::4e52:62ff:fe65:bbfd/64 scope link
       valid_lft forever preferred_lft forever
7: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether b8:ce:f6:72:b0:d4 brd ff:ff:ff:ff:ff:ff
    inet 172.31.1.30/24 brd 172.31.1.255 scope global ens1f0
       valid_lft forever preferred_lft forever
    inet6 fe80::bace:f6ff:fe72:b0d4/64 scope link
       valid_lft forever preferred_lft forever

$ helm install free5gc-v1 ./free5gc/ --set free5gc-upf.upf.securityContext.privileged=true --set free5gc-upf.upf.securityContext.capabilities=null --set global.n6network.subnetIP=172.31.1.0 --set global.n6network.gatewayIP=172.31.1.1 --set global.n6network.excludeIP=172.31.1.254 --set free5gc-upf.upf.n6if.ipAddress=172.31.1.11 --set global.n2network.masterIf=eno1 --set global.n3network.masterIf=eno1 --set global.n4network.masterIf=eno1 --set global.n6network.masterIf=ens1f0 --set global.n9network.masterIf=eno1

$ kubectl get po
NAME                                                    READY   STATUS    RESTARTS   AGE
free5gc-v1-free5gc-amf-amf-69c476d9cb-k47kz             1/1     Running   0          52s
free5gc-v1-free5gc-ausf-ausf-7675996f67-ch2bt           1/1     Running   0          51s
free5gc-v1-free5gc-dbpython-dbpython-7d4c7fd7f8-zc254   1/1     Running   0          52s
free5gc-v1-free5gc-nrf-nrf-55cffd6c79-ft4w2             1/1     Running   0          51s
free5gc-v1-free5gc-nssf-nssf-795f9f5577-79rdz           1/1     Running   0          51s
free5gc-v1-free5gc-pcf-pcf-797f677cfc-kzwk4             1/1     Running   0          51s
free5gc-v1-free5gc-smf-smf-766cddcd99-skk95             1/1     Running   0          51s
free5gc-v1-free5gc-udm-udm-684f7b8688-p4drb             1/1     Running   0          52s
free5gc-v1-free5gc-udr-udr-65cdb74d7d-tv7ch             1/1     Running   0          50s
free5gc-v1-free5gc-upf-upf-57c6c59d8b-9qw8f             1/1     Running   0          51s
free5gc-v1-free5gc-webui-webui-5b9b94bb95-x2q4n         1/1     Running   0          52s
mongodb-0                                               1/1     Running   0          51s

$ kubectl exec -it free5gc-v1-free5gc-upf-upf-57c6c59d8b-9qw8f -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if51: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default
    link/ether fe:2b:34:2a:32:a5 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 182.16.184.168/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::fc2b:34ff:fe2a:32a5/64 scope link
       valid_lft forever preferred_lft forever
5: n3@tunl0: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether 4c:52:62:65:c5:d4 brd ff:ff:ff:ff:ff:ff
    inet 10.100.50.233/29 brd 10.100.50.239 scope global n3
       valid_lft forever preferred_lft forever
    inet6 fe80::4c52:6200:165:c5d4/64 scope link
       valid_lft forever preferred_lft forever
6: n6@n4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default
    link/ether b8:ce:f6:7c:6b:58 brd ff:ff:ff:ff:ff:ff
    inet 172.31.1.11/24 brd 172.31.1.255 scope global n6
       valid_lft forever preferred_lft forever
    inet6 fe80::b8ce:f600:17c:6b58/64 scope link
       valid_lft forever preferred_lft forever
7: n4@tunl0: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether 4c:52:62:65:c5:d4 brd ff:ff:ff:ff:ff:ff
    inet 10.100.50.241/29 brd 10.100.50.247 scope global n4
       valid_lft forever preferred_lft forever
    inet6 fe80::4c52:6200:265:c5d4/64 scope link
       valid_lft forever preferred_lft forever
8: upfgtp: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1464 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none
    inet6 fe80::c9a2:a8e0:6916:9641/64 scope link stable-privacy
       valid_lft forever preferred_lft forever

Anything else we need to know?:

Environment:

  • Multus version

v3.9.3

  • Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.16", GitCommit:"c5f43560a4f98f2af3743a59299fb79f07924373", GitTreeState:"clean", BuildDate:"2023-11-15T22:39:12Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7

Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.5", GitCommit:"804d6167111f6858541cef440ccc53887fbbc96a", GitTreeState:"clean", BuildDate:"2022-12-08T10:08:09Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}

  • Primary CNI for Kubernetes cluster:

cri-o

  • OS (e.g. from /etc/os-release):

Ubuntu 20.04.1 LTS \n \l

  • File of '/etc/cni/net.d/'
$ ls /etc/cni/net.d/
00-multus.conf  10-calico.conflist  calico-kubeconfig  multus.d
  • File of '/etc/cni/multus/net.d'
$ ls /etc/cni/multus/net.d
ls: cannot access '/etc/cni/multus/net.d': No such file or directory
  • NetworkAttachment info (use kubectl get net-attach-def -o yaml)
$ kubectl get net-attach-def n3network-free5gc-v1-free5gc-upf -o yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  annotations:
    meta.helm.sh/release-name: free5gc-v1
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2024-04-04T07:39:23Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: n3network-free5gc-v1-free5gc-upf
  namespace: default
  resourceVersion: "1353009"
  uid: 5ec532e7-1348-4b2f-81d6-9f1f10221119
spec:
  config: '{ "cniVersion": "0.3.1", "plugins": [ { "type": "ipvlan", "capabilities":
    { "ips": true }, "master": "eno1", "mode": "l2", "ipam": { "type": "static", "routes":
    [ { "dst": "0.0.0.0/0", "gw": "10.100.50.238" } ] } } ] }'

$ kubectl get net-attach-def n4network-free5gc-v1-free5gc-upf -o yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  annotations:
    meta.helm.sh/release-name: free5gc-v1
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2024-04-04T07:39:23Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: n4network-free5gc-v1-free5gc-upf
  namespace: default
  resourceVersion: "1353008"
  uid: e2d1e8b4-252f-431d-8777-b211bbd645e8
spec:
  config: '{ "cniVersion": "0.3.1", "plugins": [ { "type": "ipvlan", "capabilities":
    { "ips": true }, "master": "eno1", "mode": "l2", "ipam": { "type": "static", "routes":
    [ { "dst": "0.0.0.0/0", "gw": "10.100.50.246" } ] } } ] }'

$ kubectl get net-attach-def n6network-free5gc-v1-free5gc-upf -o yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  annotations:
    meta.helm.sh/release-name: free5gc-v1
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2024-04-04T07:39:23Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: n6network-free5gc-v1-free5gc-upf
  namespace: default
  resourceVersion: "1353007"
  uid: 19e63d9b-6d5c-4544-8a2f-c3d3e021d31e
spec:
  config: '{ "cniVersion": "0.3.1", "plugins": [ { "type": "ipvlan", "capabilities":
    { "ips": true }, "master": "ens1f0", "mode": "l2", "ipam": { "type": "static",
    "routes": [ { "dst": "0.0.0.0/0", "gw": "172.31.1.1" } ] } } ] }'
  • Target pod yaml info (with annotation, use kubectl get pod <podname> -o yaml)
apiVersion: v1                                                                                                                                                     [167/1837]
kind: Pod
metadata:
  annotations:
    checksum/config: 9484d02a8722d94559244e9949507685e23a86b4b021a0cb65891810f2c8168e
    cni.projectcalico.org/containerID: c92558def4e17e02399dba61ae607a0c4246642eb16429dd1a0c3512dd6fc872
    cni.projectcalico.org/podIP: 182.16.182.141/32
    cni.projectcalico.org/podIPs: 182.16.182.141/32
    k8s.v1.cni.cncf.io/network-status: |-
      [{
          "name": "k8s-pod-network",
          "ips": [
              "182.16.182.141"
          ],
          "default": true,
          "dns": {}
      },{
          "name": "default/n3network-free5gc-v1-free5gc-upf",
          "interface": "n3",
          "ips": [
              "10.100.50.233"
          ],
          "mac": "4c:52:62:65:bb:fd",
          "dns": {}
      },{
          "name": "default/n6network-free5gc-v1-free5gc-upf",
          "interface": "n6",
          "ips": [
              "172.31.1.30"
          ],
          "mac": "b8:ce:f6:72:b0:d4",
          "dns": {}
      },{
          "name": "default/n4network-free5gc-v1-free5gc-upf",
          "interface": "n4",
          "ips": [
              "10.100.50.241"
          ],
          "mac": "4c:52:62:65:bb:fd",
          "dns": {}
      }]
    k8s.v1.cni.cncf.io/networks: '[ { "name": "n3network-free5gc-v1-free5gc-upf",                                                                                  [126/1837]
      "interface": "n3", "ips": [ "10.100.50.233/29" ], "gateway": [ "10.100.50.238"
      ] }, { "name": "n6network-free5gc-v1-free5gc-upf", "interface": "n6", "ips":
      [ "172.31.1.30/24" ], "gateway": [ "172.31.1.1" ] }, { "name": "n4network-free5gc-v1-free5gc-upf",
      "interface": "n4", "ips": [ "10.100.50.241/29" ], "gateway": [ "10.100.50.246"
      ] } ]'
    k8s.v1.cni.cncf.io/networks-status: |-
      [{
          "name": "k8s-pod-network",
          "ips": [
              "182.16.182.141"
          ],
          "default": true,
          "dns": {}
      },{
          "name": "default/n3network-free5gc-v1-free5gc-upf",
          "interface": "n3",
          "ips": [
              "10.100.50.233"
          ],
          "mac": "4c:52:62:65:bb:fd",
          "dns": {}
      },{
          "name": "default/n6network-free5gc-v1-free5gc-upf",
          "interface": "n6",
          "ips": [
              "172.31.1.30"
          ],
          "mac": "b8:ce:f6:72:b0:d4",
          "dns": {}
      },{
          "name": "default/n4network-free5gc-v1-free5gc-upf",
          "interface": "n4",
          "ips": [
              "10.100.50.241"
          ],
          "mac": "4c:52:62:65:bb:fd",
          "dns": {}
      }]
  creationTimestamp: "2024-04-04T07:39:23Z"
  generateName: free5gc-v1-free5gc-upf-upf-857ff7ddcd-
  labels:
    app.kubernetes.io/instance: free5gc-v1
    app.kubernetes.io/name: free5gc-upf
    nf: upf
    pod-template-hash: 857ff7ddcd
    project: free5gc
  name: free5gc-v1-free5gc-upf-upf-857ff7ddcd-gbdxw
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: free5gc-v1-free5gc-upf-upf-857ff7ddcd
    uid: ee82588e-c8f2-409a-819c-727ae4138ebb
  resourceVersion: "1353039"
  uid: 1e267cc9-98ec-421b-a495-1d315c8ebe00
spec:
  affinity: {}
  containers:
  - command:
    - /free5gc/config//wrapper.sh
    image: towards5gs/free5gc-upf:latest
    imagePullPolicy: Always
    name: upf
    ports:
    - containerPort: 8805
      name: n4
      protocol: UDP
    resources:
      requests:
        cpu: 500m
        memory: 512Mi
    securityContext:
      privileged: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:                                                                                                                                                   [48/1837]
    - mountPath: /free5gc/config/
      name: free5gc-v1-free5gc-upf-upf-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-mpnp5
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: machine-b1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: free5gc-v1-free5gc-upf-upf-volume
    projected:
      defaultMode: 420
      sources:
      - configMap:
          items:
          - key: upfcfg.yaml
            path: upfcfg.yaml
          - key: wrapper.sh
            mode: 511
            path: wrapper.sh
          name: free5gc-v1-free5gc-upf-upf-configmap
  - name: kube-api-access-mpnp5
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:                                                                                                                                         [5/1837]
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-04-04T07:39:23Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-04-04T07:39:27Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-04-04T07:39:27Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-04-04T07:39:23Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: cri-o://d2de0e3ea96af3feec4345ca369f8f0a07d437f99e4b04ebd19ff763fc8942b8
    image: docker.io/towards5gs/free5gc-upf:latest
    imageID: docker.io/towards5gs/free5gc-upf@sha256:09ac580458cf960168d390224afa650bd0e6d9b3b534859e973b12f5708b7829
    lastState: {}
    name: upf
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2024-04-04T07:39:26Z"
  hostIP: 172.31.16.37
  phase: Running
  podIP: 182.16.182.141
  podIPs:
  - ip: 182.16.182.141
  qosClass: Burstable
  startTime: "2024-04-04T07:39:23Z"
  • Other log outputs (if you use multus logging)
@dougbtv
Copy link
Member

dougbtv commented May 23, 2024

This one seems to be rather implementation specific! If anyone wants to try to replicate using these components, help wanted for sure. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dougbtv @InoueReo