IPVLAN vs MACVLAN at the Edge Cloud

You can get all pod and cni manifests used in the post at https://github.com/cloud-native-everything/nokia-edge-network-test-resources

Most of the Edge Cloud deployment have been done with Kubernetes. Multus is a must component in Telco CNF applications like cRAN DU/CU or 5G Core. Let’s go a very simple example, like a Security Gateway or a router. Those require multiple interfaces to connect to different slices. Newest Apps can use IPVLAN to deal with the connectivity. However, sometimes MACVLAN can be the only option if you want to keep all the IP settings under the management of the App control plane. 

• IPVLAN is a like adding additional IPs to a specific interface in the host (master interface), like you do with “ip addr add” command in linux. The pod will use this IP as its own interface independently from the main interface use as master.

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: ipvlan-251
spec:
  config: '{
  "cniVersion": "0.4.0",
  "name": "ipvlan-251",
  "plugins": [{
    "type": "ipvlan",
    "master": "VLAN-251",
    "ipam": {
      "type": "host-local",
      "subnet": "10.0.251.0/24"
    }
  },{
  "type": "enc-nws",
  "subnet": "macvlan-251",
  "additional-config": {}
  }]
}'

• MACVLAN instead, is adding MAC addresses. However, in this case, we’ll use the ‘passthru’ mode, where the pod interface is actually setting the host interface in promiscuous mode and using the same MAC address of the host. The limitation is, you can have just one pod using this interface in the same host (Kubernetes worker).

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-251
spec:
  config: '{
  "cniVersion": "0.4.0",
  "name": "macvlan-251",
  "plugins": [{
    "type": "macvlan",
    "master": "VLAN-251",
    "mode" : "passthru",
    "ipam": {
      "type": "host-local",
      "subnet": "10.0.251.0/24"
    }
  },{
  "type": "enc-nws",
  "subnet": "macvrf-251",
  "additional-config": {}
  }]
}'

The second component is used with Edge Network Controller and its Scaler Agent. You can remove it in case you don’t use that part of the automation in the underlay network.

IPVLAN can be use with as many replicas you want. However, MACVLAN ‘passthru’ should manage like a ‘DaemonSet’ to avoid more replicas in the same host, to avoid conflict with duplicated MAC addresses.

If I try to create replicas with MACVLAN, I would get issues like this:

[root@dell02 vno-scaler-22.5]# kubectl get pod
NAME                              READY   STATUS              RESTARTS   AGE
macvlan-deploy-5f79fbd558-65l46   0/1     ContainerCreating   0          7s
macvlan-deploy-5f79fbd558-6m6jz   1/1     Running             0          7s
macvlan-deploy-5f79fbd558-bjbsp   0/1     ContainerCreating   0          7s
macvlan-deploy-5f79fbd558-jmtnk   0/1     ContainerCreating   0          7s
macvlan-deploy-5f79fbd558-jthv6   1/1     Running             0          7s
macvlan-deploy-5f79fbd558-ljw4n   0/1     ContainerCreating   0          7s
macvlan-deploy-5f79fbd558-lnkwc   0/1     ContainerCreating   0          7s
macvlan-deploy-5f79fbd558-nrjbp   0/1     ContainerCreating   0          7s

Where, only one pod is allowed to use the same ‘master’ interface. If you get a ‘describe’ of the pod, you will see the following message:

Normal   AddedInterface          2m2s              multus             Add eth0 [10.244.2.34/24] from kindnet
Warning  FailedCreatePodSandBox  2m2s              kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e66fbfe53396dcd6c8cfdb8431c184d00d1278cb550427771a9e1e04dbd50bee": [default/macvlan-deploy-5f79fbd558-65l46:macvlan-251]: error adding container to network "macvlan-251": failed to create macvlan: invalid argument
Normal   AddedInterface          111s              multus             Add eth0 [10.244.2.36/24] from kindnet
Warning  FailedCreatePodSandBox  111s              kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9250fd80ce411bb183db6178063e667aa8cf39aba4172e8072bea48e9ae8154e": [default/macvlan-deploy-5f79fbd558-65l46:macvlan-251]: error adding container to network "macvlan-251": failed to create macvlan: invalid argument

Then, if you add the Pod definition, it should be something like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: py-tftpboot
data:
  http_server.py: |
    import socket, struct, fcntl
    import random
    # Defining Variables
    last_octect=str(random.randint(1,254))
    net1_ip = '10.0.251.'+last_octect
    net2_ip = '10.0.252.'+last_octect
    net3_ip = '10.0.253.'+last_octect
    net4_ip = '10.0.254.'+last_octect
    #saving var to file
    file_object=open('interfaces.txt', 'w')
    file_object.write('NET1_IP='+net1_ip+'\n')
    file_object.write('NET2_IP='+net2_ip+'\n')
    file_object.write('NET3_IP='+net3_ip+'\n')
    file_object.write('NET4_IP='+net4_ip+'\n')
    file_object.close()

    SIOCSIFADDR = 0x8916
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    def setIpAddr(iface, ip):
        bin_ip = socket.inet_aton(ip)
        ifreq = struct.pack('16sH2s4s8s', iface, socket.AF_INET,b'\x00' * 2, bin_ip,b'\x00' * 8)
        fcntl.ioctl(sock, SIOCSIFADDR, ifreq)

    setIpAddr(b'net1',net1_ip)
    setIpAddr(b'net2',net2_ip)
    setIpAddr(b'net3',net3_ip)
    setIpAddr(b'net4',net4_ip)

    import os
    from http.server import HTTPServer, BaseHTTPRequestHandler
    class SimpleHTTPRequestHandler(BaseHTTPRequestHandler):
      def do_GET(self):
          self.send_response(200)
          self.send_header("Content-type", "text/html")
          self.end_headers()
          hostname = os.environ['HOSTNAME']
          worker = os.environ['K8S_NODE_NAME']
          text = f"K8s Node: {worker} - Hostname: {hostname}\n"
          self.wfile.write(bytes(text, "utf8"))
    port = int(os.environ['HTTP_APP_PORT'])
    httpd = HTTPServer(('0.0.0.0', port), SimpleHTTPRequestHandler)
    httpd.serve_forever()

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: macvlan-deploy
spec:
  selector:
    matchLabels:
      app: macvlan-pod
  template:
    metadata:
      labels:
        app: macvlan-pod
      annotations:
        k8s.v1.cni.cncf.io/networks: |
          [
             {"name":"macvlan-251", "interface": "net1"},
             {"name":"macvlan-252", "interface": "net2"},
             {"name":"macvlan-253", "interface": "net3"},
             {"name":"macvlan-254", "interface": "net4"}
          ]
    spec:
      volumes:
      - name: tftpboot	  # The name of the volume
        configMap:
          name: py-tftpboot
      containers:
        - name: macvlan-pod
          image: python:latest
          imagePullPolicy: Never
          command: ["python3"]
          args: ["/tftpboot/http_server.py"]
          ports:
            - containerPort: 8080
              protocol: TCP
          env:
          - name: K8S_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: HTTP_APP_PORT
            value: "8080"
          securityContext:
            privileged: true
          volumeMounts:
          - name: tftpboot
            mountPath: /tftpboot

In this case, I am using a standard python image to be tested via ‘curl’ from a remote location. The pod will display the name of the pod and the k8s worker name were is hosted.

First, let’s get the interfaces IP in the pod

[root@dell02 vno-scaler-22.5]# kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
macvlan-deploy-dxr58   1/1     Running   0          101m
macvlan-deploy-qj6gp   1/1     Running   0          101m
[root@dell02 vno-scaler-22.5]# kubectl exec -ti macvlan-deploy-dxr58 -- cat interfaces.txt
NET1_IP=10.0.251.176
NET2_IP=10.0.252.176
NET3_IP=10.0.253.176
NET4_IP=10.0.254.176

And now, we can go the remote location and test then:

/ # curl http://10.0.251.176:8080
K8s Node: scaler-worker2 - Hostname: macvlan-deploy-dxr58

Let me know if you have comments: “IPVLAN vs MACVLAN at the Edge Cloud”

See ya!