We have a small server and we’re going to set up some VMs in KVM for a Kubernetes homelab. I am planing to use OpenVSwitch for my virtual network, set a proxy and a DNS, and finally, install Rancher, and create a cluster with Calico.
Over this post I am sharing my notes about installing openvswitch bridges and namespaces to build a virtual networks for the kubernetes cluster.
In the post, I will bring some details regarding:
- How to install OpenVswitch, create bridges.
- Connect an external bridge to the physical network.
- Create and internal bridge for your K8s controllers and workers
- Create a namespace to route traffic between bridges
- Have all scripts set to keep the configuration after reboot.
- I am working with Centos 7 and I will install openvswitch 2.5.10
This is one of a serie of posts with the developing of this lab. here you have the list of all:
Next picture will give you a good reference of what I am planning to do for my virtual network. In this case I will set the OVS bridges and network namespaces to route traffic between them.
Kubernetes homelab: Virtual network with OpenVSwitch and Namespaces
Build and Install OpenVSwitch
I’ve installed openvswicth downloading the code from their site and compiling it into a package. Don’t forget to disable SELINUX before anything ( set as disabled in /etc/selinux/config) sudo setenforce 0
Install packages:
sudo yum install -y openssl-devel\ gcc make python-devel openssl-devel\ kernel-devel kernel-debug-devel autoconf automake\ rpm-build redhat-rpm-config libtool selinux-policy-devel
Create ovs user and folders to prepare the package build
sudo useradd ovs sudo mkdir /home/ovs/.ssh sudo chmod 755 /home/ovs/.ssh sudo cp .ssh/authorized_keys /home/ovs/.ssh/authorized_keys sudo mkdir /home/ovs/rpmbuild sudo mkdir /home/ovs/rpmbuild/SOURCES sudo chown -R ovs:ovs /home/ovs/rpmbuild sudo chmod -R 755 /home/ovs/rpmbuild
Now it’s time to build the package. You need to run this as the ovs user:
sudo su - ovs cd /home/ovs/rpmbuild/SOURCES curl -s https://www.openvswitch.org/releases/openvswitch-2.5.10.tar.gz -o openvswitch-2.5.10.tar.gz sed 's/openvswitch-kmod, //g' openvswitch-2.5.10/rhel/openvswitch.spec > openvswitch-2.5.10/rhel/openvswitch_no_kmod.spec rpmbuild -bb --nocheck openvswitch-2.5.10/rhel/openvswitch_no_kmod.spec exit
After the package has been built, it’s time to create the configuration folder and install it.
sudo mkdir /etc/openvswitch sudo chmod 755 /etc/openvswitch sudo yum localinstall /home/ovs/rpmbuild/RPMS/x86_64/openvswitch-2.5.10-1.x86_64.rpm sudo systemctl enable openvswitch sudo systemctl start openvswitch
Configure bridges, tap interfaces and namespaces
Set variables you will us over the next step of this tutorial tutorial
NS_R="ns_r" TAP_R_INT="tap_1" TAP_R_EXT="tap_2" BR_INT="br_int" BR_EXT="br_ext" PHY_IP="192.168.1.130" PHY_MASK="255.255.255.0" PHY_INT="enp0s20f0" DF_GW="192.168.1.254" TAP_R_INT_ADD="10.10.10.1" TAP_R_EXT_ADD="192.168.1.131"
Check what namespaces you have created before with: sudo ip netns list
List all OpenvSwitch bridges sudo ovs-vsctl list-br
Create OpenvSwitch bridges:
sudo ovs-vsctl add-br $BR_INT sudo ovs-vsctl add-br $BR_EXT
Before we can do anything else, we need to connect your $BR_EXT to your external network. Then, if you have the following ip address: 192.168.1.10 in your physical interface (i.e. enp0s20f0 or eth0). You need to move it to your external bridge called $BR_EXT
I used the “ip” command, but you can use ifconfig ( it is not in the minimal Centos 7 version), you have to install the “net-tools” package for that.
Caution: don’t just copy and paste, you need to change your address for the one you are using, otherwise you will lose connectivity to your server. If you use these next commands properly, your server shouldn’t be cut off from the network
Caution: You should have those in a script to avoid connectivity issues over the process
In my case, I ran this command with “sudo” and I was disconnected from my ssh session:
#!/bin/bash # vars NS_R="ns_r" TAP_R_INT="tap_1" TAP_R_EXT="tap_2" BR_INT="br_int" BR_EXT="br_ext" PHY_IP="192.168.1.130" PHY_MASK="255.255.255.0" PHY_INT="enp0s20f0" DF_GW="192.168.1.254" # add physical interface to bridge ovs-vsctl add-port $BR_EXT $PHY_INT # remove IP address from physical interface and add it to the bridge ip addr del $PHY_IP/$PHY_MASK dev $PHY_INT & ip addr add $PHY_IP/$PHY_MASK dev $BR_EXT # add default gateway route through the new interface ip route add default via $DF_GW dev $BR_EXT
If you want this to be persistent over reboot, then you should change your ifcfg configuration file. There are tons of tutorials online you can use for that (my files are at in last section of this post)
Next, we create OVS ports, namespaces, virtual interfaces and set the IP addresses to all.
#Created those internal OVS ports sudo ovs-vsctl add-port $BR_INT $TAP_R_INT -- set Interface $TAP_R_INT type=internal sudo ovs-vsctl add-port $BR_EXT $TAP_R_EXT -- set Interface $TAP_R_EXT type=internal #Create router namespace to work between OVS bridges sudo ip netns add $NS_R #Add loopback interface to namespace sudo ip netns exec $NS_R ip address add 1.1.1.1/32 dev lo sudo ip netns exec $NS_R ip link set dev lo up #Enable forwarding in the namespace to allow route packets between bridges sudo ip netns exec $NS_R sysctl -w net.ipv4.ip_forward=1 #Attach the new interface to the namespace sudo ip link set $TAP_R_INT netns $NS_R sudo ip link set $TAP_R_EXT netns $NS_R #Enable tap interfaces in the namespace sudo ip netns exec $NS_R ip link set dev $TAP_R_INT up sudo ip netns exec $NS_R ip link set dev $TAP_R_EXT up #Configure ip addresses in the namespace sudo ip netns exec $NS_R ip addr add $TAP_R_INT_ADD/$PHY_MASK dev $TAP_R_INT sudo ip netns exec $NS_R ip addr add $TAP_R_EXT_ADD/$PHY_MASK dev $TAP_R_EXT #Create default route in the namespace sudo ip netns exec $NS_R ip route add default via $DF_GW dev $TAP_R_EXT
Time to test
We have created a namespace connected to the internal bridge to test the communication to the LAN.
sudo ovs-vsctl add-port $BR_INT tap_test -- set Interface tap_test type=internal sudo ip netns add test sudo ip link set tap_test netns test sudo ip netns exec test ip link set dev tap_test up sudo ip netns exec test ip addr add 10.10.10.100/24 dev tap_test sudo ip netns exec test ip route add default via 10.10.10.1 dev tap_test
And now the moment of true
sudo ip netns exec test ping $TAP_R_EXT_ADD
You should get a beautiful ping at this point. You shouldn’t be able to ping your $BR_EXT interface or the default gateway unless you create the required routes to the 10.10.10.0/24 network
Important note: Just a reminder. If you want those namespaces ot VMs to have access to the exterior, you have to create a route in your LAN default gateway for the network 10.10.10.0/24
Important note: Network namespaces are not persistent across system restarts. You will need to create a script that is run at startup and arrange to have it run. Check next section for more information.
Keep bridge IP and namespace after reboot
We want your Kubernetes homelab to last, and If you reboot now, you will not only lose access to your server, but also the namepace.
Then first, you should move the IP address from the physical interface to br_ext in the “network-scripts” folder. You will need to configure your physical and bridge like I show here:
# cat /etc/sysconfig/network-scripts/ifcfg-enp0s20f0 DEVICE="enp0s20f0" BOOTPROTO="none" ONBOOT="yes" HWADDR=0c:c4:7a:6d:01:e8 DEVICETYPE=ovs TYPE=OVSPort OVS_BRIDGE="br_ext" # cat /etc/sysconfig/network-scripts/ifcfg-br_ext DEVICE="br_ext" ONBOOT=yes DEVICETYPE=ovs TYPE=OVSBridge #BOOTPROTO=static OVSBOOTPROTO=static IPADDR=192.168.1.130 PREFIX=24 GATEWAY=192.168.1.254 OVSDHCPINTERFACES=enp0s20f0
Like I told, namespaces must be created back again after reboot. Then, we’ll create a service to run the script that recreates everything we have just done.
First, you have to create a service file:
#cat /etc/systemd/system/router4ovs.service Description=Create namespace to route between ovs bridges After=default.target [Service] Type=simple RemainAfterExit=yes ExecStart=/home/pinrojas/router4ovs.sh TimeoutStartSec=0 [Install] WantedBy=default.target
Create the script router4ovs.sh:
#!/bin/bash #file name: router4ovs.sh SCRIPT_NAME=$(basename -- "$0") NS_R="ns_r" TAP_R_INT="tap_1" TAP_R_EXT="tap_2" BR_INT="br_int" BR_EXT="br_ext" PHY_IP="192.168.1.130" PHY_MASK="255.255.255.0" PHY_INT="enp0s20f0" DF_GW="192.168.1.254" TAP_R_INT_ADD="10.10.10.1" TAP_R_EXT_ADD="192.168.1.131" # Create our namespace to route between bridges ip netns add $NS_R ip netns exec $NS_R ip address add 1.1.1.1/32 dev lo ip netns exec $NS_R ip link set dev lo up ip netns exec $NS_R sysctl -w net.ipv4.ip_forward=1 ip link set $TAP_R_INT netns $NS_R ip link set $TAP_R_EXT netns $NS_R ip netns exec $NS_R ip link set dev $TAP_R_INT up ip netns exec $NS_R ip link set dev $TAP_R_EXT up ip netns exec $NS_R ip addr add $TAP_R_INT_ADD/$PHY_MASK dev $TAP_R_INT ip netns exec $NS_R ip addr add $TAP_R_EXT_ADD/$PHY_MASK dev $TAP_R_EXT ip netns exec $NS_R ip route add default via $DF_GW dev $TAP_R_EXT #Create a route to reach br_int network from the host ip route add $TAP_R_INT_ADD/$PHY_MASK via $TAP_R_EXT_ADD dev br_ext #Create our namespace to test ovs-vsctl add-port $BR_INT tap_test -- set Interface tap_test type=internal ip netns add test ip link set tap_test netns test ip netns exec test ip link set dev tap_test up ip netns exec test ip addr add 10.10.10.100/24 dev tap_test ip netns exec test ip route add default via 10.10.10.1 dev tap_test wall $SCRIPT_NAME: COMPLETELY FINISHED
And then enable service:
sudo systemctl enable router4ovs.service
And you’re done
Hope this could be a good help and I hope you share or see your comments. After you got to this point it;s time to create the VMs for your first Kubernetes homelab.
See ya!
4 thoughts on “Kubernetes homelab: Virtual network with OpenVSwitch and Namespaces”