Kubernetes homelab: Virtual network with OpenVSwitch and Namespaces

We have a small server and we’re going to set up some VMs in KVM for a Kubernetes homelab. I am planing to use OpenVSwitch for my virtual network, set a proxy and a DNS, and finally, install Rancher, and create a cluster with Calico. 

Over this post I am sharing my notes about installing openvswitch bridges and namespaces to build a virtual networks for the kubernetes cluster.

In the post, I will bring some details regarding:

  • How to install OpenVswitch, create bridges.
  • Connect an external bridge to the physical network.
  • Create and internal bridge for your K8s controllers and workers
  • Create a namespace to route traffic between bridges
  • Have all scripts set to keep the configuration after reboot.
  • I am working with Centos 7 and I will install openvswitch 2.5.10

Next picture will give you a good reference of what I am planning to do for my virtual network. In this case I will set the OVS bridges and network namespaces to route traffic between them.

Kubernetes homelab: Virtual network with OpenVSwitch and Namespaces

Build and Install OpenVSwitch

I’ve  installed openvswicth downloading the code from their site and compiling it into a package. Don’t forget to disable SELINUX before anything ( set as disabled in /etc/selinux/config) sudo setenforce 0

Install packages:

sudo yum install -y openssl-devel\
gcc make python-devel openssl-devel\
kernel-devel kernel-debug-devel autoconf automake\
rpm-build redhat-rpm-config libtool selinux-policy-devel

Create ovs user and folders to prepare the package build

sudo useradd ovs
sudo mkdir /home/ovs/.ssh
sudo chmod 755 /home/ovs/.ssh
sudo cp .ssh/authorized_keys /home/ovs/.ssh/authorized_keys
sudo mkdir /home/ovs/rpmbuild
sudo mkdir /home/ovs/rpmbuild/SOURCES
sudo chown -R ovs:ovs /home/ovs/rpmbuild
sudo chmod -R 755 /home/ovs/rpmbuild

Now it’s time to build the package. You need to run this as the ovs user:

sudo su - ovs
cd /home/ovs/rpmbuild/SOURCES
curl -s https://www.openvswitch.org/releases/openvswitch-2.5.10.tar.gz -o openvswitch-2.5.10.tar.gz
sed 's/openvswitch-kmod, //g' openvswitch-2.5.10/rhel/openvswitch.spec > openvswitch-2.5.10/rhel/openvswitch_no_kmod.spec
rpmbuild -bb --nocheck openvswitch-2.5.10/rhel/openvswitch_no_kmod.spec
exit

After the package has been built, it’s time to create the configuration folder and install it.

sudo mkdir /etc/openvswitch
sudo chmod 755 /etc/openvswitch
sudo yum localinstall /home/ovs/rpmbuild/RPMS/x86_64/openvswitch-2.5.10-1.x86_64.rpm
sudo systemctl enable openvswitch
sudo systemctl start openvswitch

Configure bridges, tap interfaces and namespaces

Set variables you will us over the next step of this tutorial tutorial

NS_R="ns_r"
TAP_R_INT="tap_1"
TAP_R_EXT="tap_2"
BR_INT="br_int"
BR_EXT="br_ext"
PHY_IP="192.168.1.130"
PHY_MASK="255.255.255.0"
PHY_INT="enp0s20f0"
DF_GW="192.168.1.254"
TAP_R_INT_ADD="10.10.10.1"
TAP_R_EXT_ADD="192.168.1.131"

Check what namespaces you have created before with: sudo ip netns list

List all OpenvSwitch bridges sudo ovs-vsctl list-br

Create OpenvSwitch bridges:

sudo ovs-vsctl add-br $BR_INT
sudo ovs-vsctl add-br $BR_EXT

Before we can do anything else, we need to connect your $BR_EXT to your external network. Then, if you have the following ip address: 192.168.1.10 in your physical interface (i.e. enp0s20f0 or eth0). You need to move it to your external bridge called $BR_EXT

I used the “ip” command, but you can use ifconfig ( it is not in the minimal Centos 7 version), you have to install the “net-tools” package for that.

Caution: don’t just copy and paste, you need to change your address for the one you are using, otherwise you will lose connectivity to your server. If you use these next commands properly, your server shouldn’t be cut off from the network

Caution: You should have those in a script to avoid connectivity issues over the process

In my case, I ran this command with “sudo” and I was disconnected from my ssh session:

#!/bin/bash

# vars
NS_R="ns_r"
TAP_R_INT="tap_1"
TAP_R_EXT="tap_2"
BR_INT="br_int"
BR_EXT="br_ext"
PHY_IP="192.168.1.130"
PHY_MASK="255.255.255.0"
PHY_INT="enp0s20f0"
DF_GW="192.168.1.254"

# add physical interface to bridge
ovs-vsctl add-port $BR_EXT $PHY_INT
# remove IP address from physical interface and add it to the bridge
ip addr del $PHY_IP/$PHY_MASK dev $PHY_INT & ip addr add $PHY_IP/$PHY_MASK dev $BR_EXT
# add default gateway route through the new interface
ip route add default via $DF_GW dev $BR_EXT

If you want this to be persistent over reboot, then you should change your ifcfg configuration file. There are tons of tutorials online you can use for that (my files are at in last section of this post)

Next, we create OVS ports, namespaces, virtual interfaces and set the IP addresses to all.

#Created those internal OVS ports

sudo ovs-vsctl add-port $BR_INT $TAP_R_INT -- set Interface $TAP_R_INT type=internal
sudo ovs-vsctl add-port $BR_EXT $TAP_R_EXT -- set Interface $TAP_R_EXT type=internal

#Create router namespace to work between OVS bridges
sudo ip netns add $NS_R

#Add loopback interface to namespace
sudo ip netns exec $NS_R ip address add 1.1.1.1/32 dev lo
sudo ip netns exec $NS_R ip link set dev lo up

#Enable forwarding in the namespace to allow route packets between bridges
sudo ip netns exec $NS_R sysctl -w net.ipv4.ip_forward=1

#Attach the new interface to the namespace
sudo ip link set $TAP_R_INT netns $NS_R
sudo ip link set $TAP_R_EXT netns $NS_R

#Enable tap interfaces in the namespace
sudo ip netns exec $NS_R ip link set dev $TAP_R_INT up
sudo ip netns exec $NS_R ip link set dev $TAP_R_EXT up

#Configure ip addresses in the namespace
sudo ip netns exec $NS_R ip addr add $TAP_R_INT_ADD/$PHY_MASK dev $TAP_R_INT
sudo ip netns exec $NS_R ip addr add $TAP_R_EXT_ADD/$PHY_MASK dev $TAP_R_EXT

#Create default route in the namespace
sudo ip netns exec $NS_R ip route add default via $DF_GW dev $TAP_R_EXT

Time to test

We have created a namespace connected to the internal bridge to test the communication to the LAN.

sudo ovs-vsctl add-port $BR_INT tap_test -- set Interface tap_test type=internal
sudo ip netns add test
sudo ip link set tap_test netns test
sudo ip netns exec test ip link set dev tap_test up
sudo ip netns exec test ip addr add 10.10.10.100/24 dev tap_test
sudo ip netns exec test ip route add default via 10.10.10.1 dev tap_test

And now the moment of true

sudo ip netns exec test ping $TAP_R_EXT_ADD

You should get a beautiful ping at this point. You shouldn’t be able to ping your $BR_EXT interface or the default gateway unless you create the required routes to the 10.10.10.0/24 network

Important note: Just a reminder. If you want those namespaces ot VMs to have access to the exterior, you have to create a route in your LAN default gateway for the network 10.10.10.0/24

Important note: Network namespaces are not persistent across system restarts. You will need to create a script that is run at startup and arrange to have it run. Check next section for more information.

Keep bridge IP and namespace after reboot

We want your Kubernetes homelab to last, and If you reboot now, you will not only lose access to your server, but also the namepace.

Then first, you should move the IP address from the physical interface to br_ext in the “network-scripts” folder. You will need to configure your physical and bridge like I show here:

# cat /etc/sysconfig/network-scripts/ifcfg-enp0s20f0
DEVICE="enp0s20f0"
BOOTPROTO="none"
ONBOOT="yes"
HWADDR=0c:c4:7a:6d:01:e8
DEVICETYPE=ovs
TYPE=OVSPort
OVS_BRIDGE="br_ext"
# cat /etc/sysconfig/network-scripts/ifcfg-br_ext
DEVICE="br_ext"
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
#BOOTPROTO=static
OVSBOOTPROTO=static
IPADDR=192.168.1.130
PREFIX=24
GATEWAY=192.168.1.254
OVSDHCPINTERFACES=enp0s20f0

Like I told, namespaces must be created back again after reboot. Then, we’ll create a service to run the script that recreates everything we have just done. 

First, you have to create a service file:

#cat /etc/systemd/system/router4ovs.service
Description=Create namespace to route between ovs bridges
After=default.target

[Service]
Type=simple
RemainAfterExit=yes
ExecStart=/home/pinrojas/router4ovs.sh 
TimeoutStartSec=0

[Install]
WantedBy=default.target

Create the script router4ovs.sh:

#!/bin/bash
#file name: router4ovs.sh 

SCRIPT_NAME=$(basename -- "$0")
NS_R="ns_r"
TAP_R_INT="tap_1"
TAP_R_EXT="tap_2"
BR_INT="br_int"
BR_EXT="br_ext"
PHY_IP="192.168.1.130"
PHY_MASK="255.255.255.0"
PHY_INT="enp0s20f0"
DF_GW="192.168.1.254"
TAP_R_INT_ADD="10.10.10.1"
TAP_R_EXT_ADD="192.168.1.131"

# Create our namespace to route between bridges
ip netns add $NS_R
ip netns exec $NS_R ip address add 1.1.1.1/32 dev lo
ip netns exec $NS_R ip link set dev lo up
ip netns exec $NS_R sysctl -w net.ipv4.ip_forward=1
ip link set $TAP_R_INT netns $NS_R
ip link set $TAP_R_EXT netns $NS_R
ip netns exec $NS_R ip link set dev $TAP_R_INT up
ip netns exec $NS_R ip link set dev $TAP_R_EXT up
ip netns exec $NS_R ip addr add $TAP_R_INT_ADD/$PHY_MASK dev $TAP_R_INT
ip netns exec $NS_R ip addr add $TAP_R_EXT_ADD/$PHY_MASK dev $TAP_R_EXT
ip netns exec $NS_R ip route add default via $DF_GW dev $TAP_R_EXT

#Create a route to reach br_int network from the host
ip route add $TAP_R_INT_ADD/$PHY_MASK via $TAP_R_EXT_ADD dev br_ext

#Create our namespace to test

ovs-vsctl add-port $BR_INT tap_test -- set Interface tap_test type=internal
ip netns add test
ip link set tap_test netns test
ip netns exec test ip link set dev tap_test up
ip netns exec test ip addr add 10.10.10.100/24 dev tap_test
ip netns exec test ip route add default via 10.10.10.1 dev tap_test

wall $SCRIPT_NAME: COMPLETELY FINISHED

And then enable service:

sudo systemctl enable router4ovs.service 

And you’re done

Hope this could be a good help and I hope you share or see your comments. After you got to this point it;s time to create the VMs for your first Kubernetes homelab.

See ya!

 

4 Comments Add yours

Leave a Reply

Your email address will not be published. Required fields are marked *