Kubernetes homelab: Prepping cluster with KVM and OpenVSwitch

We have a small server and we’re going to set up some VMs in KVM for a Kubernetes homelab. I am planing to use OpenVSwitch for my virtual network, set a proxy and a DNS, and finally, install Rancher, and create a cluster with Calico. 

Over this post I am sharing my notes about installing KVM and creating the VM instances ( or KVM domains) and connecting them to a virtual network.

In the post, I will bring some details regarding:

  • How to install KVM
  • How to create VMs or KVM domains from a Centos7 cloud image
  • Configure those VMs and have them ready to start installing docker and K8s

I will create 4 instances:

  • 2 for k8s controllers. Yeah! I know 3 instances are the best for HA, but this is a lab and I just want to test to have more than one controller to check how to deal with something like with Rancher.
  • 2 for workers

Install required packages

Install KVM and guestfish:

sudo yum -y install qemu-kvm libvirt bridge-utils libguestfs-tools libvirt-python genisoimage

Download last Centos QCOW2 minimal image 

curl -s http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 -o CentOS-7-x86_64-GenericCloud.qcow2

Modify root password in the QCOW2 image

First of all, let’s set a password for root (i.e. “newpass”):

openssl passwd -1 newpass
$1$7e5esg3P$iXizEINjtaCMJOnxeRGWg.

And then let’s get into the image using guestfish and add this password to the /etc/shadow file and user root. We’ll use vi to change the file.

sudo LIBGUESTFS_BACKEND=direct guestfish --rw -a CentOS-7-x86_64-GenericCloud.qcow2

><fs> run
><fs> list-filesystems
/dev/sda1: xfs
><fs> mount /dev/sda1 /
><fs> vi /etc/shadow
><fs> exit

The top part of the former /etc/shadow file:

root:!!:18353:0:99999:7:::
bin:*:18353:0:99999:7:::
daemon:*:18353:0:99999:7:::

The file now:

root:$1$7e5esg3P$iXizEINjtaCMJOnxeRGWg:18353:0:99999:7:::
bin:*:18353:0:99999:7:::
daemon:*:18353:0:99999:7:::

Resize QCOW2 image

Now, we’ll resize the QCOW2 image to get more disk space for our pods (i.e. 20G more).

# cp CentOS-7-x86_64-GenericCloud.qcow2 centos7.qcow2
# qemu-img info centos7.qcow2 
image: centos7.qcow2
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 819M
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
# qemu-img resize centos7.qcow2 +20G

Now we need to resize the underlying filesystems using “virt-resize“. Note, however, that “virt-resize” CANNOT resize disk images in-place. So we need to make a backup copy and use it as input and use the original qcow as output. See example below.

First, we make a backup copy of the disk as shown below.

cp centos7.qcow2 centos7-orig.qcow2

NOTE: In this example /dev/sda1 is not the /boot partition. So be careful you are growing the correct partitions on your qcow.

# sudo LIBGUESTFS_BACKEND=direct virt-resize --expand /dev/sda1 centos7-orig.qcow2  centos7.qcow2 
[   0.0] Examining centos7-orig.qcow2
**********

Summary of changes:

virt-resize: warning: unknown/unavailable method for expanding the xfs 
filesystem on /dev/sda1
/dev/sda1: This partition will be resized from 8.0G to 28.0G.

**********
[   3.8] Setting up initial partition table on centos7.qcow2
[   4.2] Copying /dev/sda1
 100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00

Resize operation completed with no errors.  Before deleting the old disk,  carefully check that the resized disk boots and works correctly.

# qemu-img info centos7.qcow2 
image: centos7.qcow2
file format: qcow2
virtual size: 28G (30064771072 bytes)
disk size: 821M
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false

Important: Remember to use “xfs_growfs /” in the VM, when you connect to it, to expand the root file system later.

Finally, we verify that the filesystems have grown.

# sudo LIBGUESTFS_BACKEND=direct virt-filesystems --long -h --all -a centos7.qcow2 
Name       Type        VFS  Label  MBR  Size  Parent
/dev/sda1  filesystem  xfs  -      -    28G   -
/dev/sda1  partition   -    -      83   28G   /dev/sda
/dev/sda   device      -    -      -    28G   -

Create and configure KVM instance

Kubernetes homelab with KVM and OpenVSwitch: High level design

Create qcow2 images for all instances in your lab

sudo mkdir q=/home/qemu
for instance in {"control1","control2","worker1","worker2"} ; do \
sudo cp centos7.qcow2 /home/qemu/virt-images/k8s-$instance.qcow2; \
done 

Change ownership and permissions

sudo chown qemu:qemu -R /home/qemu

I am installing 2 kubernetes master servers and 2 kubernetes worker servers with 8GB memory each using the following commands:

for instance in {"control1","control2","worker1","worker2"} ; do \
sudo virt-install --name k8s-$instance --memory 8192 --vcpus 4 --disk /home/qemu/virt-images/k8s-$instance.qcow2 --network=bridge:br_int,model=virtio,virtualport_type=openvswitch --import --os-variant rhel7 --noautoconsole; \
done

Let’s create now the ifcfg files to configure the network

Let’s create now the ifcfg files to configure the network


cat > ifcfg-eth0-k8s-worker1 << _EOF_
DEVICE=eth0
ONBOOT=yes
NM_CONTROLLED=no
TYPE=Ethernet
PREFIX=24
BOOTPROTO=static
IPADDR=10.10.10.21
GATEWAY=10.10.10.1
_EOF_
cat > ifcfg-eth0-k8s-worker2 << _EOF_
DEVICE=eth0
ONBOOT=yes
NM_CONTROLLED=no
TYPE=Ethernet
PREFIX=24
BOOTPROTO=static
IPADDR=10.10.10.22
GATEWAY=10.10.10.1
_EOF_
cat > ifcfg-eth0-k8s-control1 << _EOF_
DEVICE=eth0
ONBOOT=yes
NM_CONTROLLED=no
TYPE=Ethernet
PREFIX=24
BOOTPROTO=static
IPADDR=10.10.10.11
GATEWAY=10.10.10.1
_EOF_
cat > ifcfg-eth0-k8s-control2 << _EOF_
DEVICE=eth0
ONBOOT=yes
NM_CONTROLLED=no
TYPE=Ethernet
PREFIX=24
BOOTPROTO=static
IPADDR=10.10.10.12
GATEWAY=10.10.10.1
_EOF_

Now, let’s copy those ifcfg file to the QCOW2 images thru guestfish

for instance in {"control1","control2","worker1","worker2"} ; do \
sudo guestfish -a /home/qemu/virt-images/k8s-$instance.qcow2 -i upload ifcfg-eth0-$instance /etc/sysconfig/network-scripts/ifcfg-eth0; \
done

Destroy instances a recreate instances with the new network configuration:

for i in `sudo virsh list --all| awk -F ' ' 'FNR>2{print $2}' | sed '/^$/d'` ; do sudo virsh destroy $i; done
for i in `sudo virsh list --all| awk -F ' ' 'FNR>2{print $2}' | sed '/^$/d'` ; do sudo virsh start $i; done

And we are done!

Checking instances and configurations in our Kubernetes homelab

Let’s check our bridge in openvswitch bridge and ports:

# sudo ovs-vsctl show
14b3eb6f-13e7-4dff-9508-6f6688bc2408
    Bridge br_ext
        Port "enp0s20f0"
            Interface "enp0s20f0"
        Port "tap_2"
            Interface "tap_2"
                type: internal
        Port br_ext
            Interface br_ext
                type: internal
    Bridge br_int
        Port "vnet2"
            Interface "vnet2"
        Port "vnet3"
            Interface "vnet3"
        Port br_int
            Interface br_int
                type: internal
        Port tap_test
            Interface tap_test
                type: internal
        Port "vnet1"
            Interface "vnet1"
        Port "vnet0"
            Interface "vnet0"
        Port "tap_1"
            Interface "tap_1"
                type: internal
    ovs_version: "2.5.10"

Check the interface name in the instance:

# sudo virsh dumpxml k8s-control1 | grep "vnet"
      <target dev='vnet0'/>

Hope this Kubernetes homelab with KVM and OpenVSwitch, could be a good help. Please, share or comment.

See ya!

3 Comments Add yours

Leave a Reply

Your email address will not be published. Required fields are marked *