On-premises Fedora34 K8s HA Cluster with kubeadm and Calico

Calico is a pure layer-3 network solution to interconnect workloads as containers or virtual servers. It relies on bird (BGP opensource project), iptables and the Linux IP stack to work. Brings also additional security features like micro-segmentation. Thanks to BGP is very scalable and also it’s tightly integrated to Kubernetes as CNI plugin.

Previous post I did it with just one controller and two workers. Now I will do it with 6 nodes cluster: three controllers and three workers. I will build this with kubeadm and install Calico CNI plugin.

On-premises Fedora34 K8s HA Cluster with kubeadm and Calico.

Prepare all your nodes.

Check first if you have all requirements in your servers in the official documentation.

In my case, i am using six  8GB memory, 4 vCPUs and 25GB of disk KVM instance with Fedora34. No swap disk. Meeting all the requirements. All nodes need different MAC addresses. Be certain about this.

I will start preparing the server. I used a clean cqow2 image of Fedora34.

I will prepare the image. In order to do that, we’ll have to reboot the server. I will use cri-o container runtime. If you prefer docker, be my guest. However, it will be a little challenging in case of fedora34. 

Here you have a picture of the topology lab. It’s using SRLinux with EVPN. To make the story short, K8s nodes only sees a subnet 192.168.101.0/24 (Layer2 domain). If you want to see more details of the SRLinux setup, check it github.

image: Topology – On-premises Fedora34 K8s HA Cluster with kubeadm and Calico

Prepare your server

Basically there some steps you have to do with a reboot process in the middle. Step-by-step process can be seen at Server World – Kubernetes : Install Kubeadm.

You can save time using the scripts I published in my previous post

LoadBalancer: Haproxy

You will need a loadbalancer for the controller. In my case I used a container with the following configuration file:

defaults
  log global
  option http-server-close
  option redispatch
  timeout http-request 10s
  timeout queue 1m
  timeout connect 10s
  timeout client 1m
  timeout server 1m
  timeout check 10s
  maxconn 3000
frontend apiserver
  bind *:6443
  default_backend apiserver
backend apiserver
  balance roundrobin
  option httpchk GET /healthz
  http-check expect status 200
  mode tcp
  option ssl-hello-chk
    server ctl-a1 192.168.101.30:6443 check
    server ctl-a2 192.168.101.31:6443 check
    server ctl-a3 192.168.101.32:6443 check

The container is the following one:

FROM python:3-alpine
RUN apk add --no-cache \
    bash \
    net-tools \
    tcpdump \
        curl \
        haproxy
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg

And then, in the container just run haproxy -f /usr/local/etc/haproxy/haproxy.cfg

In this case, the container is in the same network with the IP: 192.168.101.99

Initialize your controller node with kubeadm

Now it’s time to initialize your node. You could use –pod-network-cidr argument to the network segment you will use with Calico. It must not overlap with any other network address space in your LAN. I won’t do it in my case. I have two interfaces, eth1 is the one facing the network.  Then, I will use –apiserver-advertise-address to be sure the right ip address is used. because it’s a HA going thru the loadbalancer, you will have to add: –upload-certs –control-plane-endpoint=192.168.101.99:6443

kubeadm init --apiserver-advertise-address=192.168.101.30 --cri-socket=unix:///var/run/crio/crio.sock --kubernetes-version=1.20.5 --upload-certs --control-plane-endpoint=192.168.101.99:6443

If you get the message of “Your Kubernetes control-plane has initialized successfully!” your are on the right path. 

It’s time to set your kubectl:

# overwrite your previous config
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should see something like this:

[root@k8s-node01 ~]# kubectl get nodes
NAME         STATUS     ROLES    AGE    VERSION
k8s-node01   NotReady   master   2m6s   v1.17.9

[root@k8s-node01 ~]# journalctl -n 1 -u kubelet
-- Logs begin at Wed 2020-10-14 16:31:40 UTC, end at Wed 2020-10-14 18:18:37 UTC. --
Oct 14 18:18:37 k8s-node01 kubelet[8352]: E1014 18:18:37.304363    8352 kubelet.go:2184] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uni

Because cni plugin is not yet installed. It’s normal you get this message. It means it’s time to proceed installing your network solution: Calico.

Calico installation

I recommend to not join your other nodes until you get Calico Manifest installed!

Download the Calico networking manifest for the Kubernetes API datastore.

curl https://docs.projectcalico.org/manifests/calico.yaml -O

In this case, CIDR will be auto-detected. However, If all your nodes are in the same Layer-2 domain, you may want to disable encapsulation. In my case, I have disabled IP-n-IP:

- name: IP
  value: "autodetect"
# Enable IPIP
- name: CALICO_IPV4POOL_IPIP
  value: "Never"
# Enable or Disable VXLAN on the default IP pool.
- name: CALICO_IPV4POOL_VXLAN
  value: "Never"
# Set MTU for tunnel device used if ipip is enabled

it’s time to create your manifest

kubectl create -f calico.yaml

You should see something like this:

kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY   STATUS              RESTARTS   AGE
kube-system   calico-kube-controllers-7b4f58f565-pnn62   0/1     ContainerCreating   0    	29s
kube-system   calico-node-glznw                          0/1     Init:2/3            0    	29s
kube-system   coredns-6955765f44-2qc2k                   0/1     ContainerCreating   0    	30m
kube-system   coredns-6955765f44-spzcn                   0/1     ContainerCreating   0    	30m
kube-system   etcd-k8s-node01                            1/1     Running             0    	30m
kube-system   kube-apiserver-k8s-node01                  1/1     Running             0    	30m
kube-system   kube-controller-manager-k8s-node01         1/1     Running             0    	30m
kube-system   kube-proxy-bj4b6                           1/1     Running             0    	30m
kube-system   kube-scheduler-k8s-node01                  1/1     Running             0    	30m

After a few minutes all calico pods and coredns should be in Running status. Then, you can check if the server is Ready:

kubectl get nodes -o wide
Thu Oct 15 21:10:41 2020
NAME         STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
k8s-node01   Ready	master   87m   v1.17.9   10.10.10.101   <none>        CentOS Linux 7 (Core)   3.10.0-1127.19.1.el7.x86_64   docker://1.13.1

 

Joining other controller to the cluster

After you initialize the controller and get the calico manifesto installed and its pods running. You can join the rest of the controllers to the cluster. Do not forget to prep those servers with the script in the first section of this post. Just go to every server and run the suggested kubeadm join command you got on the kubeadm init process. Remember, it’s a controller, you need to add –control-plane –certificate-key to make work like this. Remember to use the loadbalancer IP:

kubeadm join 192.168.110.99:6443 --token cixz27.bixgtyram2dyl8lt     --discovery-token-ca-cert-hash sha256:97207054db7be454ecb48434862983891d6517f9740a890018c8132660b6b295     --control-plane --certificate-key 1a502f46ba8f2f11791faf16bf1a954aafbff62634bea0accffeb0a7a8cca236

Joining workers to the cluster

After you initialize all the controllers, you can join the rest of the servers to the cluster. Do not forget to prep those servers with the script in the first section of this post. Just go to every server and run the suggested kubeadm join command you got on the kubeadm process. Remember to use the loadbalancer IP:

kubeadm join 192.168.110.99:6443 --token cixz27.bixgtyram2dyl8lt     --discovery-token-ca-cert-hash sha256:97207054db7be454ecb48434862983891d6517f9740a890018c8132660b6b295

Then wait for the servers to be ready.

[root@ctl-b1 ~]# kubectl get nodes
NAME     STATUS   ROLES                  AGE    VERSION
ctl-b1   Ready    control-plane,master   2d9h   v1.20.5
ctl-b2   Ready    control-plane,master   2d9h   v1.20.5
ctl-b3   Ready    control-plane,master   2d9h   v1.20.5
wk-b1    Ready    <none>                 2d8h   v1.20.5
wk-b2    Ready    <none>                 2d8h   v1.20.5
wk-b3    Ready    <none>                 2d8h   v1.20.5

Leave a Reply