Calico is a pure layer-3 network solution to interconnect workloads as containers or virtual servers. It relies on bird (BGP opensource project), iptables and the Linux IP stack to work. Brings also additional security features like micro-segmentation. Thanks to BGP is very scalable and also it’s tightly integrated to Kubernetes as CNI plugin.
Previous post I did it with just one controller and two workers. Now I will do it with 6 nodes cluster: three controllers and three workers. I will build this with kubeadm and install Calico CNI plugin.
On-premises Fedora34 K8s HA Cluster with kubeadm and Calico.
Prepare all your nodes.
Check first if you have all requirements in your servers in the official documentation.
In my case, i am using six 8GB memory, 4 vCPUs and 25GB of disk KVM instance with Fedora34. No swap disk. Meeting all the requirements. All nodes need different MAC addresses. Be certain about this.
I will start preparing the server. I used a clean cqow2 image of Fedora34.
I will prepare the image. In order to do that, we’ll have to reboot the server. I will use cri-o container runtime. If you prefer docker, be my guest. However, it will be a little challenging in case of fedora34.
Here you have a picture of the topology lab. It’s using SRLinux with EVPN. To make the story short, K8s nodes only sees a subnet 192.168.101.0/24 (Layer2 domain). If you want to see more details of the SRLinux setup, check it github.
image: Topology – On-premises Fedora34 K8s HA Cluster with kubeadm and Calico
Prepare your server
Basically there some steps you have to do with a reboot process in the middle. Step-by-step process can be seen at Server World – Kubernetes : Install Kubeadm.
You can save time using the scripts I published in my previous post
LoadBalancer: Haproxy
You will need a loadbalancer for the controller. In my case I used a container with the following configuration file:
defaults
log global
option http-server-close
option redispatch
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout check 10s
maxconn 3000
frontend apiserver
bind *:6443
default_backend apiserver
backend apiserver
balance roundrobin
option httpchk GET /healthz
http-check expect status 200
mode tcp
option ssl-hello-chk
server ctl-a1 192.168.101.30:6443 check
server ctl-a2 192.168.101.31:6443 check
server ctl-a3 192.168.101.32:6443 check
The container is the following one:
FROM python:3-alpine
RUN apk add --no-cache \
bash \
net-tools \
tcpdump \
curl \
haproxy
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
And then, in the container just run haproxy -f /usr/local/etc/haproxy/haproxy.cfg
In this case, the container is in the same network with the IP: 192.168.101.99
Initialize your controller node with kubeadm
Now it’s time to initialize your node. You could use –pod-network-cidr argument to the network segment you will use with Calico. It must not overlap with any other network address space in your LAN. I won’t do it in my case. I have two interfaces, eth1 is the one facing the network. Then, I will use –apiserver-advertise-address to be sure the right ip address is used. because it’s a HA going thru the loadbalancer, you will have to add: –upload-certs –control-plane-endpoint=192.168.101.99:6443
kubeadm init --apiserver-advertise-address=192.168.101.30 --cri-socket=unix:///var/run/crio/crio.sock --kubernetes-version=1.20.5 --upload-certs --control-plane-endpoint=192.168.101.99:6443
If you get the message of “Your Kubernetes control-plane has initialized successfully!” your are on the right path.
It’s time to set your kubectl:
# overwrite your previous config mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should see something like this:
[root@k8s-node01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-node01 NotReady master 2m6s v1.17.9 [root@k8s-node01 ~]# journalctl -n 1 -u kubelet -- Logs begin at Wed 2020-10-14 16:31:40 UTC, end at Wed 2020-10-14 18:18:37 UTC. -- Oct 14 18:18:37 k8s-node01 kubelet[8352]: E1014 18:18:37.304363 8352 kubelet.go:2184] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uni
Because cni plugin is not yet installed. It’s normal you get this message. It means it’s time to proceed installing your network solution: Calico.
Calico installation
I recommend to not join your other nodes until you get Calico Manifest installed!
Download the Calico networking manifest for the Kubernetes API datastore.
curl https://docs.projectcalico.org/manifests/calico.yaml -O
In this case, CIDR will be auto-detected. However, If all your nodes are in the same Layer-2 domain, you may want to disable encapsulation. In my case, I have disabled IP-n-IP:
- name: IP value: "autodetect" # Enable IPIP - name: CALICO_IPV4POOL_IPIP value: "Never" # Enable or Disable VXLAN on the default IP pool. - name: CALICO_IPV4POOL_VXLAN value: "Never" # Set MTU for tunnel device used if ipip is enabled
it’s time to create your manifest
kubectl create -f calico.yaml
You should see something like this:
kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-7b4f58f565-pnn62 0/1 ContainerCreating 0 29s kube-system calico-node-glznw 0/1 Init:2/3 0 29s kube-system coredns-6955765f44-2qc2k 0/1 ContainerCreating 0 30m kube-system coredns-6955765f44-spzcn 0/1 ContainerCreating 0 30m kube-system etcd-k8s-node01 1/1 Running 0 30m kube-system kube-apiserver-k8s-node01 1/1 Running 0 30m kube-system kube-controller-manager-k8s-node01 1/1 Running 0 30m kube-system kube-proxy-bj4b6 1/1 Running 0 30m kube-system kube-scheduler-k8s-node01 1/1 Running 0 30m
After a few minutes all calico pods and coredns should be in Running status. Then, you can check if the server is Ready:
kubectl get nodes -o wide Thu Oct 15 21:10:41 2020 NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8s-node01 Ready master 87m v1.17.9 10.10.10.101 <none> CentOS Linux 7 (Core) 3.10.0-1127.19.1.el7.x86_64 docker://1.13.1
Joining other controller to the cluster
After you initialize the controller and get the calico manifesto installed and its pods running. You can join the rest of the controllers to the cluster. Do not forget to prep those servers with the script in the first section of this post. Just go to every server and run the suggested kubeadm join command you got on the kubeadm init process. Remember, it’s a controller, you need to add –control-plane –certificate-key to make work like this. Remember to use the loadbalancer IP:
kubeadm join 192.168.110.99:6443 --token cixz27.bixgtyram2dyl8lt --discovery-token-ca-cert-hash sha256:97207054db7be454ecb48434862983891d6517f9740a890018c8132660b6b295 --control-plane --certificate-key 1a502f46ba8f2f11791faf16bf1a954aafbff62634bea0accffeb0a7a8cca236
Joining workers to the cluster
After you initialize all the controllers, you can join the rest of the servers to the cluster. Do not forget to prep those servers with the script in the first section of this post. Just go to every server and run the suggested kubeadm join command you got on the kubeadm process. Remember to use the loadbalancer IP:
kubeadm join 192.168.110.99:6443 --token cixz27.bixgtyram2dyl8lt --discovery-token-ca-cert-hash sha256:97207054db7be454ecb48434862983891d6517f9740a890018c8132660b6b295
Then wait for the servers to be ready.
[root@ctl-b1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION ctl-b1 Ready control-plane,master 2d9h v1.20.5 ctl-b2 Ready control-plane,master 2d9h v1.20.5 ctl-b3 Ready control-plane,master 2d9h v1.20.5 wk-b1 Ready <none> 2d8h v1.20.5 wk-b2 Ready <none> 2d8h v1.20.5 wk-b3 Ready <none> 2d8h v1.20.5
