On-premises Fedora34 K8s cluster with kubeadm and Calico

Calico is a pure layer-3 network solution to interconnect workloads as containers or virtual servers. It relies on bird (BGP opensource project), iptables and the Linux IP stack to work. Brings also additional security features like micro-segmentation. Thanks to BGP is very scalable and also it’s tightly integrated to Kubernetes as CNI plugin.

Previous post I did it with just one node. Now I will do it with 3 nodes cluster: one controllers and two workers. I will build this with kubeadm and install Calico CNI plugin.

Prepare all your nodes

Check first if you have all requirements of your servers in the official documentation.

In my case, i am using three  8GB memory, 4 vCPUs and 25GB of disk KVM instance with Fedora34. No swap disk. Meeting all the requirements. All nodes need different MAC addresses. Be certain about this.

I will start preparing the server. I used a clean cqow2 image of Fedora34.

I will prepare the image. In order to do that, we’ll have to reboot the server. I will use cri-o container runtime. If you prefer docker, be my guest. However, it will be a little challenging in case of fedora34. 

Here you have a picture of the topology lab. It’s using SRLinux with EVPN. To make the story short, K8s nodes only sees a subnet 192.168.101.0/24 (Layer2 domain). If you want to see more details of the SRLinux setup, check it github.

Prepare your server

Basically there some steps you have to do with a reboot process in the middle. Step-by-step process can be seen at Server World – Kubernetes : Install Kubeadm.

Before reboot, you will have to run those scripts (replace DNS IP and domain with the one you need):

# Install base pacjages for iptables and others
dnf -y install iptables ethtool ebtables

# Enable IP packet forwarding
cat > /etc/sysctl.d/99-k8s-cri.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-ip6tables=1
EOF

echo -e overlay\\nbr_netfilter > /etc/modules-load.d/k8s.conf

# Fix cgroup version
sed -E -i 's/^GRUB_CMDLINE_LINUX="/GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0 /g' /etc/default/grub
grub2-mkconfig -o /boot/grub2/grub.cfg


touch /etc/systemd/zram-generator.conf

# Disable firewalld and system-resolv (I am using dns 172.20.20.254, you will have to replace it for the one youb need)
sed -E -i 's/^(\[main\])$/\1\ndns=default/g' /etc/NetworkManager/NetworkManager.conf
systemctl disable --now firewalld
systemctl disable --now systemd-resolved
unlink /etc/resolv.conf
touch /etc/resolv.conf
echo "nameserver 172.20.20.254" > /etc/resolv.conf
echo "search srl.nokialab.net" >> /etc/resolv.conf

# disable selinux
sudo setenforce 0
sudo sed -i 's/^SELINUX=.*$/SELINUX=disabled/' /etc/selinux/config

#######  reboot  server
echo "rebooting server in 10 seconds"
pause 10
reboot

After the server is back. You will need to run this (replace the proxy IP if you need it):

## installing packages required for kubernetes
dnf module -y install cri-o:1.20/default
systemctl enable --now cri-o
dnf -y install kubernetes-kubeadm kubernetes-node kubernetes-client cri-tools iproute-tc container-selinux
dnf -y update
dnf -y clean all

## setting hostname, maybe you dont need this line
hostnamectl set-hostname $(cat /etc/hosts | grep 192.168 | awk '{print $3}')

## setting up kubernetes variables
sed -E -i 's/^KUBELET_ADDRESS=.*$/KUBELET_ADDRESS="--address=0.0.0.0"/g' /etc/kubernetes/kubelet
sed -E -i 's/^# (KUBELET_PORT=.*)$/\1/g' /etc/kubernetes/kubelet
sed -E -i "s/^KUBELET_HOSTNAME=.*$/KUBELET_HOSTNAME=\"--hostname-override=$(hostname)\"/g" /etc/kubernetes/kubelet
sed -E -i 's/^(Environment="KUBELET_EXTRA_ARGS=.*)"$/\1 --container-runtime=remote --container-runtime-endpoint=unix:\/\/\/var\/run\/crio\/crio.sock"/g' /etc/systemd/system/kubelet.service.d/kubeadm.conf
dnf -y install iptables

## remove next  3 lines if you don't have a proxy!!!
echo 'NO_PROXY="localhost,127.0.0.1,3.0.0.0/8,192.168.0.0/16,10.0.0.0/8"'  >> /etc/sysconfig/crio
echo 'HTTP_PROXY="http://172.20.20.253:3128"'  >> /etc/sysconfig/crio
echo 'HTTPS_PROXY="http://172.20.20.253:3128"'  >> /etc/sysconfig/crio

## enable and start services
systemctl restart cri-o
systemctl enable kubelet.service
kubeadm config images pull

echo "you can use kubeadm init or join now"

Initialize your controller node with kubeadm

Now it’s time to initialize your node. You could use –pod-network-cidr argument to the network segment you will use with Calico. It must not overlap with any other network address space in your LAN. I won’t do it in my case. I have two interfaces, eth1 is the one facing the network.  Then, I will use –apiserver-advertise-address to be sure the right ip address is used.

kubeadm init --apiserver-advertise-address=192.168.101.30 --cri-socket=unix:///var/run/crio/crio.sock --kubernetes-version=1.20.5

If you get the message of “Your Kubernetes control-plane has initialized successfully!” your are on the right path. 

It’s time to set your kubectl:

# overwrite your previous config
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should see something like this:

[root@k8s-node01 ~]# kubectl get nodes
NAME         STATUS     ROLES    AGE    VERSION
k8s-node01   NotReady   master   2m6s   v1.17.9

[root@k8s-node01 ~]# journalctl -n 1 -u kubelet
-- Logs begin at Wed 2020-10-14 16:31:40 UTC, end at Wed 2020-10-14 18:18:37 UTC. --
Oct 14 18:18:37 k8s-node01 kubelet[8352]: E1014 18:18:37.304363    8352 kubelet.go:2184] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uni

Because cni plugin is not yet installed. It’s normal you get this message. It means it’s time to proceed installing your network solution: Calico.

Calico installation

I recommend to not join your other nodes until you get Calico Manifest installed!

Download the Calico networking manifest for the Kubernetes API datastore.

curl https://docs.projectcalico.org/manifests/calico.yaml -O

In this case, CIDR will be auto-detected. However, If all your nodes are in the same Layer-2 domain, you may want to disable encapsulation. In my case, I have disabled IP-n-IP:

- name: IP
  value: "autodetect"
# Enable IPIP
- name: CALICO_IPV4POOL_IPIP
  value: "Never"
# Enable or Disable VXLAN on the default IP pool.
- name: CALICO_IPV4POOL_VXLAN
  value: "Never"
# Set MTU for tunnel device used if ipip is enabled

it’s time to create your manifest

kubectl create -f calico.yaml

You should see something like this:

kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY   STATUS              RESTARTS   AGE
kube-system   calico-kube-controllers-7b4f58f565-pnn62   0/1     ContainerCreating   0    	29s
kube-system   calico-node-glznw                          0/1     Init:2/3            0    	29s
kube-system   coredns-6955765f44-2qc2k                   0/1     ContainerCreating   0    	30m
kube-system   coredns-6955765f44-spzcn                   0/1     ContainerCreating   0    	30m
kube-system   etcd-k8s-node01                            1/1     Running             0    	30m
kube-system   kube-apiserver-k8s-node01                  1/1     Running             0    	30m
kube-system   kube-controller-manager-k8s-node01         1/1     Running             0    	30m
kube-system   kube-proxy-bj4b6                           1/1     Running             0    	30m
kube-system   kube-scheduler-k8s-node01                  1/1     Running             0    	30m

After a few minutes all calico pods and coredns should be in Running status. Then, you can check if the server is Ready:

kubectl get nodes -o wide
Thu Oct 15 21:10:41 2020
NAME         STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
k8s-node01   Ready	master   87m   v1.17.9   10.10.10.101   <none>        CentOS Linux 7 (Core)   3.10.0-1127.19.1.el7.x86_64   docker://1.13.1

Joining workers to the cluster

After you initialize the controller and get the calico manifesto installed and its pods running. You can join the rest of the servers to the cluster. Do not forget to prep those servers with the script in the first section of this post. Just go to every server and run the suggested kubeadm join command you got on the kubeadm process.

kubeadm join 10.10.10.101:6443 --token ih67lm.uy4g5skeokaqbwfu \
    --discovery-token-ca-cert-hash sha256:14d3dc6bb9b4d4b50c6909214fa9bf2a2ea16ef96fd3b9aad946ee5c1f9d50c8

 

Then wait for the servers to be ready.

kubectl get nodes -o wide
Thu Oct 15 21:10:41 2020
NAME         STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
k8s-node01   Ready	master   87m   v1.17.9   10.10.10.101   <none>        CentOS Linux 7 (Core)   3.10.0-1127.19.1.el7.x86_64   docker://1.13.1
k8s-node02   Ready    <none>   74s   v1.17.9   10.10.10.102   <none>        CentOS Linux 7 (Core)   3.10.0-1127.19.1.el7.x86_64   docker://1.13.1

Time to test

This time we’ll do a deployment of 4 replicas and expose it with NodePort.

# file name: hello-node-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-node-deploy
spec:
  selector:
    matchLabels:
      run: hello-node-svc
  replicas: 4
  template:
    metadata:
      labels:
        run: hello-node-svc
    spec:
      containers:
        - name: hello-node-app
          image: pinrojas/hello-node:v1
          ports:
            - containerPort: 8080
              protocol: TCP

Let’s start creating the deployment first

kubectl create -f hello-node-deploy.yaml

You should see something like this

[root@k8s-node01 ~]# kubectl get pods --all-namespaces --output=wide
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE    IP               NODE         NOMINATED NODE   READINESS GATES
default       hello-node-deploy-8568b8dfb6-67f8x         1/1     Running   0          14m    10.140.135.129   k8s-node03   <none>           <none>
default       hello-node-deploy-8568b8dfb6-kltsv         1/1     Running   0          14m    10.140.135.130   k8s-node03   <none>           <none>
default       hello-node-deploy-8568b8dfb6-njvgc         1/1     Running   0          14m    10.140.58.193    k8s-node02   <none>           <none>
default       hello-node-deploy-8568b8dfb6-nxvtv         1/1     Running   0          14m    10.140.58.192    k8s-node02   <none>           <none>
kube-system   calico-kube-controllers-7b4f58f565-pnn62   1/1     Running   0          95m    10.140.85.194    k8s-node01   <none>           <none>
kube-system   calico-node-glznw                          1/1     Running   0          95m    10.10.10.101     k8s-node01   <none>           <none>
kube-system   calico-node-pvq8s                          1/1     Running   0          34m    10.10.10.103     k8s-node03   <none>           <none>
kube-system   calico-node-qgr5m                          1/1     Running   0          39m    10.10.10.102     k8s-node02   <none>           <none>
kube-system   coredns-6955765f44-2qc2k                   1/1     Running   0          125m   10.140.85.192    k8s-node01   <none>           <none>
kube-system   coredns-6955765f44-spzcn                   1/1     Running   0          125m   10.140.85.193    k8s-node01   <none>           <none>
kube-system   etcd-k8s-node01                            1/1     Running   0          124m   10.10.10.101     k8s-node01   <none>           <none>
kube-system   kube-apiserver-k8s-node01                  1/1     Running   0          124m   10.10.10.101     k8s-node01   <none>           <none>
kube-system   kube-controller-manager-k8s-node01         1/1     Running   0          124m   10.10.10.101     k8s-node01   <none>           <none>
kube-system   kube-proxy-bj4b6                           1/1     Running   0          125m   10.10.10.101     k8s-node01   <none>           <none>
kube-system   kube-proxy-bvx78                           1/1     Running   0          34m    10.10.10.103     k8s-node03   <none>           <none>
kube-system   kube-proxy-fmk8h                           1/1     Running   0          39m    10.10.10.102     k8s-node02   <none>           <none>
kube-system   kube-scheduler-k8s-node01                  1/1     Running   0          124m   10.10.10.101     k8s-node01   <none>           <none>

Now, you can expose it with NodePort Service

kubectl expose deployment hello-node-deploy --type=NodePort --name=hello-node-nport

Let’s check at what port was published

[root@k8s-node01 ~]# kubectl describe services hello-node-nport
Name:                     hello-node-nport
Namespace:                default
Labels:                   <none>
Annotations:              <none>
Selector:                 run=hello-node-svc
Type:                     NodePort
IP:                       10.99.66.126
Port:                     <unset>  8080/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  30083/TCP
Endpoints:                10.140.135.129:8080,10.140.135.130:8080,10.140.58.192:8080 + 1 more...
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

And let’s test it with curl

[root@k8s-node01 ~]# curl http://10.10.10.102:30083
Hello World version ONE! Host/Pod: hello-node-deploy-8568b8dfb6-njvgc
[root@k8s-node01 ~]# curl http://10.10.10.102:30083
Hello World version ONE! Host/Pod: hello-node-deploy-8568b8dfb6-kltsv
[root@k8s-node01 ~]# curl http://10.10.10.102:30083
Hello World version ONE! Host/Pod: hello-node-deploy-8568b8dfb6-kltsv
[root@k8s-node01 ~]# curl http://10.10.10.102:30083
Hello World version ONE! Host/Pod: hello-node-deploy-8568b8dfb6-67f8x
[root@k8s-node01 ~]# curl http://10.10.10.102:30083
Hello World version ONE! Host/Pod: hello-node-deploy-8568b8dfb6-nxvtv
[root@k8s-node01 ~]# curl http://10.10.10.102:30083
Hello World version ONE! Host/Pod: hello-node-deploy-8568b8dfb6-nxvtv

And we’re done. See ya!

Additional info

If you want to see what should be output you should get from kubeadm init command

[root@k8s-node01 ~]# kubeadm init --pod-network-cidr=10.140.0.0/16
W1015 19:40:51.331555   14905 version.go:101] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W1015 19:40:51.332165   14905 version.go:102] falling back to the local client version: v1.17.9
W1015 19:40:51.332876   14905 validation.go:28] Cannot validate kube-proxy config - no validator is available
W1015 19:40:51.332928   14905 validation.go:28] Cannot validate kubelet config - no validator is available
[init] Using Kubernetes version: v1.17.9
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-node01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.10.10.101]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-node01 localhost] and IPs [10.10.10.101 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-node01 localhost] and IPs [10.10.10.101 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W1015 19:42:32.993240   14905 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W1015 19:42:32.999184   14905 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 44.012968 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-node01 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-node01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: ih67lm.uy4g5skeokaqbwfu
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.10.10.101:6443 --token ih67lm.uy4g5skeokaqbwfu \
    --discovery-token-ca-cert-hash sha256:14d3dc6bb9b4d4b50c6909214fa9bf2a2ea16ef96fd3b9aad946ee5c1f9d50c8

 

If you want to see what should be output you should get from kubeadm join command

[root@k8s-node02 ~]# kubeadm join 10.10.10.101:6443 --token ih67lm.uy4g5skeokaqbwfu \
>     --discovery-token-ca-cert-hash sha256:14d3dc6bb9b4d4b50c6909214fa9bf2a2ea16ef96fd3b9aad946ee5c1f9d50c8
W1015 21:09:02.103753   22440 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

 

Leave a Reply