When we want to make Calico and MetalLB working together with BGP, we’ll run into the following issue: BGP only allows one session to be established per pair of nodes. So, if Calico has a session established with your BGP router, MetalLB cannot establish its own session – it’ll get rejected as a duplicate by BGP’s conflict resolution algorithm.
In this case, I wanted to keep using the ToR (Top of Rack) routers for the LoadBalancer instead of Spines as it’s normally proposed. Basically, because if you use overlay networks in a Fabric, Spines could be managed as router reflectors for EVPN-iBGP and transport for VxLANs.
Solution: I used a dummy interface in every kubernetes node as source-address for metallb
My Setup is using Calico and MetalLB working together with BGP in Kubernetes v1.20.5. My Cluster was installed with kubeadm with control-planes/master redundancy on Fedora 34 (Kernel version 5.13.4-200). I am using cri-o as my container runtime.
[root@ctl-a1 ~]# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ctl-a1 Ready control-plane,master 12d v1.20.5 192.168.101.30 <none> Fedora 34 (Cloud Edition) 5.13.4-200.fc34.x86_64 cri-o://1.20.0 ctl-a2 Ready control-plane,master 12d v1.20.5 192.168.101.31 <none> Fedora 34 (Cloud Edition) 5.13.4-200.fc34.x86_64 cri-o://1.20.0 ctl-a3 Ready control-plane,master 12d v1.20.5 192.168.101.32 <none> Fedora 34 (Cloud Edition) 5.13.4-200.fc34.x86_64 cri-o://1.20.0 wk-a1 Ready <none> 12d v1.20.5 192.168.101.20 <none> Fedora 34 (Cloud Edition) 5.13.4-200.fc34.x86_64 cri-o://1.20.0 wk-a2 Ready <none> 12d v1.20.5 192.168.101.21 <none> Fedora 34 (Cloud Edition) 5.13.4-200.fc34.x86_64 cri-o://1.20.0 wk-a3 Ready <none> 12d v1.20.5 192.168.101.22 <none> Fedora 34 (Cloud Edition) 5.13.4-200.fc34.x86_64 cri-o://1.20.0
This cluster was installed using this procedure at “On-premises Fedora34 K8s HA Cluster with kubeadm and Calico”
Servers are connected to two different EVPN domains (or subnets) using VLANs via the interface eth1. I have also created a dummy interface as additional loopback to be used as source-address in metalllb BGP peering.
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 52:54:00:12:15:57 brd ff:ff:ff:ff:ff:ff
altname enp0s4
altname ens4
4: VLAN-10@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:12:15:57 brd ff:ff:ff:ff:ff:ff
inet 192.168.101.30/24 brd 192.168.101.255 scope global noprefixroute VLAN-10
valid_lft forever preferred_lft forever
inet6 fe80::f9de:e5e3:ddee:5184/64 scope link noprefixroute
valid_lft forever preferred_lft forever
5: VLAN-20@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:12:15:57 brd ff:ff:ff:ff:ff:ff
inet 172.16.101.30/24 brd 172.16.101.255 scope global noprefixroute VLAN-20
valid_lft forever preferred_lft forever
inet6 fe80::2ef6:658f:a7e0:3963/64 scope link noprefixroute
valid_lft forever preferred_lft forever
14: dummy: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 46:cc:d1:82:d4:64 brd ff:ff:ff:ff:ff:ff
inet 6.4.5.30/32 scope global noprefixroute dummy
valid_lft forever preferred_lft forever
inet6 fe80::9d64:7bc8:a43c:4322/64 scope link noprefixroute
valid_lft forever preferred_lft forever
The dummy interface was created manually as follow:
nmcli connection add type dummy ifname dummy ipv4.method manual ipv4.addresses 6.4.5.30/32
The same dummy interface has been created in all k8s nodes.
# for i in {20,21,22,30,31,32}; do ssh root@172.20.20.$i ifconfig dummy | head -2 ; done
dummy: flags=195<UP,BROADCAST,RUNNING,NOARP> mtu 1500
inet 6.4.5.20 netmask 255.255.255.255 broadcast 0.0.0.0
dummy: flags=195<UP,BROADCAST,RUNNING,NOARP> mtu 1500
inet 6.4.5.21 netmask 255.255.255.255 broadcast 0.0.0.0
dummy: flags=195<UP,BROADCAST,RUNNING,NOARP> mtu 1500
inet 6.4.5.22 netmask 255.255.255.255 broadcast 0.0.0.0
dummy: flags=195<UP,BROADCAST,RUNNING,NOARP> mtu 1500
inet 6.4.5.30 netmask 255.255.255.255 broadcast 0.0.0.0
dummy: flags=195<UP,BROADCAST,RUNNING,NOARP> mtu 1500
inet 6.4.5.31 netmask 255.255.255.255 broadcast 0.0.0.0
dummy: flags=195<UP,BROADCAST,RUNNING,NOARP> mtu 1500
inet 6.4.5.32 netmask 255.255.255.255 broadcast 0.0.0.0
SRL Spine/Leaf Topology
In this case I am using Nokia SRL via Containerlab for Calico and MetalLB working together with BGP. You can see my config for every router in my repo
Next picture shows the eBGP underlay setup with Nokia SRL Spine/Leaf
Next picture shows overlay BGP-EVPN setup with Nokia SRL Spine/Leaf
Next picture shows Calico BGP setup to Nokia SRL Spine/Leaf Overlay VRF
Next picture shows Metallb BGP setup to Nokia SRL Spine/Leaf Overlay VRF
How to apply a YAML Manifest
All manifest I have in this post can be apply as follow:
kubectl apply -f <manifest yaml file>
Example: You want to install calico manifest. Then just run:
kubectl apply -f calico.yaml
Calico BGP Setup
In order to make Calico and MetalLB working together with BGP, let’s start setting up Calico:
- After installing calico using the manifest, then have to setup the BGP configuration as follow
apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: logSeverityScreen: Info nodeToNodeMeshEnabled: false serviceClusterIPs: - cidr: 10.96.0.0/12
- And now you set up the BGP peers for Calico via manifest
apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: leaf1-tor spec: peerIP: 6.5.3.1 asNumber: 65310 nodeSelector: leaf == 'leaf1' sourceAddress: "None" --- apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: leaf2-tor spec: peerIP: 6.5.3.2 asNumber: 65320 nodeSelector: leaf == 'leaf2' sourceAddress: "None"
And wait for the neighbor to establish connection to the router. After that, you should be ok to start using your pods.
Multus and calico
Multus is a CNI meta-plugin. In this case I’ve had also install multus. Check the used manifest for
multus installation for cri-o.
Installation is very simple. Just apply the manifest and you are done. Multus won’t impact any configuration in Calico. It’s just there to add more CNI plugins, like MACVLAN, and use them to add additional ports to pods. However, this post won’t cover anything else related to multus. I will bring something later in other post.
Metallb BGP Setup
After you have installed Calico. You can start installing and setting up Metallb.
- For the installation you just need to install the manifests
kubectl apply -f metallb-install-namespace.yaml kubectl apply -f metallb-install-main-image.yaml
“metallb-install-main-image.yaml” is the same manifest provided in https://metallb.universe.tf/installation/ but with metallb/controller:main and metallb/speaker:main images that have the latest changes. In this case, we need source-address option. More details of this option in Make BGP source address configurable thread.
In this case, we are using the dummy interface as the source-address to avoid conflicts with Calico.
- Now, you can setup metallb BGP peers and IP Pool with the following manifest
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
peers:
- peer-address: 6.5.3.1
peer-asn: 65310
my-asn: 65201
router-id: 6.4.5.20
source-address: 6.4.5.20
node-selectors:
- match-expressions:
- key: kubernetes.io/hostname
operator: In
values: [wk-a1]
- peer-address: 6.5.3.2
peer-asn: 65320
my-asn: 65201
router-id: 6.4.5.21
source-address: 6.4.5.21
node-selectors:
- match-expressions:
- key: kubernetes.io/hostname
operator: In
values: [wk-a2]
- peer-address: 6.5.3.1
peer-asn: 65310
my-asn: 65201
router-id: 6.4.5.22
source-address: 6.4.5.22
node-selectors:
- match-expressions:
- key: kubernetes.io/hostname
operator: In
values: [wk-a3]
- peer-address: 6.5.3.2
peer-asn: 65320
my-asn: 65201
router-id: 6.4.5.30
source-address: 6.4.5.30
node-selectors:
- match-expressions:
- key: kubernetes.io/hostname
operator: In
values: [ctl-a1]
- peer-address: 6.5.3.1
peer-asn: 65310
my-asn: 65201
router-id: 6.4.5.31
source-address: 6.4.5.31
node-selectors:
- match-expressions:
- key: kubernetes.io/hostname
operator: In
values: [ctl-a2]
- peer-address: 6.5.3.2
peer-asn: 65320
my-asn: 65201
router-id: 6.4.5.32
source-address: 6.4.5.32
node-selectors:
- match-expressions:
- key: kubernetes.io/hostname
operator: In
values: [ctl-a3]
address-pools:
- name: default
protocol: bgp
addresses:
- 10.254.254.240/28
And we are done.
Sharing results
Here you have the output of one of my leaf routers showing how we made Calico and MetalLB working together with BGP.
A:leaf1# show network-instance ip-vrf1 protocols bgp neighbor ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- BGP neighbor summary for network-instance "ip-vrf1" Flags: S static, D dynamic, L discovered by LLDP, B BFD enabled, - disabled, * slow ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- +------------------------+-----------------------------------+------------------------+--------+-------------+-------------------+-------------------+-----------------+-----------------------------------+ | Net-Inst | Peer | Group | Flags | Peer-AS | State | Uptime | AFI/SAFI | [Rx/Active/Tx] | +========================+===================================+========================+========+=============+===================+===================+=================+===================================+ | ip-vrf1 | 6.4.5.20 | metallb-bgp | D | 65201 | established | 4d:23h:43m:18s | ipv4-unicast | [4/0/33] | | ip-vrf1 | 6.4.5.22 | metallb-bgp | D | 65201 | established | 4d:23h:43m:18s | ipv4-unicast | [3/0/33] | | ip-vrf1 | 6.4.5.31 | metallb-bgp | D | 65201 | established | 4d:23h:43m:17s | ipv4-unicast | [1/0/33] | | ip-vrf1 | 192.168.101.20 | calico-bgp | D | 64512 | established | 6d:21h:23m:49s | ipv4-unicast | [5/1/10] | | ip-vrf1 | 192.168.101.22 | calico-bgp | D | 64512 | established | 6d:21h:24m:4s | ipv4-unicast | [4/1/10] | | ip-vrf1 | 192.168.101.31 | calico-bgp | D | 64512 | established | 6d:21h:23m:47s | ipv4-unicast | [2/1/10] | +------------------------+-----------------------------------+------------------------+--------+-------------+-------------------+-------------------+-----------------+-----------------------------------+ ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Summary: 0 configured neighbors, 0 configured sessions are established,0 disabled peers 6 dynamic peers
See ya!
