yml manifest or operator ?

Operator is a specific program dedicated to calico lifecyle management in k8s
yml manifest is the « old » way. It relies on deploying the calico resources on kubernetes via kubctl and a k8s yml.
yml manifest is simple but it has limitations.
Here what brings the operator way :
– automatic platform and configuration detection.
– A simplified upgrade procedure.
– Well-defined split between end-user configuration and product code.
– Resource reconciliation and lifecycle management.

Other differences :
– The manifests wa uses the kube-system namespace while the operator way use the calico-system namespace.
– Calico resources are not longer be hand-editable. Calico operator will reconcile undesired changes in order to maintain an expected state.
– Calico resources can instead be configured via the operator.tigera.io APIs.

Which datastore type ?

Two ways : etcd or kubernetes datastore.
Calico recommends the kubernetes datastore.

How to check the current datastore type value ?
Look in the calico controller pod.
For example with the manifest install way :
kubectl -n kube-system get -o yaml pod calico-kube-controllers-6d4bfc7c57-48jf6
We can see something like :

spec:
  containers:
  - env:
     ...
    - name: DATASTORE_TYPE
      value: kubernetes

How to set the datastore type value ?
Look in the calico.yml and edit it to use kubernetes DATASTORE_TYPE before to be deployed on k8s

Calico with Kubernetes : summary

– One pod running on the master node : calico-kube-controllers-XXX.
– One daemonset pod running on each node : calico-kube-node-XXX.

How to update calico ?

Upgrading an installation that uses manifests and the Kubernetes API datastore :
Download the v3.20 manifest that corresponds to your original installation method.
Calico for policy and networking:
curl https://docs.projectcalico.org/manifests/calico.yaml -O
Report your modifications on your current calico.yaml in that new calico.yaml and apply it : kubectl apply -f calico.yaml
Check that the calico pods run correctly :
kubectl get pods -n kube-system

Common issues with Calico

Problem : calico-kube-controllers-xxx pod is displayed as Running but is not read (0/1) such as :

NAME                                            READY   STATUS             RESTARTS   AGE
calico-kube-controllers-746f9d75cb-j7rzg        0/1     Running            4          24m

Symptoms : the pod is unhealthy because the readiness prob that fails with a event such as :

kube-system   25m         Warning   Unhealthy           pod/calico-kube-controllers-6d4bfc7c57-48jf6    Readiness probe failed: Failed to read status file /status/status.json: unexpected end of JSON input

The pod logs may also show these errors :

2020-09-25 17:20:16.226 [ERROR][1] status.go 129: Failed to write readiness file: open status.json: permission denied

Cause :
The calico image is flawed. It performs operation with a user that don’t have the right for.
Check the image version used by Calico controller with a command such as :
kubectl -n kube-system get -o yaml pod calico-kube-controllers-746f9d75cb-j7rzg | grep image
If the image is older than 3.20, the problem is probably there.


Solution :
– Quick and dirty : edit the running pod to update its user access rights (no tested but should work).
– Download a more recent calico.yml manifest (ensure that is uses at least the 3.20 image) and update the calico pods with that.

Problem : calico-node-xxx pods are displayed as Running but are not ready (0/1) such as :

NAME                                            READY   STATUS             RESTARTS   AGE
calico-node-9qvwl                               0/1     Running            0          3m
calico-node-zqsnp                               0/1     Running            0          2m59s

Symptoms : the pods are unhealthy because the readiness prob that fails with events such as :

  Warning  Unhealthy  51s   kubelet            Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused
  Warning  Unhealthy  41s   kubelet            Readiness probe failed: 2021-08-18 15:02:50.213 [INFO][157] confd/health.go 180: Number of node(s) with BGP peering established = 0
  Warning  Unhealthy  2m14s  kubelet            Readiness probe failed: 2021-08-18 14:49:46.535 [INFO][155] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 172.28.0.1

Note : the pod logs may also show alike errors.

Cause :
BIRD (one of components on which Calico relies on) and BGP (Border Gateway Protocol), protocol TCP on the port 179 by default, used by Calico to communicate between nodes have connectivity issues and finally doesn’t manage to establish a connection between nodes.

Solution (2 possible problems) :
– ensure that BGP ports are reachable between nodes.
For example if we have two nodes 192.168.0.2 and 192.168.0.5, we could :
executing from 192.168.0.2 : telnet 192.168.0.5 179
executing from 192.168.0.5 : telnet 192.168.0.2 179
If the checks don’t work, check that hosts are correct and that ports are well reachable from each other.
– ensure that calico uses the correct ip address for the nodes.
For example, look at :
calico/node is not ready: BIRD is not ready: BGP not established with 172.28.0.1.
Here one of node tries to communicate with another node by using the host : 172.28.0.1.
172.28.0.1 is is one of ip addresses of that node but it turns out that it is not the one that we want to use to be able to communicate with the node.
Indeed on a host, it may exist multiple physical interfaces or even multiple IP addresses configured on a physical interface.
If you fall into that issue, ensure that calico nodes always select the eth.* IP (such as eth0, eth1…) :
Update the daemonset configuration :
kubectl set env daemonset/calico-node -n kube-system IP_AUTODETECTION_METHOD=interface=eth.*
Then kill all calico-node pods.