Node Allocatable

The general idea :

	
      Node Capacity
---------------------------
|     kube-reserved       |
|-------------------------|
|     system-reserved     |
|-------------------------|
|    eviction-threshold   |
|-------------------------|
|                         |
|      allocatable        |
|   (available for pods)  |
|                         |
|                         |

Node resource information :
– kubectl describe node NODE

Capacity:
  cpu:                24
  ephemeral-storage:  4263600Ki  = 4.2GB
Allocatable:
  cpu:                24
  ephemeral-storage:  3215024Ki = 3.2GB

That ephemeral-storage info looks to be not dynamic (valued a single time at node kubelet startup)

Another way is calling the kubectl api :
First we start the proxy : kubectl proxy -p fooUnusedPort &
Then we could request the node stats :
curl "localhost:fooUnusedPort/api/v1/nodes/NODE/proxy/stats/summary"

It returns a lot of information :
– resource state for images fs
– resource state for containers fs
– resource state for fs

For example here :

"fs": {
   "time": "2020-10-01T09:26:11Z",
   "availableBytes": 3178647552,  = 3.2GB
   "capacityBytes":  4365926400,  = 4.2GB
   "usedBytes": 958107648,
   "inodesFree": 272372,
   "inodes": 278528,
   "inodesUsed": 6156
  },

node.fs.availableBytes is the space available for ephemoral storage.
It looks to be updated dynamically

How is computed node.fs.capacityBytes(storage of the node)

Several directories :
– /var/lib/kubelet
– ???

How fix fs space issues on a node

If the mount on /var/lib/kubelet doesn’t contain much spaces, K8s deploys on the node may fail
To fix that :
– either increase the space size of /var/lib/kubelet
– create a symlink from /var/lib/kubelet to a directory that has more space.

———-
Kubelet :

KUBELET find default service conf :
find / -type f -name 10-kubeadm.conf 2>/dev/null

It may at :/etc/systemd/system/kubelet.service.d/10-kubeadm.conf or still at :/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

Garbage collection for container images

Overview

Garbage collection (GC) is a function of kubelet that clean up unused images and unused containers.
Kubelet perform GC for containers every minute and GC for images every five minutes.

configuration

image-gc-high-threshold : percent of disk usage which triggers image garbage collection.
Default is 85%.
image-gc-low-threshold : percent of disk usage to which image garbage collection attempts to free.
Default is 80%.

Kubelet configurations example

Kubelet configuration are set for the current node and not the whole cluster.
So to enable a configuration in the whole cluster, we need to set the changes on each node of that !
The configuration may be set as arguments to kubelet command line or added in the /var/lib/kubelet/config.yaml file.

Example of very minimal conditions for evictions

Note : imageGCLowThresholdPercent has to be lower than imageGCHighThresholdPercent. Otherwise kubelet fails to start.

#kubelet eviction hard conditions to clean unused containers and images
evictionHard:
  imagefs.available: 1%
  memory.available: 100Mi
  nodefs.available: 1%
  nodefs.inodesFree: 1%
# kubelet rule for GC to periocaly  clean unused containers and images
imageGCHighThresholdPercent: 95
imageGCLowThresholdPercent: 94