What are Gitlab Runners
Gitlab runners execute pipeline jobs. These runners are configured and registered inside a
gitlab runner application.
That application can be installed directly on the OS or a docker container.
The runners configuration is stored into /etc/gitlab-runner/config.toml where each
runner is defined as a [[runners]] entry.
General common issues
– The job is failing with a 403 error such as:
Getting source from Git repository 00:01 Fetching changes with git depth set to 20... Reinitialized existing Git repository in /builds/basic-projects-using-shared-ci/maven-docker-basic-example1/.git/ remote: You are not allowed to download code from this project. fatal: unable to access 'http://gitlab.david.com/basic-projects-using-shared-ci/maven-docker-basic-example1.git/': The requested URL returned error: 403 ERROR: Job failed: exit code 1 |
How to fix.
Normally we don’t need to set any credential in the gitlab runner configuration to allow git
cloning.
So trying to do that is not the right way.
A common issue is the user that triggers the job is not member of the project.
Not: even if we are administrator on the gitlab it doesn’t mean we are member of all
projects,it means we need to check in the project or the group precisely.
– The job is stuck with the following error message where foo-tag… is (are) the tags set for a stage defined in the yaml pipeline :
This job is stuck because you don't have any active runners online or available with any of these tags assigned to them: foo-tag..." Go to project CI settings |
How to fix.
case 1 : no runner at all is installed matching to that tag.
Solution : as suggested,
configure/add a runner for that tag.
case 2 : a runner is installed for that tag but it is not accessible for that project.
subcase 1 : the runner was installed specifically in another project/Settings/CICD and not in a
common group which the two project belong.
solution : delete the runner installed at the project level and install it at the
group level.
subcase 2 :the runner was installed in the common group but in Settings/CICD,
the « Enable shared runners for this group » was disabled.
solution : just enable it again.
– The docker runner fails eagerly with a error message concerning caching volume.
The message can look like :
ERROR: Preparation failed: adding cache volume: set volume permissions: running permission container "9802d1f1586765cef195531e7d5571911151d5ba698c2c4d70a251e9ea2e14c5" for volume "runner-84exxmn--project-23046492-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8": starting permission container: Error response from daemon: error evaluating symlinks from mount source "/var/lib/docker/volumes/runner-84exxmn--project-23046492-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8/_data": lstat /var/lib/docker/volumes/runner-84exxmn--project-23046492-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8: no such file or directory (linux_set.go:105:0s) |
That kind of error may happen when a docker volume used for gitlab for caching resources was
dropped.
How to fix : Recreate the docker volume on the hosts with the folder _data as child
folder such as :
"9802d1f1586765cef195531e7d5571911151d5ba698c2c4d70a251e9ea2e14c5"/_data .
– gitlab runner job fails eagerly with a error message concerning disk space missing in a
subfolder of /var/lib/docker/overlay2/ while the directory contains a lot of space.
The message looks like : :
ERROR: Job failed (system failure): Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/15454/545454.... no space left on device (docker.go:787:0s)
How to fix :
-check first with df that the /var/lib/docker/overlay2/ folder of the host where the gitlab
runner is run has enough space
Double validate it with a dd to generate for example a file of 2GB.
– if no space issue detected : try by deleting/restarting the gitlab runner. If not working
restart docker too
Install a runner application running as a docker container
Create a docker runner application instance (here with a volume):
docker run -d --name gitlab-runner --restart always \ -v /var/run/docker.sock:/var/run/docker.sock \ -v gitlab-runner-config:/etc/gitlab-runner \ gitlab/gitlab-runner:latest |
The volume config contains the gitlab runner app configuration (mainly the .toml file) and we map
the docker socket of the host to allow the runners that work with docker executor to
start/stop/rm docker containers.
Register a runner instance with interactive form :
docker run --rm -it -v gitlab-runner-config:/etc/gitlab-runner gitlab/gitlab-runner:latest register |
Here we are prompted to enter :
– the gitlab url
– the registration token retrievable since the Gitlab GUI settings/CDI/Runners.
– the executor type (docker, kubernetes, custom, shell, virtualbox, …)
– the base image
Register a runner instance with non-interactive form :
Here it is an example to create a runner allowing to use the docker client via the docker socket
binding of the host :
docker run --rm -it \ -v gitlab-runner-config:/etc/gitlab-runner \ -v /var/run/docker.sock:/var/run/docker.sock \ gitlab/gitlab-runner:latest register \ --non-interactive \ --executor "docker" \ --docker-image "docker:19.03.12" \ --url "https://gitlab.com" \ --registration-token "ijEKCYos8s4hPv7J5DXW" \ --description "docker-runner" \ --tag-list "docker" \ --access-level="not_protected" |
Restart the runner application (not required generally because that performs hot
changes) :
docker restart gitlab-runner
Enable Docker commands in your CI/CD jobs
There are 4 ways to enable the use of docker build and docker run during jobs, each with their
own tradeoffs :
– The shell executor. Here job scripts are executed as the gitlab-runner user of the Gitlab
runner machine/server and it requires to install Docker Engine on that machine/server.
– The Docker executor with the Docker image (Docker-in-Docker)
– Docker socket binding
– kaniko. It is a Google tool that doesn’t depend on a Docker daemon and executes each command
within a Dockerfile completely in userspace
To use the Docker-in-Docker way or the Docker socket binding, we need to specify the
docker executor.
Here is a runner register with the docker socket binding way :
WARN : The
non-interactive form doesn’t work (in my case)! It created a docker+machine executor instead of
a docker executor. That is not visible in the config.toml but at runtime when the gitlab runner
starts : Preparing the « docker+machine » executor
docker run --rm -it \ -v gitlab-runner-config:/etc/gitlab-runner \ gitlab/gitlab-runner:latest register \ --non-interactive \ --executor "docker" \ --docker-image "docker:19.03.12" \ --docker-volumes /var/run/docker.sock:/var/run/docker.sock \ --url "https://gitlab.com" \ --registration-token "ijEKCYos8s4hPv7J5DXW" \ --description "docker-runner" \ --tag-list "docker" \ --access-level="not_protected" |
So favor the interactive form by selecting the docker executor when asked.
Finally we need to have in the config.toml a runner with a volume mount for the docker socket such as :
[[runners]] name = "docker-runner" url = "https://gitlab.com" token = "Kg-y8nNj3zsvx9Xs1DsM" executor = "docker" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure] [runners.docker] tls_verify = false image = "docker:19.03.12" privileged = false disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = false volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"] pull_policy = "if-not-present" extra_hosts = ["gitlab.david.com:172.17.0.1"] shm_size = 0 |
pull_policy:
To use local images present on the docker host (scenario of docker with socket binding), we use
that option :
[runners.docker] pull_policy = "if-not-present" |
Helpful if the docker image specified with the FROM keyword of Dockerfile or the image property
specified in the gitlab-ci is a custom image that is not in the public registry.
Without that stick, we need to add the image in a private registry and to configure the gitlab
runner to use a certificate to access to that registry (see /etc/gitlab/certs folder in the
gitlab runner)
extra_hosts:
It mimics the add-host argument of the docker run command.
It adds mapping(s) « host:ip address » in the /etc/hosts of the ephemeral container
started by the docker runner.
And here how to use docker in pipeline :
Docker Build: stage: docker-build tags: - docker # docker is the tag we specified for the runner image: docker:19.03.12 # optional, we can use the image defined in the runner script: - docker build -f docker/Dockerfile -t my-app . when: always only: refs: - master |
The docker in docker way is similar but requires two changes : no socket binding in the runner volume config, privileged mode to true and dind service conf. For now I didn’t manage to do it working.
Use docker-compose in your CI/CD jobs
There are several ways to use that.
A very simple is defining a stage relying on the docker image docker/compose:latest
(or better a specific version such as 1.27.4) and also declaring the docker dind service or a
gitlab runner mounted with the host docker socket.
Example with a gitlab runner mounted with the host docker socket :
Docker Build: stage: docker-build tags: - docker image: name: docker/compose:1.27.4 script: - docker-compose -v - docker-compose -f docker-compose-single-step.yml ps - docker-compose -f docker-compose-single-step.yml build when: always only: refs: - master |
Example with a docker dind service: …
Kubernetes executor
Use the Kubernetes executor to use Kubernetes clusters for your builds. The executor calls the Kubernetes cluster API and creates a pod for each GitLab CI job.
The Kubernetes executor divides the build into multiple steps:
Prepare: Create the Pod against the Kubernetes Cluster. This creates the containers required for the build and services to run.
Pre-build: Clone, restore cache, and download artifacts from previous stages. This step runs on a special container as part of the pod.
Build: User build.
Post-build: Create cache, upload artifacts to GitLab. This step also uses the special container as part of the pod.
Deploy to Kubernetes
Two ways to deploy on kubernetes :
– to use a runner with the Kubernetes executor and correct env variables in the runner
– to use a runner with any executor but using a kubernetes cluster configuration registered in
gitlab
Scenario 1 : gitlab ci native integration with an existing kubernetes cluster that we added in gitlab
Register an existing kubernetes cluster in gitlab
The cluster may be added at 3 distinct levels in gitlab :
– project
– group
– instance/global
In the Gitlab Kubernetes menu page, search the option « Add a Kubernetes cluster
integration ».
Here a preview with gitlab and k8s setup in local :
href= »http://myjavaadventures.com/blog/wp-content/uploads/2021/08/add-existing-k8s-in-gitlab1.png »> src= »http://myjavaadventures.com/blog/wp-content/uploads/2021/08/add-existing-k8s-in-gitlab1-1024×601.png »
alt= »add-existing-k8s-in-gitlab » width= »640″ height= »376″
class= »alignnone size-large wp-image-5309″ />
Fields to set :
– Kubernetes cluster name (required) – The name you wish to give the cluster.
– Environment scope (required) – The associated environment to this cluster.
– API URL (required) – It’s the URL that GitLab uses to access the Kubernetes
API. Kubernetes exposes several APIs, we want the “base” URL that is common to all of them. For
example, https://kubernetes.example.com rather than https://kubernetes.example.com/api/v1.
Get the API URL by running this command:
kubectl cluster-info | grep -E 'Kubernetes master|Kubernetes control plane' | awk '/http/ {print $NF}'
– CA certificate (required) – A valid Kubernetes certificate is needed to
authenticate to the cluster. We use the certificate created by default.
List the secrets with kubectl get secrets, and one should be named similar to
default-token-xxxxx. Copy that token name for use below.
Get the certificate by running this command:
kubectl get secret <secret name> -o jsonpath="{['data']['ca\.crt']}" | base64 --decode
Note : if the command returns the entire certificate chain, you must copy the Root CA
certificate and any intermediate certificates at the bottom of the chain.
– Token – GitLab authenticates against Kubernetes using service tokens, which
are scoped to a particular namespace.
The token used should belong to a service account with cluster-admin privileges.
To create this service account:
Create a file called gitlab-admin-service-account.yaml and deploy it with kubectl :
kubectl apply -f gitlab-admin-service-account.yaml
apiVersion: v1 kind: ServiceAccount metadata: name: gitlab namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: gitlab-admin roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: gitlab namespace: kube-system |
At last retrieve the token for the gitlab service account:
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep gitlab | awk '{print $1}')
– Project namespace (optional) – You don’t have to fill this in. By leaving it
blank, GitLab creates one for you.
Note:
Each project should have a unique namespace.
The project namespace is not necessarily the namespace of the secret, if you’re using a secret
with broader permissions, like the secret from default.
You should not use default as the project namespace.
If you or someone created a secret specifically for the project, usually with limited
permissions, the secret’s namespace and project namespace may be the same.
Define a pipeline in 3 stage : build a spring boot/redis app, build/tag/push a docker image to the gitlab container registry and finally deploys the image as a pod on kubernetes
Here is the complete pipeline :
stages: - maven-build - docker-build - deploy variables: MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository" CI_DEBUG_TRACE: "true" Maven Package: stage: maven-build tags: - maven3-jdk11 image: maven:3.6.3-jdk-11 cache: paths: - .m2/repository key: ${CI_COMMIT_REF_NAME} script: # we store GIT branch/commit/pipeline info into a file. Helpful for app version actuator or about->version information - echo { \"branch\":\"${CI_COMMIT_REF_NAME}\", \"commit\":\"${CI_COMMIT_SHA}\", \"pipelineId\":\"${CI_PIPELINE_ID}\"} > git-info.json - echo "git-info.json content =" && cat git-info.json - mv git-info.json ${CI_PROJECT_DIR}/src/main/resources - mvn $MAVEN_ARGS clean package # compute the the docker build version : if master, we take the maven artifact version without snapshot. Otherwise we take the take the branch name - > if [[ "${CI_COMMIT_REF_NAME}" = "master" ]]; then dockerImageVersion=$(mvn $MAVEN_ARGS -q -U -Dexpression=project.version -DforceStdout org.apache.maven.plugins:maven-help-plugin:3.2.0:evaluate) dockerImageVersion=${dockerImageVersion/-SNAPSHOT/} else dockerImageVersion=${CI_COMMIT_REF_NAME} fi - echo "dockerImageVersion=${dockerImageVersion}" - echo ${dockerImageVersion} > docker-image-version.json artifacts: name: ${CI_PROJECT_NAME}:${CI_COMMIT_REF_NAME} expire_in: 1 week paths: - ${CI_PROJECT_DIR}/target/dependency/ - ${CI_PROJECT_DIR}/docker-image-version.json when: always only: refs: - master Docker Build: stage: docker-build tags: - docker # image: docker:19.03.12 script: - cat docker-image-version.json - ls -R target/ - imageName="$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME:$(cat docker-image-version.json)" - docker build -f docker/Dockerfile -t $imageName . - docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY - docker tag $imageName $CI_REGISTRY/$imageName - docker push $CI_REGISTRY/$imageName when: always only: refs: - master Deploy in Kubernetes: stage: deploy tags: - docker # image: google/cloud-sdk:350.0.0 # image: registry.gitlab.com/gitlab-examples/kubernetes-deploy image: name: bitnami/kubectl:1.18.19 entrypoint: [""] # 1. by default the image cmd is set in the entrypoint as kubectl and gitlab runner executes and 2.gitlab runner executes the entrypoint of the image at the "step_script" stage. # "so it executes sth like "kubectl sh -c" # Which generates that error : unknown command "sh" for "kubectl" # Fix : removing the entrypoint environment: dev script: - kubectl config view --minify | grep namespace # create a secret in the current namespace (computed and created by gitlab runner) to allow to login to the container registry - kubectl create secret docker-registry gitlab-registry --docker-server="$CI_REGISTRY" --docker-username="$CI_DEPLOY_USER" --docker-password="$CI_DEPLOY_PASSWORD" --docker-email="$GITLAB_USER_EMAIL" -o yaml --dry-run=client | kubectl apply -f - # update the service account to use that secret to pull images - > kubectl -n spring-boot-docker-kubernetes-example-32-dev patch serviceaccount default -p '{"imagePullSecrets": [{"name": "gitlab-registry"}]}' - EPOCH_SEC=\"$(date +%s)\" - image_version=$(cat docker-image-version.json) - image="$CI_REGISTRY/kubernetes-projects/spring-boot-docker-kubernetes-example:${image_version}" - echo $CI_DEPLOY_USER - echo "EPOCH_SEC=$EPOCH_SEC" - echo "image_version=$image_version" - echo "image=$image" - kubectl get pods - cp k8s/deployment.yml k8s/deployment-dynamic.yml - sed -i s@__COMMIT_SHA__@$CI_COMMIT_SHA@ k8s/deployment-dynamic.yml - sed -i s@__DATE_TIME__@$EPOCH_SEC@ k8s/deployment-dynamic.yml - sed -i s@__JAR_VERSION__@$image_version@ k8s/deployment-dynamic.yml - sed -i s@__IMAGE__@$image@ k8s/deployment-dynamic.yml - sed -i s@__NAMESPACE__@$KUBE_NAMESPACE@g k8s/deployment-dynamic.yml - kubectl apply -f k8s/deployment-dynamic.yml when: always only: refs: - master |
And here the kubernetes yml that defines the resource to deploy.
Note that that is a template with some variable part that we set during the deploy stage before
effectively deploying the resources with kubectl.
apiVersion: v1 kind: Namespace metadata: name: __NAMESPACE__ --- # spring boot app apiVersion: apps/v1 kind: Deployment metadata: name: spring-boot-docker-kubernetes-example-sboot namespace: __NAMESPACE__ spec: replicas: 1 selector: #it defines how the Deployment finds which Pods to manage. matchLabels: # It may have complexer rules. Here we simply select a label that is defined just below in the pod template app: spring-boot-docker-kubernetes-example-sboot template: # pod template metadata: labels: #Optional : add a commit sha or a datime label to make the deployment to be updated by kubcetl apply even if the docker image is the same app: spring-boot-docker-kubernetes-example-sboot commitSha: __COMMIT_SHA__ dateTime: __DATE_TIME__ jarVersion: __JAR_VERSION__ spec: containers: - image: __IMAGE__ #Make imag variable name: spring-boot-docker-kubernetes-example-sboot command: ["java"] args: ["-cp", "/app:/app/lib/*", "-Djava.security.egd=file:/dev/./urandom", "-Xdebug", "-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=0.0.0.0:8888", "davidxxx.SpringBootAndRedisApplication"] imagePullPolicy: Always # useful in dev because the source code may change but that we reuse the same dev docker image # imagePullPolicy: IfNotPresent # prevent pulling from a docker registry readinessProbe: httpGet: scheme: HTTP # We use the default spring boot health check readiness (spring boot 2.2 or +) path: /actuator/health/readiness port: 8090 initialDelaySeconds: 15 timeoutSeconds: 5 livenessProbe: httpGet: scheme: HTTP # We use the default spring boot health check liveness (spring boot 2.2 or +) path: /actuator/health/liveness port: 8090 initialDelaySeconds: 15 timeoutSeconds: 15 # it is informational (only ports declared in service are required). It is alike docker EXPOSE instruction # ports: # - containerPort: 8090 # most of the time : it has to be the same than NodePort.targetPort --- apiVersion: v1 kind: Service metadata: labels: app: spring-boot-docker-kubernetes-example-sboot # service name matters. It provides the way which other pods of the cluster may communicate with that pod name: spring-boot-docker-kubernetes-example-sboot namespace: __NAMESPACE__ spec: type: NodePort ports: - name: "application-port" # port.name is just informational targetPort: 8090 # port of the running app port: 8090 # Cluster IP Port nodePort: 30000 # External port (has to be unique in the cluster). By default Kubernetes allocates a node port from a range (default: 30000-32767) - name: "debug-port"# port.name is just informational targetPort: 8888 # port of the running app port: 8888 # Cluster IP Port nodePort: 30001 # External port (has to be unique in the cluster). By default Kubernetes allocates a node port from a range (default: 30000-32767) selector: app: spring-boot-docker-kubernetes-example-sboot ###------- REDIS---------### # Redis app --- apiVersion: apps/v1 kind: Deployment metadata: name: spring-boot-docker-kubernetes-example-redis namespace: __NAMESPACE__ spec: replicas: 1 selector: #it defines how the Deployment finds which Pods to manage. matchLabels: # It may have complexer rules. Here we simply select a label that is defined just below in the pod template app: spring-boot-docker-kubernetes-example-redis template: # pod template metadata: labels: app: spring-boot-docker-kubernetes-example-redis commitSha: __COMMIT_SHA__ #Optional : add a commit sha label to make the deployment to be updated by kubcetl apply even if the docker image is the same dateTime: __DATE_TIME__ spec: # Optional : hard constraint : force to deploy on a node that has as tag redis-data=true affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: redis-data operator: In values: - "true" # alternative : soft constraint : try to deploy on a node that has as tag redis-data=true # preferredDuringSchedulingIgnoredDuringExecution: # - weight: 1 # preference: # matchExpressions: # - key: redis-data # operator: In # values: # - "true" containers: - image: redis:6.0.9-alpine3.12 name: spring-boot-docker-kubernetes-example-redis # --appendonly flag allows to persist data into a file command: ["redis-server"] args: ["--appendonly", "yes"] # it is informational (only ports declared in service are required). It is alike docker EXPOSE instruction # ports: # - containerPort: 6379 # name: foo-redis volumeMounts: - mountPath: /data name: redis-data volumes: - name: redis-data hostPath: path: /var/kub-volume-mounts/spring-boot-docker-kubernetes-example-redis/_data --- apiVersion: v1 kind: Service metadata: labels: app: spring-boot-docker-kubernetes-example-redis # service name matters. It provides the way which other pods of the cluster may communicate with that pod name: spring-boot-docker-kubernetes-example-redis namespace: __NAMESPACE__ spec: type: ClusterIP ports: - name: "application-port" # port.name is just informational targetPort: 6379 # port of the running app port: 6379 # Cluster IP Port selector: app: spring-boot-docker-kubernetes-example-redis |
Pipeline explanations
Note : no explanation on the two first part
(maven build and docker build/tag/push) that is not the matter here.
Here how Gitlab ci native Kubernetes features works when a pipeline is executed.
1) The Gitlab runner relies on the native Kubernetes features only if the stage definition
define the environment property. It defines in which kubernetes « conceptual »
environment the stage is currently executing.
2) The Gitlab runner creates a namespace (if not existing) for the project.
The namespace is a concatenation of project name + project id + environment
value we set in the stage.
For example a project named spring-boot-docker-kubernetes-example with as id 32
and an environment property set to dev will make gitlab ci to
create/use the kubernetes namespace
spring-boot-docker-kubernetes-example-32-dev.
3) The Gitlab runner uses that namespace to interact and deploy any resources on kubectl.
Kubernetes basic behavior :
When a namespace is created, Kubernetes creates automatically a service account for it, named
« default ».
We are so able to apply resources on that namespace (pod, deployment, service, secret,
volume…).
Kubernetes interaction with container registry :
To pull images from a private registry that requires an authentication, Kubernetes needs some
configuration .
Either by setting a imagePullSecrets field in the pod resource or by configuring the service
account of the current namespace with that information.
We will use the second way that is more generic and practical.
So in the first lines of the deploy stage we :
– create in that namespace a secret resource that allows to access to the registry
– update the service account of that namespace to use that secret to pull images.
Common issues with kubernetes
Problem : kubectl apply -f ... doesn’t fail and Kubernetes creates
the pod but the pod is not READY and keeps as status : ImagePullBackOff along a « denied: access
forbidden » event .
kubectl describe pod foo-pod returns something like :
Normal Scheduled 56s default-scheduler Successfully assigned spring-boot-docker-kubernetes-example-32-dev/spring-boot-docker-kubernetes-example-sboot-7b964b78b5-5wjps to david-virtual-machine Normal BackOff 24s (x3 over 53s) kubelet Back-off pulling image "registry.david.com:5050/kubernetes-projects/spring-boot-docker-kubernetes-example:1.0.0" Warning Failed 24s (x3 over 53s) kubelet Error: ImagePullBackOff Normal Pulling 10s (x3 over 54s) kubelet Pulling image "registry.david.com:5050/kubernetes-projects/spring-boot-docker-kubernetes-example:1.0.0" Warning Failed 10s (x3 over 53s) kubelet Failed to pull image "registry.david.com:5050/kubernetes-projects/spring-boot-docker-kubernetes-example:1.0.0": rpc error: code = Unknown desc = Error response from daemon: Head https://registry.david.com:5050/v2/kubernetes-projects/spring-boot-docker-kubernetes-example/manifests/1.0.0: denied: access forbidden Warning Failed 10s (x3 over 53s) kubelet Error: ErrImagePull |
Solution :
Create a secret docker-registry with kubectl.
Problem : gitlab runner looks for in the official docker registry
while we don’t want to
If we don’t use a docker registry for kube, the gitlab job with k8s may contact the official
docker registry access
and fail if not reachable.
Symptom :
Look for events in the gitlab runner process/container that shows a failed image pull.
Solution :
Manually pull the image on the docker node that runs the pod and set the
imagePullPolicy attribute of the deployment/pod to never to prevent
any error related to pulling.
Problem :
the kubernetes master contacted by the gitlab runner doesn’t manage to make the pod running. It
sticks to the pending state
Symptom :
ci pipeline is blocked with a error that loops forever like (where gitlab is the k8s namespace
used by the k8s serviceaccount to deploy a pod from the gitlab runner):
Waiting for pod gitlab/runner-t3jt2q9b-project-230-concurrent-aser6h to be running, status is Pending Waiting for pod gitlab/runner-t3jt2q9b-project-230-concurrent-aser6h to be running, status is Pending .... |
We can also see in the events of the runner namespace error events referencing a timeout issue :
51s Warning FailedCreatePodSandBox pod/runner-xxxxxxxx-project-115-concurrent-yyyyy Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod "runner-t3jt2q6a-project-115-concurrent-0mfkf8": operation timeout: context deadline exceeded 50s Normal SandboxChanged pod/runner-xxxxxxxx-project-115-concurrent-yyyyy Pod sandbox changed, it will be killed and re-created. |
How to fix :
Step one: Diagnostic issues with below checks
Step two :
if something looks to prevent pod running (memory issue, node toleration issue, and so for…)
or appears blocking at the cluster level, fix that.
– Check details and logs of the pod (it has two containers : build and helper):
kubectl -n gitlab describe pod gitlab/runner-t3jt2q9b-project-230-concurrent-aser6h
kubectl -n gitlab logs gitlab/runner-t3jt2q9b-project-230-concurrent-aser6h helper
kubectl -n gitlab logs gitlab/runner-t3jt2q9b-project-230-concurrent-aser6h build
– Check events on the gitlab namespace and on the whole cluster
kubectl -n gitlab get events
kubectl get -A events
– Check the state of the kube-system components
kubectl -n kube-system logs kube-scheduler-hostnameofmaster
kubectl -n kube-system logs kube-apiserver-hostnameofmaster
kubectl -n kube-system logs kube-controller-manager-hostnameofmaster
and so for…
Problem : we encounter some logs such as below means that there
is an issue with creds/certs to access to the cluster:
error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log kube-scheduler-hostnameofmaster) |
To fix that kind of issues:
– check the certificates validity
– restart kubelet
– if not enough kill kube-XXX docker containers (scheduler, apiserver, controller-manager).
Killing docker child containers should be enough (that is those not prefixed with « k8s_POD_ »).
Gitlab runner cache
Phsyical Location
Its location depends on the way which the runner application was installed : locally or with
docker.
Local install : stored under the gitlab-runner user’s home directory: /home/gitlab-runner/cache/<user>/<project>/<cache-key>/cache.zip.
Docker install , stored on the docker host as a volume : /var/lib/docker/volumes/<volume-id>/_data/<user>/<project>/<cache-key>/cache.zip .
About docker way : A distinct cache is created by user/project/cache-key and so a distinct
volume too. Volume ids have long and generated name such as : runner-84exxmn--project-23046492-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
Caches/Volumes are created/read on the fly by the gitlab runner during the CI execution. So
these cache volumes are never mounted on the gitlab runner container.
Enable caching
No caching is enabled by default. We enable it thanks to the cache: keyword.
Cache
is shared between pipelines of a same project. Caches are indeed not shared across
projects.
Caching availability for jobs
Caching may be declared/usable at pipeline level to make it to be available at every stage or at
some specific stage levels to make it to be available at some specific stages.
It depends on the declaration location of the cache: keyword : stage(s) or root
level.
Caching scope
A Cache may be created for a specific branch or across different or even all branches.
We define it thanks to the key attribute.
Caching content
We define directories and/or files to cache with the paths: that accepts a list of
elements.
Example : caching the .m2/repository folder with a distinct
cache for each branch/tag name and usable only in that specific stage :
Maven Package: stage: maven-build tags: - maven image: maven:3.6.3-jdk-11 cache: paths: - .m2/repository key: ${CI_COMMIT_REF_NAME} |