GitLab components
Summarize
– ngnix : web server in front of the UI application (serve static resources and proxy to
unicorn). We could have multiple instances of it : for unicor and for the container
registry
– unicorn : the UI application (ruby). Has its own server
– redis : temporary storage (ex: incoming job, job information and some metadata)
– PostgreSQL : persistent database (ex : users, permissions, issues and some other metadata).
Has its own server
——————————————————————
Components (from the official doc)
A typical install of GitLab will be on GNU/Linux. It uses Nginx or Apache as a web front end to
proxypass the Unicorn web server. By default, communication between Unicorn and the front end is
via a Unix domain socket but forwarding requests via TCP is also supported. The web front end
accesses /home/git/gitlab/public bypassing the Unicorn server to serve static pages, uploads
(e.g. avatar images or attachments), and precompiled assets. GitLab serves web pages and a
GitLab API using the Unicorn web server. It uses Sidekiq as a job queue which, in turn, uses
redis as a non-persistent database backend for job information, meta data, and incoming jobs.
We also support deploying GitLab on Kubernetes using our GitLab Helm chart.
The GitLab web app uses PostgreSQL for persistent database information (e.g. users, permissions,
issues, other meta data). GitLab stores the bare Git repositories it serves in
/home/git/repositories by default. It also keeps default branch and hook information with the
bare repository.
When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve
authorization and access as well as serving Git objects.
The add-on component GitLab Shell serves repositories over SSH. It manages the SSH keys within
/home/git/.ssh/authorized_keys which should not be manually edited. GitLab Shell accesses the
bare repositories through Gitaly to serve Git objects and communicates with redis to submit jobs
to Sidekiq for GitLab to process. GitLab Shell queries the GitLab API to determine authorization
and access.
Gitaly executes Git operations from GitLab Shell and the GitLab web app, and provides an API to
the GitLab web app to get attributes from Git (e.g. title, branches, tags, other meta data), and
to get blobs (e.g. diffs, commits, files).
You may also be interested in the production architecture of GitLab.com
——————————————————————
Update Gitlab server
1) Follow the upgrade path : https://docs.gitlab.com/ee/update/index.html#upgrade-paths
2) Update your package manager : apt-get update
3) Look for the wished version : sudo apt list -a gitlab-ee | less
Trick with Hyper-V, XRDP and Ubuntu image provided by windows
The GitLab configure command relies on the init cycle of our Linux OS. More particularly, the
gitlab-runsvdir.service started by Chef. The unit starts after the
multi-user.target of the system be enabled and running. But it will never happen.
So the runsvdir.service will wait forever.
The files that contains that setup are :
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/files/default/gitlab-runsvdir.service:3:After=basic.target>
/opt/gitlab/embedded/cookbooks/package/files/default/gitlab-runsvdir.service:3:After=basic.target
They contain something like that :
[Unit] Description=GitLab Runit supervision process After=basic.target .. |
We can retrieve these files in this way :
sudo grep /opt/gitlab/ -rn -e "After=multi-user.target" 2>/dev/null
The workaround is replacing multi-user.target by basic.target in these
files.
And of course to avoid surprises, we should set the gitlab package as not updatable any longer
Gitlab token
Personal access tokens
How to create them ?
Go on the top right corner (profile->settings), then menu « Access Tokens » on the left
menu.
What ?
Allow to authenticate with:
– GitLab API.
– GitLab repositories (it means projects).
– GitLab registry (container registry).
These tokens inherit permissions from the user who created them.
Oauth 2 token
Clasic oauth 2 process
Deploy token
How to create them ?
Go to the project (or group) you want to create deploy tokens for.
Go to Settings > Repository.
Expand the Deploy tokens section.
Save the deploy token somewhere safe because after you leave or refresh the page, you can’t
access it again.
What ?
– download a project (git clone)
– push and pull packages and container registry images of a project without having a user and a
password.
These are managed by maintainers only.
These cannot be used with the GitLab API.
Deploy tokens are tied to the project and stay
enabled even when the user who created the token is removed from the project.
Use deploy token in projects of a group
First step :
we create two variables on the settings > CICD > variable
menu.
We could name them CI_DEPLOY_USER and CI_DEPLOY_PASSWORD.
The name of these variables don’t matter because we still need to specify them in the pipeline
scripts.
CI_DEPLOY_USER value must be equal to the deploy token’s username and
CI_DEPLOY_PASSWORD must be equal to the deploy token’s password.
Note :
The deploy token’s name matters only to benefit from automatic injection of
CI_DEPLOY_USER and CI_DEPLOY_PASSWORD in the pipeline.
For that it works, the deploy token has to be limited to a project and not a entire group
and the deploy token’s name has to be
gitlab-deploy-token.
Using the token in a gitlab pipeline
We could login to the docker registry in that way :
docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY
CI_REGISTRY doesn’t need to be declared as a CICD variable. Gitlab injects it.
We could clone a git project in that way :
git clone https://$CI_DEPLOY_USER>:$CI_DEPLOY_PASSWORD@gitlab.example.com/awesome_project.git
Gitlab administration
Locations of important stuffs
Source configuration file :
/etc/gitlab/gitlab.rb
That template generates the configuration of most gitlab components (see below and reconfigure
command)
Generated configuration files :
/var/opt/gitlab/*
Example : /var/opt/gitlab/nginx/, /var/opt/gitlab/postgresql/, /var/opt/gitlab/registry/…
GitLab logs :
/var/log/gitlab/***
Common admin commands
Display all GitLab environment information :
sudo gitlab-rake gitlab:env:info
Tail logs of all GitLab enabled services :
sudo gitlab-ctl tail
Tail logs of a specific GitLab service :
sudo gitlab-ctl tail foo-service
Restart all GitLab services :
sudo gitlab-ctl restart
Restart a specific GitLab component (for example nginx) :
sudo gitlab-ctl restart nginx
Get the list of configured GitLab services :
gitlab-ctl service-list
Service with a * character are enabled.
Get the status of GitLab services (running or not) :
gitlab-ctl status
Those that run have a PID info and these that don’t manage to have a label aside such as:
« wanted to start »
Reconfigure GitLab (required when you change the gitlab.rb configuration file):
gitlab-ctl reconfigure
It will generates and overwrite gitlab services configurations and restart the services.
If « Let’s Encrypt fails » message for a chosen URL, the website https://letsdebug.net can test
that and help to diagnostic.
Show the configuration that would be generated by executing reconfigure (helpful to check that
our gitlab.rb changes are what we expect o) :
gitlab-ctl show-config
Customize the gitlab application
Change the GitLab URL :
Edit in gitlab.rb :
external_url "http://gitlab.example.com:myPort"
And think of adding the mapping 127.0.0.1:gitlab.example.com in
etc/hosts
Rails console
Reset a git user password :
1) Open a Rails console :
gitlab-rails console
2) Find the user by username :
user = User.find_by_username('fooUser')
by id :
user = User.find('fooUserId')
by email :
user = User.find_by(email: 'user@example.com')
Important: to change the password of the root user we can use the id 1.
3) Reset the password :
user.password = 'secret_pass'
user.password_confirmation = 'secret_pass'
4) (informational). Communicate to the user that the password was changed by the admin :
user.send_only_admin_changed_your_password_notification!
5) By default, a confirmation by mail is required by the user.
To skip that :
user.skip_reconfirmation!
6) Save the changes :
user.save!
Common issues
Problem :
Some services (gitlab server url, gitlab registry…) returns a 502 HTTP error.
A typical symptom is the gitlab ui that shows the error :
502
Whoops, GitLab is taking too much time to respond. |
Solution :
– Case 1 : you just restarted some gitlab services, that is expected. Gitlab takes a couple
minutes to be fully working.
– Case 2 : some services are down (intentionally or not). Check the services status with gitlab-ctl status and checks logs of
services that fail to run.
– Case 3 : all services are up. In that case, we need to search in logs of gitlab services
.
A naive but very often simple way is to search common error message in the last minutes.
For example : we query the gitlab server url, we get the 502 error response.
Then from /var/log/gitlab/ : we search log files modified in the last 5 minutes
with the insensitive case « error » word :
find -mmin -5 -exec grep -ri "error" {} \;
If not enough we could hit broader by searching other words (-P for Perl regex) :
find -mmin -5 -exec grep -r -Pi "error|permission|denied|critical|fatal" {} \;