Prometheus with Docker or Kubernetes

Prometheus config

Example

Here is a sample configuration that defines several jobs. All don’t require credentials but workers that require it.

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
 
# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093
 
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
 
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
 
  - job_name: 'docker'
    static_configs:
      - targets: ['foo.host:9323']
      - targets: ['foo.host:9323']
      - targets: ['foo.host:9323']
 
  - job_name: 'spring-boot-master'
    metrics_path: 'actuator/prometheus'
    static_configs:
      - targets: ['spring-foo-host:8080']
 
  - job_name: 'workers'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['bar1-host:8080']
      - targets: ['bar2-host:8080']
      - targets: ['bar3-host:8080']      
    basic_auth:
      username: 'user'
      password: 'pass'

Storage configuration

Prometheus has several flags that allow configuring the local storage.
The most important ones are :

--storage.tsdb.path : Where Prometheus writes its database. Defaults to data/.

--storage.tsdb.retention.time : When to remove old data. Defaults to 15d.
Overrides storage.tsdb.retention if this flag is set to anything other than default.

--storage.tsdb.retention.size : [EXPERIMENTAL] The maximum number of bytes that storage blocks can use (note that this does not include the WAL size, which can be substantial).
Oldest data will be removed first.
Defaults to 0 or disabled.
Units supported: B, KB, MB, GB, TB, PB, EB. Ex: « 512MB »

--storage.tsdb.retention: Deprecated in favor of storage.tsdb.retention.time.

--storage.tsdb.wal-compression: Enables compression of the write-ahead log (WAL).
Depending on your data, you can expect the WAL size to be halved with little extra cpu load.
This flag was introduced in 2.11.0 and enabled by default in 2.20.0.
Note that once enabled, downgrading Prometheus to a version below 2.11.0 will require deleting the WAL.

Deploy Prometheus with Docker

We defined 1 named volume for the data and a 1 bind mount for the config :
docker run --rm --name prom -d -p 9095:9090 -v /home/david/workspace-gitlab/prometheus/prometheus-conf:/etc/prometheus -v prometheus-data:/prometheus prom/prometheus:v2.24.1

Deploy Prometheus with Kubernetes

We defined 2 hostPath volumes (for the data and for the config) because we want to use a node with a specific tag to deploy Prometheus.
It may make sense if only a machine has access to all others.
We want to enable --web.enable-lifecycle to allow hot refresh of prometehus when its config changes.
But as we override the args of the container, we also need to specify the --config.file arg.

apiVersion: v1
kind: Namespace
metadata:
  name: my-tools
---
# spring boot app
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: my-tools
spec:
  replicas: 1
  selector: #it defines how the Deployment finds which Pods to manage.
    matchLabels: 
      app: prometheus
  template: # pod template
    metadata:
      labels:
        app: prometheus
    spec:
      # prefer to be deployed only on a specific node (because volume not shared between node )
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: prometheus
                    operator: In
                    values:
                    - "true"
      containers:
        - image: prom/prometheus:v2.24.1
          name: prometheus
          args: ["--web.enable-lifecycle", "--config.file=/etc/prometheus/prometheus-conf.yml"]
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: prometheus-config
              mountPath: /etc/prometheus
            - name: prometheus-data
              mountPath: /prometheus
      volumes:
        - name: prometheus-config
          hostPath:
             path: /var/prometheus/config
        - name: prometheus-data
          hostPath:
             path: /var/prometheus/data # the folder rights may make the app fails to start. Try with chmod u=rwx,g=rwx,o=rx -R data
 
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: prometheus
  name: prometheus
  namespace: my-tools
spec:
  type: NodePort
  ports:
    - name: "prom-ui"
      targetPort: 9090 # port of the running app
      port: 9090 # Cluster IP Port
      nodePort: 31000 # External port (has to be unique in the cluster). By default Kubernetes allocates  a node port from a range (default: 30000-32767)
  selector:
    app: prometheus
---

FAQ Errors

Error at container startup :

level=error ts=2021-01-28T11:17:23.696Z caller=query_logger.go:87 component=activeQueryTracker msg=&quot;Error opening query log file&quot; file=data/queries.active err=&quot;open data/queries.active: no such file or directory&quot;

Possible cause :
The folder rights on the host for the data folder may not enough. Try with chmod u=rwx,g=rwx,o=rx -R data

Services exposed by prometheus

* Reload prometheus config on the fly :
curl -X POST -v "$(hostname):9090/-/reload"
or
send signal to the process

Ce contenu a été publié dans Non classé. Vous pouvez le mettre en favoris avec ce permalien.

Laisser un commentaire

Votre adresse de messagerie ne sera pas publiée. Les champs obligatoires sont indiqués avec *