Prometheus config
Example
Here is a sample configuration that defines several jobs. All don’t require credentials but workers that require it.
# my global config global: scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. # scrape_timeout is set to the global default (10s). # Alertmanager configuration alerting: alertmanagers: - static_configs: - targets: # - alertmanager:9093 # Load rules once and periodically evaluate them according to the global 'evaluation_interval'. rule_files: # - "first_rules.yml" # - "second_rules.yml" # A scrape configuration containing exactly one endpoint to scrape: # Here it's Prometheus itself. scrape_configs: # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'docker' static_configs: - targets: ['foo.host:9323'] - targets: ['foo.host:9323'] - targets: ['foo.host:9323'] - job_name: 'spring-boot-master' metrics_path: 'actuator/prometheus' static_configs: - targets: ['spring-foo-host:8080'] - job_name: 'workers' metrics_path: '/metrics' static_configs: - targets: ['bar1-host:8080'] - targets: ['bar2-host:8080'] - targets: ['bar3-host:8080'] basic_auth: username: 'user' password: 'pass' |
Storage configuration
Prometheus has several flags that allow configuring the local storage.
The most important ones are :
--storage.tsdb.path
: Where Prometheus writes its database. Defaults to data/
.
--storage.tsdb.retention.time
: When to remove old data. Defaults to 15d.
Overrides storage.tsdb.retention
if this flag is set to anything other than default.
--storage.tsdb.retention.size
: [EXPERIMENTAL] The maximum number of bytes that storage blocks can use (note that this does not include the WAL size, which can be substantial).
Oldest data will be removed first.
Defaults to 0 or disabled.
Units supported: B, KB, MB, GB, TB, PB, EB. Ex: « 512MB »
--storage.tsdb.retention
: Deprecated in favor of storage.tsdb.retention.time
.
--storage.tsdb.wal-compression
: Enables compression of the write-ahead log (WAL).
Depending on your data, you can expect the WAL size to be halved with little extra cpu load.
This flag was introduced in 2.11.0 and enabled by default in 2.20.0.
Note that once enabled, downgrading Prometheus to a version below 2.11.0 will require deleting the WAL.
Deploy Prometheus with Docker
We defined 1 named volume for the data and a 1 bind mount for the config :
docker run --rm --name prom -d -p 9095:9090 -v /home/david/workspace-gitlab/prometheus/prometheus-conf:/etc/prometheus -v prometheus-data:/prometheus prom/prometheus:v2.24.1
Deploy Prometheus with Kubernetes
We defined 2 hostPath volumes (for the data and for the config) because we want to use a node with a specific tag to deploy Prometheus.
It may make sense if only a machine has access to all others.
We want to enable --web.enable-lifecycle
to allow hot refresh of prometehus when its config changes.
But as we override the args of the container, we also need to specify the --config.file
arg.
apiVersion: v1 kind: Namespace metadata: name: my-tools --- # spring boot app apiVersion: apps/v1 kind: Deployment metadata: name: prometheus namespace: my-tools spec: replicas: 1 selector: #it defines how the Deployment finds which Pods to manage. matchLabels: app: prometheus template: # pod template metadata: labels: app: prometheus spec: # prefer to be deployed only on a specific node (because volume not shared between node ) affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: prometheus operator: In values: - "true" containers: - image: prom/prometheus:v2.24.1 name: prometheus args: ["--web.enable-lifecycle", "--config.file=/etc/prometheus/prometheus-conf.yml"] imagePullPolicy: IfNotPresent volumeMounts: - name: prometheus-config mountPath: /etc/prometheus - name: prometheus-data mountPath: /prometheus volumes: - name: prometheus-config hostPath: path: /var/prometheus/config - name: prometheus-data hostPath: path: /var/prometheus/data # the folder rights may make the app fails to start. Try with chmod u=rwx,g=rwx,o=rx -R data --- apiVersion: v1 kind: Service metadata: labels: app: prometheus name: prometheus namespace: my-tools spec: type: NodePort ports: - name: "prom-ui" targetPort: 9090 # port of the running app port: 9090 # Cluster IP Port nodePort: 31000 # External port (has to be unique in the cluster). By default Kubernetes allocates a node port from a range (default: 30000-32767) selector: app: prometheus --- |
FAQ Errors
Error at container startup :
level=error ts=2021-01-28T11:17:23.696Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=data/queries.active err="open data/queries.active: no such file or directory" |
Possible cause :
The folder rights on the host for the data folder may not enough. Try with chmod u=rwx,g=rwx,o=rx -R data
Services exposed by prometheus
* Reload prometheus config on the fly :
curl -X POST -v "$(hostname):9090/-/reload"
or
send signal to the process