Docker logging drivers

How to know the default Logging driver ?

docker info --format '{{.LoggingDriver}}'

The default : the json-file logging driver

Each Docker daemon has a default logging driver, which each container uses unless you configure it to use a different logging driver.
As a default, Docker uses the json-file logging driver, which caches container logs as JSON internally.
In addition to logging drivers included with Docker, you can also implement and use logging driver plugins.

use the “local” logging driver to prevent disk-exhaustion

The default logging driver : json-file performs no log-rotation.
That can cause a significant amount of disk space to be used for containers.
The json-file logging driver (without log-rotation) is currently as default for two reasons :
– backward compatibility with older versions of Docker
– cases where Docker is used as runtime for Kubernetes.

For other situations, the “local” logging driver is recommended as it performs log-rotation by default, and uses a more efficient file format.

Configure the default logging driver

in /etc/docker/daemon.json, set the value of log-driver json property to the name of the logging driver to use.
Example : set the default logging driver to the local driver with minimal options:

{
  "log-driver": "local"
}

Example : set the default logging driver to json-file with some options:
We specify the property log-opts to define log options.

{
 "log-driver": "json-file",
 "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "labels": "production_status",
    "env": "os,customer"
 }
}

Warn : any change performed in daemon.json require a restart of the docker service.

Supported logging drivers

Kubernetes logging with Docker containers

– The logging of docker containers managed by Kubernetes occurs at the node level where the container runs.
– Kubernetes is not responsible for rotating logs.
To rotate logs, we have two choices : either specifying max-size and if needed max-file in daemon.json file of docker of each node or implementing our own rotation mechanism on each node.

Common errors/problems

Problem :
kubectl logs foo-container … fails with a message explaning that the log file is missing, such as  :
failed to try resolving symlinks in path « /var/log/pods/my-apps_spring-boot-docker-kubernetes-example-sboot-78c4cf9469-hgqqd_e9064376-92ce-4ca9-8a77-8ac2e2b04c2f/spring-boot-docker-kubernetes-example-sboot/0.log »: lstat /var/log/pods/my-apps_spring-boot-docker-kubernetes-example-sboot-78c4cf9469-hgqqd_e9064376-92ce-4ca9-8a77-8ac2e2b04c2f/spring-boot-docker-kubernetes-example-sboot/0.log: no such file or directory
Causes :
– we use a Docker logging driver not compatible with Kubernetes.
Solution :
Use the json-file logging driver.

Default logs locations

Docker : stdout of each container is stored in /var/lib/docker/containers/foo_container_id.

Kubernetes : it creates symbolic links to these Docker logs.

First :
/var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/
-> sym link to
/var/lib/docker/containers.

Second :
/var/log/containers/<pod_name>_<namespace>_<container_id>.log
-> sym link to
/var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/