Environment variables

Compose CLI environment variables

These configure the docker-compose command line execution.


COMPOSE_PROJECT_NAME
Sets the project name. This value is prepended along with the service name to the container on start up.
ex : COMPOSE_PROJECT_NAME=fooapp
If the template declares two services db and web, containers will be fooapp_db_1 and fooapp_web_1.

COMPOSE_FILE
Specify the path to the Compose file instead of docker-compose.yml in the current directory.
ex: COMPOSE_FILE: foo-docker/foo-docker-compose.yml

DOCKER_HOST
Sets the URL of the docker daemon instead of default to unix:///var/run/docker.sock
ex: DOCKER_HOST: foodomain:2375

Environment variables in compose template

Substitute environment variables in Compose files :

web:
  image: "webapp:${TAG}"

These env variables may be either present on the current shell or may be passed dynamically with the flag --env-file such as :
docker-compose --env-file ./config/.env.dev ...

Set environment variables in a container :

web:
  environment:
    - DEBUG=1

Pass environment variables to a container :

web:
  environment:
    - DEBUG

Here the variable value comes from the variable in the shell in which Compose is run.

The “.env” file :
It sets default values for any environment variables referenced in the Compose file in an environment file named .env.
Example :

$ cat .env
TAG=v1.5
$ cat docker-compose.yml
version: '3'
services:
  web:
    image: "webapp:${TAG}"

Flags before subcommand

Docker-like flags

-H FOO_HOST : to execute the command on the specified docker host.

Prefix of containers/services

By default, all docker object created by docker-compose (services, volume, network…) are prefixed with the current directory name such as currentDir_.
To override that prefix :
docker-compose -p otherPrefix
When we use that process to run a docker-compose project, we also need to specify that same suffix for other commands related to that project such as : docker-compose -p otherPrefix ps to show current services.

Basic commands

docker-compose build

Usage : build [options] [--build-arg key=val...] [SERVICE...]

* Build or rebuild services declared in the docker-compose file.
By default, all declared services are built. But we could also specify exactly those to build.

Helpful flags :

    --no-cache              Do not use cache when building the image.
    --no-rm                 Do not remove intermediate containers after a successful build.
    --parallel              Build images in parallel.
    --progress string       Set type of progress output (auto, plain, tty).
                            EXPERIMENTAL flag for native builder.

docker-compose up

Usage : up [options] [--scale SERVICE=NUM...] [SERVICE...]

* Create and start containers for declared services declared in the docker-compose file.
It does that for not yet existing containers and that re-create and re-start them in case of existing containers which the image or the docker-compose configuration has changed.
Be aware : by default, that doesn’t build the images.
By default, all declared services are handled. But we could also specify exactly those to handle.

Helpful flags :

-d, --detach               Detached mode (deamon)
--build                    Build images before starting containers
--no-deps                  Don't start linked services.
--force-recreate           Recreate containers even if their configuration
                           and image haven't changed
--abort-on-container-exit  Stops all containers if any container was
                           stopped. Incompatible with -d.
--exit-code-from SERVICE   Return the exit code of the selected service
                           container. Implies --abort-on-container-exit.

Examples :
* Build, create if needed and start in detached mode all services declared in the docker-compose file :
docker-compose up -d --build

* Build, create if needed and start a specific service declared in the docker-compose file :
docker-compose up --build my-service

* Build, force the recreation and start a specific service declared in the docker-compose file :
docker-compose up --build --force-recreate my-service

Combine docker-compose build and up to get a full build and a restart of  containers

Sometimes, we changed things in multiple places : the applications, the DockerImages, the configuration/docker-compose templates.
In that case, a full build (no cache) and a stop/restart of all containers makes sense :
docker-compose build --no-cache && docker-compose up -d --force-recreate
Note that here the subcommand up --build is not enough because the build still uses the image layers cache. Only the subcommand build --no-cache will do that.
And my really favorite combination command is the same and by following logs :
docker-compose build --no-cache && docker-compose up -d --force-recreate && docker-compose logs -f

docker-compose pull : useful for snapshot images

Latest and development docker image tags are going to be overwritten.
To ensure docker compose always uses the last version of our images, we can execute first docker-compose pull.
Finally quite similar to what we do with docker pull.

docker-compose down

Usage: down [options]
Stops containers and clean them, that is removes containers, networks, volumes, and images created by up.
By default, the only things removed are:
– Containers for services defined in the Compose file
– Networks defined in the `networks` section of the Compose file
– The default network, if one is used

Helpful flags :
-v or –volumes : remove names and anonymous volumes attached to containers.

docker-compose run

Usage: run [options] [-v VOLUME...] [-p PORT...] [-e KEY=VAL...] [-l KEY=VALUE...] SERVICE [COMMAND] [ARGS...]
Run a one-off command on a service.
By default, linked services will be started, unless they are already running.
That command can be seen as a swiss-knife. That allow to execute a command on a service but not only. That is also a way to start and run a specific service with further options that the up command doesn’t provide.
In that use, we can see that as the symmetric command to docker run in the context of docker-compose.
Note that by default, ports are not mapped to the host even if the docker-compose template declare that.

Helpful flags (almost all of them) :

    -d, --detach          Detached mode: Run container in the background, print
                          new container name.
    --name NAME           Assign a name to the container
    --entrypoint CMD      Override the entrypoint of the image.
    -e KEY=VAL            Set an environment variable (can be used multiple times)
    -u, --user=""         Run as specified username or uid
    --no-deps             Don't start linked services.
    --rm                  Remove container after run. Ignored in detached mode.
    -p, --publish=[]      Publish a container's port(s) to the host
    --service-ports       Run command with the service's ports enabled and mapped
                          to the host.
    -v, --volume=[]       Bind mount a volume (default [])
    -T                    Disable pseudo-tty allocation. By default `docker-compose run`
                          allocates a TTY.
    -w, --workdir=""      Working directory inside the container

Examples :
* Run a service by enabling the port mapping, disabling the dependencies services start and by changing the entrypoint of the image by bash :
docker-compose run --no-deps --service-ports --entrypoint bash fooservice

Declare a named volume

For example for a volume with as directory /app/data on the image/container and mounted/named app-data on the host :

version: '2.3'
services:
  my-app:
    ...
    volumes:
      - app-data:/app/data
 
volumes:
  app-data:

Use a user-bridge and a none network

You can rely on that.

Handle multiple environments

My general idea

– don’t rely too much on the .env file because that is not specific to a service or to an environment because that way binds all properties declared in into every containers and you cannot have an .env file by environment.
– the .env file that we can add in the directory of the docker-compose template and the service: env_file yaml property that we define in the docker-compose.yml are very distinct things.
The .env file allows to provide a properties file (with key=value) that are usable both in the build and the execution time for every containers while the service: env_file yaml property allows to provide a properties file (with key=value) that are usable only at the execution time and for a specific (or more) container
– Use the service: env_file yaml property to define runtime properties needed for a container
– Use the service: build: args yaml property to define buildtime properties needed for a container
– Define a base docker-compose.yml template and define a specific docker-compose.yml for each environment with only the things that differ from the base template.

Define a docker-compose.yml template base and define a custom template by environment

Here a simple use case : we have 2 backend services that communicate and 2 environments to containerize : integration, production.

1) We define the base docker-compose.yml template that is common for any environment (while we may still override that).

version: '3.5'
 
services: 
  quizz-spring-boot:
    build:
      context: ../quizz/quizz-backend
      dockerfile: docker/Dockerfile-quizz-spring-boot
    volumes:
      - quizz-data:/usr/quizz/db
    networks:
      - network-quizz-app
 
  hall-of-fame-spring-boot:
    build:
      context: ../hall-of-fame/hall-of-fame-backend
      dockerfile: docker/Dockerfile-hall-of-fame-spring-boot
    volumes:
      - quizz-data:/usr/quizz/db
    networks:
      - network-quizz-app
 
volumes:
  quizz-data:
 
networks:
  network-quizz-app:
    driver: bridge

2) We define in the same directory a docker-compose.override.yml template for the integration environment.

Docker override docker-compose.override.yml with that file by default if it is found.
We could add new properties or override these defined in the base template. Here we only add new properties, that is rather a good practice because in a general way, we should avoid to move in the base template things that are overridden for about each environment template : that’s makes things unclear and error-prone.

In the example, we can see some interesting things :
– the ports are specified (makes sense : it depends on the actual environment)
– in quizz-spring-boot, we define an argument for the build of the quizz-spring-boot service : APPLICATION_PROPERTIES_FILENAME=application-integration.properties.
That is a simple way to define a set of properties usable in the build of a specific container/service and for a specific environment, here the quizz-spring-boot service and the integration environment.
– in hall-of-frame-spring-boot, we provide environment properties coming from the integration.env file : ../hall-of-fame/hall-of-fame-backend/docker/envs/integration.env usable at runtime of that specific container and for a specific environment, here the quizz-spring-boot service and the integration environment.

version: '3.5'
services:
  quizz-spring-boot:
    build:
      args:
        - APPLICATION_PROPERTIES_FILENAME=application-integration.properties
    ports:
      - "8282:8080"
 
  hall-of-fame-spring-boot:
    ports:
      - "8283:8080"
    env_file:
      - ../hall-of-fame/hall-of-fame-backend/docker/envs/integration.env

The application-integration.properties is a classical properties file that we use with spring boot. We could pass it in the application such as.
While the integration.env is also a properties file but with a more general way to name properties since here we will usethe environment variables directly in the Dockerfile.
Here is a snippet of these.
integration.env :

LOGGING_LEVEL_ROOT=INFO
LOGGING_LEVEL_SECURITY=DEBUG

application-integration.properties :

....
logging.level.root=INFO
spring.datasource.url = jdbc:postgresql:database
....

Here how we use at build time in the Dockerfile the arg passed :

RUN echo "application-properties used : $APPLICATION_PROPERTIES_FILENAME"
ARG APPLICATION_PROPERTIES_FILENAME
RUN test -n "$APPLICATION_PROPERTIES_FILENAME"
COPY config/$APPLICATION_PROPERTIES_FILENAME ./

The code declares the arg to be able to get its value, then it checks that the variable is valued (string length > 0) and then we copy the file on the image currently built.

And here how we use at runtime in the Dockerfile the environment properties passed :

CMD java -cp /app:/app/lib/* mypackage.myApplication \
             --logging.level.root=$LOGGING_LEVEL_ROOT \
             --logging.level.org.springframework.security=$LOGGING_LEVEL_SECURITY \

3) We define another docker-compose template for the production environment :

version: '3.5'
services:
  quizz-spring-boot:
    build:
      args:
        - APPLICATION_PROPERTIES_FILENAME=application-production.properties
    ports:
      - "8282:8080"
 
  hall-of-fame-spring-boot:
    ports:
      - "8283:8080"
    env_file:
      - ../hall-of-fame/hall-of-fame-backend/docker/envs/production.env

We can see that the template is very close of the integration template but with env properties and args specific to the production.
Of course, even if not done here, we could have overridden properties defined in the base template as Dockerfile or anything…

4) Commands to execute the docker-compose according to the environment.

* Build and run for the production env (need to declare base template and then the production template) :
docker-compose -f docker-compose.yml -f docker-compose.production.yml up --build -d

* Build and run for the production env (need to declare nothing if we are in the right directory) :
docker-compose up --build -d

Dependency between services

Express dependency between services

We do it by adding a depends_on attribute in the service declaration that depends on another service.
The idea behind is that docker-compose will create services in the defined order, and ensure that up and stop sub-commands honor that relationship.
Example :

version: "2.4"
services:
  web:
    build: .
    depends_on:
      - db
      - redis
  redis:
    image: redis
  db:
    image: postgres

Go further : express a dependency condition for starting a services

The thing is that in the above example, the services are created in the defined order but the web service doesn’t wait for db and redis to be ready, they only wait for these to be started.
To set that constraint, we need two things :
– specify the condition: service_healthy in the service declaration that depends on another service
By default the value is condition: service_started. That’s the behavior that we had previously.
– specify a healthcheck attribute under the service that is defined as a dependency.
service_healthy condition relies on that healthcheck.

Previous example refined :
web service starts when redis is started (as previously) and db is healthy.

version: "2.4"
services:
  web:
    build: .
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_started
  redis:
    image: redis
  db:
    image: postgres
    healthcheck:
      test: "exit 0"