Prep: Things to do before we get started.

1. Open these slides: https://tampa.bretfisher.com/

2. Get a server: I provisioned one for each. Ask me for the IPs.

3. Access your server over SSH ssh docker@w.x.y.z or WebSSH (http://w.x.y.z:8080)

   • username: docker | password: training

4. Let me know if you can't get in, we have multiple backup options!

Note

• This is hands on. You'll want to do most of these commands with me.

• These slides are take-home.

logistics-bret.md

Introductions

• Hello! I'm Bret Fisher (@bretfisher), a fan of 🐳 🏖 🥃 👾 ✈️ 🐶

• I'm a DevOps Consultant+Trainer (300k students), OSS maintainer, and Docker Captain.

• 👉 Watch: My weekly cloud native DevOps live show with guests. Join us on Thursdays!

• 👉 Listen: That show turns into a podcast called "DevOps and Docker Talk."

• 👉 Read: You can get my weekly updates in my Newsletter.

• 👉 Chat: Join 12k DevOps pros in my Discord server devops.fan logistics-bret.md

Accessing these slides now

• We recommend that you open these slides in your browser:

  https://tampa.bretfisher.com/

• This is a public URL, you're welcome to share it with others!

• Use arrows to move to next/previous slide

  (up, down, left, right, page up, page down)

• Type a slide number + ENTER to go to that slide

• The slide number is also visible in the URL bar

  (e.g. .../#123 for slide 123)

shared/about-slides.md

These slides are open source

• The sources of these slides are available in a public GitHub repository:

  https://github.com/bretfisher/container.training

• These slides are written in Markdown

• You are welcome to share, re-use, re-mix these slides

• Typos? Mistakes? Questions? Feel free to hover over the bottom of the slide ...

👇 Try it! The source file will be shown and you can view it on GitHub and fork and edit it.

shared/about-slides.md

Accessing these slides later

• Slides will remain online so you can review them later if needed

  (let's say we'll keep them online at least 1 year, how about that?)

• You can download the slides using that URL:

  https://tampa.bretfisher.com/slides.zip

  (then open the file docker.yml.html)

• You can also generate a PDF of the slides

  (by printing them to a file; but be patient with your browser!)

shared/about-slides.md

These slides are constantly updated

• https://container.training

• Upstream repo https://github.com/jpetazzo/container.training

shared/about-slides.md

Part 1

• Our training environment

• Our first containers

• Background containers

• Understanding Docker images

• Building Docker images with a Dockerfile

• CMD and ENTRYPOINT

• Copying files during the build

• Exercise — writing Dockerfiles

(auto-generated TOC)

Part 2

• Container networking basics

• The Container Network Model

• Service discovery with containers

• Local development workflow with Docker

• Compose for development stacks

• Exercise — writing a Compose file

(auto-generated TOC)

Part 3

• (Extra Docker content)
• Tips for efficient Dockerfiles
• Dockerfile examples
• Reducing image size
• Multi-stage builds
• Exercise — writing better Dockerfiles
• Getting inside a container
• Restarting and attaching to containers
• Naming and inspecting containers
• Labels
• Advanced Dockerfile Syntax
• Container network drivers

(auto-generated TOC)

shared/toc.md

What is Docker, the idea?

• Docker Inc makes many tools to build, deploy, and run containers.

• They invented the modern way to run "containers". (previously jails, chroot, zones, etc.)

What is Docker Inc., the company?

• Docker Inc was quicly formed after Docker the tool/project was created in 2013.

• Previously, Docker Inc. focused on Dev and Ops tooling (2013-2019).

What is docker the tool?

• "Installing Docker" really means "Installing the Docker Engine and CLI".

• The Docker Engine is a daemon (a service running in the background).

Didn't Kubernetes replace Docker?

• This is a common misconception.

Did you know Docker has its own Orchestration?

• Docker Swarm "mode" is still a thing

• And might be having a renaissance

• (I have a course on that too!) Todays_Agenda.md

Our training environment

• If you are attending a tutorial or workshop:

  • a VM has been provisioned for each student

• If you are doing or re-doing this course on your own, you can:

  • install Docker locally (as explained in the chapter "Installing Docker")

  • install Docker on e.g. a cloud VM

  • use https://www.play-with-docker.com/ to instantly get a training environment

containers/Training_Environment.md

Checking your Virtual Machine

Once logged in, make sure that you can run a basic Docker command:

$ docker version
Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:02:46 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:00:51 2022
  OS/Arch:          linux/amd64
  Experimental:     false
...

If this doesn't work, raise your hand so that an instructor can assist you!

Objectives

At the end of this lesson, you will have:

• Seen Docker in action.

• Started your first containers.

containers/First_Containers.md

Hello World

In your Docker environment, just run the following command:

$ docker run busybox echo hello world
hello world

(If your Docker install is brand new, you will also see a few extra lines, corresponding to the download of the busybox image.)

containers/First_Containers.md

That was our first container!

• We used one of the smallest, simplest images available: busybox.

• busybox is typically used in embedded systems (phones, routers...)

• We ran a single process and echo'ed hello world.

containers/First_Containers.md

A more useful container

Let's run a more exciting container:

$ docker run -it ubuntu
root@04c0bb0a6c07:/#

• This is a brand new container.

• It runs a bare-bones, no-frills ubuntu system.

• -it is shorthand for -i -t.

  • -i tells Docker to connect us to the container's stdin.

  • -t tells Docker that we want a pseudo-terminal.

containers/First_Containers.md

Do something in our container

Try to run figlet in our container.

root@04c0bb0a6c07:/# figlet hello
bash: figlet: command not found

Alright, we need to install it.

containers/First_Containers.md

Install a package in our container

We want figlet, so let's install it:

root@04c0bb0a6c07:/# apt-get update
...
Fetched 1514 kB in 14s (103 kB/s)
Reading package lists... Done
root@04c0bb0a6c07:/# apt-get install figlet
Reading package lists... Done
...

One minute later, figlet is installed!

containers/First_Containers.md

Try to run our freshly installed program

The figlet program takes a message as parameter.

root@04c0bb0a6c07:/# figlet hello
 _          _ _       
| |__   ___| | | ___  
| '_ \ / _ \ | |/ _ \ 
| | | |  __/ | | (_) |
|_| |_|\___|_|_|\___/

Beautiful! 😍

containers/First_Containers.md

Host and containers are independent things

• We ran an ubuntu container on an Linux/Windows/macOS host.

• They have different, independent packages.

• Installing something on the host doesn't expose it to the container.

• And vice-versa.

• Even if both the host and the container have the same Linux distro!

• We can run any container on any host.

  (One exception: Windows containers can only run on Windows hosts; at least for now.)

containers/First_Containers.md

Where's our container?

• Our container is now in a stopped state.

• It still exists on disk, but all compute resources have been freed up.

• We will see later how to get back to that container.

containers/First_Containers.md

Starting another container

What if we start a new container, and try to run figlet again?

$ docker run -it ubuntu
root@b13c164401fb:/# figlet
bash: figlet: command not found

• We started a brand new container.

• The basic Ubuntu image was used, and figlet is not here.

containers/First_Containers.md

Where's my container?

• Can we reuse that container that we took time to customize?

  We can, but that's not the default workflow with Docker.

• What's the default workflow, then?

  Always start with a fresh container.
  If we need something installed in our container, build a custom image.

• That seems complicated!

  We'll see that it's actually pretty easy!

• And what's the point?

  This puts a strong emphasis on automation and repeatability. Let's see why ...

containers/First_Containers.md

Pets vs. Cattle

• In the "pets vs. cattle" metaphor, there are two kinds of servers.

• Pets:

  • have distinctive names and unique configurations

  • when they have an outage, we do everything we can to fix them

• Cattle:

  • have generic names (e.g. with numbers) and generic configuration

  • configuration is enforced by configuration management, golden images ...

  • when they have an outage, we can replace them immediately with a new server

• What's the connection with Docker and containers?

containers/First_Containers.md

Local development environments

• When we use local VMs (with e.g. VirtualBox or VMware), our workflow looks like this:

  • create VM from base template (Ubuntu, CentOS...)

  • install packages, set up environment

  • work on project

  • when done, shut down VM

  • next time we need to work on project, restart VM as we left it

  • if we need to tweak the environment, we do it live

• Over time, the VM configuration evolves, diverges.

• We don't have a clean, reliable, deterministic way to provision that environment.

containers/First_Containers.md

Local development with Docker

• With Docker, the workflow looks like this:

  • create container image with our dev environment

  • run container with that image

  • work on project

  • when done, shut down container

  • next time we need to work on project, start a new container

  • if we need to tweak the environment, we create a new image

• We have a clear definition of our environment, and can share it reliably with others.

• Let's see in the next chapters how to bake a custom image with figlet!

Objectives

Our first containers were interactive.

We will now see how to:

• Run a non-interactive container.
• Run a container in the background.
• List running containers.
• Check the logs of a container.
• Stop a container.
• List stopped containers.

containers/Background_Containers.md

A non-interactive container

We will run a small custom container.

This container just displays the time every second.

$ docker run jpetazzo/clock
Fri Feb 20 00:28:53 UTC 2015
Fri Feb 20 00:28:54 UTC 2015
Fri Feb 20 00:28:55 UTC 2015
...

• This container will run forever.
• To stop it, press ^C.
• Docker has automatically downloaded the image jpetazzo/clock.
• This image is a user image, created by jpetazzo.
• We will hear more about user images (and other types of images) later.

containers/Background_Containers.md

When ^C doesn't work...

Sometimes, ^C won't be enough.

Why? And how can we stop the container in that case?

containers/Background_Containers.md

What happens when we hit ^C

SIGINT gets sent to the container, which means:

• SIGINT gets sent to PID 1 (default case)

• SIGINT gets sent to foreground processes when running with -ti

But there is a special case for PID 1: it ignores all signals!

• except SIGKILL and SIGSTOP

• except signals handled explicitly

TL,DR: there are many circumstances when ^C won't stop the container.

containers/Background_Containers.md

How to stop these containers, then?

• Start another terminal and forget about them

  (for now!)

• We'll shortly learn about docker kill

containers/Background_Containers.md

Run a container in the background

Containers can be started in the background, with the -d flag (daemon mode):

$ docker run -d jpetazzo/clock
47d677dcfba4277c6cc68fcaa51f932b544cab1a187c853b7d0caf4e8debe5ad

• We don't see the output of the container.
• But don't worry: Docker collects that output and logs it!
• Docker gives us the ID of the container.

containers/Background_Containers.md

List running containers

How can we check that our container is still running?

With docker ps, just like the UNIX ps command, lists running processes.

$ docker ps
CONTAINER ID  IMAGE           ...  CREATED        STATUS        ...
47d677dcfba4  jpetazzo/clock  ...  2 minutes ago  Up 2 minutes  ...

Docker tells us:

• The (truncated) ID of our container.
• The image used to start the container.
• That our container has been running (Up) for a couple of minutes.
• Other information (COMMAND, PORTS, NAMES) that we will explain later.

containers/Background_Containers.md

Starting more containers

Let's start two more containers.

$ docker run -d jpetazzo/clock
57ad9bdfc06bb4407c47220cf59ce21585dce9a1298d7a67488359aeaea8ae2a

$ docker run -d jpetazzo/clock
068cc994ffd0190bbe025ba74e4c0771a5d8f14734af772ddee8dc1aaf20567d

Check that docker ps correctly reports all 3 containers.

containers/Background_Containers.md

Viewing only the last container started

When many containers are already running, it can be useful to see only the last container that was started.

This can be achieved with the -l ("Last") flag:

$ docker ps -l
CONTAINER ID  IMAGE           ...  CREATED        STATUS        ...
068cc994ffd0  jpetazzo/clock  ...  2 minutes ago  Up 2 minutes  ...

containers/Background_Containers.md

View only the IDs of the containers

Many Docker commands will work on container IDs: docker stop, docker rm...

If we want to list only the IDs of our containers (without the other columns or the header line), we can use the -q ("Quiet", "Quick") flag:

$ docker ps -q
068cc994ffd0
57ad9bdfc06b
47d677dcfba4

containers/Background_Containers.md

Combining flags

We can combine -l and -q to see only the ID of the last container started:

$ docker ps -lq
068cc994ffd0

At a first glance, it looks like this would be particularly useful in scripts.

However, if we want to start a container and get its ID in a reliable way, it is better to use docker run -d, which we will cover in a bit.

(Using docker ps -lq is prone to race conditions: what happens if someone else, or another program or script, starts another container just before we run docker ps -lq?)

containers/Background_Containers.md

View the logs of a container

We told you that Docker was logging the container output.

Let's see that now.

$ docker logs 068
Fri Feb 20 00:39:52 UTC 2015
Fri Feb 20 00:39:53 UTC 2015
...

• We specified a prefix of the full container ID.
• You can, of course, specify the full ID.
• The logs command will output the entire logs of the container.
  (Sometimes, that will be too much. Let's see how to address that.)

containers/Background_Containers.md

View only the tail of the logs

To avoid being spammed with eleventy pages of output, we can use the --tail option:

$ docker logs --tail 3 068
Fri Feb 20 00:55:35 UTC 2015
Fri Feb 20 00:55:36 UTC 2015
Fri Feb 20 00:55:37 UTC 2015

• The parameter is the number of lines that we want to see.

containers/Background_Containers.md

Follow the logs in real time

Just like with the standard UNIX command tail -f, we can follow the logs of our container:

$ docker logs --tail 1 --follow 068
Fri Feb 20 00:57:12 UTC 2015
Fri Feb 20 00:57:13 UTC 2015
^C

• This will display the last line in the log file.
• Then, it will continue to display the logs in real time.
• Use ^C to exit.

containers/Background_Containers.md

Stop our container

There are two ways we can terminate our detached container.

• Killing it using the docker kill command.
• Stopping it using the docker stop command.

The first one stops the container immediately, by using the KILL signal.

The second one is more graceful. It sends a TERM signal, and after 10 seconds, if the container has not stopped, it sends KILL.

Reminder: the KILL signal cannot be intercepted, and will forcibly terminate the container.

containers/Background_Containers.md

Stopping our containers

Let's stop one of those containers:

$ docker stop 47d6
47d6

This will take 10 seconds:

• Docker sends the TERM signal;
• the container doesn't react to this signal (it's a simple Shell script with no special signal handling);
• 10 seconds later, since the container is still running, Docker sends the KILL signal;
• this terminates the container.

containers/Background_Containers.md

Killing the remaining containers

Let's be less patient with the two other containers:

$ docker kill 068 57ad
068
57ad

The stop and kill commands can take multiple container IDs.

Those containers will be terminated immediately (without the 10-second delay).

Let's check that our containers don't show up anymore:

$ docker ps

containers/Background_Containers.md

List stopped containers

We can also see stopped containers, with the -a (--all) option.

$ docker ps -a
CONTAINER ID  IMAGE           ...  CREATED      STATUS
068cc994ffd0  jpetazzo/clock  ...  21 min. ago  Exited (137) 3 min. ago
57ad9bdfc06b  jpetazzo/clock  ...  21 min. ago  Exited (137) 3 min. ago
47d677dcfba4  jpetazzo/clock  ...  23 min. ago  Exited (137) 3 min. ago
5c1dfd4d81f1  jpetazzo/clock  ...  40 min. ago  Exited (0) 40 min. ago
b13c164401fb  ubuntu          ...  55 min. ago  Exited (130) 53 min. ago

Objectives

In this section, we will explain:

• What is an image.

• What is a layer.

• The various image namespaces.

• How to search and download images.

• Image tags and when to use them.

containers/Initial_Images.md

What is an image?

• Image = files + metadata

• These files form the root filesystem of our container.

• The metadata can indicate a number of things, e.g.:

  • the author of the image
  • the command to execute in the container when starting it
  • environment variables to be set
  • etc.

• Images are made of layers, conceptually stacked on top of each other.

• Each layer can add, change, and remove files and/or metadata.

• Images can share layers to optimize disk usage, transfer times, and memory use.

containers/Initial_Images.md

Example for a Java webapp

Each of the following items will correspond to one layer:

• CentOS base layer
• Packages and configuration files added by our local IT
• JRE
• Tomcat
• Our application's dependencies
• Our application code and assets
• Our application configuration

(Note: app config is generally added by orchestration facilities.)

containers/Initial_Images.md

Differences between containers and images

• An image is a read-only filesystem.

• A container is an encapsulated set of processes,

  running in a read-write copy of that filesystem.

• To optimize container boot time, copy-on-write is used instead of regular copy.

• docker run starts a container from a given image.

containers/Initial_Images.md

Wait a minute...

If an image is read-only, how do we change it?

• We don't.

• We create a new container from that image.

• Then we make changes to that container.

• When we are satisfied with those changes, we transform them into a new layer.

• A new image is created by stacking the new layer on top of the old image.

containers/Initial_Images.md

A chicken-and-egg problem

• The only way to create an image is by "freezing" a container.

• The only way to create a container is by instantiating an image.

• Help!

containers/Initial_Images.md

Creating the first images

There is a special empty image called scratch.

• It allows to build from scratch.

The docker import command loads a tarball into Docker.

• The imported tarball becomes a standalone image.
• That new image has a single layer.

Note: you will probably never have to do this yourself.

containers/Initial_Images.md

Creating other images

docker commit

• Saves all the changes made to a container into a new layer.
• Creates a new image (effectively a copy of the container).

docker build (used 99% of the time)

• Performs a repeatable build sequence.
• This is the preferred method!

We will explain both methods in a moment.

containers/Initial_Images.md

Images namespaces

There are three namespaces:

• Official images

  e.g. ubuntu, busybox ...

• User (and organizations) images

  e.g. bretfisher/clock

• Self-hosted images

  e.g. registry.example.com:5000/my-private/image

Let's explain each of them.

containers/Initial_Images.md

Root namespace

The root namespace is for official images.

They are gated by Docker Inc.

They are generally authored and maintained by third parties.

Those images include:

• Small, "swiss-army-knife" images like busybox.

• Distro images to be used as bases for your builds, like ubuntu, fedora...

• Ready-to-use components and services, like redis, postgresql...

• Over 150 at this point!

containers/Initial_Images.md

User namespace

The user namespace holds images for Docker Hub users and organizations.

For example:

bretfisher/clock

The Docker Hub user is:

bretfisher

The image name is:

clock

containers/Initial_Images.md

Self-hosted namespace

This namespace holds images which are not hosted on Docker Hub, but on third party registries.

They contain the hostname (or IP address), and optionally the port, of the registry server.

For example:

localhost:5000/wordpress

• localhost:5000 is the host and port of the registry
• wordpress is the name of the image

Other examples:

quay.io/coreos/etcd
gcr.io/google-containers/hugo

containers/Initial_Images.md

How do you store and manage images?

Images can be stored:

• On your Docker host.
• In a Docker registry.

You can use the Docker client to download (pull) or upload (push) images.

To be more accurate: you can use the Docker client to tell a Docker Engine to push and pull images to and from a registry.

containers/Initial_Images.md

Showing current images

Let's look at what images are on our host now.

$ docker images
REPOSITORY       TAG       IMAGE ID       CREATED         SIZE
fedora           latest    ddd5c9c1d0f2   3 days ago      204.7 MB
centos           latest    d0e7f81ca65c   3 days ago      196.6 MB
ubuntu           latest    07c86167cdc4   4 days ago      188 MB
redis            latest    4f5f397d4b7c   5 days ago      177.6 MB
postgres         latest    afe2b5e1859b   5 days ago      264.5 MB
alpine           latest    70c557e50ed6   5 days ago      4.798 MB
debian           latest    f50f9524513f   6 days ago      125.1 MB
busybox          latest    3240943c9ea3   2 weeks ago     1.114 MB
training/namer   latest    902673acc741   9 months ago    289.3 MB
jpetazzo/clock   latest    12068b93616f   12 months ago   2.433 MB

containers/Initial_Images.md

Downloading images

There are two ways to download images.

• Explicitly, with docker pull.

• Implicitly, when executing docker run and the image is not found locally.

containers/Initial_Images.md

Pulling an image

$ docker pull debian:jessie
Pulling repository debian
b164861940b8: Download complete
b164861940b8: Pulling image (jessie) from debian
d1881793a057: Download complete

• As seen previously, images are made up of layers.

• Docker has downloaded all the necessary layers.

• In this example, :jessie indicates which exact version of Debian we would like.

  It is a version tag.

containers/Initial_Images.md

Image and tags

• Images can have tags.

• Tags define image versions or variants.

• docker pull ubuntu will refer to ubuntu:latest.

• The :latest tag is generally updated often.

containers/Initial_Images.md

When to (not) use tags

Don't specify tags:

• When doing rapid testing and prototyping.
• When experimenting.
• When you want the latest version.

Do specify tags:

• When recording a procedure into a script.
• When going to production.
• To ensure that the same version will be used everywhere.
• To ensure repeatability later.

This is similar to what we would do with pip install, npm install, etc.

containers/Initial_Images.md

Section summary

We've learned how to:

• Understand images and layers.
• Understand Docker image namespacing.
• Search and download images.

Objectives

We will build a container image automatically, with a Dockerfile.

At the end of this lesson, you will be able to:

• Write a Dockerfile.

• Build an image from a Dockerfile.

containers/Building_Images_With_Dockerfiles.md

Dockerfile overview

• A Dockerfile is a build recipe for a Docker image.

• It contains a series of instructions telling Docker how an image is constructed.

• The docker build command builds an image from a Dockerfile.

containers/Building_Images_With_Dockerfiles.md

Writing our first Dockerfile

Our Dockerfile must be in a new, empty directory.

1. Create a directory to hold our Dockerfile.

$ mkdir myimage

1. Create a Dockerfile inside this directory.

$ cd myimage
$ vim Dockerfile

Of course, you can use any other editor of your choice.

containers/Building_Images_With_Dockerfiles.md

Type this into our Dockerfile...

FROM ubuntu
RUN apt-get update
RUN apt-get install figlet

• FROM indicates the base image for our build.

• Each RUN line will be executed by Docker during the build.

• Our RUN commands must be non-interactive.
  (No input can be provided to Docker during the build.)

• In many cases, we will add the -y flag to apt-get.

containers/Building_Images_With_Dockerfiles.md

Build it!

Save our file, then execute:

$ docker build -t figlet .

• -t indicates the tag to apply to the image.

• . indicates the location of the build context.

We will talk more about the build context later.

To keep things simple for now: this is the directory where our Dockerfile is located.

containers/Building_Images_With_Dockerfiles.md

What happens when we build the image?

It depends if we're using BuildKit or not!

If there are lots of blue lines and the first line looks like this:

[+] Building 1.8s (4/6)

... then we're using BuildKit.

If the output is mostly black-and-white and the first line looks like this:

Sending build context to Docker daemon  2.048kB

... then we're using the "classic" or "old-style" builder.

containers/Building_Images_With_Dockerfiles.md

To BuildKit or Not To BuildKit

Classic builder:

• copies the whole "build context" to the Docker Engine

• linear (processes lines one after the other)

• requires a full Docker Engine

BuildKit:

• only transfers parts of the "build context" when needed

• will parallelize operations (when possible)

• can run in non-privileged containers (e.g. on Kubernetes)

containers/Building_Images_With_Dockerfiles.md

With the classic builder

The output of docker build looks like this:

docker build -t figlet .
Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM ubuntu
 ---> f975c5035748
Step 2/3 : RUN apt-get update
 ---> Running in e01b294dbffd
(...output of the RUN command...)
Removing intermediate container e01b294dbffd
 ---> eb8d9b561b37
Step 3/3 : RUN apt-get install figlet
 ---> Running in c29230d70f9b
(...output of the RUN command...)
Removing intermediate container c29230d70f9b
 ---> 0dfd7a253f21
Successfully built 0dfd7a253f21
Successfully tagged figlet:latest

• The output of the RUN commands has been omitted.
• Let's explain what this output means.

containers/Building_Images_With_Dockerfiles.md

Sending the build context to Docker

Sending build context to Docker daemon 2.048 kB

• The build context is the . directory given to docker build.

• It is sent (as an archive) by the Docker client to the Docker daemon.

• This allows to use a remote machine to build using local files.

• Be careful (or patient) if that directory is big and your link is slow.

• You can speed up the process with a .dockerignore file

  • It tells docker to ignore specific files in the directory

  • Only ignore files that you won't need in the build context!

containers/Building_Images_With_Dockerfiles.md

Executing each step

Step 2/3 : RUN apt-get update
 ---> Running in e01b294dbffd
(...output of the RUN command...)
Removing intermediate container e01b294dbffd
 ---> eb8d9b561b37

• A container (e01b294dbffd) is created from the base image.

• The RUN command is executed in this container.

• The container is committed into an image (eb8d9b561b37).

• The build container (e01b294dbffd) is removed.

• The output of this step will be the base image for the next one.

containers/Building_Images_With_Dockerfiles.md

With BuildKit

[+] Building 7.9s (7/7) FINISHED
 => [internal] load build definition from Dockerfile                                                 0.0s
 => => transferring dockerfile: 98B                                                                  0.0s
 => [internal] load .dockerignore                                                                    0.0s
 => => transferring context: 2B                                                                      0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                     1.2s
 => [1/3] FROM docker.io/library/ubuntu@sha256:cf31af331f38d1d7158470e095b132acd126a7180a54f263d386  3.2s
 => => resolve docker.io/library/ubuntu@sha256:cf31af331f38d1d7158470e095b132acd126a7180a54f263d386  0.0s
 => => sha256:cf31af331f38d1d7158470e095b132acd126a7180a54f263d386da88eb681d93 1.20kB / 1.20kB       0.0s
 => => sha256:1de4c5e2d8954bf5fa9855f8b4c9d3c3b97d1d380efe19f60f3e4107a66f5cae 943B / 943B           0.0s
 => => sha256:6a98cbe39225dadebcaa04e21dbe5900ad604739b07a9fa351dd10a6ebad4c1b 3.31kB / 3.31kB       0.0s
 => => sha256:80bc30679ac1fd798f3241208c14accd6a364cb8a6224d1127dfb1577d10554f 27.14MB / 27.14MB     2.3s
 => => sha256:9bf18fab4cfbf479fa9f8409ad47e2702c63241304c2cdd4c33f2a1633c5f85e 850B / 850B           0.5s
 => => sha256:5979309c983a2adeff352538937475cf961d49c34194fa2aab142effe19ed9c1 189B / 189B           0.4s
 => => extracting sha256:80bc30679ac1fd798f3241208c14accd6a364cb8a6224d1127dfb1577d10554f            0.7s
 => => extracting sha256:9bf18fab4cfbf479fa9f8409ad47e2702c63241304c2cdd4c33f2a1633c5f85e            0.0s
 => => extracting sha256:5979309c983a2adeff352538937475cf961d49c34194fa2aab142effe19ed9c1            0.0s
 => [2/3] RUN apt-get update                                                                         2.5s
 => [3/3] RUN apt-get install figlet                                                                 0.9s
 => exporting to image                                                                               0.1s
 => => exporting layers                                                                              0.1s
 => => writing image sha256:3b8aee7b444ab775975dfba691a72d8ac24af2756e0a024e056e3858d5a23f7c         0.0s
 => => naming to docker.io/library/figlet                                                            0.0s

containers/Building_Images_With_Dockerfiles.md

Understanding BuildKit output

• BuildKit transfers the Dockerfile and the build context

  (these are the first two [internal] stages)

• Then it executes the steps defined in the Dockerfile

  ([1/3], [2/3], [3/3])

• Finally, it exports the result of the build

  (image definition + collection of layers)

containers/Building_Images_With_Dockerfiles.md

The caching system

If you run the same build again, it will be instantaneous. Why?

• After each build step, Docker takes a snapshot of the resulting image.

• Before executing a step, Docker checks if it has already built the same sequence.

• Docker uses the exact strings defined in your Dockerfile, so:

  • RUN apt-get install figlet cowsay
    is different from
    RUN apt-get install cowsay figlet

  • RUN apt-get update is not re-executed when the mirrors are updated

You can force a rebuild with docker build --no-cache ....

containers/Building_Images_With_Dockerfiles.md

Running the image

The resulting image is not different from the one produced manually.

$ docker run -ti figlet
root@91f3c974c9a1:/# figlet hello
 _          _ _       
| |__   ___| | | ___  
| '_ \ / _ \ | |/ _ \ 
| | | |  __/ | | (_) |
|_| |_|\___|_|_|\___/

Yay! 🎉

containers/Building_Images_With_Dockerfiles.md

Using image and viewing history

The history command lists all the layers composing an image.

For each layer, it shows its creation time, size, and creation command.

When an image was built with a Dockerfile, each layer corresponds to a line of the Dockerfile.

$ docker history figlet
IMAGE         CREATED            CREATED BY                     SIZE
f9e8f1642759  About an hour ago  /bin/sh -c apt-get install fi  1.627 MB
7257c37726a1  About an hour ago  /bin/sh -c apt-get update      21.58 MB
07c86167cdc4  4 days ago         /bin/sh -c #(nop) CMD ["/bin   0 B
<missing>     4 days ago         /bin/sh -c sed -i 's/^#\s*\(   1.895 kB
<missing>     4 days ago         /bin/sh -c echo '#!/bin/sh'    194.5 kB
<missing>     4 days ago         /bin/sh -c #(nop) ADD file:b   187.8 MB

containers/Building_Images_With_Dockerfiles.md

Shell syntax vs exec syntax

Dockerfile commands that execute something can have two forms:

• plain string, or shell syntax:
  RUN apt-get install figlet

• JSON list, or exec syntax:
  RUN ["apt-get", "install", "figlet"]

We are going to change our Dockerfile to see how it affects the resulting image.

containers/Building_Images_With_Dockerfiles.md

Using exec syntax in our Dockerfile

Let's change our Dockerfile as follows!

FROM ubuntu
RUN apt-get update
RUN ["apt-get", "install", "figlet"]

Then build the new Dockerfile.

$ docker build -t figlet .

containers/Building_Images_With_Dockerfiles.md

History with exec syntax

Compare the new history:

$ docker history figlet
IMAGE         CREATED            CREATED BY                     SIZE
27954bb5faaf  10 seconds ago     apt-get install figlet         1.627 MB
7257c37726a1  About an hour ago  /bin/sh -c apt-get update      21.58 MB
07c86167cdc4  4 days ago         /bin/sh -c #(nop) CMD ["/bin   0 B
<missing>     4 days ago         /bin/sh -c sed -i 's/^#\s*\(   1.895 kB
<missing>     4 days ago         /bin/sh -c echo '#!/bin/sh'    194.5 kB
<missing>     4 days ago         /bin/sh -c #(nop) ADD file:b   187.8 MB

• Exec syntax specifies an exact command to execute.

• Shell syntax specifies a command to be wrapped within /bin/sh -c "...".

containers/Building_Images_With_Dockerfiles.md

When to use exec syntax and shell syntax

• shell syntax:

  • is easier to write
  • interpolates environment variables and other shell expressions
  • creates an extra process (/bin/sh -c ...) to parse the string
  • requires /bin/sh to exist in the container

• exec syntax:

  • is harder to write (and read!)
  • passes all arguments without extra processing
  • doesn't create an extra process
  • doesn't require /bin/sh to exist in the container
  • Typically only used for CMD at end of Dockerfile

containers/Building_Images_With_Dockerfiles.md

Objectives

In this lesson, we will learn about two important Dockerfile commands:

CMD and ENTRYPOINT.

These commands allow us to set the default command to run in a container.

containers/Cmd_And_Entrypoint.md

Defining a default command

When people run our container, we want to greet them with a nice hello message, and using a custom font.

For that, we will execute:

figlet -f script hello

• -f script tells figlet to use a fancy font.

• hello is the message that we want it to display.

containers/Cmd_And_Entrypoint.md

Adding CMD to our Dockerfile

Our new Dockerfile will look like this:

FROM ubuntu
RUN apt-get update
RUN ["apt-get", "install", "figlet"]
CMD figlet -f script hello

• CMD defines a default command to run when none is given.

• It can appear at any point in the file.

• Each CMD will replace and override the previous one.

• As a result, while you can have multiple CMD lines, it is useless.

containers/Cmd_And_Entrypoint.md

Build and test our image

Let's build it:

$ docker build -t figlet .
...
Successfully built 042dff3b4a8d
Successfully tagged figlet:latest

And run it:

$ docker run figlet
 _          _   _       
| |        | | | |      
| |     _  | | | |  __  
|/ \   |/  |/  |/  /  \_
|   |_/|__/|__/|__/\__/

containers/Cmd_And_Entrypoint.md

Overriding CMD

If we want to get a shell into our container (instead of running figlet), we just have to specify a different program to run:

$ docker run -it figlet bash
root@7ac86a641116:/#

• We specified bash.

• It replaced the value of CMD.

containers/Cmd_And_Entrypoint.md

Using ENTRYPOINT

We want to be able to specify a different message on the command line, while retaining figlet and some default parameters.

In other words, we would like to be able to do this:

$ docker run figlet salut
           _            
          | |           
 ,   __,  | |       _|_ 
/ \_/  |  |/  |   |  |  
 \/ \_/|_/|__/ \_/|_/|_/

We will use the ENTRYPOINT verb in Dockerfile.

containers/Cmd_And_Entrypoint.md

Adding ENTRYPOINT to our Dockerfile

Our new Dockerfile will look like this:

FROM ubuntu
RUN apt-get update
RUN ["apt-get", "install", "figlet"]
ENTRYPOINT ["figlet", "-f", "script"]

• ENTRYPOINT defines a base command (and its parameters) for the container.

• The command line arguments are appended to those parameters.

• Like CMD, ENTRYPOINT can appear anywhere, and replaces the previous value.

Why did we use JSON syntax for our ENTRYPOINT?

containers/Cmd_And_Entrypoint.md

Implications of JSON vs string syntax

• When CMD or ENTRYPOINT use string syntax, they get wrapped in sh -c.

• To avoid this wrapping, we can use JSON syntax.

What if we used ENTRYPOINT with string syntax?

$ docker run figlet salut

This would run the following command in the figlet image:

sh -c "figlet -f script" salut

containers/Cmd_And_Entrypoint.md

Build and test our image

Let's build it:

$ docker build -t figlet .
...
Successfully built 36f588918d73
Successfully tagged figlet:latest

And run it:

$ docker run figlet salut
           _            
          | |           
 ,   __,  | |       _|_ 
/ \_/  |  |/  |   |  |  
 \/ \_/|_/|__/ \_/|_/|_/

containers/Cmd_And_Entrypoint.md

Using CMD and ENTRYPOINT together

What if we want to define a default message for our container?

Then we will use ENTRYPOINT and CMD together.

• ENTRYPOINT will define the base command for our container.

• CMD will define the default parameter(s) for this command.

• They both have to use JSON syntax.

containers/Cmd_And_Entrypoint.md

CMD and ENTRYPOINT together

Our new Dockerfile will look like this:

FROM ubuntu
RUN apt-get update
RUN ["apt-get", "install", "figlet"]
ENTRYPOINT ["figlet", "-f", "script"]
CMD ["hello world"]

• ENTRYPOINT defines a base command (and its parameters) for the container.

• If we don't specify extra command-line arguments when starting the container, the value of CMD is appended.

• Otherwise, our extra command-line arguments are used instead of CMD.

containers/Cmd_And_Entrypoint.md

Build and test our image

Let's build it:

$ docker build -t myfiglet .
...
Successfully built 6e0b6a048a07
Successfully tagged myfiglet:latest

Run it without parameters:

$ docker run myfiglet
 _          _   _                             _        
| |        | | | |                           | |    |  
| |     _  | | | |  __             __   ,_   | |  __|  
|/ \   |/  |/  |/  /  \_  |  |  |_/  \_/  |  |/  /  |  
|   |_/|__/|__/|__/\__/    \/ \/  \__/    |_/|__/\_/|_/

containers/Cmd_And_Entrypoint.md

Overriding the image default parameters

Now let's pass extra arguments to the image.

$ docker run myfiglet hola mundo
 _           _                                               
| |         | |                                      |       
| |     __  | |  __,     _  _  _           _  _    __|   __  
|/ \   /  \_|/  /  |    / |/ |/ |  |   |  / |/ |  /  |  /  \_
|   |_/\__/ |__/\_/|_/    |  |  |_/ \_/|_/  |  |_/\_/|_/\__/

We overrode CMD but still used ENTRYPOINT.

containers/Cmd_And_Entrypoint.md

Overriding ENTRYPOINT

What if we want to run a shell in our container?

We cannot just do docker run myfiglet bash because that would just tell figlet to display the word "bash."

We use the --entrypoint parameter:

$ docker run -it --entrypoint bash myfiglet
root@6027e44e2955:/#

containers/Cmd_And_Entrypoint.md

CMD and ENTRYPOINT recap

• docker run myimage executes ENTRYPOINT + CMD

• docker run myimage args executes ENTRYPOINT + args (overriding CMD)

• docker run --entrypoint prog myimage executes prog (overriding both)

containers/Cmd_And_Entrypoint.md

When to use ENTRYPOINT vs CMD

ENTRYPOINT is great for "containerized binaries".

Example: docker run consul --help

(Pretend that the docker run part isn't there!)

CMD is great for images with multiple binaries.

Example: docker run busybox ifconfig

(It makes sense to indicate which program we want to run!)

Objectives

So far, we have installed things in our container images by downloading packages.

We can also copy files from the build context to the container that we are building.

Remember: the build context is the directory containing the Dockerfile.

In this chapter, we will learn a new Dockerfile keyword: COPY.

containers/Copying_Files_During_Build.md

Build some C code

We want to build a container that compiles a basic "Hello world" program in C.

Here is the program, hello.c:

int main () {
  puts("Hello, world!");
  return 0;
}

Let's create a new directory, and put this file in there.

Then we will write the Dockerfile.

containers/Copying_Files_During_Build.md

The Dockerfile

On Debian and Ubuntu, the package build-essential will get us a compiler.

When installing it, don't forget to specify the -y flag, otherwise the build will fail (since the build cannot be interactive).

Then we will use COPY to place the source file into the container.

FROM ubuntu
RUN apt-get update
RUN apt-get install -y build-essential
COPY hello.c /
RUN make hello
CMD /hello

Create this Dockerfile.

containers/Copying_Files_During_Build.md

Testing our C program

• Create hello.c and Dockerfile in the same directory.

• Run docker build -t hello . in this directory.

• Run docker run hello, you should see Hello, world!.

Success!

containers/Copying_Files_During_Build.md

COPY and the build cache

• Run the build again.

• Now, modify hello.c and run the build again.

• Docker can cache steps involving COPY.

• Those steps will not be executed again if the files haven't been changed.

containers/Copying_Files_During_Build.md

Details

• We can COPY whole directories recursively

• It is possible to do e.g. COPY . .

  (but it might require some extra precautions to avoid copying too much)

• In older Dockerfiles, you might see the ADD command; consider it deprecated

  (it is similar to COPY but can automatically extract archives)

• If we really wanted to compile C code in a container, we would:

  • place it in a different directory, with the WORKDIR instruction

  • even better, use the gcc official image

containers/Copying_Files_During_Build.md

Exercise — writing Dockerfiles

Let's write Dockerfiles for an existing application!

1. Check out the code repository

2. Read all the instructions

3. Write Dockerfiles

4. Build and test them individually

containers/Exercise_Dockerfile_Basic.md

Code repository

Clone the repository available at:

https://github.com/jpetazzo/wordsmith

It should look like this:

├── LICENSE
├── README
├── db/
│   └── words.sql
├── web/
│   ├── dispatcher.go
│   └── static/
└── words/
    ├── pom.xml
    └── src/

containers/Exercise_Dockerfile_Basic.md

Instructions

The repository contains instructions in English and French.
For now, we only care about the first part (about writing Dockerfiles).
Place each Dockerfile in its own directory, like this:

├── LICENSE
├── README
├── db/
│   ├── Dockerfile
│   └── words.sql
├── web/
│   ├── Dockerfile
│   ├── dispatcher.go
│   └── static/
└── words/
    ├── Dockerfile
    ├── pom.xml
    └── src/

containers/Exercise_Dockerfile_Basic.md

Build and test

Build and run each Dockerfile individually.

For db, we should be able to see some messages confirming that the data set was loaded successfully (some INSERT lines in the container output).

For web and words, we should be able to see some message looking like "server started successfully".

That's all we care about for now!

Bonus question: make sure that each container stops correctly when hitting Ctrl-C.

Done for today. On Friday:

• Docker Networking, Docker Compose, & Workflows local Dev & Test

• Remember to signup for the Udemy courses (info on front page)

• Do the example Dockerfile!

• Dig into Bonus Sections

next.md

Objectives

We will now run network services (accepting requests) in containers.

At the end of this section, you will be able to:

• Run a network service in a container.

• Connect to that network service.

• Find a container's IP address.

containers/Container_Networking_Basics.md

Running a very simple service

• We need something small, simple, easy to configure

  (or, even better, that doesn't require any configuration at all)

• Let's use the official NGINX image (named nginx)

• It runs a static web server listening on port 80

• It serves a default "Welcome to nginx!" page

containers/Container_Networking_Basics.md

Running an NGINX server

$ docker run -d -P nginx
66b1ce719198711292c8f34f84a7b68c3876cf9f67015e752b94e189d35a204e

• Docker will automatically pull the nginx image from the Docker Hub

• -d / --detach tells Docker to run it in the background

• P / --publish-all tells Docker to publish all ports

  (publish = make them reachable from other computers)

• ...OK, how do we connect to our web server now?

containers/Container_Networking_Basics.md

Finding our web server port

• First, we need to find the port number used by Docker

  (the NGINX container listens on port 80, but this port will be mapped)

• We can use docker ps:

  $ docker ps
  CONTAINER ID  IMAGE  ...  PORTS                  ...
  e40ffb406c9e  nginx  ...  0.0.0.0:12345->80/tcp  ...

• This means:

  port 12345 on the Docker host is mapped to port 80 in the container

• Now we need to connect to the Docker host!

containers/Container_Networking_Basics.md

Finding the address of the Docker host

• When running Docker on your Linux workstation:

  use localhost, or any IP address of your machine

• When running Docker on a remote Linux server:

  use any IP address of the remote machine

• When running Docker Desktop on Mac or Windows:

  use localhost

• In other scenarios (docker-machine, local VM...):

  use the IP address of the Docker VM

containers/Container_Networking_Basics.md

Connecting to our web server (GUI)

Point your browser to the IP address of your Docker host, on the port shown by docker ps for container port 80.

containers/Container_Networking_Basics.md

Connecting to our web server (CLI)

You can also use curl directly from the Docker host.

Make sure to use the right port number if it is different from the example below:

$ curl localhost:12345
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

containers/Container_Networking_Basics.md

How does Docker know which port to map?

• There is metadata in the image telling "this image has something on port 80".

• We can see that metadata with docker inspect:

$ docker inspect --format '{{.Config.ExposedPorts}}' nginx
map[80/tcp:{}]

• This metadata was set in the Dockerfile, with the EXPOSE keyword.

• We can see that with docker history:

$ docker history nginx
IMAGE               CREATED             CREATED BY
7f70b30f2cc6        11 days ago         /bin/sh -c #(nop)  CMD ["nginx" "-g" "…
<missing>           11 days ago         /bin/sh -c #(nop)  STOPSIGNAL [SIGTERM]
<missing>           11 days ago         /bin/sh -c #(nop)  EXPOSE 80/tcp

containers/Container_Networking_Basics.md

Why can't we just connect to port 80?

• Our Docker host has only one port 80

• Therefore, we can only have one container at a time on port 80

• Therefore, if multiple containers want port 80, only one can get it

• By default, containers do not get "their" port number, but a random one

  (not "random" as "crypto random", but as "it depends on various factors")

• We'll see later how to force a port number (including port 80!)

containers/Container_Networking_Basics.md

Finding the web server port in a script

Parsing the output of docker ps would be painful.

There is a command to help us:

$ docker port <containerID> 80
0.0.0.0:12345

containers/Container_Networking_Basics.md

Manual allocation of port numbers

If you want to set port numbers yourself, no problem:

$ docker run -d -p 80:80 nginx
$ docker run -d -p 8000:80 nginx
$ docker run -d -p 8080:80 -p 8888:80 nginx

• We are running three NGINX web servers.
• The first one is exposed on port 80.
• The second one is exposed on port 8000.
• The third one is exposed on ports 8080 and 8888.

Note: the convention is port-on-host:port-on-container.

containers/Container_Networking_Basics.md

Plumbing containers into your infrastructure

There are many ways to integrate containers in your network.

• Start the container, letting Docker allocate a public port for it.
  Then retrieve that port number and feed it to your configuration.

• Pick a fixed port number in advance, when you generate your configuration.
  Then start your container by setting the port numbers manually.

• Use an orchestrator like Kubernetes or Swarm.
  The orchestrator will provide its own networking facilities.

Orchestrators typically provide mechanisms to enable direct container-to-container communication across hosts, and publishing/load balancing for inbound traffic.

containers/Container_Networking_Basics.md

Finding the container's IP address

We can use the docker inspect command to find the IP address of the container.

$ docker inspect --format '{{ .NetworkSettings.IPAddress }}' <yourContainerID>
172.17.0.3

• docker inspect is an advanced command, that can retrieve a ton of information about our containers.

• Here, we provide it with a format string to extract exactly the private IP address of the container.

containers/Container_Networking_Basics.md

Pinging our container

Let's try to ping our container from another container.

docker run alpine ping <ipaddress>
PING 172.17.0.X (172.17.0.X): 56 data bytes
64 bytes from 172.17.0.X: seq=0 ttl=64 time=0.106 ms
64 bytes from 172.17.0.X: seq=1 ttl=64 time=0.250 ms
64 bytes from 172.17.0.X: seq=2 ttl=64 time=0.188 ms

When running on Linux, we can even ping that IP address directly!

(And connect to a container's ports even if they aren't published.)

containers/Container_Networking_Basics.md

How often do we use -p and -P ?

• When running a stack of containers, we will often use Compose

• Compose will take care of exposing containers

  (through a ports: section in the docker-compose.yml file)

• It is, however, fairly common to use docker run -P for a quick test

• Or docker run -p ... when an image doesn't EXPOSE a port correctly

containers/Container_Networking_Basics.md

Section summary

We've learned how to:

• Expose a network port.

• Connect to an application running in a container.

• Find a container's IP address.

Objectives

We will learn about the CNM (Container Network Model).

At the end of this lesson, you will be able to:

• Create a private network for a group of containers.

• Use container naming to connect services together.

• Dynamically connect and disconnect containers to networks.

• Set the IP address of a container.

We will also explain the principle of overlay networks and network plugins.

containers/Container_Network_Model.md

The Container Network Model

Docker has "networks".

We can manage them with the docker network commands; for instance:

$ docker network ls
NETWORK ID          NAME                DRIVER
6bde79dfcf70        bridge              bridge
8d9c78725538        none                null
eb0eeab782f4        host                host
4c1ff84d6d3f        blog-dev            overlay
228a4355d548        blog-prod           overlay

New networks can be created (with docker network create).

(Note: networks none and host are special; let's set them aside for now.)

containers/Container_Network_Model.md

What's a network?

• Conceptually, a Docker "network" is a virtual switch

  (we can also think about it like a VLAN, or a WiFi SSID, for instance)

• By default, containers are connected to a single network

  (but they can be connected to zero, or many networks, even dynamically)

• Each network has its own subnet (IP address range)

• A network can be local (to a single Docker Engine) or global (span multiple hosts)

• Containers can have network aliases providing DNS-based service discovery

  (and each network has its own "domain", "zone", or "scope")

containers/Container_Network_Model.md

Service discovery

• A container can be given a network alias

  (e.g. with docker run --net some-network --net-alias db ...)

• The containers running in the same network can resolve that network alias

  (i.e. if they do a DNS lookup on db, it will give the container's address)

• We can have a different db container in each network

  (this avoids naming conflicts between different stacks)

• When we name a container, it automatically adds the name as a network alias

  (i.e. docker run --name xyz ... is like docker run --net-alias xyz ...

containers/Container_Network_Model.md

Network isolation

• Networks are isolated

• By default, containers in network A cannot reach those in network B

• A container connected to both networks A and B can act as a router or proxy

• Published ports are always reachable through the Docker host address

  (docker run -P ... makes a container port available to everyone)

containers/Container_Network_Model.md

How to use networks

• We typically create one network per "stack" or app that we deploy

• More complex apps or stacks might require multiple networks

  (e.g. frontend, backend, ...)

• Networks allow us to deploy multiple copies of the same stack

  (e.g. prod, dev, pr-442, ....)

• If we use Docker Compose, this is managed automatically for us

containers/Container_Network_Model.md

Creating a network

Let's create a network called dev.

$ docker network create dev
4c1ff84d6d3f1733d3e233ee039cac276f425a9d5228a4355d54878293a889ba

The network is now visible with the network ls command:

$ docker network ls
NETWORK ID          NAME                DRIVER
6bde79dfcf70        bridge              bridge
8d9c78725538        none                null
eb0eeab782f4        host                host
4c1ff84d6d3f        dev                 bridge

containers/Container_Network_Model.md

Placing containers on a network

We will create a named container on this network.

It will be reachable with its name, es.

$ docker run -d --name es --net dev elasticsearch:2
8abb80e229ce8926c7223beb69699f5f34d6f1d438bfc5682db893e798046863

containers/Container_Network_Model.md

Communication between containers

Now, create another container on this network.

$ docker run -ti --net dev alpine sh
root@0ecccdfa45ef:/#

From this new container, we can resolve and ping the other one, using its assigned name:

/ # ping es
PING es (172.18.0.2) 56(84) bytes of data.
64 bytes from es.dev (172.18.0.2): icmp_seq=1 ttl=64 time=0.221 ms
64 bytes from es.dev (172.18.0.2): icmp_seq=2 ttl=64 time=0.114 ms
64 bytes from es.dev (172.18.0.2): icmp_seq=3 ttl=64 time=0.114 ms
^C
--- es ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.114/0.149/0.221/0.052 ms
root@0ecccdfa45ef:/#

containers/Container_Network_Model.md

Service discovery with containers

• Let's try to run an application that requires two containers.

• The first container is a web server.

• The other one is a redis data store.

• We will place them both on the dev network created before.

containers/Container_Network_Model.md

Running the web server

• The application is provided by the container image jpetazzo/trainingwheels.

• We don't know much about it so we will try to run it and see what happens!

Start the container, exposing all its ports:

$ docker run --net dev -d -P jpetazzo/trainingwheels

Check the port that has been allocated to it:

$ docker ps -l

containers/Container_Network_Model.md

Test the web server

• If we connect to the application now, we will see an error page:

• This is because the Redis service is not running.
• This container tries to resolve the name redis.

Note: we're not using a FQDN or an IP address here; just redis.

containers/Container_Network_Model.md

Start the data store

• We need to start a Redis container.

• That container must be on the same network as the web server.

• It must have the right network alias (redis) so the application can find it.

Start the container:

$ docker run --net dev --net-alias redis -d redis

containers/Container_Network_Model.md

Test the web server again

• If we connect to the application now, we should see that the app is working correctly:

• When the app tries to resolve redis, instead of getting a DNS error, it gets the IP address of our Redis container.

containers/Container_Network_Model.md

A few words on scope

• Container names are unique (there can be only one --name redis)

• Network aliases are not unique

• We can have the same network alias in different networks:

  docker run --net dev --net-alias redis ...
  docker run --net prod --net-alias redis ...

• We can even have multiple containers with the same alias in the same network

  (in that case, we get multiple DNS entries, aka "DNS round robin")

containers/Container_Network_Model.md

Good to know ...

• Docker will not create network names and aliases on the default bridge network.

• Therefore, if you want to use those features, you have to create a custom network first.

• Network aliases are not unique on a given network.

• i.e., multiple containers can have the same alias on the same network.

• In that scenario, the Docker DNS server will return multiple records.
  (i.e. you will get DNS round robin out of the box.)

• Enabling Swarm Mode gives access to clustering and load balancing with IPVS.

• Creation of networks and network aliases is generally automated with tools like Compose.

containers/Container_Network_Model.md

Network drivers

• A network is managed by a driver.

• The built-in drivers include:

  • bridge (default)
  • none
  • host
  • macvlan
  • overlay (for Swarm clusters)

• More drivers can be provided by plugins (OVS, VLAN...)

• A network can have a custom IPAM (IP allocator).

containers/Container_Network_Model.md

Overlay networks

• The features we've seen so far only work when all containers are on a single host.

• If containers span multiple hosts, we need an overlay network to connect them together.

• Docker ships with a default network plugin, overlay, implementing an overlay network leveraging VXLAN, enabled with Swarm Mode.

• Other plugins (Weave, Calico...) can provide overlay networks as well.

• Once you have an overlay network, all the features that we've used in this chapter work identically across multiple hosts.

containers/Container_Network_Model.md

Connecting and disconnecting dynamically

• So far, we have specified which network to use when starting the container.

• The Docker Engine also allows connecting and disconnecting while the container is running.

• This feature is exposed through the Docker API, and through two Docker CLI commands:

  • docker network connect <network> <container>

  • docker network disconnect <network> <container>

containers/Container_Network_Model.md

Dynamically connecting to a network

• We have a container named es connected to a network named dev.

• Let's start a simple alpine container on the default network:

  $ docker run -ti alpine sh
  / #

• In this container, try to ping the es container:

  / # ping es
  ping: bad address 'es'

  This doesn't work, but we will change that by connecting the container.

containers/Container_Network_Model.md

Finding the container ID and connecting it

• Figure out the ID of our alpine container; here are two methods:

  • looking at /etc/hostname in the container,

  • running docker ps -lq on the host.

• Run the following command on the host:

  $ docker network connect dev <container_id>

containers/Container_Network_Model.md

Checking what we did

• Try again to ping es from the container.

• It should now work correctly:

  / # ping es
  PING es (172.20.0.3): 56 data bytes
  64 bytes from 172.20.0.3: seq=0 ttl=64 time=0.376 ms
  64 bytes from 172.20.0.3: seq=1 ttl=64 time=0.130 ms
  ^C

• Interrupt it with Ctrl-C.

containers/Container_Network_Model.md

Looking at the network setup in the container

We can look at the list of network interfaces with ifconfig, ip a, or ip l:

/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
18: eth0@if19: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
20: eth1@if21: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:14:00:04 brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.4/16 brd 172.20.255.255 scope global eth1
       valid_lft forever preferred_lft forever
/ #

Each network connection is materialized with a virtual network interface.

As we can see, we can be connected to multiple networks at the same time.

containers/Container_Network_Model.md

Disconnecting from a network

• Let's try the symmetrical command to disconnect the container:

  $ docker network disconnect dev <container_id>

• From now on, if we try to ping es, it will not resolve:

  / # ping es
  ping: bad address 'es'

• Trying to ping the IP address directly won't work either:

  / # ping 172.20.0.3
  ... (nothing happens until we interrupt it with Ctrl-C)

containers/Container_Network_Model.md

Objectives

At the end of this section, you will be able to:

• Share code between container and host.

• Use a simple local development workflow.

containers/Local_Development_Workflow.md

Local development in a container

We want to solve the following issues:

• "Works on my machine"

• "Not the same version"

• "Missing dependency"

By using Docker containers, we will get a consistent development environment.

containers/Local_Development_Workflow.md

Working on the "namer" application

• We have to work on some application whose code is at:

  https://github.com/bretfisher/namer.

• What is it? We don't know yet!

• Let's download the code.

$ git clone https://github.com/bretfisher/namer

containers/Local_Development_Workflow.md

Looking at the code

$ cd namer
$ ls -1
company_name_generator.rb
config.ru
docker-compose.yml
Dockerfile
Gemfile

Looking at the Dockerfile

FROM ruby
COPY . /src
WORKDIR /src
RUN bundler install
CMD ["rackup", "--host", "0.0.0.0"]
EXPOSE 9292

• This application is using a base ruby image.
• The code is copied in /src.
• Dependencies are installed with bundler.
• The application is started with rackup.
• It is listening on port 9292.

containers/Local_Development_Workflow.md

Building and running the "namer" application

• Let's build the application with the Dockerfile!

Connecting to our application

• Point our browser to our Docker node, on the port allocated to the container.

Making changes to the code

Option 1:

• Edit the code locally
• Rebuild the image
• Re-run the container

Option 2:

• Enter the container (with docker exec)
• Install an editor
• Make changes from within the container

Option 3:

• Use a bind mount to share local files with the container
• Make changes locally
• Changes are reflected in the container

containers/Local_Development_Workflow.md

Our first volume

We will tell Docker to map the current directory to /src in the container.

$ docker run -d -v $(pwd):/src -P namer

• -d: the container should run in detached mode (in the background).

• -v: the following host directory should be mounted inside the container.

• -P: publish all the ports exposed by this image.

• namer is the name of the image we will run.

• We don't specify a command to run because it is already set in the Dockerfile via CMD.

Note: on Windows, replace $(pwd) with %cd% (or ${pwd} if you use PowerShell).

containers/Local_Development_Workflow.md

Mounting volumes inside containers

The -v flag mounts a directory from your host into your Docker container.

The flag structure is:

[host-path]:[container-path]:[rw|ro]

• [host-path] and [container-path] are created if they don't exist.

• You can control the write status of the volume with the ro and rw options.

• If you don't specify rw or ro, it will be rw by default.

containers/Local_Development_Workflow.md

Testing the development container

• Check the port used by our new container.

$ docker ps -l
CONTAINER ID  IMAGE  COMMAND  CREATED        STATUS  PORTS                   NAMES
045885b68bc5  namer  rackup   3 seconds ago  Up ...  0.0.0.0:32770->9292/tcp ...

• Open the application in your web browser.

containers/Local_Development_Workflow.md

Making a change to our application

Our customer really doesn't like the color of our text. Let's change it.

$ vi company_name_generator.rb

And change

color: royalblue;

To:

color: red;

containers/Local_Development_Workflow.md

Viewing our changes

• Reload the application in our browser.

Understanding volumes

• Volumes are not copying or synchronizing files between the host and the container

• Changes made in the host are immediately visible in the container (and vice versa)

• When running on Linux:

  • volumes and bind mounts correspond to directories on the host

  • if Docker runs in a Linux VM, these directories are in the Linux VM

• When running on Docker Desktop:

  • volumes correspond to directories in a small Linux VM running Docker

  • access to bind mounts is translated to host filesystem access
    (a bit like a network filesystem)

containers/Local_Development_Workflow.md

Trash your servers and burn your code

(This is the title of a 2013 blog post by Chad Fowler, where he explains the concept of immutable infrastructure.)

Immutable infrastructure in a nutshell

• Instead of updating a server, we deploy a new one.

• This might be challenging with classical servers, but it's trivial with containers.

• In fact, with Docker, the most logical workflow is to build a new image and run it.

• If something goes wrong with the new image, we can always restart the old one.

• We can even keep both versions running side by side.

If this pattern sounds interesting, you might want to read about blue/green deployment and canary deployments.

containers/Local_Development_Workflow.md

Recap of the development workflow

1. Write a Dockerfile to build an image containing our development environment.
   (Rails, Django, ... and all the dependencies for our app)

2. Start a container from that image.
   Use the -v flag to mount our source code inside the container.

3. Edit the source code outside the container, using familiar tools.
   (vim, emacs, textmate...)

4. Test the application.
   (Some frameworks pick up changes automatically.
   Others require you to Ctrl-C + restart after each modification.)

5. Iterate and repeat steps 3 and 4 until satisfied.

6. When done, commit+push source code changes.

containers/Local_Development_Workflow.md

Section summary

We've learned how to:

• Share code between container and host.

• Set our working directory.

• Use a simple local development workflow.

Compose for development stacks

Dockerfile = great to build one container image.

What if we have multiple containers?

What if some of them require particular docker run parameters?

How do we connect them all together?

... Compose solves these use-cases (and a few more).

containers/Compose_For_Dev_Stacks.md

Life before Compose

Before we had Compose, we would typically write custom scripts to:

• build container images,

• run containers using these images,

• connect the containers together,

• rebuild, restart, update these images and containers.

containers/Compose_For_Dev_Stacks.md

Life with Compose

Compose enables a simple, powerful onboarding workflow:

1. Checkout our code.

2. Run docker compose up.

3. Our app is up and running!

containers/Compose_For_Dev_Stacks.md

Life after Compose

(Or: when do we need something else?)

• Compose is not an orchestrator

• It isn't designed to need to run containers on multiple nodes

  (it can, however, work with Docker Swarm Mode)

• Compose isn't ideal if we want to run containers on Kubernetes

  • it uses different concepts (Compose services ≠ Kubernetes services)

  • it needs a Docker Engine (although containerd support might be coming)

  • there are projects to convert Compose files, like Kompose and Podman

containers/Compose_For_Dev_Stacks.md

First rodeo with Compose

1. Write Dockerfiles

2. Describe our stack of containers in a YAML file called docker-compose.yml

3. docker compose up (or docker compose up -d to run in the background)

4. Compose pulls and builds the required images, and starts the containers

5. Compose shows the combined logs of all the containers

   (if running in the background, use docker compose logs)

6. Hit Ctrl-C to stop the whole stack

   (if running in the background, use docker compose stop)

containers/Compose_For_Dev_Stacks.md

Iterating

After making changes to our source code, we can:

1. docker compose build to rebuild container images

2. docker compose up to restart the stack with the new images

We can also combine both with docker compose up --build

Compose will be smart, and only recreate the containers that have changed.

When working with interpreted languages:

• don't rebuild each time

• leverage a volumes section instead

containers/Compose_For_Dev_Stacks.md

Launching Our First Stack with Compose

First step: clone the source code for the app we will be working on.

git clone https://github.com/bretfisher/trainingwheels
cd trainingwheels

Second step: start the app.

docker compose up

Watch Compose build and run the app.

That Compose stack exposes a web server on port 8000; try connecting to it.

containers/Compose_For_Dev_Stacks.md

Launching Our First Stack with Compose

We should see a web page like this:

Each time we reload, the counter should increase.

containers/Compose_For_Dev_Stacks.md

Stopping the app

When we hit Ctrl-C, Compose tries to gracefully terminate all of the containers.

After ten seconds (or if we press ^C again) it will forcibly kill them.

containers/Compose_For_Dev_Stacks.md

The (short) history of Docker Compose

• 2013-2015, "fig" was created in Python by a small team outside Docker.

• 2015, Docker hired them, and renamed fig to Docker Compose.

• It's CLI, docker-compose, reigned as the easiest tool for running containers with YAML.

• But, it was a separate pip install, and was not Go (golang) like every other Docker tool.

• 2020, Docker rebuilt it in Go, making it faster and easier to install.

• They call it Compose V2, and it has many new features (video walk-through)

• Replace all your docker-compose keystrokes with docker compose.

• It should have 100% backward compatibility. containers/Compose_For_Dev_Stacks.md

The docker-compose.yml file

Here is the file used in the demo:

services:
  www:
    build: www
    ports:
      - ${PORT-8000}:5000
    user: nobody
    environment:
      DEBUG: 1
    command: python counter.py
    volumes:
      - ./www:/src
  redis:
    image: redis

containers/Compose_For_Dev_Stacks.md

Compose file structure

A Compose file has multiple sections:

• services is mandatory. Each service corresponds to one or more containers from the same image (replicas).

• networks is optional and indicates to which networks containers should be connected.
  (By default, containers will be connected on a private, per-compose-file network.)

• volumes is optional and can define volumes to be used and/or shared by the containers.

containers/Compose_For_Dev_Stacks.md

Containers in docker-compose.yml

Each service in the YAML file must contain either build, or image.

• build indicates a path containing a Dockerfile.

• image indicates an image name (local, or on a registry).

• If both are specified, an image will be built from the build directory and named image.

The other parameters are optional.

They encode the parameters that you would typically add to docker run.

Sometimes they have several minor improvements.

containers/Compose_For_Dev_Stacks.md

Container parameters

• command indicates what to run (like CMD in a Dockerfile).

• ports translates to one (or multiple) -p options to map ports.
  You can specify local ports (i.e. x:y to expose public port x).

• volumes translates to one (or multiple) -v options.
  You can use relative paths here.

Bookmark this reference doc! https://docs.docker.com/compose/compose-file/

containers/Compose_For_Dev_Stacks.md

Environment variables

• We can use environment variables in Compose files

  (like $THIS or ${THAT})

• We can provide default values, e.g. ${PORT-8000}

• Compose will also automatically load the environment file .env

  (it should contain VAR=value, one per line)

• This is a great way to customize build and run parameters

  (base image versions to use, build and run secrets, port numbers...)

containers/Compose_For_Dev_Stacks.md

Configuring a Compose stack

• Follow 12-factor app configuration principles

  (configure the app through environment variables)

• Provide (in the repo) a default environment file suitable for development

  (no secret or sensitive value)

• Copy the default environment file to .env and tweak it

  (or: provide a script to generate .env from a template) containers/Compose_For_Dev_Stacks.md

Running multiple copies of a stack

• Copy the stack in two different directories, e.g. front and frontcopy

• Compose prefixes images and containers with the directory name:

  front_www, front_www_1, front_db_1

  frontcopy_www, frontcopy_www_1, frontcopy_db_1

• Alternatively, use docker compose -p frontcopy

  (to set the --project-name of a stack, which default to the dir name)

• Each copy is isolated from the others (runs on a different network)

containers/Compose_For_Dev_Stacks.md

Checking stack status

We have ps, docker ps, and similarly, docker compose ps:

$ docker compose ps
Name                      Command             State           Ports          
----------------------------------------------------------------------------
trainingwheels_redis_1   /entrypoint.sh red   Up      6379/tcp               
trainingwheels_www_1     python counter.py    Up      0.0.0.0:8000->5000/tcp

Shows the status of all the containers of our stack.

Doesn't show the other containers.

containers/Compose_For_Dev_Stacks.md

Cleaning up (1)

If you have started your application in the background with Compose and want to stop it easily, you can use the kill command:

$ docker compose kill

Likewise, docker compose rm will let you remove containers (after confirmation):

$ docker compose rm
Going to remove trainingwheels_redis_1, trainingwheels_www_1
Are you sure? [yN] y
Removing trainingwheels_redis_1...
Removing trainingwheels_www_1...

containers/Compose_For_Dev_Stacks.md

Cleaning up (2)

Alternatively, docker compose down will stop and remove containers.

It will also remove other resources, like networks that were created for the application.

$ docker compose down
Stopping trainingwheels_www_1 ... done
Stopping trainingwheels_redis_1 ... done
Removing trainingwheels_www_1 ... done
Removing trainingwheels_redis_1 ... done

Use docker compose down -v to remove everything including volumes.

containers/Compose_For_Dev_Stacks.md

Special handling of volumes

• When an image gets updated, Compose automatically creates a new container

• The data in the old container is lost...

• ...Except if the container is using a volume

• Compose will then re-attach that volume to the new container

  (and data is then retained across database upgrades)

• All good database images use volumes

  (e.g. all official images)

containers/Compose_For_Dev_Stacks.md

Gotchas with volumes

• Unfortunately, Docker volumes don't have labels or metadata

• Compose tracks volumes thanks to their associated container

• If the container is deleted, the volume gets orphaned

• Example: docker compose down && docker compose up

  • the old volume still exists, detached from its container

  • a new volume gets created

• docker compose down -v/--volumes deletes volumes

  (but not docker compose down && docker compose down -v!)

containers/Compose_For_Dev_Stacks.md

Managing volumes explicitly

Option 1: named volumes

services:
  app:
    volumes:
    - data:/some/path
volumes:
  data:

• Volume will be named <project>_data

• It won't be orphaned with docker compose down

• It will correctly be removed with docker compose down -v

containers/Compose_For_Dev_Stacks.md

Managing volumes explicitly

Option 2: relative paths

services:
  app:
    volumes:
    - ./data:/some/path

• Makes it easy to colocate the app and its data

  (for migration, backups, disk usage accounting...)

• Won't be removed by docker compose down -v

containers/Compose_For_Dev_Stacks.md

Managing complex stacks

• Compose provides multiple features to manage complex stacks

  (with many containers)

• -f/--file/$COMPOSE_FILE can be a list of Compose files

  (separated by : and merged together)

• Services can be assigned to one or more profiles

• --profile/$COMPOSE_PROFILE can be a list of comma-separated profiles

  (see Using service profiles in the Compose documentation)

• These variables can be set in .env containers/Compose_For_Dev_Stacks.md

Dependencies

• A service can have a depends_on section

  (listing one or more other services)

• This is used when bringing up individual services

  (e.g. docker compose up blah or docker compose run foo)

• It can even wait for a service to be up and ready for connections (healthy)

services:
  node:
    depends_on:
      db:
        condition: service_healthy
  db:
    image: postgres
    healthcheck:
      test: /healthchecks/postgres-healthcheck

containers/Compose_For_Dev_Stacks.md

Exercise — writing a Compose file

Let's write a Compose file for the wordsmith app!

The code is at: https://github.com/jpetazzo/wordsmith

containers/Exercise_Composefile.md

(Extra Docker content)

docker.yml

Tips for efficient Dockerfiles

We will see how to:

• Reduce the number of layers.

• Leverage the build cache so that builds can be faster.

• Embed unit testing in the build process.

containers/Dockerfile_Tips.md

Reducing the number of layers

• Each line in a Dockerfile creates a new layer.

• Build your Dockerfile to take advantage of Docker's caching system.

• Combine commands by using && to continue commands and \ to wrap lines.

Note: it is frequent to build a Dockerfile line by line:

RUN apt-get install thisthing
RUN apt-get install andthatthing andthatotherone
RUN apt-get install somemorestuff

And then refactor it trivially before shipping:

RUN apt-get install thisthing andthatthing andthatotherone somemorestuff

containers/Dockerfile_Tips.md

Avoid re-installing dependencies at each build

• Classic Dockerfile problem:

  "each time I change a line of code, all my dependencies are re-installed!"

• Solution: COPY dependency lists (package.json, requirements.txt, etc.) by themselves to avoid reinstalling unchanged dependencies every time.

containers/Dockerfile_Tips.md

Example "bad" Dockerfile

The dependencies are reinstalled every time, because the build system does not know if requirements.txt has been updated.

FROM python
WORKDIR /src
COPY . .
RUN pip install -qr requirements.txt
EXPOSE 5000
CMD ["python", "app.py"]

containers/Dockerfile_Tips.md

Fixed Dockerfile

Adding the dependencies as a separate step means that Docker can cache more efficiently and only install them when requirements.txt changes.

FROM python
WORKDIR /src
COPY requirements.txt .
RUN pip install -qr requirements.txt
COPY . .
EXPOSE 5000
CMD ["python", "app.py"]

containers/Dockerfile_Tips.md

Be careful with chown, chmod, mv

• Layers cannot store efficiently changes in permissions or ownership.

• Layers cannot represent efficiently when a file is moved either.

• As a result, operations like chown, chown, mv can be expensive.

• For instance, in the Dockerfile snippet below, each RUN line creates a layer with an entire copy of some-file.

  COPY some-file .
  RUN chown www-data:www-data some-file
  RUN chmod 644 some-file
  RUN mv some-file /var/www

• How can we avoid that?

containers/Dockerfile_Tips.md

Put files on the right place

• Instead of using mv, directly put files at the right place.

• When extracting archives (tar, zip...), merge operations in a single layer.

  Example:

  ...
  RUN wget http://.../foo.tar.gz \
   && tar -zxf foo.tar.gz \
   && mv foo/fooctl /usr/local/bin \
   && rm -rf foo foo.tar.gz
  ...

containers/Dockerfile_Tips.md

Use COPY --chown

• The Dockerfile instruction COPY can take a --chown parameter.

  Examples:

  ...
  COPY --chown=1000 some-file .
  COPY --chown=1000:1000 some-file .
  COPY --chown=www-data:www-data some-file .

• The --chown flag can specify a user, or a user:group pair.

• The user and group can be specified as names or numbers.

• When using names, the names must exist in /etc/passwd or /etc/group.

  (In the container, not on the host!)

containers/Dockerfile_Tips.md

Set correct permissions locally

• Instead of using chmod, set the right file permissions locally.

• When files are copied with COPY, permissions are preserved.

containers/Dockerfile_Tips.md

Embedding unit tests in the build process

FROM <baseimage>
RUN <install dependencies>
COPY <code>
RUN <build code>
RUN <install test dependencies>
COPY <test data sets and fixtures>
RUN <unit tests>
FROM <baseimage>
RUN <install dependencies>
COPY <code>
RUN <build code>
CMD, EXPOSE ...

• The build fails as soon as an instruction fails
• If RUN <unit tests> fails, the build doesn't produce an image
• If it succeeds, it produces a clean image (without test libraries and data)

containers/Dockerfile_Tips.md

Dockerfile examples

There are a number of tips, tricks, and techniques that we can use in Dockerfiles.

But sometimes, we have to use different (and even opposed) practices depending on:

• the complexity of our project,

• the programming language or framework that we are using,

• the stage of our project (early MVP vs. super-stable production),

• whether we're building a final image or a base for further images,

• etc.

We are going to show a few examples using very different techniques.

containers/Dockerfile_Tips.md

When to optimize an image

When authoring official images, it is a good idea to reduce as much as possible:

• the number of layers,

• the size of the final image.

This is often done at the expense of build time and convenience for the image maintainer; but when an image is downloaded millions of time, saving even a few seconds of pull time can be worth it.

RUN apt-get update && apt-get install -y libpng12-dev libjpeg-dev && rm -rf /var/lib/apt/lists/* \
    && docker-php-ext-configure gd --with-png-dir=/usr --with-jpeg-dir=/usr \
    && docker-php-ext-install gd
...
RUN curl -o wordpress.tar.gz -SL https://wordpress.org/wordpress-${WORDPRESS_UPSTREAM_VERSION}.tar.gz \
    && echo "$WORDPRESS_SHA1 *wordpress.tar.gz" | sha1sum -c - \
    && tar -xzf wordpress.tar.gz -C /usr/src/ \
    && rm wordpress.tar.gz \
    && chown -R www-data:www-data /usr/src/wordpress

(Source: Wordpress official image)

containers/Dockerfile_Tips.md

When to not optimize an image

Sometimes, it is better to prioritize maintainer convenience.

In particular, if:

• the image changes a lot,

• the image has very few users (e.g. only 1, the maintainer!),

• the image is built and run on the same machine,

• the image is built and run on machines with a very fast link ...

In these cases, just keep things simple!

(Next slide: a Dockerfile that can be used to preview a Jekyll / github pages site.)

containers/Dockerfile_Tips.md

FROM debian:sid
RUN apt-get update -q
RUN apt-get install -yq build-essential make
RUN apt-get install -yq zlib1g-dev
RUN apt-get install -yq ruby ruby-dev
RUN apt-get install -yq python-pygments
RUN apt-get install -yq nodejs
RUN apt-get install -yq cmake
RUN gem install --no-rdoc --no-ri github-pages
COPY . /blog
WORKDIR /blog
VOLUME /blog/_site
EXPOSE 4000
CMD ["jekyll", "serve", "--host", "0.0.0.0", "--incremental"]

containers/Dockerfile_Tips.md

Multi-dimensional versioning systems

Images can have a tag, indicating the version of the image.

But sometimes, there are multiple important components, and we need to indicate the versions for all of them.

This can be done with environment variables:

ENV PIP=9.0.3 \
    ZC_BUILDOUT=2.11.2 \
    SETUPTOOLS=38.7.0 \
    PLONE_MAJOR=5.1 \
    PLONE_VERSION=5.1.0 \
    PLONE_MD5=76dc6cfc1c749d763c32fff3a9870d8d

(Source: Plone official image)

containers/Dockerfile_Tips.md

Entrypoints and wrappers

It is very common to define a custom entrypoint.

That entrypoint will generally be a script, performing any combination of:

• pre-flights checks (if a required dependency is not available, display a nice error message early instead of an obscure one in a deep log file),

• generation or validation of configuration files,

• dropping privileges (with e.g. su or gosu, sometimes combined with chown),

• and more.

containers/Dockerfile_Tips.md

A typical entrypoint script

 #!/bin/sh
 set -e
 # first arg is '-f' or '--some-option'
 # or first arg is 'something.conf'
 if [ "${1#-}" != "$1" ] || [ "${1%.conf}" != "$1" ]; then
     set -- redis-server "$@"
 fi
 # allow the container to be started with '--user'
 if [ "$1" = 'redis-server' -a "$(id -u)" = '0' ]; then
     chown -R redis .
     exec su-exec redis "$0" "$@"
 fi
 exec "$@"

(Source: Redis official image)

containers/Dockerfile_Tips.md

Factoring information

To facilitate maintenance (and avoid human errors), avoid to repeat information like:

• version numbers,

• remote asset URLs (e.g. source tarballs) ...

Instead, use environment variables.

ENV NODE_VERSION 10.2.1
...
RUN ...
    && curl -fsSLO --compressed "https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION.tar.xz" \
    && curl -fsSLO --compressed "https://nodejs.org/dist/v$NODE_VERSION/SHASUMS256.txt.asc" \
    && gpg --batch --decrypt --output SHASUMS256.txt SHASUMS256.txt.asc \
    && grep " node-v$NODE_VERSION.tar.xz\$" SHASUMS256.txt | sha256sum -c - \
    && tar -xf "node-v$NODE_VERSION.tar.xz" \
    && cd "node-v$NODE_VERSION" \
...

(Source: Nodejs official image)

containers/Dockerfile_Tips.md

Overrides

In theory, development and production images should be the same.

In practice, we often need to enable specific behaviors in development (e.g. debug statements).

One way to reconcile both needs is to use Compose to enable these behaviors.

Let's look at the trainingwheels demo app for an example.

containers/Dockerfile_Tips.md

Production image

This Dockerfile builds an image leveraging gunicorn:

FROM python
RUN pip install flask
RUN pip install gunicorn
RUN pip install redis
COPY . /src
WORKDIR /src
CMD gunicorn --bind 0.0.0.0:5000 --workers 10 counter:app
EXPOSE 5000

(Source: trainingwheels Dockerfile)

containers/Dockerfile_Tips.md

Development Compose file

This Compose file uses the same image, but with a few overrides for development:

• the Flask development server is used (overriding CMD),

• the DEBUG environment variable is set,

• a volume is used to provide a faster local development workflow.

services:
  www:
    build: www
    ports:
      - 8000:5000
    user: nobody
    environment:
      DEBUG: 1
    command: python counter.py
    volumes:
      - ./www:/src

(Source: trainingwheels Compose file)

containers/Dockerfile_Tips.md

How to know which best practices are better?

• The main goal of containers is to make our lives easier.

• In this chapter, we showed many ways to write Dockerfiles.

• These Dockerfiles use sometimes diametrically opposed techniques.

• Yet, they were the "right" ones for a specific situation.

• It's OK (and even encouraged) to start simple and evolve as needed.

• Feel free to review this chapter later (after writing a few Dockerfiles) for inspiration!

Reducing image size

• In the previous example, our final image contained:

  • our hello program

  • its source code

  • the compiler

• Only the first one is strictly necessary.

• We are going to see how to obtain an image without the superfluous components.

containers/Multi_Stage_Builds.md

Can't we remove superfluous files with RUN?

What happens if we do one of the following commands?

• RUN rm -rf ...

• RUN apt-get remove ...

• RUN make clean ...

Removing files with an extra layer

When downloading an image, all the layers must be downloaded.

Therefore, RUN rm does not reduce the size of the image or free up disk space.

containers/Multi_Stage_Builds.md

Removing unnecessary files

Various techniques are available to obtain smaller images:

• collapsing layers,

• adding binaries that are built outside of the Dockerfile,

• squashing the final image,

• multi-stage builds.

Let's review them quickly.

containers/Multi_Stage_Builds.md

Collapsing layers

You will frequently see Dockerfiles like this:

FROM ubuntu
RUN apt-get update && apt-get install xxx && ... && apt-get remove xxx && ...

Or the (more readable) variant:

FROM ubuntu
RUN apt-get update \
 && apt-get install xxx \
 && ... \
 && apt-get remove xxx \
 && ...

This RUN command gives us a single layer.

The files that are added, then removed in the same layer, do not grow the layer size.

containers/Multi_Stage_Builds.md

Collapsing layers: pros and cons

Pros:

• works on all versions of Docker

• doesn't require extra tools

Cons:

• not very readable

• some unnecessary files might still remain if the cleanup is not thorough

• that layer is expensive (slow to build)

containers/Multi_Stage_Builds.md

Building binaries outside of the Dockerfile

This results in a Dockerfile looking like this:

FROM ubuntu
COPY xxx /usr/local/bin

Of course, this implies that the file xxx exists in the build context.

That file has to exist before you can run docker build.

For instance, it can:

• exist in the code repository,
• be created by another tool (script, Makefile...),
• be created by another container image and extracted from the image.

See for instance the busybox official image or this older busybox image.

containers/Multi_Stage_Builds.md

Building binaries outside: pros and cons

Pros:

• final image can be very small

Cons:

• requires an extra build tool

• we're back in dependency hell and "works on my machine"