Using Docker in CI/CD

Transcript

Hello, welcome to this tech talk session about using Docker and CI/CD. So, about Docker and CI/CD, a quick introduction: Docker is a platform for developing, shipping and running applications in a consistent environment. That means you can run your project everywhere on your laptop, on a server, on the cloud, and so on. CI/CD is continuous integration and continuous delivery and deployment. It’s a set of practices that consist to automate everything. For example, you update your code, you commit it, and push it, and then the build and testing are done automatically as well in deployment.

Table of Contents

• Why use Docker in CI/CD (0:41) (0:21)
• Development Workflow & CI/CD (1:08)
• Improve your pipelines (3:55)
• Improve the DX with Docker Compose (7:14)
• Make your deliverables efficient (8:32)
• Building images from the CI/CD (10:52)
• Help your CI (11:12)
• Learn more

Why Use Docker in CI/CD (0:41)

I said you can run your project everywhere. You can stop it, restart it in a glitch, and do that a hundred times a day. So you can do the same with your CI/CD system. And then you add the guarantee you can reproduce easily your running environment at each step of your CI/CD pipeline. And with automation, it’s easy to test, retry, test again, and test again and retire again. But first, let’s have a look at a development workflow.

Development Workflow & CI/CD (1:08)

This is an example of a development workflow. I call it feature flow, others call it GitLab flow. It works on GitLab, GitLab, Bitbucket, and so on. The main branch is the production branch. As a developer, when you want to do a feature or a fix, you will create a feature branch from the main branch. Add every commit and update of your pull request, or merge request if you are using GitHub. The CI will be triggered with a batch of quality built-in security chips. And at the end, when you will mount on the main branch, the CI will be triggered again for example, to package and deploy the application. When you commit, the CI pipeline needs to bootstrap an environment at every step. When you merge, the CI will need to bootstrap a preproduction and production environment. And you must avoid differences in all these environments. And often, it’s complicated and very long, depending on the CI/CD platform or source system. Keeping everything in sync is hard. The feedback loop is very hard and very complicated, and the developer experience could be terrible.

This is a quick reminder of GitLab CI. GitLab CI is the name of the CI/CD feature of GitLab, it’s like Jenkins or GitHub Action. The GitLab runners are the servers that execute the workflow. You can install the runners everywhere, even locally, or on a server, or on Kubernetes for example. The operation is very simple. The developer commits and push the changes to the GitLab instance. The runners are polling the GitLab instance. When a runner detects there is something to do, it Git Clone the project. Then the runner executes the command described in the .gitlab-ci.yml file, this file is in the repository. And the runners send feedback to the GitLab instance.

Every job of a pipeline is executed by an executor. It’s a golang program installed in the runner. The executor can execute the command of a job on a bare metal platform, a VM and container. In this list, I have two jobs belonging to the same stage. For both of these two jobs, I’m telling the executor that I will use the Ubuntu image and then all the commands will be run into a Ubuntu container. And I can install what I want.

So, what is important to remember? You can run pipeline jobs in containers. Each job can use a different image. And then it’s easy to provide a ready-to-use environment with all your dependencies for every job. And of course, it facilitates the test.

Improve your pipelines (3:55)

Now, let’s see how Docker can help to improve the CI. You can choose the image you need for a job. And you have complete control of the operating system for every job. Remember, it’s bad to use the latest tag. In this example, I start from a base image. Ubuntu, for example. But sometimes, it could be interesting to provide an image with all the tools you need. In this example, I use one of my personal images with go and node js already installed. Then you don’t need to install these tools. They are already installed into the image. And then you will streamline, smooth and accelerate the pipeline. And you have the guarantee that everyone is using the same tools. With the same version, with the same behavior. You will accelerate the pipeline by using ready-to-use images with all the tools preinstalled.

In my example, before using the tools image, the duration was around 2 minutes and 30 seconds. And now with my tools image, I moved on to just a minute. But when you are creating an image, you can be cleverer than me. And don’t use a huge generic image that can do everything. Select relevant based images for every job and choose a smaller image. In this example, I selected for every job an image dedicated to a specific context. I know that these images embed only the necessary binaries and dependencies. So these images are optimized and lighter. So you will accelerate your pipeline. And you will make it easier to reuse jobs and images for other pipelines. This time, with appropriate images, I moved on to 40 seconds. If you run the jobs very often it makes the difference.

You can do better. I selected a smaller golang image then the pool of the image will be faster. So this time, I moved on to 30 seconds. And in addition, by reducing the attack surface with a smaller image, we have improved the security of the images. So keep these best practices in mind and create your own best practices with the experience. Use or build small images. This will speed up the load time. It’s the topic of the Image Deep Dive tech talk presentation. If you use the multi-stage build technique, you will produce smaller images and it will facilitate the deployment. Don’t forget to version your images and to use the versioned images to maintain consistency and avoid surprises.

By using Docker in CI/CD, you will gain several benefits. Portability, if you need to move your CI on a new environment or even change your CI system, your pain will be less. Isolation, you will avoid the side effect. Efficiency, you will do the same task with less energy. Scalability, if you need to add a component in the chain, it will be easier and more manageable. And of course, security.

Improve the DX with Docker Compose (7:14)

And to help even more on this, we have a nice Docker feature– Docker Compose. For example, I’m working on a Compose project and I use NodeJS as the application server and POSTGRESQL as the database. With Docker Compose, bootstrapping on demand, this kind of project is pretty straightforward. You only need a single and simple Compose file. And it’s really, really easy to use with GitLab CI and other CI/CD systems. You can use a Docker image with the Docker tools for a job. Of course, pin a specific version of the image to avoid compatibility problems. To use Docker in the job script, you should include the Docker DND service, Docker in Docker. And now you can forget the complicated plumbing of the pipeline. This solution streamlines the management of multiple containers and their interactions into a GitLab CI pipeline. With this example, I will run the unit tests of the application. And you will be able to follow the test result with the output of the runner or with the test report of GitLab if you generate the appropriate artifact.

Make your deliverables efficient (8:32)

Now, let’s talk about some best practices. First, do not include the secrets into the images, mount the secret as a volume, use environment variables or use a vault. Use only trusted content from Docker and from your organization. For example, at Docker, we have three kinds of trusted content. Docker official image, verify publisher and sponsored open source project. Please don’t use the latest tag when you use an image. It’s too dangerous.

Keep your Docker images slim and efficient. Only include what is absolutely necessary for your application and organize the image layers wisely to make them reusable. This not only shrinks the image size but also makes your builder faster. Plus, embrace multi-stage builds, they are a great tool to keep everything streamlined and simple.

So, let’s talk a bit more about the multi-stage builds. This is my second favorite feature of Docker, the first one is Compose. Using multi-stage builds is like defining a pipeline into a Dockerfile. The Dockerfile on this slide is split in two steps. At the first step, it builds a golang application with the official golang image. The size of this image is about 800 megabytes. Then, once the application built, we use a minimal scratch image to copy the binary of the first stage into the final image. In the end, the size of the image is only about 20 megabytes. Of course, the size can change depending on your application and complexity. Thanks to this, you will have benefits. Reduced image size. So, it improves the load times and storage efficiency. You will have faster rebuilds. And you will improve security because a smaller image has minimal attack surface. And security is a crucial point with CI/CD. You need to be sure that your images do not have any vulnerabilities.

Building images from the CI/CD (10:52)

Regarding the build and the publication of Docker images from CI/CD, the job is pretty straightforward. I won’t spend time on this part because GitLab did a nice documentation to explain the various ways to build the Docker images from GitLab CI.

Help your CI (11:12)

A few more words about CI. So, about cache management. Build Kits speeds up builds by caching results. Build Kits is the default built engine for Docker images. It saves previous work to avoid repetition. Then you can store the cache externally for later use. This is especially useful in CI/CD environment where builds start from scratch each time, but need to stay fast.

Here you can see some cache examples. So, you can build and push an image and store the cache in a registry. So, you will store a part of the build in a registry and you will be able to reuse this part at the next build. The second example is the usage of an object storage, the S3 bucket, for example, with the Docker build.

And that’s the end, thank you for your attention

Learn more

• Check out the docs on Docker Build and Docker Compose.
• New to Docker? Get started.
• Deep dive into Docker products with free learning paths.
• Subscribe to the Docker Newsletter.
• Get the latest release of Docker Desktop.
• Have questions? The Docker community is here to help.