Search by Tags

Speed up Build of Debian Based Images for Torizon

 
Applicable for

Article updated at 23 Apr 2019
Subscribe for this article updates

Introduction

If you need to build a debian container and want to reduce and customize the set of packages you install using apt-get this can be time consuming, because every time you re-run docker build the packages will be re-downloaded from scratch.

You can mitigate the issue by adding multiple apt-get commands in the container, keeping in the docker cache those that you already executed. This has two negative side effects:

  1. The Dockerfile will get more complex, requires more build/commit steps and getting a full list of the packages you add can be complicated.
  2. If you don't clear /var/lib/apt/lists/* the size of your container will increase and if you do it after every apt-get execution you should also re-execute apt-get update at every new step.

An alternative solution is to run a proxy on your host PC. In this way apt-get will download packages from the local cache, speeding up the whole operation and allowing you to keep experimenting with a single apt-get update / install / cache cleanup statement inside your container.

Prerequisites

The following prerequisites are necessary for this article:

The instructions are applicable for Debian-based machines. Proceedings may vary for other distros.

Install Proxy Server

On a debian-based machine you can just install the squid-deb-proxy package:

$ sudo apt-get install squid-deb-proxy

If you are running Ubuntu or another debian-based distro that does not download packages from debian.org you have to change squid-deb-proxy configuration.

Change the following code snippet from:

/etc/squid-deb-proxy/squid-deb-proxy.conf
# this file contains the archive mirrors by default, # if you use a different mirror, add it there acl to_archive_mirrors dstdomain "/etc/squid-deb-proxy/autogenerated/mirror-dstdomain.acl"

To:

/etc/squid-deb-proxy/squid-deb-proxy.conf
# this file contains the archive mirrors by default, # if you use a different mirror, add it there acl to_archive_mirrors dstdomain "/etc/squid-deb-proxy/mirror-dstdomain.acl"

Add the following lines to /etc/squid-deb-proxy/mirror-dstdomain.acl:

/etc/squid-deb-proxy/mirror-dstdomain.acl
#debian .debian.org

Restart squid-deb-proxy:

 
$ sudo service squid-deb-proxy restart
 

You will also need to add some code do your Dockerfile to redirect apt-get requests to the proxy.
This sample uses a command line argument named APT_PROXY.

At the beginning of your Dockerfile you need to add the argument and check if it has a value, this will allow your container to be safely built also on systems that don't have the proxy installed.

Dockerfile
ARG APT_PROXY #if argument APT_PROXY is configured, it will be used to speed-up download of deb packages RUN if [ "$APT_PROXY" != "" ]; then \ echo "Acquire::http::Proxy \"http://$APT_PROXY:8000\";" > /etc/apt/apt.conf.d/30proxy ;\ echo "Acquire::http::Proxy::ppa.launchpad.net DIRECT;" >> /etc/apt/apt.conf.d/30proxy ; \ echo "squid-deb-proxy configured"; \ else \ echo "no squid-deb-proxy configured"; \ fi

Then you can run apt-get and install your packages, typically in this way:

Dockerfile
RUN apt-get -q -y update && apt-get -q -y --no-install-recommends install systemd dbus && rm -rf /var/lib/apt/lists/*

This command will update the local package database, install required packages (limiting the dependencies to the mandatory ones) and then cleanup cache folder.
After this we should remove our proxy configuration to avoid issues if apt-get is run inside the container on a different host.

Dockerfile
# remove proxy so container will run on any host RUN if [ "$APT_PROXY" != "" ]; then rm /etc/apt/apt.conf.d/30proxy; fi

To pass the argument to command line you should add a --build-arg entry to your docker build command line:

 
$ docker build --rm -t myimagename:latest --build-arg APT_PROXY=$APT_PROXY_ARG .
 

If you build your container using a shell script (this is anyway a good idea to avoid mistakes like misspelling the tags or forgetting some additional parms) you can add this code to automatically detect if squid-deb-proxy is running on your machine and set the argument accordingly:

build.sh
# detects if we have squid-deb-proxy running and set APT_PROXY_ARG LOCAL_IP=$(hostname -I | awk '{print $1}') echo "HEAD /" | nc $LOCAL_IP 8000 | grep squid-deb-proxy >> /dev/null # proxy is running if [ $? -eq 0 ] then echo "squid-deb-proxy is running" APT_PROXY_ARG=$LOCAL_IP else echo "squid-deb-proxy is not running, enable it if you want to cache packages used in containers." APT_PROXY_ARG="" fi