Creating a Conan package using a Docker runner¶
Warning
This feature is experimental and subject to breaking changes. See the Conan stability section for more information.
First of all you need to have the Docker daemon installed and running, plus Conan and the docker Python package. This tutorial assumes that you are running Conan inside a Python virtual environment, skip the first line if you already have the docker Python package installed in your virtual environment.
# install docker in your virtual environment if you don't have it already installed
$ pip install conan docker
$ docker ps
$ CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
Now we are going to create create simple cmake_lib Conan template to later run inside Docker using the runner feature. Let’s create the Conan package and a Dockerfile inside our project folder.
$ cd </my/runner/folder>
$ mkdir mylib
$ cd mylib
$ conan new cmake_lib -d name=mylib -d version=0.1
$ tree
.
├── CMakeLists.txt
├── conanfile.py
├── include
│ └── mylib.h
├── src
│ └── mylib.cpp
└── test_package
├── CMakeLists.txt
├── conanfile.py
└── src
└── example.cpp
Dockerfile
FROM ubuntu:22.04
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
build-essential \
cmake \
python3 \
python3-pip \
python3-venv \
&& rm -rf /var/lib/apt/lists/*
RUN pip install conan
$ cd </my/runner/folder>/mylib
$ tree
.
...
├── Dockerfile
...
Now, we need to define two new profiles inside the conan profiles folder. Replace </my/runner/folder> with your real project folder path.
docker_example_host profile
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux
[runner]
type=docker
dockerfile=</my/runner/folder>/mylib
cache=copy
remove=true
docker_example_build profile
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux
We are going to start from a totally clean environment, without any containers, images or conan package.
$ conan list "*:*"
Found 0 pkg/version recipes matching * in local cache
$ docker ps --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
Now, it’s time to create our library mylib using our new runner definition.
$ conan create . --version 0.1 -pr:h docker_example_host -pr:b docker_example_build
If we split and analyze the command output, we can see what is happening and where the commands are being executed.
1. Standard conan execution.
======== Exporting recipe to the cache ========
mylib/0.1: Exporting package recipe: </my/runner/folder>/mylib/conanfile.py
mylib/0.1: Copied 1 '.py' file: conanfile.py
mylib/0.1: Copied 1 '.txt' file: CMakeLists.txt
mylib/0.1: Copied 1 '.h' file: mylib.h
mylib/0.1: Copied 1 '.cpp' file: mylib.cpp
mylib/0.1: Exported to cache folder: /Users/davidsanfal/.conan2/p/mylib4abd06a04bdaa/e
mylib/0.1: Exported: mylib/0.1#8760bf5a311f01cc26f3b95428203210 (2024-07-08 12:22:01 UTC)
======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux
Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux
2. Build docker image
**********************************************
* Building the Docker image: my-conan-runner *
**********************************************
Dockerfile path: '</my/runner/folder>/mylib/Dockerfile'
Docker build context: '</my/runner/folder>/mylib'
Step 1/4 : FROM ubuntu:22.04
...
---> 2bcf70201cce
Successfully built 2bcf70201cce
Successfully tagged conan-runner-default:latest
3. Save the local cache running conan cache save.
***********************************************************************************
* Save host cache in: </my/runner/folder>/mylib/.conanrunner/local_cache_save.tgz *
***********************************************************************************
Found 1 pkg/version recipes matching * in local cache
Saving mylib/0.1: mylib4abd06a04bdaa
4. Create and initialize the docker container.
*********************************
* Creating the docker container *
*********************************
*****************************************
* Container conan-runner-docker running *
*****************************************
5. Check if the container has a conan version with the runner feature.
*******************************************
* Running in container: "conan --version" *
*******************************************
Conan version 2.5.0
6. Initialize the container conan cache using the host copy running conan cache restore.
***********************************************************************************************************
* Running in container: "conan cache restore "/root/conanrunner/mylib/.conanrunner/local_cache_save.tgz"" *
***********************************************************************************************************
Restore: mylib/0.1 in mylib4abd06a04bdaa
Local Cache
mylib
mylib/0.1
revisions
8760bf5a311f01cc26f3b95428203210 (2024-07-08 12:22:19 UTC)
packages
recipe_folder: mylib4abd06a04bdaa
7. Run the conan create inside the container and build “mylib”.
*********************************************************************************************************************************************************
* Running in container: "conan create /root/conanrunner/mylib --version 0.1 -pr:h docker_example_host -pr:b docker_example_build -f json > create.json" *
*********************************************************************************************************************************************************
======== Exporting recipe to the cache ========
mylib/0.1: Exporting package recipe: /root/conanrunner/mylib/conanfile.py
mylib/0.1: Copied 1 '.py' file: conanfile.py
mylib/0.1: Copied 1 '.txt' file: CMakeLists.txt
mylib/0.1: Copied 1 '.cpp' file: mylib.cpp
mylib/0.1: Copied 1 '.h' file: mylib.h
mylib/0.1: Exported to cache folder: /root/.conan2/p/mylib4abd06a04bdaa/e
mylib/0.1: Exported: mylib/0.1#8760bf5a311f01cc26f3b95428203210 (2024-07-08 12:22:20 UTC)
======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux
Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux
======== Computing dependency graph ========
Graph root
cli
Requirements
mylib/0.1#8760bf5a311f01cc26f3b95428203210 - Cache
======== Computing necessary packages ========
mylib/0.1: Forced build from source
Requirements
mylib/0.1#8760bf5a311f01cc26f3b95428203210:8631cf963dbbb4d7a378a64a6fd1dc57558bc2fe - Build
======== Installing packages ========
-------- Installing package mylib/0.1 (1 of 1) --------
...
[ 50%] Building CXX object CMakeFiles/example.dir/src/example.cpp.o
[100%] Linking CXX executable example
[100%] Built target example
======== Testing the package: Executing test ========
mylib/0.1 (test package): Running test()
mylib/0.1 (test package): RUN: ./example
mylib/0.1: Hello World Release!
mylib/0.1: __x86_64__ defined
mylib/0.1: _GLIBCXX_USE_CXX11_ABI 1
mylib/0.1: __cplusplus201703
mylib/0.1: __GNUC__11
mylib/0.1: __GNUC_MINOR__4
mylib/0.1 test_package
8. Copy just the package created inside the container using the pkglist.json info from the previous conan create, restore this new package inside the host cache running a conan cache save and remove the container.
************************************************************************************************************************************
* Running in container: "conan cache save --list=pkglist.json --file "/root/conanrunner/mylib"/.conanrunner/docker_cache_save.tgz" *
************************************************************************************************************************************
Saving mylib/0.1: mylib4abd06a04bdaa
Saving mylib/0.1:8631cf963dbbb4d7a378a64a6fd1dc57558bc2fe: b/mylib503035e4ee8ae/p
Saving mylib/0.1:8631cf963dbbb4d7a378a64a6fd1dc57558bc2fe metadata: b/mylib503035e4ee8ae/d/metadata
Local Cache
mylib
mylib/0.1
revisions
8760bf5a311f01cc26f3b95428203210 (2024-07-08 12:22:20 UTC)
packages
8631cf963dbbb4d7a378a64a6fd1dc57558bc2fe
revisions
ded6547554ff2306db5250451340fa43
package_folder: b/mylib503035e4ee8ae/p
metadata_folder: b/mylib503035e4ee8ae/d/metadata
info
settings
os: Linux
arch: x86_64
compiler: gcc
compiler.cppstd: gnu17
compiler.libcxx: libstdc++11
compiler.version: 11
build_type: Release
options
fPIC: True
shared: False
recipe_folder: mylib4abd06a04bdaa
******************************************************************************************
* Restore host cache from: </my/runner/folder>/mylib/.conanrunner/docker_cache_save.tgz *
******************************************************************************************
Restore: mylib/0.1 in mylib4abd06a04bdaa
Restore: mylib/0.1:8631cf963dbbb4d7a378a64a6fd1dc57558bc2fe in b/mylib503035e4ee8ae/p
Restore: mylib/0.1:8631cf963dbbb4d7a378a64a6fd1dc57558bc2fe metadata in b/mylib503035e4ee8ae/d/metadata
**********************
* Stopping container *
**********************
**********************
* Removing container *
**********************
If we now check the status of our conan and docker cache, we will see the new mylib package compile for Linux and the new docker image but we don’t have any container because we define remove=true
$ conan list "*:*"
Found 1 pkg/version recipes matching * in local cache
Local Cache
mylib
mylib/0.1
revisions
8760bf5a311f01cc26f3b95428203210 (2024-07-08 12:33:28 UTC)
packages
8631cf963dbbb4d7a378a64a6fd1dc57558bc2fe
info
settings
arch: x86_64
build_type: Release
compiler: gcc
compiler.cppstd: gnu17
compiler.libcxx: libstdc++11
compiler.version: 11
os: Linux
options
fPIC: True
shared: False
$ docker ps --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
my-conan-runner latest 2bcf70201cce 11 minutes ago 531MB
What we have just done is to compile a library from scratch inside a Docker container without running any Docker command and retrieve the generated packages in a totally transparent and easily debuggable way thanks to our terminal output.
In this way, we can work as we have always done regardless of whether it is on our machine or in a container, without several open terminals and having the result of each operation in the same cache, being able to reuse the compiled packages from a previous compilation in another container automatically and transparently.