Conan: A Python package manager

Conan is a C and C++ package manager, and to deal with the vast variability of C and C++ build systems, compilers, configurations, etc., it was designed to be extremely flexible, to allow users the freedom to configure builds in virtually any manner required. This is one of the reasons to use Python as the scripting language for Conan package recipes.

With this flexibility, Conan is able to do very different tasks: package Visual Studio modules, package Go code, build packages from sources or from binaries retrieved from elsewhere, etc.

Python code can be reused and packaged with Conan to share functionalities or tools among conanfile.py files. Here we can see a full example of Conan as a Python package manager.

A full Python and C/C++ package manager

The real utility of this is that Conan is a C and C++ package manager. So, for example, you are able to create a Python package that wraps the functionality of the Poco C++ library. Poco itself has transitive (C/C++) dependencies, but they are already handled by Conan. Furthermore, a very interesting thing is that nothing has to be done in advance for that library, thanks to useful tools such as pybind11, that lets you easily create Python bindings.

So let’s build a package with the following files:

conanfile.py: The package recipe.
__init__.py: A required file which should remain blank.
pypoco.cpp: The C++ code with the pybind11 wrapper for Poco that generates a Python extension (a shared library that can be imported from Python).
CMakeLists.txt: The CMake build file that is able to compile pypoco.cpp into a Python extension (pypoco.pyd in Windows, pypoco.so in Linux)
poco.py: A Python file that makes use of the pypoco Python binary extension built with pypoco.cpp.
test_package/conanfile.py: A test consumer “convenience” recipe to create and test the package.

The pypoco.cpp file can be coded easily thanks to the elegant pybind11 library:

pypoco.cpp

 #include <pybind11/pybind11.h>
 #include "Poco/Random.h"

 using Poco::Random;
 namespace py = pybind11;

 PYBIND11_PLUGIN(pypoco) {
     py::module m("pypoco", "pybind11 example plugin");
     py::class_<Random>(m, "Random")
         .def(py::init<>())
         .def("nextFloat", &Random::nextFloat);
     return m.ptr();
 }

And the poco.py file is straightforward:

poco.py

 import sys
 import pypoco

 def random_float():
     r = pypoco.Random()
     return r.nextFloat()

The conanfile.py is a bit longer, but is still quite easy to understand:

conanfile.py

 from conans import ConanFile, tools, CMake

 class PocoPyReuseConan(ConanFile):
     name = "PocoPy"
     version = "0.1"
     requires = "Poco/1.9.0@pocoproject/stable", "pybind11/any@memsharded/stable"
     settings = "os", "compiler", "arch", "build_type"
     exports = "*"
     generators = "cmake"
     build_policy = "missing"

     def build(self):
         cmake = CMake(self)
         pythonpaths = "-DPYTHON_INCLUDE_DIR=C:/Python27/include -DPYTHON_LIBRARY=C:/Python27/libs/python27.lib"
         self.run('cmake %s %s -DEXAMPLE_PYTHON_VERSION=2.7' % (cmake.command_line, pythonpaths))
         self.run("cmake --build . %s" % cmake.build_config)

     def package(self):
         self.copy('*.py*')
         self.copy("*.so")

     def package_info(self):
         self.env_info.PYTHONPATH.append(self.package_folder)

The recipe now declares 2 requires that will be used to create the binary extension: the Poco library and the pybind11 library.

As we are actually building C++ code, there are a few important things that we need:

Input settings that define the OS, compiler, version and architecture we are using to build our extension. This is necessary because the binary we are building must match the architecture of the Python interpreter that we will be using.
The build() method is actually used to invoke CMake. You may see that we had to hardcode the Python path in the example, as the CMakeLists.txt call to find_package(PythonLibs) didn’t find my Python installation in C:/Python27, even though that is a standard path. I have also added the cmake generator to be able to easily use the declared requires build information inside my CMakeLists.txt.
The CMakeLists.txt is not posted here, but is basically the one used in the pybind11 example with just 2 lines to include the cmake file generated by Conan for dependencies. It can be inspected in the GitHub repo.
Note that we are using Python 2.7 as an input option. If necessary, more options for other interpreters/architectures could be easily provided, as well as avoiding the hardcoded paths. Even the Python interpreter itself could be packaged in a Conan package.

The above recipe will generate a different binary for different compilers or versions. As the binary is being wrapped by Python, we could avoid this and use the same binary for different setups, modifying this behavior with the conan_info() method.

$ conan export . memsharded/testing
$ conan install PocoPy/0.1@memsharded/testing -s arch=x86 -g virtualenv
$ activate
$ python
>>> import poco
>>> poco.random_float()
0.697845458984375

Now, the first invocation of conan install will retrieve the dependencies and build the package. The next invocation will use the cached binaries and be much faster. Note how we have to specify -s arch=x86 to match the architecture of the Python interpreter to be used, in our case, 32 bits.

The output of the conan install command also shows us the dependencies that are being pulled:

Requirements
    OpenSSL/1.0.2l@conan/stable from conan.io
    Poco/1.9.0@pocoproject/stable from conan.io
    PocoPy/0.1@memsharded/testing from local
    pybind11/any@memsharded/stable from conan.io
    zlib/1.2.11@conan/stable from conan.io

This is one of the great advantages of using Conan for this task, because by depending on Poco, other C and C++ transitive dependencies are retrieved and used in the application.

For a deeper look into the code of these examples, please refer to this github repo. The above examples and code have only been tested on Win10, VS14u2, but may work on other configurations with little or no extra work.