Conan: A Python package manager

Conan is a C and C++ package manager, and to deal with the vast variability of C and C++ build systems, compilers, configurations, etc., it was designed with a great flexibility in mind, trying to let the users do almost what they want. This is one of the reasons to use Python as the scripting language for Conan package recipes.

With this flexibility, Conan is able to do very different tasks: package Visual Studio modules, package Go code, build packages from sources or from binaries retrieved from elsewhere, etc.

Python code can be reused and packaged with Conan to share functionalities or tools among conanfile.py files. Here we can see a full example of Conan as a Python package manager.

A full Python and C/C++ package manager

The real utility of this is that Conan is a C and C++ package manager. So if we want to create a python package that wraps the functionality of, lets say the Poco C++ library, it can be easily done. Poco itself has transitive (C/C++) dependencies, but they are already handled by Conan. Furthermore, a very interesting thing is that nothing has to be done in advance for that library, thanks to useful tools as pybind11, that allows to create python bindings easily.

So let’s build a package with the following files:

conanfile.py: The package recipe.
__init__.py: necessary file, blank.
pypoco.cpp: The C++ code with the pybind11 wrapper for Poco that generates a python extension (a shared library that can be imported from python).
CMakeLists.txt: cmake build file that is able to compile pypoco.cpp into a Python extension (pypoco.pyd in Windows, pypoco.so in Linux)
poco.py: A Python file that makes use of the pypoco Python binary extension built with pypoco.cpp.
test_package/conanfile.py: A test consumer recipe for convenience to create and test the package.

The pypoco.cpp file can be coded easily thanks to the elegant pybind11 library:

pypoco.cpp

 #include <pybind11/pybind11.h>
 #include "Poco/Random.h"

 using Poco::Random;
 namespace py = pybind11;

 PYBIND11_PLUGIN(pypoco) {
     py::module m("pypoco", "pybind11 example plugin");
     py::class_<Random>(m, "Random")
         .def(py::init<>())
         .def("nextFloat", &Random::nextFloat);
     return m.ptr();
 }

And the poco.py file is straigthforward:

poco.py

 import sys
 import pypoco

 def random_float():
     r = pypoco.Random()
     return r.nextFloat()

The conanfile.py has a few more lines than the above, but it is still quite easy to understand:

conanfile.py

 from conans import ConanFile, tools, CMake

 class PocoPyReuseConan(ConanFile):
     name = "PocoPy"
     version = "0.1"
     requires = "Poco/1.9.0@pocoproject/stable", "pybind11/any@memsharded/stable"
     settings = "os", "compiler", "arch", "build_type"
     exports = "*"
     generators = "cmake"
     build_policy = "missing"

     def build(self):
         cmake = CMake(self)
         pythonpaths = "-DPYTHON_INCLUDE_DIR=C:/Python27/include -DPYTHON_LIBRARY=C:/Python27/libs/python27.lib"
         self.run('cmake %s %s -DEXAMPLE_PYTHON_VERSION=2.7' % (cmake.command_line, pythonpaths))
         self.run("cmake --build . %s" % cmake.build_config)

     def package(self):
         self.copy('*.py*')
         self.copy("*.so")

     def package_info(self):
         self.env_info.PYTHONPATH.append(self.package_folder)

The recipe now declares 2 requires, the Poco library and the pybind11 library that we will be using to create the binary extension.

As we are actually building some C++ code, we need a few important things:

Input settings that define the OS, compiler, version and architecture we are using to build our extension. This is necessary because the binary we are building must match the architecture of the python interpreter that we will be using.
The build() method is used actually to invoke CMake. See we had to hardcode the python path in the example, as the CMakeLists.txt call to find_package(PythonLibs) didn’t find my python installed in C:/Python27, quite a standard path. I have added the cmake generator too to be able to easily use the declared requires build information inside my CMakeLists.txt.
The CMakeLists.txt is not posted here, but is basically the one used in the pybind11 example with just 2 lines to include the conan generated cmake file for dependencies. It can be inspected in the github repo.
Note that we are using Python 2.7 as an input option. If necessary, more options for other interpreters/architectures could be easily provided, as well as avoiding the hardcoded paths. Even the Python interpreter itself could be packaged in a conan package.

The above recipe will generate a different binary for different compilers or versions. As the binary is being wrapped by python, we could avoid this and use the same binary for different setups, modifying this behavior with the conan_info() method.

$ conan export . memsharded/testing
$ conan install PocoPy/0.1@memsharded/testing -s arch=x86 -g virtualenv
$ activate
$ python
>>> import poco
>>> poco.random_float()
0.697845458984375

Now the first invocation of conan install will build retrieve the dependencies and build the package. The next invocation will use the cached binaries and be much faster. Note how we have to specify -s arch=x86 to build matching the architecture of the python interpreter to be used, in our case, 32 bits.

We can also read in the output of the conan install the dependencies that are being pulled:

Requirements
    OpenSSL/1.0.2l@conan/stable from conan.io
    Poco/1.9.0@pocoproject/stable from conan.io
    PocoPy/0.1@memsharded/testing from local
    pybind11/any@memsharded/stable from conan.io
    zlib/1.2.11@conan/stable from conan.io

This is the great thing about using Conan for this task, by depending on Poco, other C and C++ transitive dependencies are being retrieved and used in the application.

If you want to have a further look to the code of these examples, you can check this github repo. The above examples and code have been tested only in Win10, VS14u2, but might work with other configurations with little or no extra work.