Packages in editable mode

Warning

This is an experimental feature subject to breaking changes in future releases.

When working in big projects with several functionalities interconnected it is recommended to avoid the one-and-only huge project approach in favor of several libraries, each one specialized in a set of common tasks, even maintained by dedicated teams. This approach helps to isolate and reusing code helps with compiling times and reduces the likelihood of including files that not correspond to the API of the required library.

Nevertheless, in some case, it is useful to work in several libraries at the same time and see how the changes in one of them are propagated to the others. Following the local workflow an user can execute the commands conan source, conan install, conan build and conan package, but in order to get the changes ready for a consumer library, it is needed the conan create that will actually trigger a build to generate the binaries in the cache or to run conan export-pkg to copy locally built artifacts into the conan cache and make them available to consumers.

With the editable packages, you can tell Conan where to find the headers and the artifacts ready for consumption in your local working directory. There is no need to package.

Let’s see this feature over an practical example; the code can be found in the conan examples repository:

$ git clone https://github.com/conan-io/examples.git

In the project examples/features/editable/cmake a developer is creating the app hello but at the same time they want to work on say/0.1@user/channel library which is tightly coupled to the app.

The package say/0.1@user/channel is already working, the developer has the sources in a local folder and they are using whatever method to build and develop locally and can perform a conan create . say/0.1@user/channel to create the package.

Also, there is a conanfile.txt (or a more complex recipe) for the application hello that has say/0.1@user/channel among its requirements. When building this application, the resources of say are used from the Conan local cache.

Put a package in editable mode

To avoid creating the package say/0.1@user/channel in the cache for every change, we are going to put that package in editable mode, creating a link from the reference in the cache to the local working directory:

$ conan editable add examples/features/editable/cmake/say say/0.1@user/channel
# you could do "cd <examples/features/editable/cmake/say> && conan editable add . say/0.1@user/channel"

That is it. Now, every usage of say/0.1@user/channel, by any other Conan package or project, will be redirected to the examples/features/editable/cmake/say user folder instead of using the package from the conan cache.

The Conan package recipes define a package “layout” in their package_info() methods. The default one, if nothing is specified is equivalent to:

def package_info(self):
    # default behavior, doesn't need to be explicitly defined in recipes
    self.cpp_info.includedirs = ["include"]
    self.cpp_info.libdirs = ["lib"]
    self.cpp_info.bindirs = ["bin"]
    self.cpp_info.resdirs = ["res"]

That means that conan will use the path examples/features/editable/cmake/say/include for locating the headers of the say package, the examples/features/editable/cmake/say/lib to locate the libraries of the package, and so on.

That might not be very useful, as typically while editing the source code and doing incremental builds, the development layout is different from that final “package” layout. While it is possible to run a conan package local command to execute the packaging in the user folder and it will achieve that final layout, that is not very elegant as it should be run after every modification.

In order to populate cpp_info.libs, the usage of tools.collect_libs() is discouraged as it won’t find any library when the package is in editable mode and it hasn’t been compiled yet. This empty list will be written to files by generators and it won’t be updated after working on the editable package.

Editable packages layouts

The custom layout of a package while it is in editable mode can be defined in different ways:

Recipe layout() method

The layout() method allows to declare cpp_info for the self.source_folder and self.build_folder. You can read more about it in the package layout documentation page. If the method layout() is used, any other layout declaration will be ignored.

Example:

from conans import ConanFile

class Pkg(ConanFile):

    def layout(self):
        self.cpp.source.includedirs = ["include"]
        self.cpp.build.libdirs = ["."]
        self.cpp.build.libs = ["mylib"]
        self.cpp.build.includedirs = ["gen_include"]
        self.cpp.package.libs = ["mylib"]

NOTE: The layout() feature will be fully functional only in the new build system integrations (in the conan.tools space). If you are using other integrations, they might not fully support this feature.

Recipe package_info()

A recipe can define a custom layout when it is not living in the local cache, in its package_info() method, something like:

from conans import ConanFile

class Pkg(ConanFile):
    settings = "build_type"
    def package_info(self):
        if not self.in_local_cache:
            d = "include_%s" % self.settings.build_type
            self.cpp_info.includedirs = [d.lower()]

That will map the include directories to examples/features/editable/cmake/say/include_debug when working with build_type=Debug conan setting, and to examples/features/editable/cmake/say/include_release if build_type=Release. In the same way, other directories (libdirs, bindirs, etc) can be customized, with any logic, different for different OS, build systems, etc.

from conans import ConanFile

class Pkg(ConanFile):
    settings = "os", "compiler", "arch", "build_type"
    def package_info(self):
        if not self.in_local_cache:
            if self.settings.compiler == "Visual Studio":
                # NOTE: Use the real layout used in your VS projects, this is just an example
                self.cpp_info.libdirs = ["%s_%s" % (self.settings.build_type, self.settings.arch)]

That will define the libraries directories to examples/features/editable/cmake/say/Release_x86_64, for example. That is only an example, the real layout used by VS would be different.

Layout external files

Instead of changing the recipe file to match the local layout, it’s possible to define the layout in a separate file. This is especially useful if you have a large number of libraries with the same structure so you can write it once and use it for several packages.

Layout files are ini files, but before parsing them Conan uses the Jinja2 template engine passing the settings, options and current reference objects, so you can add logic to the files:

[build_folder]
build/{{settings.build_type}}

[source_folder]
src

[includedirs]
src

[libdirs]
build/{{settings.build_type}}/lib

You can have a look at the Jinja2 documentation to know more about its powerful syntax.

This file can use the package reference to customize logic for a specific package:

[say/0.1@user/channel:build_folder]
{% if settings.compiler == "Visual Studio" %}
build
{% else %}
build/{{settings.build_type}}
{% endif %}

[build_folder]
build/{{settings.arch}}/{{settings.build_type}}

[source_folder]
src

[includedirs]
src

[libdirs]
build/{{settings.build_type}}/lib

[bindirs]
build/{{settings.build_type}}/bin

This layout will define the src include directory for the say and for other packages in editable mode. Also, the build_folder has a condition only for say/0.1@user/channel package. It will use a specific path, according the compiler.

In every case the directories that will be affected by the editable mode will be includedirs, libdirs, bindirs, resdirs, srcdirs and builddirs, all of them declared in the cpp_info dictionary; the rest of values in that dictionary won’t be modified. So cflags, defines, library names in libs defined in package_info() will still be used.

By default all folders paths are relative to the directory where the conanfile.py of the editable package is (which is the path used to create the link), though they also allow absolute paths.

Specifying layout files

Layout files are specified in the conan editable add command, as an extra argument:

$ cd examples/features/editable/cmake/say
$ conan editable add . say/0.1@user/channel --layout=layout_vs

That layout_vs file will be first looked for relative to the current directory (the path can be absolute too). If it is found, that will be used. It is possible to add those layouts in the source repositories, so they are always easy to find after a clone.

If the specified layout is not found relative to the current directory, it will be looked for in the conan cache, in the .conan/layouts folder. This is very convenient to have a single definition of layouts that can be shared with the team and installed with conan config install.

If no argument is specified, the conan editable add command will try to use a .conan/layouts/default layout from the local cache.

You can switch layout files by passing a different argument to new calls to conan editable add.

Evaluation order and priority

It is important to understand the evaluation order and priorities regarding the definitions of layouts:

  • If the layout() method is declared in the recipe, it will ignore the rest of the information regarding the layout. Only the self.cpp.source and self.cpp.build objects will be taken into account.

Otherwise:

  • The first thing that will always execute is the recipe package_info(). That will define the flags, definitions, as well as some values for the layout folders: includedirs, libdirs, etc.

  • If a layout file is defined, either explicitly or using the implicit .conan/layouts/default, conan will look for matches, based on its package reference.

  • If a match is found, either because of global definitions like [includedirs] or because a match like [pkg/version@user/channel:includedirs], then the layout folders (includedirs, libdirs, resdirs, builddirs, bindirs), will be invalidated and replaced by the ones defined in the file.

  • If a specific match like [pkg/version@user/channel:includedirs] is found, it is expected to have defined also its specific [pkg/version@user/channel:libdirs], etc. The global layout folders specified without package reference won’t be applied once a match is found.

  • It no match is found, the original values for the layout folders defined in package_info() will be respected.

  • The layout file to be used is defined at conan editable add time. If a .conan/layouts/default file is added after the conan editable add, it will not be used at all.

Using a package in editable mode

Once a reference is in editable mode it is used system wide (for every set of settings and options) by Conan (by every Conan client that uses the same cache), no changes are required in the consumers. Every conan install command that requires our editable say/0.1@user/channel package will use the paths to the local directory and the changes made to this project will be taken into account by the packages using its headers or linking against it.

To summarize, consumption of packages in editable mode is transparent to their consumers. To try that it is working, the following flow should work:

  • Get sources of say/0.1@user/channel: git/svn clone... && cd folder

  • Put package in editable mode: conan editable add . say/0.1@user/channel --layout=layout_gcc

  • Work with it and build using any tool. Check that your local layout is reflected in the layout file layout_gcc specified in the previous step.

  • Go to the consumer project: hello

  • Build it using any local flow: conan install and build

  • Go back to say/0.1@user/channel source folder, do some changes, and just build. No Conan commands necessary

  • Go to the consumer project: hello and rebuild. It should get the changes from the say library.

In that way, it is possible to be developing both the say library and the hello application, at the same time, without any Conan command.

Note

When a package is in editable mode, most of the commands will not work. It is not possible to conan upload, conan export or conan create when a package is in editable mode.

Revert the editable mode

In order to revert the editable mode just remove the link using:

$ conan editable remove say/0.1@user/channel

It will remove the link (the local directory won’t be affected) and all the packages consuming this requirement will get it from the cache again.

Warning

Packages that are built consuming an editable package in its graph upstreams can generate binaries and packages incompatible with the released version of the editable package. Avoid uploading these packages without re-creating them with the in-cache version of all the libraries.