Rebar Plugin Locking: An Insight into the Kickstarter Development Process ​

Recently we hit our base goal regarding the Rebar kickstarter to develop Version 4.

And we are not stopping there.

One major point in the next stretch goal is locking Rebar plugins. At first glance, this might sound like a small extension to Rebar's existing dependency locking. On a closer look, it reveals surprising complexities in dependency resolution and plugin execution.

This post provides insight into the recent research and planning process.

What It's About ​

Every Erlang developer working with Rebar3 knows the rebar.lock file that appears whenever an application is compiled. Usually, it gets checked in together with the application code into a version-controlled repository.

This file is the source of truth regarding the dependency versions. Rebar generates it whenever it fetches dependencies and provides explicit command line tasks for unlocking and upgrading them.

Its aim is to make builds reproducible across machines and across times.

One important task related to locking is version picking. An application can depend on different versions of the same dependency:

root_app
├─ dep1
│   └─ dep2 v1.0
└─ dep3
    └─ dep2 v2.0

Since all dependencies will run in the same Erlang VM, we need to decide which version to use.

Rebar follows two simple rules to pick a winning version:

  • Higher level wins: Dependencies closest to the root define the winning version. This allows developers to select the version, by configuring it at the root level in the rebar.config file.

  • On same level first one in the configuration file wins: In the case of different versions of a dependency on the same level the version is picked based on which comes first in the configuration. This is an arbitrary choice that still provides repeatability.

Consistency across repeated builds is not achieved by locking application dependencies alone. It also depends on the plugins involved and their dependencies.

Root-level plugin versions can be manually locked in the rebar.config. This requires manual upgrades in the configuration and is not 100% reliable, Git tags, for example, can be moved. Locking should use unique references like the Git commit hash or package checksums.

Locking the dependencies of plugins as well would provide more transparency of the tools being used. This is relevant, for example, to make sure one does not use a version with a security issue.

An automated locking mechanism would enforce these benefits and reduce the effort to manually check them by the developer.

Why It Does Not Already Exist ​

Although the idea of plugin locking is the same as for dependencies, there are edge cases that make it impossible to just reuse the same mechanisms.

The main point is that we cannot separate the generation of the dependency tree from the compilation. Consider the following example:

root
└─ plugin1
    ├─ plugin2 // required to fetch dep1 
    │   └─ dep2
    └─ dep1
        └─ dep2

Here we need the following order of steps in Rebar:

  1. Fetch plugin1.
  2. Fetch plugin2.
  3. Fetch dep2.
  4. Compile plugin2 with dep2.
  5. Fetch dep1 using plugin2.
  6. Compile plugin1 with dep1.

The complete dependency tree of plugin1 is only detectable at the end and requires compilation of plugin2 in between.

For ordinary dependencies, Rebar constructs the dependency tree ahead of compilation, this pattern cannot be applied to plugins making the implementation of locking and version picking more complicated.

What We Are Going to do About It ​

Thanks to the support from our Kickstarter backers, we will put in the effort required to implement locking for plugins with the following design decisions:

  • Keep ordinary application dependencies and plugin dependencies separated.

  • Review the existing version picking for plugin dependencies that is done during fetching and compilation and ensure in the final plugin dependency tree:

    • At the same level, plugin dependencies win over ordinary dependencies (Note that this is only regarding the plugin dependency tree and plugins can depend on other plugins or other applications using plugins or deps in their rebar.config).

    • Higher level wins.

    • On the same level, first in configuration file wins.

  • Lock the implicitly defined dependency tree of plugins in the state and extend the lock file.

  • Extend command line interface for an unlock feature for plugins.

We hope that version picking can be implemented without overly complex code changes, even when considering tricky examples like::

root
├─ plugin1
│   ├─ plugin2 // required to fetch dep1
│   │   └─ dep2 v1.0
│   ├─ dep1
│   │   └─ dep2 v2.0
│   └─ dep2 v1.5
└─ dep2 v2.1

The goal is that it leads to dependency trees where dep2 v1.5 wins the version picking for plugins and dep2 v2.1 stays the winning version for ordinary dependencies:

root (plugins)
└─ plugin1
    ├─ plugin2 // required to fetch dep1
    │   └─ dep2 v1.5
    ├─ dep1
    │   └─ dep2 v1.5
    └─ dep2 v1.5

root (ordinary dependencies)
└─ dep2 v2.1

The new lock file shape should separate ordinary dependencies from plugin dependencies. We will use the lock file versioning and increase the major version to 2.0.0. The current idea is that it will look like:

{"2.0.0",[
  {deps, [{<<"dep2">>,{pkg,<<"dep2">>,<<"v2.1">>},0}]},
  {plugins, [
    {<<"plugin1">>,{pkg,<<"plugin1">>,<<"1.2.0">>},0},
    {<<"plugin2">>,{git,"https://github.com/example/plugin2.git ",
                            {ref,"{Git Commit Hash}"}},1},
    {<<"dep2">>, {pkg, <<"dep2">>, <<"v1.5">>}, 1}
  ]}
]}.
[
  {pkg_hash,[
    {deps, [{<<"dep2">>, <<"{dep2_inner_checksum_v2.1}">>]}},
    {plugins, [
        {<<"plugin1">>, <<"{plugin1_inner_checksum}">>},
        {<<"dep2">>, <<"{dep2_inner_checksum_v1.5}">>}
    ]}
  ]},
  {pkg_hash_ext,[
    {deps, [{<<"dep2">>, <<"{dep2_outer_checksum_v2.1}">>}]},
    {plugins, [
        {<<"plugin1">>, <<"{plugin1_outer_checksum}">>},
        {<<"dep2">>, <<"{dep2_outer_checksum_v1.5}">>}
    ]}
  ]}
].

In the end, the whole plugin dependency tree will be locked whenever plugins are fetched and compiled, for example during rebar compile, and can be unlocked with

rebar plugins unlock <plugin>
rebar plugins unlock --all

Thanks ​

Special thanks to Fred Hebert, who pointed us to the difficult edge cases that we need to consider and explained why it's not as easy as it sounds in the first place.

Thank you to all the Kickstarter backers who are funding this work.