Using Nix for binary software distribution

Mar 02, 2021 — Travis Athougies

Distributing Linux binaries is hard. You have to make sure the target system has the right libraries, the right versions of those libraries, the right directory structure to use those libraries, and so on. Things get complicated really fast. To make things worse, error messages are rarely helpful and cryptic – not exactly the end user experience you want.

There are a few solutions to this problem:

Produce build packages for every distribution (or at least all the popular ones). Unfortunately, this quickly becomes untenable, and eventually some user will want to use your software on a distribution that you’ve never heard of.
Only distribute your package as source, and let the distribution packages handle it. This works if your package is popular, but good luck convincing packagers (who are usually volunteers) to maintain your package. Of course, this doesn’t work at all for proprietary software. Even for open-source software, users may not be willing to build from source (building a modern browser from source can take hours, for example).
Statically link everything together. This can work (Go does it), but not everything likes static linking, and there may be licensing concerns as well. Moreover, most Linux systems, by virtue of using glibc, are going to end up dynamic linking at some point.
The final solution, is to simply ship all your libraries with your application. I call this the ‘bundle-everything’ pattern. This is common in the Windows world, where applications often come with an ungodly number of DLLs. Of course, you end up with large numbers of duplicated libraries all necessary to support single applications. Systems like the Docker image format have been developed to alleviate some of this duplication. Unfortunately, Docker breaks the Unix paradigm. Docker applications run in their own isolated namespace, which is okay for application services but not a good fit for a core system service. Moreover, the layered approach breaks when you have two images that use slightly different revisions of a base image. They may share >50% of their image data, but because of the hierarchical nature of the layering, it’ll still be duplicated.

A better solution: Nix

In this post, I wanted to write about a pattern I’ve seen used successfully at several companies I’ve worked at as well as for our own products (like TunnelHound. It’s a variation of the ‘bundle-everything’ pattern but with a twist: use the Nix package manager to build your application against a known set of dependencies and then distribute that.

This has all the advantages of the ‘bundle-everything’ pattern and more. Due to the nature of Nix derivations, applications that share libraries will not end up duplicating disk usage. This preserves the Linux standard of re-using shared libraries, while still allowing us to pin dependencies.

For those of you who do not know what Nix is, it’s basically a declarative, immutable package manager. Every Nix package is uniquely identified by a hash of the build steps used to build it. That means that everything from how GCC was invoked to the libraries linked against it are all hashed together in a Merkle-tree-like structure. Nix can identify every version of the dependencies our software was built against, down to their binary representation. The Nix tools allow us to copy over software and its dependencies in a format called a closure. By distributing closures to end users, we can provide them with copies of our software. Sounds easy? It is!

Let’s see how this works in practice.

Step 1: Create a derivation for your application

We won’t walk through the entire process of creating a Nix derivation, but here’s a good overview of what to do.

Once you have your derivation, make sure it builds with nix-build. This produces a nix store path like /nix/store/<hash>-<pkg-name>.

In order to produce a version with known dependencies, it’s a good idea to pin NixPkgs or use Nix flakes to ensure that your dependencies are versioned along with your application.

Step 2: Sign your derivation

Nix allows you to sign your build outputs using your private key. This gives end users re-assurance that the derivations were built by you and prevents bad actors injecting malware into the Nix store. You can generate a Nix signing key by:

nix-store --generate-binary-cache-key <company name>-1 /path/to/private/key /path/to/public-key

Now, you can sign your derivation and all its dependencies by doing:

nix sign-paths -r -k /path/to/private/key /nix/store/<path to my build>

Step 3: Upload your builds to a binary cache

A Nix binary cache is a /nix/store served over a network, typically via SSH or HTTP. I recommend using HTTP for binary software distribution.

An HTTP Nix binary cache is just a collection of files. You can use any static web host to host your binary cache. You can also serve your /nix/store directly using the nix-serve service.

Other options include hosting your binary cache on Amazon S3 or using a dedicated service like Cachix.

Typically, you deploy software to your binary cache by running

nix copy /nix/store/<path to my build> --to <remote host/s3 path/etc>

Step 4: Configure user systems

Assuming you now have your binary cache served on HTTP somewhere, you can now instruct your users to pull your software.

First, they have to install Nix. Nix can be installed on any Linux distribution.

Then, they need to trust your public key and add the binary cache (also known as a substituter). They can do this by modifying /etc/nix/nix.conf to add your public key. For example, replace the following lines in nix.conf

trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
substituters = https://cache.nixos.org/

with

trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= my-org-1:MyPuBlIcKeY...
substituters = https://cache.nixos.org/ https://myorg.com/cache/

Note that you can skip modifying the substituters step if you’re going to follow step 6 below.

If your users are using NixOS, they can modify their /etc/nixos/configuration.nix:

nix.binaryCachePublicKeys = [ "my-org-1:MyPuBlIcKeY" ];
nix.binaryCaches = [ "https://myorg.com/cache/" ];

and do a nixos-rebuild switch

Typically, you would automate this step via shell script or other installer program. If you chose to use Cachix, then users can simply use the cachix use <binary-cache-name> command to complete all the steps above.

Step 5: Distribute your software

Now, whenever you have a new version of your software built and pushed to your cache, all you need to do is distribute the /nix/store path to your users, and your users can download your software by doing:

nix-store --realise /nix/store/release-path

Assuming the keys are installed correctly, your users will now have your release binary along with all dependencies! That’s it. They can invoke your program binary directly now:

/nix/store/release-path/bin/my-executable

In order to keep your software around for a while, it’s best to add garbage-collector root. This is a path on your filesystem that is a symlink to your nix software path. The existence of the symlink ensures that the Nix garbage collector will not get rid of the path.

nix-store --realise /nix/store/release-path --add-root --indirect /opt/my-org/software

Now, users can invoke your application by using the /opt path:

/opt/my-org/software/bin/my-executable

Step 6 (Bonus): Create a nix channel for your software

Nix channels allow you to distribute your .nix source files to end-users who may want to build your software from source, or make modifications to your build (for example, changing dependencies). If your software source is available, you can offer your users a channel for maximum flexibility. A nix channel is just as easy as a nix binary cache. It’s simply a directory on a static HTTP website with a file named nixexprs.tar.xz.

For example, if you want to set up the nix channel https://acme.org/releases/v1 to distribute version 1 of your software, simply tar up your Nix files using tar:

tar -cJf /path/to/releases/v1/nixexprs.tar.xz /my/nix/files \
      --transform "s,^,${PWD##*/}/," \
      --owner=0 --group=0 --mtime="1970-01-01 00:00:0 UTC"

Of course, users may still want to pull binaries from your cache. You can associate your nix-channel with a binary cache by adding a file binary-cache-url to the directory. This is just a text file containing the URL of your nix cache

echo `<URL of cache>` > /path/to/releases/v1/binary-cache-url

Now upload both this nixexprs.tar.xz and binary-cache-url to the https://acme.org/releases/v1 dir.

Users can use your channel by doing:

nix-channel --add https://acme.org/releases/v1 acme

As long as your key is in their trusted-public-keys, Nix will automatically pull derivation outputs from your binary cache. If users modify your channel, then any unaffected dependencies will be pulled as well. If your key is not trusted, then Nix will build everything from scratch.

Extra goodies

Distributing system images

That’s it! Not only can you distribute software this way, you can also distribute entire system configurations. A NixOS system is just a Nix derivation output with a /nix/store/abcdef..-name../bin/switch-to-configuration script that can be invoked with the same arguments as nixos-rebuild.

For example, if /nix/store/by0gaxprnv2fapzil9rfi5vhlx8rpbaf-nixos-system-travis-nixos-20.03.3081.629fe7b1450 is the output of a NixOS derivation (in this case, the one powering my laptop), then running

/nix/store/by0gaxprnv2fapzil9rfi5vhlx8rpbaf-nixos-system-travis-nixos-20.03.3081.629fe7b1450/bin/switch-to-configuration switch

Will immediately, and atomically, turn the current system into one configured exactly like my laptop. And

/nix/store/by0gaxprnv2fapzil9rfi5vhlx8rpbaf-nixos-system-travis-nixos-20.03.3081.629fe7b1450/bin/switch-to-configuration boot

Will turn the current system into one configured exactly like my laptop on the next boot. The switch-to-configuration program is completely idempotent, so it’s safe to call any time.

This is the mechanism we use in TunnelHound to distribute appliance updates. We simply push our latest builds to our binary cache in S3, and then TunnelHound instances are configured to pull images from S3 and switch to them, whenever the user decides to upgrade. By using Nix, we avoid the problem of stale configuration, out-of-date dependencies borking upgrades, or leaving systems in a half-baked state. The end-user experience is great, and there is little chance of broken upgrades. This is perfect for a small development shop like F Omega.

Cleaning up old versions

Of course, upgrading to a new software version should also result in the old version being deleted. However, the commands above simply add more binaries to the nix store. Because the store is immutable, old versions (and all their dependencies are still hanging around). Luckily, it’s pretty easy to get rid of them by doing

nix gc

Please note that, unless you’ve configured a GC root as above or are distributing a full NixOS image, then this could potentially get rid of the latest software version as well.

Conclusion

With a bit of shell-scripting, you can develop entire software distribution systems. At previous startups I’ve worked at, this allowed an incredibly small team to distribute incredibly complex system configurations for network appliances with large numbers of dependent services. This would have been a nightmare using anything other than Nix. Using a more traditional distribution as our base would have opened up the possibility of partial upgrades which could have brought down the entire appliance and led to angry customers. Luckily, with Nix, updates are atomic, and you never end up with overwritten dependencies that are difficult to recover from.

I hope that this post motivates you to use Nix for software distribution, whether binary-only or open-source. Nix provides a number of advantages over competing solutions like Docker, such as the ability to distribute systems software, as well as a better deduplication story. Nix binary caches are easy to host on any static HTTP web host. Cryptographic keys ensure the authenticity of your builds to your end-users. Not only can you ship software, but entire system configurations.

If you want to learn more about distributing software with Nix, I highly recommend the Nix Pills and the nix.dev tutorials.