Intro

During the last few years I’ve seen a lot of Nix based projects and many of them seem to struggle with the fact that often you have a few internal “packages” (e.g. some build step) that can depend on each other.

Often overlays are used to add internal packages into pkgs. IMHO this is an anti-pattern as long as you are not trying to modify packages that are part of nixpkgs. Example: You might want to stick to a specific version of Python 3 in your project but you almost never want to do python3 = self.python38; in the overlay. Why is that? Well, if you do override it like that you are also overriding the python3 attribute that is used for build-time (only) packages within nixpkgs. This means you’ll do a needless rebuild of a huge amount of packages whenever you update your nixpkgs snapshot. Even worse is that it might not always build as nobody (but you) tested it with that flavor of the package.

Almost all projects that I’ve seen will just want a local package set that contains internal packages while still making use of the vast packages in nixpkgs. By keeping these two separate beasts it gets a lot easier to reason about what in the package set is part of your CI/release build and which packages must compile.

In nixpkgs we have the ability to use callPackage to inject required dependencies into an expression based on the entire package set available.

A package in nixpkgs might depend on stdenv and python3 and thus declare it like this at the top of the expression:

{ stdenv, python3 }:
…

callPackage will then inspect the expression and provide the intersection of attributes from the function and those declared in the global package set (commonly named pkgs within nixpkgs).

I believe this is also what you should be using in your projects. The “big” difference is that you want your packages accessible via this mechanism as well.

It is trivially possible to extend the above example into your own package set like so:

my-packages.nix:

let
   nixpkgs = import <nixpkgs> {};
   myPackages = pkgs: {
     myPackageA = pkgs.callPackage ./a {};
     myPackageB = pkgs.callPackage ./b {};
   };
in
nixpkgs.lib.makeScope nixpkgs.newScope myPackages

Now the package myPackageB can depend on myPackageA by declaring it as an input in its expression:

./b/default.nix:

{ stdenv, python3, myPackageB }:
…

Running nix-build my-packages.nix -A myPackageB will also build myPackageA as that is a dependency of the package. It will also pull in python3 and stdenv from nixpkgs as we haven’t provide “more specific” packages with those attribute names.

Building the entire package set will only build myPackageA and myPackageB as there are no other packages declared in the set.

In a very simple scenario your CI script could look just like this: nix-build my-packages.nix

With the above code we already manged to have a dedicated set of packages or in other words a set of deliverables for our product.

In small examples and with names such as myPackage the above example might be fine and is IMHO already an improvement over what is being done in the wild.

We still have an issue with naming collisions. If you have an internal package that has the same name as a package within nixpkgs while depending on both of them (in different parts or your project) you can’t refer to one of them without renaming it. The one in your package set “shadows” the one from nixpkgs. This can be solved by always referring to nixpkgs explicitly in your expressions.

my-packages.nix:

let
   nixpkgs = import <nixpkgs> {};
   myPackages = pkgs: {
     myPackageA = pkgs.callPackage ./a {};
     myPackageB = pkgs.callPackage ./b {};
   };
  self =  nixpkgs.lib.makeScope ({
    callPackage = pkgs.lib.callPackageWith (nixpkgs // self);
  }) myPackages
in self

Summary

Regardless of flakes and if they fix some of these issue I generally think we should have a few guidelines:

0. Pin all your inputs by content hash (niv, flakes, custom fetchers, …).

1. Do not mix nixpkgs, your packages & additional external dependencies in one attribute set. It will eventually become confusing what you control and what you depend on.

2. Internal packages should be able to rely on the callPackage mechanism just as much as you would do in nixpkgs.

3. Apply overrides to external dependencies (nixpkgs, 3rd-party, …) at the place where you introduce them. This avoid you accidentally using the wrong version. This also includes that you should have (as much as possible) just one version of a package so you do not end up building against the wrong versions.

4. The root of your project (the default.nix, flake.nix or whatever you prefer) should only expose those expressions that are part of your project.