Why I started using Hydra as CI

Hydra is the Perl + C++ stack that evaluates and runs builds described by Nix expressions. It provides a usually good-enough (but not great) web interface and more importantly uploads (intermediary) build results into a binary cache (local storage, S3, …). The instance for NixOS, Nix and nixpkgs is running at https://hydra.nixos.org.

I’ve always been on the fence about “Hydra is the CI” when it comes to development on PR based workflows. Hydra doesn’t really have a very strong story around integration into those. There are some integrations but usually it feels bulky and isn’t really a first-class citizen.

Yet I’ve come to use it for my small scale projects that aren’t (public) on any of the forges. Why you might ask? If the experience isn’t great, why would you use it?

Well, in my case I am working on pretty large builds that have many intermediary steps. Verifying that everything (still) works usually involves bootstrapping GCC, glibc and all their dependencies.

I could (and from time to time still do) that locally but that doesn’t give me a nice way to manage build logs and intermediary build results. Whenever I push something to my hydra machine I can go back in time and look at build logs, dependency graphs and also retrieve a cached version of the build.

All of that could be done locally but it would require more disk space, custom tooling to keep old derivations and their logs around etc. There are already enough projects on my list that I’ll never finish..

Another neat side-effect of the workflow is that I must commit my changes and thus gain checkpoints on my journey. More often than not I would otherwise just hack on an expression “until it works” without having a good way of going back in time. These intermediary commits then serve me as a log of work that went into the change I am working on. I can squash it (into a few chunks) that make sense. I can also write down my learnings from the intermediary steps.

This isn’t so much a statement how great Hydra is but perhaps a statement what is lacking in terms of Nix developer tooling. Building expressions works, reproducing something that another developer ran into also works (most of the time) but really diving into build graphs and logs of builds (that didn’t happen on my machine) isn’t a great experience. We do have graphviz based tooling to inspect dependency graphs but that isn’t usually fun to deal with.

I think I can now finally understand the “Hydra is the CI” comment that was made by Eelco on IRC years ago. The system isn’t that bad and it is the best we have for some workflows.