Andreas Rammhold

Why I started using Hydra as CI

Sat, 27 Nov 2021 19:00:00 +0000

Hydra is the Perl + C++ stack that evaluates and runs builds described by Nix expressions. It provides a usually good-enough (but not great) web interface and more importantly uploads (intermediary) build results into a binary cache (local storage, S3, …). The instance for NixOS, Nix and nixpkgs is running at https://hydra.nixos.org.

I’ve always been on the fence about “Hydra is the CI” when it comes to development on PR based workflows. Hydra doesn’t really have a very strong story around integration into those. There are some integrations but usually it feels bulky and isn’t really a first-class citizen.

Yet I’ve come to use it for my small scale projects that aren’t (public) on any of the forges. Why you might ask? If the experience isn’t great, why would you use it?

Well, in my case I am working on pretty large builds that have many intermediary steps. Verifying that everything (still) works usually involves bootstrapping GCC, glibc and all their dependencies.

I could (and from time to time still do) that locally but that doesn’t give me a nice way to manage build logs and intermediary build results. Whenever I push something to my hydra machine I can go back in time and look at build logs, dependency graphs and also retrieve a cached version of the build.

All of that could be done locally but it would require more disk space, custom tooling to keep old derivations and their logs around etc. There are already enough projects on my list that I’ll never finish..

Another neat side-effect of the workflow is that I must commit my changes and thus gain checkpoints on my journey. More often than not I would otherwise just hack on an expression “until it works” without having a good way of going back in time. These intermediary commits then serve me as a log of work that went into the change I am working on. I can squash it (into a few chunks) that make sense. I can also write down my learnings from the intermediary steps.

This isn’t so much a statement how great Hydra is but perhaps a statement what is lacking in terms of Nix developer tooling. Building expressions works, reproducing something that another developer ran into also works (most of the time) but really diving into build graphs and logs of builds (that didn’t happen on my machine) isn’t a great experience. We do have graphviz based tooling to inspect dependency graphs but that isn’t usually fun to deal with.

I think I can now finally understand the “Hydra is the CI” comment that was made by Eelco on IRC years ago. The system isn’t that bad and it is the best we have for some workflows.

Distributed Forges

Sun, 07 Mar 2021 13:30:00 +0000

People that know me are very likely to know my stance on GitHub.com, GitLab.com or any other centralized software forges.

I have the strong believe that the future of developing open and free software must be done on open and free platforms that allow users and contributors to interact with the forge without requiring centralized data storage or identity management.

Today I came across a blog post that covers this well: Decentralized forge: distributing the means of digital production (archive.org)

It also links to a wide variety of other (tangential) topics that I can highly recommend to anyone interested in F/OSS and privacy.

Blog deployment via git-receive hooks

Thu, 08 Oct 2020 15:00:00 +0200

While quickly going through the options for hosting this blog I decided to just do it myself.. Serving some static content (for a small audience) isn’t that hard and I’m running a few (mostly idle) machines anyway.

My requirements were just to have IPv6, some way to tune the webserver config and some kind of “CI” that converts the website from markdown to HTML (preferably with Nix).

Actually pushing the content to a server is trivial if you already have SSH access. So I could use either rsync or SSH to do it. Since the blog is already in a git repo I wanted to treat the target server as a git remote. Git has a bunch of built-in hooks that can be used for many things. In this case I was interested in the post-receive hook.

Pushing the content (in it’s raw format) to a server is already trivial if you have SSH access. I would just commit and push the latest version to another remote.

For building the static assets I could be using git post-receive hooks that just call nix-build … -o $documentRoot.

And that is what I did.

All that was required to make this happen was adding a new file to the NixOS configuration of the server this is running on:

{ config, pkgs, ... }:
let
home = "/var/lib/blog";
env = pkgs.buildEnv {
name = "post-receive-env";
paths = [
pkgs.git
pkgs.nix
pkgs.coreutils
pkgs.gnutar
pkgs.xz
];
};
post-receive = pkgs.writeShellScript "post-receive" ''
 export PATH=${env}/bin
 set -ex

 GIT_DIR=$(git rev-parse --git-dir 2>/dev/null)
 if [ -z "$GIT_DIR" ]; then
 echo >&2 "fatal: post-receive: GIT_DIR not set"
 exit 1
 fi

 TMPDIR=$(mktemp -d)
 function cleanup() {
 rm -rf "$TMPDIR"
 }
 trap cleanup EXIT

 git clone "$GIT_DIR" "$TMPDIR"
 unset GIT_DIR
 cd "$TMPDIR"
 exec nix-build "$TMPDIR" -A build -o ${home}/current
 '';
in
{
config = {
services.nginx.virtualHosts."andreas.rammhold.de" = {
enableACME = true;
forceSSL = true;
root = "/var/lib/blog/current";
};
users.users.blog = {
inherit home;
createHome = true;
isNormalUser = true;
openssh = config.users.users.andi.openssh;
};
systemd.services."blog-prepare-git-repo" = {
wantedBy = [ "multi-user.target" ];
path = [
pkgs.git
];
script = ''
 set -ex
 cd ${home}
 chmod +rX ${home} # no secrets in that home dir, everyone can read my public blog
 test -e repo || git init --bare repo
 ln -nsf ${post-receive} repo/hooks/post-receive
 '';
serviceConfig = {
Kind = "one-shot";
User = "blog";
};
};
};
}

What options are there to build Rust projects with Nix?

Tue, 29 Sep 2020 23:00:00 +0000

This post is for those that are already convinced that they want to use Nix for building their Rust applications. Over the course of the last years a few different approaches to the problem have evolved. Each comes with up- & downsides and the right choice depends on your requirements. I most likely forgot one or more tools that I haven’t used. Feel free to shoot me an email and I’ll do my best of adding it.

buildRustPackage

Within Nixpkgs there is a builder for Rust packages called buildRustPackage.There is not much secret sauce in the idea. The build is basically split into to phases. The first phase downloads all the dependencies using cargo-download. The second phase invokes cargo to build & optionally run the tests.

Here is how you use it:

{ nixpkgs ? <nixpkgs> }:
let pkgs = import nixpkgs {}; in
pkgs.buildRustPackage {
pname = "my-package";
version = "1.0";
src = ./.;
cargoSha256 = "0000000000000000000000000000000000000000000000000000000000000000";
}

As you can see the usage is really straight forward and you do not need to generate any files in the repository. That being said you’ll have to update the cargoSha256 field whenever you change any of your dependencies.

The obvious downside is that fixed output derivation can be hard to deal with and this usage is, to some degree, an abuse of the FOD’s that were intended for simple fetching of source archives. In the past buildRustPackage builds had issues across difference architectures as cargo-vendor would generate different outputs.

Pro:

Really low bar for entry
No generated files to check into the repo

Con:

Slow for big projects as it compiles everything in one go.
Fixed output hashes have to be maintained.

import-cargo

Eelco Dolstra wrote cargo-import which, compared to the previous contender, doesn’t require any fixed output updating or manual setps after the initial setup. Just like buildRustPackage, it is doing all the compilation as a single large build without any support for incremental recompilation.

The usage, as described in the repo, requires you to use the experimental flakes support in nix:

{
description = "My Rust project";
inputs = {
nixpkgs.url = github:NixOS/nixpkgs/nixos-20.03;
import-cargo.url = github:edolstra/import-cargo;
};
outputs = { self, nixpkgs, import-cargo }: let
inherit (import-cargo.builders) importCargo;
in {
defaultPackage.x86_64-linux =
with import nixpkgs { system = "x86_64-linux"; };
stdenv.mkDerivation {
name = "testrust";
src = self;
nativeBuildInputs = [
# setupHook which makes sure that a CARGO_HOME with vendored dependencies
# exists
(importCargo { lockFile = ./Cargo.lock; inherit pkgs; }).cargoHome
# Build-time dependencies
rustc cargo
];
buildPhase = ''
 cargo build --release --offline
 '';
installPhase = ''
 install -Dm775 ./target/release/testrust $out/bin/testrust
 '';
};
};
}

(example taken from the upstream repo)

For anyone that is familiar with regular stdenv.mkDerivation builds this should read very familiar. You can integrate this easily into poly-language projects that might invoke cargo at some point during the build.

These days I still actively avoid experimental flakes for most of my projects. I’ve extracted the logic into a Gist that you can just callPackage.

Pro:

No generated files to check into the repo
No manual updating of files even when dependencies change

Cons:

Upstream version requires usage of experimental Nix features
Slow for big projects as it compiles everything in one go.

naersk

naersk is very similar to import-cargo but it has a major upside: It builds dependencies independent from the actual target. This gives you caching of dependencies and reduces the main build to just building your code and linking it to the dependencies.

While it is doing much better than any of the alternatives already it still isn’t perfect for large repositories with multiple rust projects that share a large number of overlapping dependencies. I’m not holding this against it. It is an excellent piece nonetheless.

Instead of quoting a usage example I’d like to refer you to the upstream documentation which is really well done and offers more options than I would consider covering here.

Pro:

Incremental builds with cached dependencies
No generated files to check into the repo
No manual updating of files even when dependencies change

Cons: Not really any unless you have plenty of in-house crates that share dependencies (or depend on each other).

crate2nix

Last but not least, there is also crate2nix, which is the logical replacement of carnix (an early pioneer in the era that is effectively dead). Back in the days carnix introduced buildRustCrate to nixpkgs. The main motivation here is to build each crate as their own derivation. This leads to very good incremental builds. You will rebuild a sub-tree of the dependencies when you change one instead of all of them. It also allows you to re-use already compiled dependencies across rust builds which furthermore cuts down the compile times in some situations.

The downside to all the other approaches is that you’ll have to generate Nix expressions and check them into your repository. These expressions can be quite large but for some the benefits might outweigh the costs.

In order to use it you’ll have to make the crate2nix tool available in your shell. Once you’ve done that it comes down to invoking crate2nix generate followed by nix-build -A rootCrate.build.

Pro:

Suitable for large projects where build closures are overlapping as each dependency is it’s own derivation

Cons:

Large generated files have to be checked into the repo

Comparison table

	buildRustPackage	import-cargo	naersk	crate2nix
Requires Generated Files	No	No	No	Yes
Manual FOD hashes	Yes	No	No	No
Requires experimental Nix	No	Yes (but see comment below)	No	No
Incremental build	No	No	Yes, but not for dependencies	Yes, for all crates

ranz2nix - A new approach to packaging node packages with Nix

Tue, 14 Jul 2020 12:00:00 +0000

A few weeks ago I was asked to create a simple online gallery for my sisters wedding pictures. The whole process of selecting the right software probably deserves a blog post on it’s own. In the end I decided to go with photoprism as it looked like the best solution (on multiple scales).

The unfortunate fact was that they are using Node.js to build their frontend application which meant that I had to use node2nix, yarn2nix, … or to roll my own solution. A few attempts into using node2nix (paired with previous experiences) I decided that it isn’t worth my time. I don’t mean to discredit the work that was done there but for me it simply doesn’t solve the problem.

Mostly I am not a fan of generating mostly duplicate Nix code out of some other source repository when all the information (the native package manager requires) are already available. I could never figure out how to properly patch the generated node2nix expressions without resorting to manually calling patch as part of the upgrade process. It also doesn’t really work without checking out the projects source code and running the code generation within it.

After all I just wanted to use niv to pin and update it without requiring further recurring manual steps.

I started looking at the contents of the package-lock.json to see if it contains anything of use for an alternative approach and was pleasantly surprised.

Unlike many other lock file formats it is just plain JSON that contains both the SRI hash and the source URI of each of the locked dependencies:

{
"name": "photoprism",
"version": "1.0.0",
"lockfileVersion": 1,
"requires": true,
"dependencies": {
"@babel/cli": {
"version": "7.10.4",
"resolved": "https://registry.npmjs.org/@babel/cli/-/cli-7.10.4.tgz",
"integrity": "sha512-xX99K4V1BzGJdQANK5cwK+EpF1vP9gvqhn+iWvG+TubCjecplW7RSQimJ2jcCvu6fnK5pY6mZMdu6EWTj32QVA==",
…

This means I can just fetch all of the dependencies with Nix without writing a single line of Node.js or caring about internals of npm like dependency resolution (and circular dependencies).

But how does that help?

Well, iff NPM uses a “fetcher” that supports file:// URI’s I can rewrite the lockfile in such a way that the resolved values point to nix store paths without invalidating the integrity hashes. By doing this we no longer need network access during the build and everything can be provided within the build sandbox. The sources will be made available since the patched lockfile will carry them as dependencies.

At this stage all that is left is actually rewriting the contents of the file. Since Nix has builtin support for JSON so it was straight forward to recursively updating all the dependencies in the file:

{
lockFilePath
}:
let
# …
lockFile = builtins.fromJSON (builtins.readFile lockFilePath);
# …
patchedLockfile = pkgs.writeTextFile {
name = "package-lock.json";
destination = "/package-lock.json";
text = (
let
patchDep = sourcesList: name: v:
let
dsources = lib.mapAttrs (mkSource) (v.dependencies or { });
src = findBestSource sourcesList name;
in
(v // {
dependencies = lib.mapAttrs (patchDep ([ dsources ] ++ sourcesList)) (v.dependencies or { });
}) // (if src == null then packageOverride name v else {
resolved = "file://" + (toString src);
}
);
file = lockFile // {
dependencies = lib.mapAttrs (patchDep [ sources ]) (lockFile.dependencies or { });
};
in
builtins.toJSON file
);
};
# …
}

(Full source)

Now we can use the patched lockfile and create a new source tree with the lockfile being replaced. All that is left then is to call npm --offline install to install all the dependencies Node.js projects.

{
# …
patchedBuildRoot = pkgs.symlinkJoin {
name = "patched-root-${lockFile.name}";
paths = [
patchedLockfile
sourcePath
];
};
patchedBuild = pkgs.stdenv.mkDerivation {
name = lockFile.name;
src = patchedBuildRoot;
buildInputs = [ nodejs ];
buildPhase = ''
 export HOME=$(mktemp -d)
 chmod -R +rw .
 npm --offline install
 '';
installPhase = ''
 ls -la
 mkdir $out
 cp -rv node_modules $out/node_modules
 '';
passthru.lockFile = patchedLockfile + "/package-lock.json";
};
# …
}

(Full source)

With the node_modules folder being handled we can move on to the actual build process. For the case of photoprism there was one dependency that didn’t have any sources in the lockfile (namely minimist). For that purpose I quickly added a packageOverride attribute to the lockfile patching code. It allows specifying sources for all those where sources are missing. I expected there to be more of these edge cases but apparently that was all that was required. The entire frontend build expression turned out to be fairly simple:

{
# …
frontend = let
noderanz = callPackage ranz2nix {
nodejs = nodejs-12_x;
sourcePath = src + "/frontend";
packageOverride = name: spec: if name == "minimist" && spec ? resolved && spec.resolved == "" then {
resolved = "file://" + (
toString (
fetchurl {
url = "https://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz";
sha256 = "0w7jll4vlqphxgk9qjbdjh3ni18lkrlfaqgsm7p14xl3f7ghn3gc";
}
)
);
} else {};
};
node_modules = noderanz.patchedBuild;
in
stdenv.mkDerivation {
name = "photoprism-frontend";
nativeBuildInputs = [ nodejs-12_x ];
inherit src;
sourceRoot = "photoprism-src/frontend";
postUnpack = ''
 chmod -R +rw .
 '';
NODE_ENV = "production";
buildPhase = ''
 export HOME=$(mktemp -d)
 ln -sf ${node_modules}/node_modules node_modules
 ln -sf ${node_modules.lockFile} package-lock.json
 npm run build
 '';
installPhase = ''
 cp -rv ../assets/static/build $out
 '';
};
# …
}

(Full source)

And that is all! Hurray!

When compared to any other Node.js approach this was actually fun to use and to develop. With a bit of dedicated time and brainstorming this little PoC could evolve into a more sophisticated solution. For now I am done with this project as it just does what I need and I can finally get back to deploying the gallery. :-)

EDIT: Since all of the above was just a PoC and was still very rough on the edges I started working on a cleaner version of this approach over at npmlock2nix.

NixOS IPv6 Prefix Delegation

Fri, 01 May 2020 21:00:00 +0000

Note: This used to be a draft that I had lying around for more than a year now. I’ve decided to just publish it now.

My home internet connection comes with proper dual-stack support. I get a public IPv4 address via DHCP, a /128 (IA_NA) and a /48 IPv6 prefix (IA_PD) via DHCPv6.

Traditionally you would configure your networking via iproute2 and then fork-off a DHCP client to configure the external addresses.

Usually all of that is being hidden from you through wrappers (like Debian’s ifupdown). The configuration would be set, the daemons fired off and hopefully everything would go well.

The limitations of the system become visible once you have a more dynamic set of interfaces that have to be initialized in some order, some VPN device that depend on the uplink connection etc. While systems like ifupdown have employed hooks of all sorts that you still end up writing a bunch of (inlined) shell scripts that deal with some little details of your setup. Adding sleep statements at worst. Things get tricky when one interface going up changes things on another interface or even system wide (think sysctl, iptables, starting a VPN daemon, …).

As soon as your configuration grows (over days, months, years, …) it becomes harder to verify that everything still works after a clean reboot of the system. After all you’ve been modifying the system with each “restart” of the network configuration script/tooling.

I’ve had systems work reliably after a reboot with huge networking scripts on these systems. Was it fun? Was it straightforward? No, not really. You really had to know what you were doing. Having to manually fiddle with some parameters after a reboot wasn’t uncommon. Sometimes services wouldn’t come up properly because the network was not up when they were being scheduled.

Here is where we should start talking about systemd-networkd (or just networkd). networkd allows you to declare all (or really almost everything) that you previously encoded in lines and lines of shell-like configuration files as structured configuration. That doesn’t just include the (static) addresses, routes, VLANs, VRFs, bridges, policy routing, wirguard, … but also runtime configuration like DHCPv4 and DHCPv6 and prefix delegation. You can configure when the system is to be considered “online” and which interfaces (in which states) contribute to that.¹

This makes for a very nice property: You now have a single directory of plain text files that define all your network configurations. As long as you have networkd installed and a way to render your /etc/systemd/network configuration you are good. Rolling back a change now just means rolling back that directory. Versioning text files is what we are good at, no?

Ontop of that we can make use of Nix and specifically the NixOS module system to generate the NixOS configuration and also manage the services on our system in the very same way.

And this gets me back to the topic of this post. My residential internet connection and configuring my CPE aka router. I was planning to do the migration from my old venerable Debian system to a shiny new NixOS machine but I didn’t just want to pull the plug and fiddle until everything is seeminlgy working again.

I decided to just simulate a simplified version of the relevant pieces of the ISP infra and test my poential configuration against that. Since the termination just happens on an (isolated) Ethernet L2 all I really worried about was getting the DHCPv6 client configuration right. Especially the prefix delegation. For this blog post I trimmed it down to just use the DHCPv6 prefix delegation part.

This is the high level definition of the test:

let
makeTest = import (<nixpkgs> + "/nixos/tests/make-test-python.nix");
in
makeTest (
{ pkgs, lib, ... }:
{
nodes = {
# The `isp` node will represent my upstream ISP. It will run a
# traditional network stack with isc's dhcpd4 and dhcpd6
isp = {
virtualisation.vlans = [ 1 ];
imports = [ ./isp.nix ];
};
# My CPE / router that will handle internal as well as external
# DHCPv{4,6}, RA etc..
router = {
virtualisation.vlans = [ 1 2 ];
# import the custom module that contains all the implementation details
imports = [ ./router.nix ];
};
# A test client that will verify the result
client = {
virtualisation.vlans = [ 2 ];
imports = [ ./client.nix ];
};
};
testScript = builtins.readFile ./test.py;
}
)

First the makeTest function is being imported from <nixpkgs>. It serves as a way to construct NixOS VM tests from the package set. Each test contains a number of machine definitions and a test script.

Each of the nodes can be in multiple virtual networks. The ISP and the Router node will share one VLAN (1), the Router and the Client node will share another (2). This simulates the missing wires/switches/wireless between the devices.

Each of the nodes imports a single file that contains the detailed configuration.

The next step was filling in the details about how the ISPs side of things operates. I mostly just copied in some files that I had lying around from a previous setup. Here is the commented isp.nix:

{ lib, pkgs, ... }:
{
# The ISP's routers job is to delegate IPv6 prefixes via DHCPv6. Like with
# regular IPv6 auto-configuration it will also emit IPv6 router
# advertisements (RAs). Those RA's will not carry a prefix but in contrast
# just set the "Other" flag to indicate to the receiving nodes that they
# should attempt DHCPv6.
#
# Note: On the ISPs device we don't really care if we are using networkd in
# this example. That being said we can't use it (yet) as networkd doesn't
# implement the serving side of DHCPv6. We will use ISC's well aged dhcpd6
# for that task.
networking = {
useDHCP = false;
firewall.enable = false;
interfaces.eth1.ipv4.addresses = lib.mkForce []; # no need for legacy IP
interfaces.eth1.ipv6.addresses = lib.mkForce [
{ address = "2001:DB8::"; prefixLength = 64; }
];
};
# Since we want to program the routes that we delegate to the "customer"
# into our routing table we must have a way to gain the required privs.
# This security wrapper will do in our test setup.
#
# DO NOT COPY THIS TO PRODUCTION AS IS. Think about it at least twice.
# Everyone on the "isp" machine will be able to add routes to the kernel.
security.wrappers.add-dhcpd-lease = {
source = pkgs.writeShellScript "add-dhcpd-lease" ''
 exec ${pkgs.iproute}/bin/ip -6 route replace "$1" via "$2"
 '';
capabilities = "cap_net_admin+ep";
};
services = {
# Configure the DHCPv6 server
#
# We will hand out /48 prefixes from the subnet 2001:DB8:F000::/36.
# That gives us ~8k prefixes. That should be enough for this test.
#
# Since (usually) you will not receive a prefix with the router
# advertisements we also hand out /128 leases from the range
# 2001:DB8:0000:0000:FFFF::/112.
dhcpd6 = {
enable = true;
interfaces = [ "eth1" ];
extraConfig = ''
 subnet6 2001:DB8::/36 {
 range6 2001:DB8:0000:0000:FFFF:: 2001:DB8:0000:0000:FFFF::FFFF;
 prefix6 2001:DB8:F000:: 2001:DB8:FFFF:: /48;
 }

 # This is the secret sauce. We have to extract the prefix and the
 # next hop when commiting the lease to the database. dhcpd6
 # (rightfully) has not concept of adding routes to the systems
 # routing table. It really depends on the setup.
 #
 # In a production environment your DHCPv6 server is likely not the
 # router. You might want to consider BGP, custom NetConf calls, …
 # in those cases.
 on commit {
 set IP = pick-first-value(binary-to-ascii(16, 16, ":", substring(option dhcp6.ia-na, 16, 16)), "n/a");
 set Prefix = pick-first-value(binary-to-ascii(16, 16, ":", suffix(option dhcp6.ia-pd, 16)), "n/a");
 set PrefixLength = pick-first-value(binary-to-ascii(10, 8, ":", substring(suffix(option dhcp6.ia-pd, 17), 0, 1)), "n/a");
 log(concat(IP, " ", Prefix, " ", PrefixLength));
 execute("/run/wrappers/bin/add-dhcpd-lease", concat(Prefix,"/",PrefixLength), IP);
 }
 '';
};
# Finally we have to set up the router advertisements. While we could be
# using networkd or bird for this task `radvd` is probably the most
# venerable of them all. It was made explicitly for this purpose and
# the configuration is much more straightforward than what networkd
# requires.
# As outlined above we will have to set the `Managed` flag as otherwise
# the clients will not know if they should do DHCPv6. (Some do
# anyway/always)
radvd = {
enable = true;
config = ''
 interface eth1 {
 AdvSendAdvert on;
 AdvManagedFlag on;
 AdvOtherConfigFlag off; # we don't really have DNS or NTP or anything like that to distribute
 prefix ::/64 {
 AdvOnLink on;
 AdvAutonomous on;
 };
 };
 '';
};
};
}

Next the router.nix:

# This will be our (residential) router that receives the IPv6 prefix (IA_PD)
# and /128 (IA_NA) allocation.
#
# Here we will actually start using networkd.
{
systemd.services.systemd-networkd.environment.SYSTEMD_LOG_LEVEL = "debug";
boot.kernel.sysctl = {
# we want to forward packets from the ISP to the client and back.
"net.ipv6.conf.all.forwarding" = 1;
};
networking = {
useNetworkd = true;
useDHCP = false;
# Consider enabling this in production and generating firewall rules
# for fowarding/input from the configured interfaces so you do not have
# to manage multiple places
firewall.enable = false;
};
systemd.network = {
networks = {
# Configuration of the interface to the ISP.
# We must request accept RAs and request the PD prefix.
"01-eth1" = {
name = "eth1";
networkConfig = {
Description = "ISP interface";
IPv6AcceptRA = true;
#DHCP = false; # no need for legacy IP
};
linkConfig = {
# We care about this interface when talking about being "online".
# If this interface is in the `routable` state we can reach
# others and they should be able to reach us.
RequiredForOnline = "routable";
};
# This configures the DHCPv6 client part towards the ISPs DHCPv6 server.
dhcpV6Config = {
# We have to include a request for a prefix in our DHCPv6 client
# request packets.
# Otherwise the upstream DHCPv6 server wouldn't know if we want a
# prefix or not. Note: On some installation it makes sense to
# always force that option on the DHPCv6 server since there are
# certain CPEs that are just not setting this field but happily
# accept the delegated prefix.
PrefixDelegationHint = "::/48";
};
ipv6PrefixDelegationConfig = {
# Let networkd know that we would very much like to use DHCPv6
# to obtain the "managed" information. Not sure why they can't
# just take that from the upstream RAs.
Managed = true;
};
};
# Interface to the client. Here we should redistribute a /64 from
# the prefix we received from the ISP.
"01-eth2" = {
name = "eth2";
networkConfig = {
Description = "Client interface";
# the client shouldn't be allowed to send us RAs, that would be weird.
IPv6AcceptRA = false;
# Just delegate prefixes from the DHCPv6 PD pool.
# If you also want to distribute a local ULA prefix you want to
# set this to `yes` as that includes both static prefixes as well
# as PD prefixes.
IPv6PrefixDelegation = "dhcpv6";
};
# finally "act as router" (according to systemd.network(5))
ipv6PrefixDelegationConfig = {
RouterLifetimeSec = 300; # required as otherwise no RA's are being emitted
# In a production environment you should consider setting these as well:
#EmitDNS = true;
#EmitDomains = true;
#DNS= = "fe80::1"; # or whatever "well known" IP your router will have on the inside.
};
# This adds a "random" ULA prefix to the interface that is being
# advertised to the clients.
# Not used in this test.
# ipv6Prefixes = [
# {
# ipv6PrefixConfig = {
# AddressAutoconfiguration = true;
# PreferredLifetimeSec = 1800;
# ValidLifetimeSec = 1800;
# };
# }
# ];
};
# finally we are going to add a static IPv6 unique local address to
# the "lo" interface. This will serve as ICMPv6 echo target to
# verify connectivity from the client to the router.
"01-lo" = {
name = "lo";
addresses = [
{ addressConfig.Address = "FD42::1/128"; }
];
};
};
};
# make the network-online target a requirement, we wait for it in our test script
systemd.targets.network-online.wantedBy = [ "multi-user.target" ];
}

And finally the most simple of them all the client.nix:

# This is the client behind the router. We should be receving router
# advertisements for both the ULA and the delegated prefix.
# All we have to do is boot with the default (networkd) configuration.
{
systemd.services.systemd-networkd.environment.SYSTEMD_LOG_LEVEL = "debug";
networking = {
useNetworkd = true;
useDHCP = false;
};
# make the network-online target a requirement, we wait for it in our test script
systemd.targets.network-online.wantedBy = [ "multi-user.target" ];
}

While this looks pretty verbose the actual information density is pretty high once you remove all the comments.

Now that all the system configuraitons have been layed out we can start thinking about testing the setup.

The key results we want to gather from executing this test are:

Does our router configure all the interfaces as we expect?
Do we receive router advertisements on the client?
Does the router request a prefix and advertise a subnet from that to the client?
Is forwarding from / to the delegated prefix working?

We can do that fairly straightforward by filling in the previously left blank testScript attribute.

As a first step we start the router node and ensure that systemd-networkd had a chance to configure the interfaces and is ready to send out router advertisements.

# First start the router and wait for it it reach a state where we are
# certain networkd is up and it is able to send out RAs
router.start()
router.wait_for_unit("systemd-networkd.service")

Afterwards we can start the client and wait for it’s network stack to reache the online target. That means the system has brought up all the configured interfaces and whatever interface was configured as uplink has reached the desired state (routeable) by waiting for the network-online.target:

client.start()
client.wait_for_unit("network-online.target")

Since we previously configured a static address on the router node (FD42::1) we can verify that we got a valid default router via the router by attempting to reach said address.

# the static address on the router should become reachable within a few seconds
client.wait_until_succeeds("ping -6 -c 1 FD42::1")

We did use the wait_until_succeeds function since it might take a few seconds for RAs to arrive, everything being setup etc.

Once this succeeded we know that we have a working L3 connectivity between the router and the client. That likely means that our objective 2. is fullfilled, likely 1. is fine as well but we will continue being skeptical.

At this point the upstream router is still powered off and we shouldn’t be able to reach any of the addresses configured on there. Just to make sure we didn’t screw up elsewhere in the setup we can verify that:

# the global IP of the ISP router should still not be a reachable
router.fail("ping -6 -c 1 2001:DB8::")

Now that we are somehwat certain that the router and client setup is kinda working we can start the isp node and finally start checking the router delegation.

isp.start()
# Since for the ISP "being online" should have no real meaning we just
# wait for the target where all the units have been started.
# It probably still takes a few more seconds for all the RA timers to be
# fired etc..
isp.wait_for_unit("multi-user.target")

Since the router considers his uplink interface to be required to be online (as without uplink there is no external connectivity) we can now just wait for our router to reach the online state. Once that is reached we know that the ISP did hand out an IPv6 address and potentially an IA_PD prefix:

# wait until the uplink interface has a good status
router.wait_for_unit("network-online.target")
router.wait_until_succeeds("ping -6 -c1 2001:DB8::")

This now tells us that the router node has obtained a default route and some address that is routeable between it and the isp node. Shortly after that the client should obtain a public routeable addresss from the assigned prefix delegation. Yet again we give it a few seconds to propagate and thus we just use the wait_until_succeeds:

client.wait_until_succeeds("ping -6 -c1 2001:DB8::")

Now that we know that we have connectivity all the way client <-> router <-> isp we do a final check on the prefix we got assigned on the client. We expect an IPv6 prefix that has global scope and starts with the documentation prefix (2001:DB8:/32):

# verify that we got a globally scoped address in eth1 from the
# documentation prefix
ip_output = client.succeed("ip --json -6 address show dev eth1")
import json
ip_json = json.loads(ip_output)[0]
assert any(
addr["local"].upper().startswith("2001:DB8:")
for addr in ip_json["addr_info"]
if addr["scope"] == "global"
)

The code from the examples can be retrieved here

This example is based on the NixOS test that I contributed. For my actual deployment I am using a custom router module that is being tested as part of my development work on the networkd DHCPv6 subnet id PR.

These days there are (as far as I know) helpers that try to move these semantics over to the traditional wrappers. I’ve not seen any of them in years tho.. ↩︎

IPv6.watch - Sometimes the results are off

Thu, 09 Apr 2020 16:30:00 +0000

A while ago I was asked to implement prometheus support for ipv6.watch. Ever since then I am occasionally looking at the graphs to see how the world is doing. Two days ago I realized that my graphs were taking nose dives for a few hours at a time.

Initially I thought it would be some temporary issue with Hetzner (where the VM is hosted) and their networking. So I briefly looked through the enormous amounts of (unhepful) logs the service generated, did some manual poking and after not seeing any apparent errors on my side declared it transient and closed the case.

This afternoon I decided to give it another look.

I fired up my local Grafana development environment that spawns a local (ephemeral) clone of the production Grafana instance and started tinkering with the data.

My first attempt was to figure out if some specific site (aka a website) is impacting the overall results. Most of the available visualisations didn’t really look like a good fit for my needs. Looking through the Grafana plugins page and I eventually found the statusmap plugin and gave it a shot:

While that graph looks fancy there isn’t much you can take away from it with a quick glance. It has waaaaay too much information. Next try was to aggregate the data over the providers that I am querying:

This time the results look a lot more helpful. So something seems to be up with Level3’s public DNS service. Let’s take a closer look at how individual resolvers are doing:

Aha! So there is this one resolver 4.2.2.2 that is very flaky with it’s sister 4.2.2.1 that also isn’t doing great at all times.

I took a brief look at today’s RIPE BGPlay history for 4.0.0.0/9 but that looked pretty stable. It probably isn’t some global routing issue.

Since I didn’t want to invest more time I setup two RIPE Atlas measurements to see if those errors are occurring for others (from 100 random places on the planet) as well:

One of them checks the DNS resolution of youtube.com by 4.2.2.2: https://atlas.ripe.net/measurements/24653529/
The other one checks of 4.2.2.2 can be reached via ICMP echo: https://atlas.ripe.net/measurements/24653530/

Next time this issue bugs me I can hopefully just pull up those measurements and have a look at the results.

Introducing broken.sh

Sat, 04 Apr 2020 12:00:00 +0000

A few years ago, I started working on broken.sh¹, a security issue tracker for NixOS and Nixpkgs. Ever since I moved most of my private computing into the Nix ecosystem, I’ve noticed a lack of work in the field of security issues. As I’ve been trying to improve the situation (by contributing to Nixpkgs), it became obvious that we lack proper tooling in the area. I wasn’t just interested in which packages were broken on my machines. I wanted to tackle all the issues throughout Nixpkgs.

While broken.sh started out as a tool to just give me a short list of actionable items, it has somewhat grown. Right now I like to see it as an attempt at creating something other distributions like Debian, Arch Linux, Gentoo and many more have had for many many years as part of their infrastructure. NixOS should have something similar. I hope this isn’t the final answer to that question, but a (first?) step in the right direction.

While people aren’t usually that bad on tracking new highly visibile issues and fixing them, there are still plenty of not-so-famous issues that just never get any attention. A part of the problem is likely to be their discoverability, or the lack of it. The only issue tracking done for NixOS is one big lump of issues in the GitHub issue tracker. However, GitHub issues cannot really be categorized into packages, releases, maintainers, commits, etc. I figured out early on that creating an overview outside of GitHub would be the best way forward.

In the very early days, broken.sh was just a static HTML page providing an overview of one specific revision, listing all the issues that I managed to identify. That was okay as a one-shot report but did not fit my long-term vision.

Below you can see the output of the very first working version. Back then the project was still called an “update tracker”.

I also tried to provide information on which upstream versions are available for a given package - hoping to eventually tackle that issue, too (hah!). I eventually decided to focus on the security issues.

How does it work?

In a nutshell, I am looking at every revision of a NixOS channel and extract all the package versions and patches. For each of these, I try to figure out if a recent snapshot of the NVD database contains relevant information for them. If there is a positive match, it will be shown on the website.

My current approach consists of multiple steps. Each step is available as a subcommand of the nix-vuln-scanner binary:

$ ./result/bin/nix-vuln-scanner
USAGE:
nix-vuln-scanner [FLAGS] <SUBCOMMAND>
FLAGS:
-d, --debug enable debug logging
-h, --help Prints help information
-V, --version Prints version information
SUBCOMMANDS:
channel-report Generates reports for all "bumps" in a channel
help Prints this message or the help of the given subcommand(s)
notes List, create and edit notes
process-db Process the output of the report command and generate an sqlite database from it
report Generate a new issue report file
serve serves the contents of the DB via HTTP.

You do not always have to run all parts in order to obtain and use the results. For example, if all you need is a brief ad-hoc information, you shouldn’t have to run a web server or even deal with a (sqlite) database (file). Instead you can just run the report subcommand like this:

$ nix-vuln-scanner report -p packages.json -s nixos-20.03 -o my-report.json
[2020-04-04][21:53:42][nix_vuln_scanner::git::clone][INFO] Locking "nixpkgs/repo/68b00da0eb30d2acc9c57208a4f921c1/clone.lock"
[2020-04-04][21:53:42][nix_vuln_scanner::git::clone][INFO] Initial cloning of git://github.com/nixos/nixpkgs to "nixpkgs/repo/68b00da0eb30d2acc9c57208a4f921c1/clone"
Cloning into bare repository 'nixpkgs/repo/68b00da0eb30d2acc9c57208a4f921c1/clone'...
remote: Enumerating objects: 129, done.
remote: Counting objects: 100% (129/129), done.
remote: Compressing objects: 100% (109/109), done.
remote: Total 1871146 (delta 5), reused 42 (delta 2), pack-reused 1871017
Receiving objects: 100% (1871146/1871146), 1.07 GiB | 40.51 MiB/s, done.
Resolving deltas: 100% (1289234/1289234), done.
[2020-04-04][21:54:52][nix_vuln_scanner::git::clone][INFO] Locking "nixpkgs/repo/68b00da0eb30d2acc9c57208a4f921c1/clone.lock"
[2020-04-04][21:54:52][nix_vuln_scanner::git::clone][INFO] Fetching from origin in "nixpkgs/repo/68b00da0eb30d2acc9c57208a4f921c1/clone"
From git://github.com/nixos/nixpkgs
* branch HEAD -> FETCH_HEAD
[2020-04-04][21:54:53][nix_vuln_scanner::git::clone][INFO] Locking "nixpkgs/repo/68b00da0eb30d2acc9c57208a4f921c1/nix-vuln-scanner/nixos-20.03.lock"
[2020-04-04][21:54:53][nix_vuln_scanner::git::clone][INFO] Initial cloning of git://github.com/nixos/nixpkgs to "nixpkgs/repo/68b00da0eb30d2acc9c57208a4f921c1/nix-vuln-scanner/nixos-20.03"
Cloning into 'nixpkgs/repo/68b00da0eb30d2acc9c57208a4f921c1/nix-vuln-scanner/nixos-20.03'...
Updating files: 100% (21392/21392), done.
…
Switched to a new branch 'nixos-20.03'
[2020-04-04][21:54:58][nix_vuln_scanner::report][INFO] Using cached data only. If no local data is available it will still be fetched.
[2020-04-04][21:54:58][nix_vuln_scanner::report][INFO] Fetching the local nixos packages
[2020-04-04][21:54:58][nix_vuln_scanner::fs_cache][INFO] File packages.json not found: Os { code: 2, kind: NotFound, message: "No such file or directory" }
[2020-04-04][21:55:06][nix_vuln_scanner::report::nix][INFO] Removing all occurences of /home/andi/dev/…/nixpkgs/repo/68b00da0eb30d2acc9c57208a4f921c1/nix-vuln-scanner/nixos-20.03 in meta.position
[2020-04-04][21:55:06][nix_vuln_scanner::report][INFO] Fetching NVD CVE database
[2020-04-04][21:55:06][nix_vuln_scanner::fs_cache][INFO] Read nvd-2002.json.gz from local cache.
…
[2020-04-04][21:55:06][nix_vuln_scanner::fs_cache][INFO] Read nvd-2019.json.gz from local cache.
[2020-04-04][21:55:10][nix_vuln_scanner::fs_cache][INFO] File nvd-2020.json.gz not found: Os { code: 2, kind: NotFound, message: "No such file or directory" }
[2020-04-04][21:55:10][nix_vuln_scanner::fs_cache][INFO] Filling local cache with https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2020.json.gz.
[2020-04-04][21:55:11][nix_vuln_scanner::report][INFO] Filtering package list...
[2020-04-04][21:55:17][nix_vuln_scanner::report::nix_patches][INFO] failed to execute nix eval for ruby_2_5: error: value is a string with context while a set was expected, at (string):11:89
[2020-04-04][21:55:17][nix_vuln_scanner::report::nix_patches][INFO] failed to execute nix eval for ruby: error: value is a string with context while a set was expected, at (string):11:89
[2020-04-04][21:55:18][nix_vuln_scanner::report::nix_patches][INFO] failed to execute nix eval for ruby_2_7: error: value is a string with context while a set was expected, at (string):11:89
[2020-04-04][21:55:18][nix_vuln_scanner::report::nix_patches][INFO] failed to execute nix eval for plasma-desktop: error: value is a string with context while a set was expected, at (string):11:89
[2020-04-04][21:55:18][nix_vuln_scanner::report::nix_patches][INFO] failed to execute nix eval for e2fsprogs: error: value is null while a set was expected, at (string):11:89
[2020-04-04][21:55:20][nix_vuln_scanner::report][INFO] Found 1667 interesting packages.

The above command produces a my-report.json file containing information about the analyzed revision and the potential issues that have been found:

{
"version": "1",
"repo": "git://github.com/nixos/nixpkgs",
"revision": "nixos-20.03",
"channel_name": "local",
"date_time": "2020-04-04 19:55:23.055948016 UTC",
"commit_time": null,
"advance_time": null,
"packages": {
…
"cairo": {
"attribute_name": "cairo",
"package_name": "cairo",
"version": "1.16.0",
"cves": [
{
"name": "CVE-2018-19876",
"patched": true
},
{
"name": "CVE-2019-6461",
"patched": false
},
{
"name": "CVE-2019-6462",
"patched": false
}
],
"patches": [
"CVE-2018-19876.patch"
]
},
…
}
}

Once you generated the JSON file, you can run all sorts of queries on it. For example, you can list all the open (unpatched) CVEs:

$ jq -r '.packages[]|select(.cves[].patched == false)|[.attribute_name, (.cves|map(.name)|join(","))] | @tsv' < my-report.json | uniq | column -t | head -n5
libmad CVE-2018-7263
gnupg1orig CVE-2018-12020
ansible_2_6 CVE-2020-10684,CVE-2020-1733,CVE-2020-1735,CVE-2020-1736,CVE-2020-1738,CVE-2020-1739,CVE-2020-1740
ncompress CVE-2001-1413
lepton CVE-2017-7448,CVE-2017-8891,CVE-2018-12108

(If you are a jq wizard and have a better way to obtain the results above, please let me know :))

To generate the data displayed on broken.sh, the above procedure is run for every channel revision (meaning all those commits have passed hydra and arrived on the channel). Instead of analyzing each of them manually there is a handy subcommand that downloads the list of revisions from Graham Christensen’s archive over at https://channels.nix.gsc.io.

Once all the revisions have been analyzed, they are imported into the sqlite database through the process-db subcommand. As of now, the database only serves the purpose of providing a data backend for the web interface.

At the time of writing I am updating all revisions of NixOS 18.03, NixOS 18.09, NixOS 19.03, NixOS 19.09, NixOS 20.03 and NixOS unstable at least once every day. (More frequently isn’t really possible without optimizing the code further. It already runs for almost a full day.). That means even if we gain knowledge of new CVEs today we will be able tell since when we have been exposed to the vulnerabilities.

What you can do right now

broken.sh might not yet be very user-friendly and also not very mature (in many regards), but it can be used for a bunch of useful information:

Overview across all the channels on the index:

$ http --json https://broken.sh
{ "statistics": [
[
"nixos-unstable",
{
"unpatched": 941,
"patched": 118
}
],
[
"nixos-20.03",
{
"unpatched": 1072,
"patched": 120
}
],
[
"nixos-19.09",
{
"unpatched": 1340,
"patched": 183
}
],
[
"nixos-19.03",
{
"unpatched": 2304,
"patched": 195
}
],
[
"nixos-18.09",
{
"unpatched": 5311,
"patched": 120
}
],
[
"nixos-18.03",
{
"unpatched": 5482,
"patched": 107
}
]
]
}

List of open issues in a channel in a revision:

$ http --json https://broken.sh/channels/nixos-unstable
{
"channel": {
"id": 1,
"name": "nixos-unstable"
},
"commit": {
"id": 841,
"revision": "57f2ea5ca13ee68bdf9f2e9965ee251301b7b8e8",
"commit_time": "2020-02-27 10:34:00 UTC"
},
"packages_with_issues": [
{
"name": "a2ps",
"attribute_name": "a2ps",
"versions": [
{
"version": "4.14",
"issues": [
{
"id": 429,
"identifier": "CVE-2001-1593"
},
{
"id": 430,
"identifier": "CVE-2014-0466"
},
{
"id": 431,
"identifier": "CVE-2015-8107"
}
],
"patches": [
{
"id": 176,
"name": "09_CVE-2001-1593.diff"
},
{
"id": 177,
"name": "CVE-2014-0466.diff"
}
]
…

History of a CVE and which channel revisions are affected by it:

$ http --json https://broken.sh/issues/CVE-2001-1593
{
"issue": { "id": 429, "identifier": "CVE-2001-1593" },
"packages": [
{
"package": { "id": 286, "name": "a2ps", "attribute_name": "a2ps" },
"channels": [
{
"channel": { "id": 2, "name": "nixos-18.03" },
"bumps_with_versions": [
{
"channel_bump": { "id": 146, "channel_id": 2, "commit_id": 146, "channel_bump_date": "2019-02-20 10:25:14 UTC" },
"commit": { "id": 146, "revision": "cb0e20d6db96fe09a501076c7a2c265359982814", "commit_time": "2018-08-11 14:42:43 UTC"
},
"versions": [
{
"version": "4.14",
"patches": [
{
"id": 176,
"name": "09_CVE-2001-1593.diff"
},
{
"id": 177,
"name": "CVE-2014-0466.diff"
},
{
"id": 178,
"name": "fix-format-security.diff"
}
]
}
]
},

**Diff of two revisions to see what issues have been resolved, remained the

same or been fixed**: Currently there is no UI to select which revisions to compare. The format of the URLs should be self-explanatory:

$ http --json https://broken.sh/diff/revisions/27a5ddcf747fb2bb81ea9c63f63f2eb3eec7a2ec/4cd2cb43fb3a87f48c1e10bb65aee99d8f24cb9d
{
"old": { "id": 655, "revision": "27a5ddcf747fb2bb81ea9c63f63f2eb3eec7a2ec", "commit_time": "2019-10-14 19:23:11 UTC" },
"new": { "id": 678, "revision": "4cd2cb43fb3a87f48c1e10bb65aee99d8f24cb9d", "commit_time": "2019-10-23 18:19:15 UTC" },
"issues": {
"common": [
{
"attribute_name": "a2ps",
"version": "4.14",
"issue": "CVE-2015-8107"
},
…
],
"new": [
{
"attribute_name": "clamav",
"version": "0.102.0",
"issue": "CVE-2020-3123"
},
…
],
…

Future work

Integration with ofBorg: I have been thinking about integrating this with ofBorg. The idea is that reviewers, submitters and commiters will make better informed decisions if they know what the potential security impact of a proposed change is.

I imagine having a PR flagged as security critical could draw more attention to it. This is especially relevant when individual contributors try to fix security-relevant changes but they do not have any means to draw attention to it. They basically disappear in the flood of PRs that are hitting Nixpkgs constantly.
Paper trail: We frequently get questions about our patching policies, why something wasn’t fixed yet, etc. Having some kind of paper trail where we (try to) document findings would help users, developers and any interested party figure out why a specific issue wasn’t fixed on NixOS.

For instance, there are often issues in a sort of undefined state. That might be due to projects being dead, nobody having written a fix, the scope of exploitation being too narrow or any other reason and a combination of these. We should be able to document findings (and solutions, comments, …) about an issue in a way that allows for automated analysis.

More than once have I started working through the entire list of open issues. Sometimes I am running into the very same issues I saw six months ago, but totally forgot about. I tried adding a way to take notes on open and resolved issues with the notes subcommand but haven’t made any (or much) progress.

The tricky part is to get a proper structure that is editable manually and that can be versioned with something like Git. I have my opinions on the topic and over the years there have been several discussions in #nixos-security. One of the (other) long term goals in this regard is being able to produce NixOS Security Advisories (again). But that is probably material for a dedicated blog entry…

To sum it up…

Have I managed everything I was initially targeting? No, but most of it.

broken.sh is already quite usable and provides value as is. In the worst case it just provides me with a list of issues that I should go investigate when bored.

For the most part of a year I haven’t really changed much of the code. It just runs unattended without causing me any extra work. That is really nice and I am actually proud of it. When all of this started I neither really understood Nix nor Rust. More senior Rust developers are probably disgusted by some of the code that I produced but that is fine by me. It was my first “real” rust project and I would probably do things differently these days.

There is plenty of room for improvement. I just reached a point where fighting this alone doesn’t feel fruitful anymore. While working on broken.sh I went from many Nixpkgs PRs a day to just a few a week. Being exposed to the sheer volume of Nixpkgs' issues is tiresome. I often end up diving into some rabbit hole and lose track of time. By the time I am done fixing it, it’s well past bed time and my evening is gone.

I occasionally receive feedback that encourages me to continue the work. That is what made me consider writing this blog post. Conversations are lossy and aren’t easily replicated, so here is an overview of my project, in the hope that it is, too, useful to you.

If you have a neat idea or some matter that you want to discuss, please do not hesitate to contact me (mail, IRC).

Many thanks to Ana, Felix and Florian for reviewing and proof-reading this post. 💗

The name broken.sh is just a stupid pun on how broken everything is. Not really a statement about how broken NixOS is. Just a nice domain that I did come up with at the time. :) ↩︎

SaltStack, Ansible, Puppet, … how can we share data between servers?

Sat, 22 Apr 2017 16:50:00 +0000

WARNING: This is still a draft somewhat..

Lately I’ve been depressed by the lack of automation in my life or really in the projects I take part in.

Most of my stuff is not hosted on AWS, Azure, Digitalocean or other big “cloud” players. That is due to multiple reasons:

We may have hardware, rack, power and network capacity available which is basically for free.
We might need physical connectivity to the machine - e.g. connecting to a local WiFi mesh network, a beamer, some audio equipment and so on…
We want to remain in control of our data. That means we are not going to trust some random internet business.
It is cheaper, even if you’ve to pay for the servers that are running 24/7. This might not be true for a website that sells boat trips on the Atlantic Ocean and gets high traffic spikes every 2nd Sunday from April to December but for our scenarios it works quite well.

At the office I’ve been using puppet more then six years and I’m quite happy with it. In the local Freifunk-Community (https://darmstadt.freifunk.net) we are using SaltStack. For my personal infrastructure (E-Mail, Webhosting, random docker containers, pastebin, dn42, …) I’m using Ansible (https://ansible.com).

Neither of those deployments are very trivial, they are probably just the average project for either of those tools.

A few weeks back I and a few others started writing SaltStack states for an IRC-Network that we are running. If you think about automating the deployment of IRCd’s it is rather easier. Create a VM, install your favorite flavor of linux, install build dependencies for the IRCd of your choice and generate the configuration. That’s basically it.

We’ve been using TLS in our network for about 10 years now. (Both user and server2server communication). Since it wasn’t and still is not desirable to buy wildcard certificates or share certificates between servers we are running our own CA (including OCSP service etc.). With the switch to automated server configuration we also want to deploy LetsEncrypt - actually LetsEncrypt encouraged us to start using some kind of automation since humans aren’t reliable and having to jupe a server every 3 months because the admin still didn’t configure a LetsEncrypt cronjob is not what we are after.

So I started out writing some states and a Vagrantfile (https://gist.github.com/andir/2188da38904557a58cd7df29fd277275) for easier testing of the whole stack. After tackling a few issues with Vagrant and different boxes (as vagrant calls VM images) everything looked fine.

We basically have a zoo of VMs on our laptops wich simulate the production network. At least so we thought.

For the links between the IRCDs we are using TLS. So far we did exchange fingerprints, manually add them to the ircd.conf, /quote rehash and off you go.

When you automate the deployment you also automate the configuration of links between your servers. While you can configure send_password and receive_password since what I do know when that isn’t what we want to rely on when sending chat messages around the globe. So we still need to exchange certificate fingerprints. One might say that TLSA records are made for that but I wonder why I can’t use the already verified connection between my Salt Master and the Salt Minion. If you query DNS for TLSA records it requires your DNS to be up, DNSSEC to work, your local resolver to have a copy of the root dns zone or an upstream resolver that (hopefully) verifies DNSSEC - unlike OpenDNS.

Since we are also running our own DNS Servers adding them as a dependency isn’t really a great idea. We might be recovering from the death of some parts of the infrastructure. The only reasonable approach to me is to use the authenticated relationship from minion and master to exchange public keys, fingerprints etc.

While I’m currently focusing on exchanging certificate fingerprints and/or public keys this applies to a bunch of scenarios you might come acress when you decide to drop all shared secrets and generate them on your servers. A few of those could be:

Generating and exchanging TSIG keys for nsupdate between your DNS Servers and minions.
Communicating a shared secret for database access, borg backups, …
Distributing SSHFPs between your servers (local verification, publishing them to DNS as SSHFP records,…)
Trust bootstrapping (after minion key verification) for any kind of internal secret database

With SaltStack you’ve a few options to transfer data from a minion to the master / other minions:

use the Mine to export data
use grains
use pillars (with pre-generated certificates, keys, …) -> not the desired solution
trigger events with custom context
use a vault

In puppet you could use exported ressources, in Ansible you could write to some kind of custom api without much overhead or query/configure the other side on-demand since a single run isn’t restricted to a single server.

Puppet has solved the issue for many years - people tend to not like puppet for various reasons. Maybe because it feels like programming?

Using the Salt Mine

The Salt Mine is a handy construct when it comes to exporting data from a node as long as that data only exists as some kind of list or with a very low cardinality. Also it only makes sense if the data is mostly static and doesn’t change. Changes to the exported data (e.g. another information, removal of information, …) require changes to the minion configuration. This isn’t really practical if you are trying to write modular states where not everything is hard-wired with another state.

For reference this is how you would export information about a certificate:

mine_functions:
my_awesome_certificate:
- mine_function: tls.cert_info
- /etc/letsencrypt/live/.../cert.pem

This has to be deployed on each minion and you probably also have to restart the minion when changeing the file.

More information on the salt mine can be found at https://docs.saltstack.com/en/latest/topics/mine/.

Trying to export data via Grains

I’m not going to get into the details or provide a example configuration here. I’ve written a bunch of grains and they just work. The limitation is that you can’t really attribute them. So you can’t tell your custom grains what to export without changing the code or introducing custom configuration files, adding stuff to the minion config etc.. While this approach would automatically propagate new certificate fingerprints, public keys, … you still have to use the Salt Mine to actually export them. Which isn’t that bad.

The lack of configuration options for custom grains kills this approach for me.

Pillars

LOLNOPE. Using pillars would require manual collection of fingerprints or (even worse) central management of all the certificates.

This simply doesn’t work for me.

Using a custom event

This is what seems to be a promising but ephemeral approach to the issue.

The basic idea is:

Create the public-/private key pair on the minion.
On changes to that fail (creation, key rollover, new certificate, …) execute a script with the filename as argument
The (bash) script then extracts the information we want from the file. (using openssl or other command line tools) and publishes those information using something like
```
salt-call event.send my/custom/event/certificate-changed '{"certfp": "ABCDEF01234567890…"}'
```

In order to use the “published” fingerprint other minions must be up, running and listening to those events. Otherwise the information is lost and nobody cares.

This approach only works if we figure out how to store the fingerprints - after receiving an event.

It basically sucks as bad as the others but we might be able to configure the links after randomly restarting salt-minions and running salt '*' state.apply …

Using a vault

At the time of this writing this seems to be a valid option. I’ve not tried it yet. There will be an update on this soon (tm).

Reading of the vault seems to be rather easy. You can also write to it but only using infromation that is availbe during state-rendering. So I’m not sure what the benefit is. One could probably combine this with the event approach to store the public keys of the minions in the vault by listening to key events on the master.

Conclusion

This world sucks. We need better tools :/

Juniper MCD decided to coredump on commit and rollback

Tue, 08 Nov 2016 15:00:00 +0000

On a Juniper EX3300 a colleague of mine entered an invalid statement:

 interface-range some_ports {
member ge0/0/2;
member-range ge-0/0/0 to ge-0/0/1;
}

The member ge0/0/2 is missing a - between ge and 0/0/2. Juniper (for whatever reason) accepted the input but mcd decided to segfault when asked to delete or rollback the configuration.

 pid 75982 (mgd), uid 0: exited on signal 11 (core dumped)

Recovering from that case is actually not that hard. You just know the right command(s) ;-)

You can load an older configuration via the load override <config file> command. I did also try the load replace <config file> command but that also segfaulted..


andi@foo# load override ?
Possible completions:
<filename> Filename (URL, local, remote, or floppy)
db/ Last changed: Oct 27 10:44:35
juniper.conf+.gz Size: 6000, Last changed: Nov 08 15:12:55
juniper.conf.1.gz Size: 5913, Last changed: Jun 30 13:36:52
juniper.conf.2.gz Size: 5881, Last changed: Jun 30 12:59:25
juniper.conf.3.gz Size: 5280, Last changed: Jun 30 12:54:02
[...]
{master:0}[edit]
andi@foo# load override juniper.conf+.gz
load complete

In my case the juniper.conf+.gz was the desired config file. I recommend inspecting those files before loading them (they are stored in /config/).

Using VRFs with linux and systemd-networkd

Sun, 09 Oct 2016 13:00:00 +0000

While working on a systemd-networkd patch to implement (at least basic) VRF interfaces I did write :doc:my other post <linux-ip-vrf>. This post should give you a brief example on how you can create a VRF with systemd-networkd.

At this point it really only created the interfaces and enslaves potential customer interfaces to a given VRF.

You still have to implement all the ip rule-stuff yourself. For example a systemd.unit file might be the right approach which is executed/started after the network is “up”.

First you’ve to create the systemd.netdev vrf-customer1.netdev file:

After restarting systemd-networkd with systemctl restart systemd-networkd you should see the corresponding interface:

$ ip -d link show dev vrf-customer1
9: vrf-customer1: <NOARP,MASTER> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 02:74:c7:e1:de:64 brd ff:ff:ff:ff:ff:ff promiscuity 0
vrf table 42 addrgenmode eui64 numtxqueues 1 numrxqueues 1

Note the last line which states vrf table 42.

To add an interface to the VRF you’ll have to modify/create the corresponding .network file. This is how the file /etc/systemd/network/enp0s31f6.network would look on my notebook:

Restarting systemd-networkd again and checking the status using ip -d link gives us:

$ip -d link show dev enp0s31f6
3: enp0s31f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master vrf-customer1 state DOWN mode DEFAULT group default qlen 1000
link/ether 50:7b:9d:cf:34:dc brd ff:ff:ff:ff:ff:ff promiscuity 0
vrf_slave table 42 addrgenmode eui64 numtxqueues 1 numrxqueues 1

Again note the last line which states vrf_slave table 42. Also in the first line you can see that it belongs to the VRF vrf-customer.

And that is all for now. You still have to add the ip rule commands in some way or another (there is no support in systemd-networkd yet and I did not have a good idea without inventing ip rule management in systemd).

Using VRFs with linux

Tue, 14 Jun 2016 17:00:00 +0000

Ever since I’ve heard about VRF support (or VRF-lite like it is called in Documentation/network/vrf.txt) I wanted to start tinkering with it. Since the topic is currently only covered in the previously mentioned linux kernel documentation I thought it would be a good idea to post some notes.

It basically boils down to adding an VRF interface and creating two ip rule-entries.

I’m using a local VM with ArchLinux since the VRF feature seems to require a rather recent kernel. My experience with kernels below version 4.6 weren’t that great.

$ ip -br link # this is where we are start off
lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
ens3 DOWN 52:54:00:12:34:56 <BROADCAST,MULTICAST>

Now are adding a new interface named vrf-customer1 with the table customer1 assigned to it. The table parameter is used to place routes from your devices within the VRF into the right routing table

$ ip link add vrf-customer1 type vrf table customer1
$ ip -d link show vrf-customer1 # verify that the interface indeed exists and has the correct table assigned to it
4: vrf-customer1: <NOARP,MASTER> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether ca:22:59:ba:05:da brd ff:ff:ff:ff:ff:ff promiscuity 0
vrf table 100 addrgenmode eui64

Next: redirect the traffic from and to the vrf to the customer1 table and verify the rules are indeed as expected:

$ ip -4 rule add oif vrf-customer1 lookup customer1
$ ip -4 rule add iif vrf-customer1 lookup customer1
$ ip -6 rule add oif vrf-customer1 lookup customer1
$ ip -6 rule add iif vrf-customer1 lookup customer1
$ ip -4 rule
0: from all lookup local
32764: from all iif vrf-customer1 lookup customer1
32765: from all oif vrf-customer1 lookup customer1
32766: from all lookup main
32767: from all lookup default
$ ip -6 rule
0: from all lookup local
32764: from all oif vrf-customer1 lookup customer1
32765: from all iif vrf-customer1 lookup customer1
32766: from all lookup main

To make any use of our VRF we will have to add an device to it. In my case I’ll add the only available “physical” device ens3.

$ ip link set ens3 master vrf-customer1
$ # verify the interface is indeed a member of the VRF
$ ip -br link show master vrf-customer1
ens3 DOWN 52:54:00:12:34:56 <BROADCAST,MULTICAST>

Now that we’ve an interface to receive send send packets with it we should consider adding an IP-Address to it. Since IPv6 is enabled per default we don’t need to configure a LL-Address for that protocol.

 $ # add an IP to the interface
$ ip addr add 10.0.0.1/24 dev ens3
$ ip route show table customer1
local 10.0.0.1 dev ens3 proto kernel scope host src 10.0.0.1

Seeing a route like that might confuse the average linux user. Those routes usually exist within the local table which you can check via ip route show table local

The route to the /24 we’ve added is still missing from the interface. Why is that? You’ll have to change the state of the interface to “UP”:

 $ ip link set ens3 up
$ ip route show table customer1
broadcast 10.0.0.0 dev ens3 proto kernel scope link src 10.0.0.1
10.0.0.0/24 dev ens3 proto kernel scope link src 10.0.0.1
local 10.0.0.1 dev ens3 proto kernel scope host src 10.0.0.1
broadcast 10.0.0.255 dev ens3 proto kernel scope link src 10.0.0.1
$ ip -6 route show table customer1
local fe80::5054:ff:fe12:3456 dev lo proto none metric 0 pref medium
fe80::/64 dev ens3 proto kernel metric 256 pref medium
ff00::/8 dev vrf-customer1 metric 256 pref medium
ff00::/8 dev ens3 metric 256 pref medium

suddenly routes \o/

Using multiple client classes with ISC DHCPd

Mon, 23 May 2016 11:00:00 +0000

Since the internet is lacking examples of how to use multiple classes with a single pool here is one:

 class "mac-filtered-clients"
{
match binary-to-ascii (16, 8, ":", substring(hardware, 1, 6));
}
subclass "mac-filtered-clients" "50:7b:00:00:00:00"; # some cool host!
class "J-client" {
spawn with option agent.circuit-id;
match if (substring(option agent.circuit-id, 30, 9) = "foo-bar");
lease limit 1;
}
subnet 192.168.0.0 255.255.0.0 {
pool {
range 192.168.0.10 192.168.0.150;
allow members of "J-client";
allow members of "mac-filtered-clients";
}
}

This isn’t very special compared to a setup with just a single class but it can be confusing since debugging classes is a pita.. One pitfall I did run into was using the byte representation of the mac-addresses (without the quotes) and using match hardware;. The example above works for me (tm).

Postgresql-tmpfs with systemd.socket-activation for local (ephemeral) data during development

Fri, 22 Apr 2016 09:50:09 +0000

During development of database related stuff you commonly run into the “issue” (or non-issue depending on your taste) of running a local database server - or multiple of those.

In my case I have to run a local postgresql server on my notebook. I asked myself: I’m not always developing on that piece of software, and I do not always require or want a local postgresql server. What can I do about that?!?

On top of that using my precious SSD to store data I am going to delete anyway souds like a waste (or money). In my development environment I can and want to safely wipe the data often. Also most of the database load comes from running test cases anyway. That stuff doesn’t need to end up on my (slow, compared to RAM) disk. Using a tmpfs for that kind of stuff sounds much saner to me.

The part of running a repetitive clean database setup sounded like the use case for a container based thing. These days docker is pretty “hot” and it solves the issue of distributing re-useable images. There is an official postgresql image on docker hub for various versions of postgresql. I’ve simply build a new image based on that. It is available on docker hub (https://hub.docker.com/r/andir/postgresql-tmpfs/) or if you prefer to build it on your own you can download the Dockerfile on GitHub (https://github.com/andir/postgresql-tmpfs).

Now that we are past the introductional blabla here are the systemd unit files I’m using to achieve this:

You can either put those unit files in /etc/systemd/system or install them as systemd-user units in ~/.config/systemd/user.

systemctl daemon-reload
systemctl enable postgresql-docker.socket

If you try to connect to the postgresql server (nc 127.0.0.1 5432) you can observe the container while it is starting (journalctl -f).

The default username, password and database name is postgres. You can change that by modifying the startup arguments of the docker container. Those are documented at https://hub.docker.com/_/postgres/.

Happy data trashing \o/

P.S.: If you’ve an idea on how to stop the service after x minutes of inactivity please let me know. Stopping the service manually isn’t really what I’m after.

About

Mon, 01 Jan 0001 00:00:00 +0000

I like hacking on software projects, taking pictures or just listening to music. These days I spent my days (and most evenings) with Nix releated topics.

Some of my projects:

npmlock2nix npm-lockfile.json based Node build tooling for Nix
broken.sh NixOS and Nixpkgs vulnerability database
ipv6.watch IPv6 deployment status on some (random) public websites
glaskugel.io Searx instance that tunnels all search traffic through tor
darmstadt.io Public transit map of my home town (Darmstadt)

(Incomplete, probably forgot a few)

Contact

You can reach me via

IRC (nick: andi-, networks: libera, hackint, …),
XMPP (andi@kack.it),
Matrix (@andi:kack.it) or
email (andreas (@) rammhold.de).