Introducing broken.sh
A few years ago, I started working on broken.sh1, a security issue tracker for NixOS and Nixpkgs. Ever since I moved most of my private computing into the Nix ecosystem, I’ve noticed a lack of work in the field of security issues. As I’ve been trying to improve the situation (by contributing to Nixpkgs), it became obvious that we lack proper tooling in the area. I wasn’t just interested in which packages were broken on my machines. I wanted to tackle all the issues throughout Nixpkgs.
While broken.sh started out as a tool to just give me a short list of actionable items, it has somewhat grown. Right now I like to see it as an attempt at creating something other distributions like Debian, Arch Linux, Gentoo and many more have had for many many years as part of their infrastructure. NixOS should have something similar. I hope this isn’t the final answer to that question, but a (first?) step in the right direction.
While people aren’t usually that bad on tracking new highly visibile issues and fixing them, there are still plenty of not-so-famous issues that just never get any attention. A part of the problem is likely to be their discoverability, or the lack of it. The only issue tracking done for NixOS is one big lump of issues in the GitHub issue tracker. However, GitHub issues cannot really be categorized into packages, releases, maintainers, commits, etc. I figured out early on that creating an overview outside of GitHub would be the best way forward.
In the very early days, broken.sh was just a static HTML page providing an overview of one specific revision, listing all the issues that I managed to identify. That was okay as a one-shot report but did not fit my long-term vision.
Below you can see the output of the very first working version. Back then the project was still called an “update tracker”.
I also tried to provide information on which upstream versions are available for a given package - hoping to eventually tackle that issue, too (hah!). I eventually decided to focus on the security issues.
How does it work?
In a nutshell, I am looking at every revision of a NixOS channel and extract all the package versions and patches. For each of these, I try to figure out if a recent snapshot of the NVD database contains relevant information for them. If there is a positive match, it will be shown on the website.
My current approach consists of multiple steps. Each step is available as a
subcommand of the nix-vuln-scanner binary:
$ ./result/bin/nix-vuln-scanner
USAGE:
nix-vuln-scanner [FLAGS] <SUBCOMMAND>
FLAGS:
-d, --debug enable debug logging
-h, --help Prints help information
-V, --version Prints version information
SUBCOMMANDS:
channel-report Generates reports for all "bumps" in a channel
help Prints this message or the help of the given subcommand(s)
notes List, create and edit notes
process-db Process the output of the report command and generate an sqlite database from it
report Generate a new issue report file
serve serves the contents of the DB via HTTP.
You do not always have to run all parts in order to obtain and use the results.
For example, if all you need is a brief ad-hoc information, you shouldn’t have
to run a web server or even deal with a (sqlite) database (file). Instead you
can just run the report subcommand like this:
$ nix-vuln-scanner report -p packages.json -s nixos-20.03 -o my-report.json
[2020-04-04][21:53:42][nix_vuln_scanner::git::clone][INFO] Locking "nixpkgs/repo/68b00da0eb30d2acc9c57208a4f921c1/clone.lock"
[2020-04-04][21:53:42][nix_vuln_scanner::git::clone][INFO] Initial cloning of git://github.com/nixos/nixpkgs to "nixpkgs/repo/68b00da0eb30d2acc9c57208a4f921c1/clone"
Cloning into bare repository 'nixpkgs/repo/68b00da0eb30d2acc9c57208a4f921c1/clone'...
remote: Enumerating objects: 129, done.
remote: Counting objects: 100% (129/129), done.
remote: Compressing objects: 100% (109/109), done.
remote: Total 1871146 (delta 5), reused 42 (delta 2), pack-reused 1871017
Receiving objects: 100% (1871146/1871146), 1.07 GiB | 40.51 MiB/s, done.
Resolving deltas: 100% (1289234/1289234), done.
[2020-04-04][21:54:52][nix_vuln_scanner::git::clone][INFO] Locking "nixpkgs/repo/68b00da0eb30d2acc9c57208a4f921c1/clone.lock"
[2020-04-04][21:54:52][nix_vuln_scanner::git::clone][INFO] Fetching from origin in "nixpkgs/repo/68b00da0eb30d2acc9c57208a4f921c1/clone"
From git://github.com/nixos/nixpkgs
* branch HEAD -> FETCH_HEAD
[2020-04-04][21:54:53][nix_vuln_scanner::git::clone][INFO] Locking "nixpkgs/repo/68b00da0eb30d2acc9c57208a4f921c1/nix-vuln-scanner/nixos-20.03.lock"
[2020-04-04][21:54:53][nix_vuln_scanner::git::clone][INFO] Initial cloning of git://github.com/nixos/nixpkgs to "nixpkgs/repo/68b00da0eb30d2acc9c57208a4f921c1/nix-vuln-scanner/nixos-20.03"
Cloning into 'nixpkgs/repo/68b00da0eb30d2acc9c57208a4f921c1/nix-vuln-scanner/nixos-20.03'...
Updating files: 100% (21392/21392), done.
…
Switched to a new branch 'nixos-20.03'
[2020-04-04][21:54:58][nix_vuln_scanner::report][INFO] Using cached data only. If no local data is available it will still be fetched.
[2020-04-04][21:54:58][nix_vuln_scanner::report][INFO] Fetching the local nixos packages
[2020-04-04][21:54:58][nix_vuln_scanner::fs_cache][INFO] File packages.json not found: Os { code: 2, kind: NotFound, message: "No such file or directory" }
[2020-04-04][21:55:06][nix_vuln_scanner::report::nix][INFO] Removing all occurences of /home/andi/dev/…/nixpkgs/repo/68b00da0eb30d2acc9c57208a4f921c1/nix-vuln-scanner/nixos-20.03 in meta.position
[2020-04-04][21:55:06][nix_vuln_scanner::report][INFO] Fetching NVD CVE database
[2020-04-04][21:55:06][nix_vuln_scanner::fs_cache][INFO] Read nvd-2002.json.gz from local cache.
…
[2020-04-04][21:55:06][nix_vuln_scanner::fs_cache][INFO] Read nvd-2019.json.gz from local cache.
[2020-04-04][21:55:10][nix_vuln_scanner::fs_cache][INFO] File nvd-2020.json.gz not found: Os { code: 2, kind: NotFound, message: "No such file or directory" }
[2020-04-04][21:55:10][nix_vuln_scanner::fs_cache][INFO] Filling local cache with https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2020.json.gz.
[2020-04-04][21:55:11][nix_vuln_scanner::report][INFO] Filtering package list...
[2020-04-04][21:55:17][nix_vuln_scanner::report::nix_patches][INFO] failed to execute nix eval for ruby_2_5: error: value is a string with context while a set was expected, at (string):11:89
[2020-04-04][21:55:17][nix_vuln_scanner::report::nix_patches][INFO] failed to execute nix eval for ruby: error: value is a string with context while a set was expected, at (string):11:89
[2020-04-04][21:55:18][nix_vuln_scanner::report::nix_patches][INFO] failed to execute nix eval for ruby_2_7: error: value is a string with context while a set was expected, at (string):11:89
[2020-04-04][21:55:18][nix_vuln_scanner::report::nix_patches][INFO] failed to execute nix eval for plasma-desktop: error: value is a string with context while a set was expected, at (string):11:89
[2020-04-04][21:55:18][nix_vuln_scanner::report::nix_patches][INFO] failed to execute nix eval for e2fsprogs: error: value is null while a set was expected, at (string):11:89
[2020-04-04][21:55:20][nix_vuln_scanner::report][INFO] Found 1667 interesting packages.
The above command produces a my-report.json file containing information about
the analyzed revision and the potential issues that have been found:
{
"version": "1",
"repo": "git://github.com/nixos/nixpkgs",
"revision": "nixos-20.03",
"channel_name": "local",
"date_time": "2020-04-04 19:55:23.055948016 UTC",
"commit_time": null,
"advance_time": null,
"packages": {
…
"cairo": {
"attribute_name": "cairo",
"package_name": "cairo",
"version": "1.16.0",
"cves": [
{
"name": "CVE-2018-19876",
"patched": true
},
{
"name": "CVE-2019-6461",
"patched": false
},
{
"name": "CVE-2019-6462",
"patched": false
}
],
"patches": [
"CVE-2018-19876.patch"
]
},
…
}
}
Once you generated the JSON file, you can run all sorts of queries on it. For example, you can list all the open (unpatched) CVEs:
$ jq -r '.packages[]|select(.cves[].patched == false)|[.attribute_name, (.cves|map(.name)|join(","))] | @tsv' < my-report.json | uniq | column -t | head -n5
libmad CVE-2018-7263
gnupg1orig CVE-2018-12020
ansible_2_6 CVE-2020-10684,CVE-2020-1733,CVE-2020-1735,CVE-2020-1736,CVE-2020-1738,CVE-2020-1739,CVE-2020-1740
ncompress CVE-2001-1413
lepton CVE-2017-7448,CVE-2017-8891,CVE-2018-12108
(If you are a jq wizard and have a better way to obtain the results above,
please let me know :))
To generate the data displayed on broken.sh, the above procedure is run for every channel revision (meaning all those commits have passed hydra and arrived on the channel). Instead of analyzing each of them manually there is a handy subcommand that downloads the list of revisions from Graham Christensen’s archive over at https://channels.nix.gsc.io.
Once all the revisions have been analyzed, they are imported into the sqlite
database through the process-db subcommand. As of now, the database only
serves the purpose of providing a data backend for the web interface.
At the time of writing I am updating all revisions of NixOS 18.03, NixOS 18.09, NixOS 19.03, NixOS 19.09, NixOS 20.03 and NixOS unstable at least once every day. (More frequently isn’t really possible without optimizing the code further. It already runs for almost a full day.). That means even if we gain knowledge of new CVEs today we will be able tell since when we have been exposed to the vulnerabilities.
What you can do right now
broken.sh might not yet be very user-friendly and also not very mature (in many regards), but it can be used for a bunch of useful information:
Overview across all the channels on the index:
$ http --json https://broken.sh { "statistics": [ [ "nixos-unstable", { "unpatched": 941, "patched": 118 } ], [ "nixos-20.03", { "unpatched": 1072, "patched": 120 } ], [ "nixos-19.09", { "unpatched": 1340, "patched": 183 } ], [ "nixos-19.03", { "unpatched": 2304, "patched": 195 } ], [ "nixos-18.09", { "unpatched": 5311, "patched": 120 } ], [ "nixos-18.03", { "unpatched": 5482, "patched": 107 } ] ] }List of open issues in a channel in a revision:
$ http --json https://broken.sh/channels/nixos-unstable { "channel": { "id": 1, "name": "nixos-unstable" }, "commit": { "id": 841, "revision": "57f2ea5ca13ee68bdf9f2e9965ee251301b7b8e8", "commit_time": "2020-02-27 10:34:00 UTC" }, "packages_with_issues": [ { "name": "a2ps", "attribute_name": "a2ps", "versions": [ { "version": "4.14", "issues": [ { "id": 429, "identifier": "CVE-2001-1593" }, { "id": 430, "identifier": "CVE-2014-0466" }, { "id": 431, "identifier": "CVE-2015-8107" } ], "patches": [ { "id": 176, "name": "09_CVE-2001-1593.diff" }, { "id": 177, "name": "CVE-2014-0466.diff" } ] …History of a CVE and which channel revisions are affected by it:
$ http --json https://broken.sh/issues/CVE-2001-1593 { "issue": { "id": 429, "identifier": "CVE-2001-1593" }, "packages": [ { "package": { "id": 286, "name": "a2ps", "attribute_name": "a2ps" }, "channels": [ { "channel": { "id": 2, "name": "nixos-18.03" }, "bumps_with_versions": [ { "channel_bump": { "id": 146, "channel_id": 2, "commit_id": 146, "channel_bump_date": "2019-02-20 10:25:14 UTC" }, "commit": { "id": 146, "revision": "cb0e20d6db96fe09a501076c7a2c265359982814", "commit_time": "2018-08-11 14:42:43 UTC" }, "versions": [ { "version": "4.14", "patches": [ { "id": 176, "name": "09_CVE-2001-1593.diff" }, { "id": 177, "name": "CVE-2014-0466.diff" }, { "id": 178, "name": "fix-format-security.diff" } ] } ] },**Diff of two revisions to see what issues have been resolved, remained the
same or been fixed**: Currently there is no UI to select which revisions to compare. The format of the URLs should be self-explanatory:
$ http --json https://broken.sh/diff/revisions/27a5ddcf747fb2bb81ea9c63f63f2eb3eec7a2ec/4cd2cb43fb3a87f48c1e10bb65aee99d8f24cb9d { "old": { "id": 655, "revision": "27a5ddcf747fb2bb81ea9c63f63f2eb3eec7a2ec", "commit_time": "2019-10-14 19:23:11 UTC" }, "new": { "id": 678, "revision": "4cd2cb43fb3a87f48c1e10bb65aee99d8f24cb9d", "commit_time": "2019-10-23 18:19:15 UTC" }, "issues": { "common": [ { "attribute_name": "a2ps", "version": "4.14", "issue": "CVE-2015-8107" }, … ], "new": [ { "attribute_name": "clamav", "version": "0.102.0", "issue": "CVE-2020-3123" }, … ], …
Future work
Integration with ofBorg: I have been thinking about integrating this with ofBorg. The idea is that reviewers, submitters and commiters will make better informed decisions if they know what the potential security impact of a proposed change is.
I imagine having a PR flagged as security critical could draw more attention to it. This is especially relevant when individual contributors try to fix security-relevant changes but they do not have any means to draw attention to it. They basically disappear in the flood of PRs that are hitting Nixpkgs constantly.
Paper trail: We frequently get questions about our patching policies, why something wasn’t fixed yet, etc. Having some kind of paper trail where we (try to) document findings would help users, developers and any interested party figure out why a specific issue wasn’t fixed on NixOS.
For instance, there are often issues in a sort of undefined state. That might be due to projects being dead, nobody having written a fix, the scope of exploitation being too narrow or any other reason and a combination of these. We should be able to document findings (and solutions, comments, …) about an issue in a way that allows for automated analysis.
More than once have I started working through the entire list of open issues. Sometimes I am running into the very same issues I saw six months ago, but totally forgot about. I tried adding a way to take notes on open and resolved issues with the
notessubcommand but haven’t made any (or much) progress.The tricky part is to get a proper structure that is editable manually and that can be versioned with something like Git. I have my opinions on the topic and over the years there have been several discussions in #nixos-security. One of the (other) long term goals in this regard is being able to produce NixOS Security Advisories (again). But that is probably material for a dedicated blog entry…
To sum it up…
Have I managed everything I was initially targeting? No, but most of it.
broken.sh is already quite usable and provides value as is. In the worst case it just provides me with a list of issues that I should go investigate when bored.
For the most part of a year I haven’t really changed much of the code. It just runs unattended without causing me any extra work. That is really nice and I am actually proud of it. When all of this started I neither really understood Nix nor Rust. More senior Rust developers are probably disgusted by some of the code that I produced but that is fine by me. It was my first “real” rust project and I would probably do things differently these days.
There is plenty of room for improvement. I just reached a point where fighting this alone doesn’t feel fruitful anymore. While working on broken.sh I went from many Nixpkgs PRs a day to just a few a week. Being exposed to the sheer volume of Nixpkgs' issues is tiresome. I often end up diving into some rabbit hole and lose track of time. By the time I am done fixing it, it’s well past bed time and my evening is gone.
I occasionally receive feedback that encourages me to continue the work. That is what made me consider writing this blog post. Conversations are lossy and aren’t easily replicated, so here is an overview of my project, in the hope that it is, too, useful to you.
If you have a neat idea or some matter that you want to discuss, please do not hesitate to contact me (mail, IRC).
Many thanks to Ana, Felix and Florian for reviewing and proof-reading this post. 💗
The name
broken.shis just a stupid pun on how broken everything is. Not really a statement about how broken NixOS is. Just a nice domain that I did come up with at the time. :) ↩︎