ranz2nix - A new approach to packaging node packages with Nix
A few weeks ago I was asked to create a simple online gallery for my sisters wedding pictures. The whole process of selecting the right software probably deserves a blog post on it’s own. In the end I decided to go with photoprism as it looked like the best solution (on multiple scales).
The unfortunate fact was that they are using Node.js to build their frontend application which meant that I had to use node2nix, yarn2nix, … or to roll my own solution. A few attempts into using node2nix (paired with previous experiences) I decided that it isn’t worth my time. I don’t mean to discredit the work that was done there but for me it simply doesn’t solve the problem.
Mostly I am not a fan of generating mostly duplicate Nix code out of some other
source repository when all the information (the native package manager
requires) are already available. I could never figure out how to properly patch
the generated node2nix expressions without resorting to manually calling
patch as part of the upgrade process. It also doesn’t really work without
checking out the projects source code and running the code generation within
it.
After all I just wanted to use niv to pin and update it without requiring further recurring manual steps.
I started looking at the contents of the
package-lock.json
to see if it contains anything of use for an alternative approach and was pleasantly surprised.
Unlike many other lock file formats it is just plain JSON that contains both the SRI hash and the source URI of each of the locked dependencies:
{
"name": "photoprism",
"version": "1.0.0",
"lockfileVersion": 1,
"requires": true,
"dependencies": {
"@babel/cli": {
"version": "7.10.4",
"resolved": "https://registry.npmjs.org/@babel/cli/-/cli-7.10.4.tgz",
"integrity": "sha512-xX99K4V1BzGJdQANK5cwK+EpF1vP9gvqhn+iWvG+TubCjecplW7RSQimJ2jcCvu6fnK5pY6mZMdu6EWTj32QVA==",
…
This means I can just fetch all of the dependencies with Nix
without writing a single line of Node.js or caring about internals of npm
like dependency resolution (and circular dependencies).
But how does that help?
Well, iff NPM uses a “fetcher” that supports file:// URI’s I can rewrite the
lockfile in such a way that the resolved values point to nix store paths
without invalidating the integrity hashes. By doing this we no longer need
network access during the build and everything can be provided within the build
sandbox. The sources will be made available since the patched lockfile will
carry them as dependencies.
At this stage all that is left is actually rewriting the contents of the file. Since Nix has builtin support for JSON so it was straight forward to recursively updating all the dependencies in the file:
{
lockFilePath
}:
let
# …
lockFile = builtins.fromJSON (builtins.readFile lockFilePath);
# …
patchedLockfile = pkgs.writeTextFile {
name = "package-lock.json";
destination = "/package-lock.json";
text = (
let
patchDep = sourcesList: name: v:
let
dsources = lib.mapAttrs (mkSource) (v.dependencies or { });
src = findBestSource sourcesList name;
in
(v // {
dependencies = lib.mapAttrs (patchDep ([ dsources ] ++ sourcesList)) (v.dependencies or { });
}) // (if src == null then packageOverride name v else {
resolved = "file://" + (toString src);
}
);
file = lockFile // {
dependencies = lib.mapAttrs (patchDep [ sources ]) (lockFile.dependencies or { });
};
in
builtins.toJSON file
);
};
# …
}
Now we can use the patched lockfile and create a new source tree with the
lockfile being replaced. All that is left then is to call npm --offline install to install all the dependencies Node.js projects.
{
# …
patchedBuildRoot = pkgs.symlinkJoin {
name = "patched-root-${lockFile.name}";
paths = [
patchedLockfile
sourcePath
];
};
patchedBuild = pkgs.stdenv.mkDerivation {
name = lockFile.name;
src = patchedBuildRoot;
buildInputs = [ nodejs ];
buildPhase = ''
export HOME=$(mktemp -d)
chmod -R +rw .
npm --offline install
'';
installPhase = ''
ls -la
mkdir $out
cp -rv node_modules $out/node_modules
'';
passthru.lockFile = patchedLockfile + "/package-lock.json";
};
# …
}
With the node_modules folder being handled we can move on to the
actual build process. For the case of photoprism there was one dependency
that didn’t have any sources in the lockfile (namely minimist). For that
purpose I quickly added a packageOverride attribute to the lockfile patching
code. It allows specifying sources for all those where sources are missing. I
expected there to be more of these edge cases but apparently that was all that
was required. The entire frontend build expression turned out to be fairly
simple:
{
# …
frontend = let
noderanz = callPackage ranz2nix {
nodejs = nodejs-12_x;
sourcePath = src + "/frontend";
packageOverride = name: spec: if name == "minimist" && spec ? resolved && spec.resolved == "" then {
resolved = "file://" + (
toString (
fetchurl {
url = "https://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz";
sha256 = "0w7jll4vlqphxgk9qjbdjh3ni18lkrlfaqgsm7p14xl3f7ghn3gc";
}
)
);
} else {};
};
node_modules = noderanz.patchedBuild;
in
stdenv.mkDerivation {
name = "photoprism-frontend";
nativeBuildInputs = [ nodejs-12_x ];
inherit src;
sourceRoot = "photoprism-src/frontend";
postUnpack = ''
chmod -R +rw .
'';
NODE_ENV = "production";
buildPhase = ''
export HOME=$(mktemp -d)
ln -sf ${node_modules}/node_modules node_modules
ln -sf ${node_modules.lockFile} package-lock.json
npm run build
'';
installPhase = ''
cp -rv ../assets/static/build $out
'';
};
# …
}
And that is all! Hurray!
When compared to any other Node.js approach this was actually fun to use and to develop. With a bit of dedicated time and brainstorming this little PoC could evolve into a more sophisticated solution. For now I am done with this project as it just does what I need and I can finally get back to deploying the gallery. :-)
EDIT: Since all of the above was just a PoC and was still very rough on the edges I started working on a cleaner version of this approach over at npmlock2nix.