Persisting the escape hatch

Here’s a heresy for a NixOS blog: sometimes I just want to install a binary.

Not declare it. Not open the flake, find the right module, add it to home.packages, re-run the whole eval, and switch, all so I can run some CLI tool exactly once to see if it does the thing I hoped it does. Sometimes I want the dirty escape hatch. nix-env -iA nixos.whatever, use it, forget about it.

The problem is that I run this host on a tmpfs root that gets wiped every boot. So the escape hatch works right up until I reboot, at which point the tool evaporates along with everything else I didn’t explicitly bless. Which is the whole point of the setup, and also, in this one narrow case, deeply annoying.

Fine. I’ll persist the profile. How hard could it be.

The obvious wrong move

nix-env installs land in your user profile, and everybody knows where that is:

~/.nix-profile

So I did the thing that has worked a hundred times for a hundred other directories on this box. I added it to the impermanence list:

home.persistence."/persist".directories = [
  # ...
  ".nix-profile"
];

make build, clean. Evaluation is happy. The flake checks pass. Everything is green and I am, briefly, a genius.

Then make switch walks up and dies on the doorstep.

The build was never the problem. Evaluation doesn’t care about the shape of your home directory; it’s just producing a derivation. The lie only surfaces at activation, when something actually has to reconcile the config with the messy filesystem reality. That reality looks like this:

$ ls -la ~/.nix-profile
~/.nix-profile -> ~/.local/state/nix/profiles/profile

~/.nix-profile is not a directory. It’s a symlink. It has always been a symlink. I have been looking at it for years and never once actually looked at it.

Two things you can’t do at once

You cannot persist a symlink as if it were a directory. Impermanence’s directories mechanism wants to own that path: create a home for it on /persist, surface it back into $HOME, and do all that assuming the thing is a plain directory it can manage. But ~/.nix-profile already has an owner. Nix made that symlink. Nix maintains it. Now two systems both believe they’re in charge of the same path, and activation is where they have their fight. I don’t get to referee. I just get the error and a session that won’t switch.

This is the same family of bug I wrote about back in July, the “mount path is not canonical (contains a symlink)” trap, just wearing a different hat. Last time I created the offending symlink by hand. This time Nix created it for me and I tried to persist it anyway. Same lesson, second verse: the persistence layer and symlinks have opinions about each other, and you need to know whose symlink it is.

The thing I was saving wasn’t the thing that mattered

Even if persisting ~/.nix-profile had worked, even if impermanence had happily bind-mounted it, I’d have saved the wrong thing. Follow the pointer:

~/.nix-profile
  -> ~/.local/state/nix/profiles/profile
     -> profile-1-link
        -> /nix/store/…-user-environment

~/.nix-profile is a signpost. It points at the profile. The profile is a signpost too. It points at the current generation. The generations themselves (the actual profile-N-link files, the manifest, the record of what nix-env installed) all live in:

~/.local/state/nix/profiles/

That directory is a real directory. It holds the state I care about. And it was sitting on the tmpfs, getting incinerated every boot, while I fussed over the symlink pointing at it.

The store paths themselves? Already safe, because /nix/store is persistent. So the only thing that ever needed saving was the little pile of generation symlinks and the manifest. Everything else is derivable.

I was trying to persist the sign instead of the town.

The fix, in three moves

One: persist the town. The canonical generation store, not the symlink:

home.persistence."/persist".directories = [
  # ...
  ".local/state/nix/profiles"   # nix-env generations + manifest live here
];

Two: recreate the sign, declaratively. ~/.nix-profile is a convenience symlink Nix makes lazily, and on a tmpfs root it vanishes every boot along with the rest of $HOME. If I want nix-env-installed binaries on PATH the instant I log in, without having to poke Nix first to regenerate the link, I pin it myself:

home.file.".nix-profile" = {
  source = config.lib.file.mkOutOfStoreSymlink
    "${config.home.homeDirectory}/.local/state/nix/profiles/profile";
  force = true;
};

Note the flip. I stopped asking impermanence to persist the symlink and started asking home-manager to recreate it, pointing at the persisted store. The stable profiles/profile target always resolves to whatever the current generation is, exactly like Nix would do it. I’m just doing it on a schedule I control (every activation) instead of waiting for Nix to get around to it.

Three: seed the town before you move in. This one bites if you skip it. On the first switch, impermanence creates an empty directory on /persist and surfaces it over ~/.local/state/nix/profiles. If your live profile is still sitting on the tmpfs at that moment, you just shadowed it with nothing and orphaned every generation you had. So copy the real thing across first, symlinks and all:

cp -a ~/.local/state/nix/profiles /persist/home/lowcache/.local/state/nix/profiles

Then switch. readlink ~/.nix-profile points where it should, nix-env -iA nixos.hello && hello prints its cheerful little greeting, and (the whole point) it’s still there after a reboot.

What I actually learned

Two things, and neither is really about Nix.

First: when the build passes and the switch fails, stop trusting the build. Evaluation tells you the spec is coherent. It tells you nothing about whether the spec can survive contact with the filesystem you already have. The gap between those two is where activation errors live, and they’re always about state the pure part of the system can’t see.

Second, and I keep re-learning this one: follow the symlink before you try to save it. I spent an embarrassing amount of energy trying to preserve ~/.nix-profile when ~/.nix-profile was never anything but a pointer to the thing worth preserving. Persist the data. Regenerate the signposts. Don’t confuse the two.

The escape hatch works now. nix-env -iA for the throwaway stuff, the flake for anything I actually mean. And on the next boot, the throwaway stuff that I decided to keep is still there, while everything I truly threw away stays gone.

Which, come to think of it, is just impermanence working as designed. I only had to stop persisting the wrong end of the arrow.

Config: lowcache/volnixos