NixOS System-Wide CFLAGS

2023/06/15

Background

After spending a good chunk of time (mostly) stabilizing my NixOS setup, I’d like to validate my waste of time reap some optimizations.

The default binary cache uses universal, generally safe compilation options. Instead, can the system be built with optimizations specific to the host’s architecture? And if so, then it’s free to decide on additional optimization and hardening flags.

Is the cost of not being able to use upstream caches and instead compiling a whole system from source worth it? Probably not, really. But, at least for me, since I really only ever nixos-rebuild switch on new nixpkgs releases anymore (upgrading to 23.05 is my excuse now), the upfront build time is only experienced on the (bi-yearly) stable release schedule.

Dead Ends

This section contains dead ends and caveats for alternatives that might be useful reference, skip to Final NixOS Configuration if not interested.

First attempt, nixpkgs { overlays = [ ... ] } might work? Quick search suggests withCFlags. Using -march=znver2 as a running example,

    overlays = [
      (self: super: {
        stdenv = super.withCFlags [ "-march=znver2" ]
          super.stdenv;
      })
    ];

This is the approach I found online, but fails at building the first package with an error about -march=znver2 being an invalid option, suggesting instead znver1. This could work for flags like -O3, but newer architecture-specific flags like znver2 are too new for… whatever version of GCC is used by default.

First link from a search:

This is still tricky and there is no clear comprehensive documentation for overriding compiler in various cases.

Foreboding. Well, back into the compiler mines…

Attempting a predefined stdenv with a new enough GCC,

    overlays = [
      (self: super: {
        stdenv = super.withCFlags [
          # ...
        ] super.gcc13Stdenv;
      })
    ];

Oops,

       at /nix/store/vhq11h949l5zycaw07acphv53ifq4p2c-source/pkgs/top-level/all-packages.nix:14925:3:

        14924|   gcc12Stdenv = overrideCC gccStdenv buildPackages.gcc12;
        14925|   gcc13Stdenv = overrideCC gccStdenv buildPackages.gcc13;
             |   ^
        14926|

       error: infinite recursion encountered

Yeah, this infinite recursion is understandable: stdenv -> gcc13Stdenv -> gccStdenv -> stdenv. A few more efforts in this genre results in failing at the same spot with infinite recursion or bad configuration,

        at /nix/store/vhq11h949l5zycaw07acphv53ifq4p2c-source/pkgs/top-level/all-packages.nix:16739:67:

        16738|       callPackage ../build-support/cc-wrapper (let self = {
        16739|     nativeTools = stdenv.targetPlatform == stdenv.hostPlatform && stdenv.cc.nativeTools or false;
             |                                                                   ^

What does nativeTools mean? Found definition in Armijn Hemel’s Masters Thesis on NixOS (2006), but a thesis is where I call this a dead end.

Discourse has a good answer. Seems close!

{ pkgs ? import <nixpkgs> {
  config.replaceStdenv = { pkgs, ... }: pkgs.gcc12Stdenv;
} }:
# ...

Oh… It’s that easy? replaceStdenv is exactly what I want. (Doesn’t look to be included in relevant documentation, as far as I can tell…)

nixpkgs.config = {
  replaceStdenv = { pkgs, ... }:
    (pkgs.withCFlags [
      "-march=znver2" # does this work?
    ] pkgs.gcc13Stdenv);
};

Finally kicks off a build!

Final NixOS Configuration

This overlay can go wherever nixpkgs gets pulled in. Since there’s a potential for sharing across machines, I’m including this inline at top-level flake.nix:

    nixosConfigurations = {
      znver2 = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux";
        # ...
        modules = [
          # ...

          # architecture-specific + hardening CFLAGS
          ({ pkgs, lib, ... }: {
            nixpkgs.config = {
              replaceStdenv = { pkgs, ... }:
                (pkgs.withCFlags [
                  "-O3"
                  "-fstack-protector-all"
                  "-march=znver2"
                  "-mtune=znver2"
                ] pkgs.gcc12Stdenv);
            };
          })
        ];
      };
    };

Remote Builds

Rebuilding the whole system may be prohibitively expensive for weaker computers like netbooks, devboards, &c.

To use a remote build host, there are dedicated solutions like deploy-rs, but just nixos-rebuild can work:

$ nixos-rebuild switch \
    --fast \
    --build-host desktop \ # --use-remote-sudo
    --flake .#$(hostname)

What CFLAGS?

Some good resources to start: