Nix Pills

Preface
1. Why You Should Give it a Try
1.1. Introduction
1.2. Rationale for this series
1.3. Not being purely functional
1.4. Being purely functional
1.5. Mutable vs. immutable
1.6. Conclusion
1.7. Next pill...
2. Install on Your Running System
2.1. Installation
2.2. The beginnings of the Nix store
2.3. The Nix database
2.4. The first profile
2.5. Nixpkgs expressions
2.6. FAQ: Can I change /nix to something else?
2.7. Conclusion
2.8. Next pill...
3. Enter the Environment
3.1. Enter the environment
3.2. Install something
3.3. Path merging
3.4. Rolling back and switching generation
3.5. Querying the store
3.6. Closures
3.7. Dependency resolution
3.8. Recovering the hard way
3.9. Channels
3.10. Conclusion
3.11. Next pill
4. The Basics of the Language
4.1. Value types
4.2. Identifier
 4.3. Strings
4.4. Lists
4.5. Attribute sets
4.6. If expressions
4.7. Let expressions
4.8. With expression
4.9. Laziness
4.10. Next pill
5. Functions and Imports
5.1. Nameless and single parameter
5.2. More than one parameter
5.3. Arguments set
5.4. Default and variadic attributes
5.5. Imports
5.6. Next pill
6. Our First Derivation
6.1. The derivation function
6.2. Digression about .drv files
6.3. What's in a derivation set
6.4. Referring to other derivations
6.5. An almost working derivation
6.6. When is the derivation built
6.7. Conclusion
6.8. Next pill
7. Working Derivation
7.1. Introduction
7.2. Using a script as a builder
7.3. The builder environment
7.4. The .drv contents
7.5. Packaging a simple C program
7.6. Explanation
7.7. Enough of nix repl
7.8. Next pill
8. Generic Builders
8.1. Packaging GNU hello world
8.2. A generic builder
8.3. A more convenient derivation function
 8.4. Conclusion
8.5. Next pill
9. Automatic Runtime Dependencies
9.1. Build dependencies
9.2. Digression about NAR files
9.3. Runtime dependencies
9.4. Another phase in the builder
9.5. Conclusion
9.6. Next pill
10. Developing with nix-shell
10.1. What's nix-shell
10.2. A builder for nix-shell
10.3. Conclusion
10.4. Next pill
11. Garbage Collector
11.1. How does it work
11.2. Playing with the GC
11.3. Indirect roots
11.4. Cleanup everything
11.5. Conclusion
11.6. Next pill
12. Inputs Design Pattern
12.1. Repositories in Nix
12.2. The single repository pattern
12.3. Packaging graphviz
12.4. Digression about gcc and ld wrappers
12.5. Completing graphviz with gd
12.6. The repository expression
12.7. The inputs pattern
12.8. Conclusion
12.9. Next pill
13. Callpackage Design Pattern
13.1. The callPackage convenience
13.2. Implementing callPackage
13.3. Use callPackage to simplify the repository
13.4. Conclusion
13.5. Next pill
14. Override Design Pattern
14.1. About composability
14.2. The override pattern
14.3. The override implementation
14.4. Conclusion
14.5. Next pill
15. Nix Search Paths
15.1. The NIX_PATH
15.2. Fake it a little
15.3. The path to repository
15.4. A big word about nix-env
15.5. Conclusion
15.6. Next pill
16. Nixpkgs Parameters
16.1. The default.nix expression
16.2. The system parameter
16.3. The config parameter
16.4. About .nix functions
16.5. Conclusion
16.6. Next pill
17. Nixpkgs Overriding Packages
17.1. Overriding a package
17.2. In an imperative world...
17.3. Fixed point
17.3.1. Overriding a set with fixed point
17.4. Overriding nixpkgs packages
17.5. The ~/.nixpkgs/config.nix file
17.6. Conclusion
17.7. Next pill
18. Nix Store Paths
18.1. Source paths
18.1.1. Step 1, compute the hash of the file
18.1.2. Step 2, build the string description
18.1.3. Step 3, compute the final hash
18.2. Output paths
18.3. Fixed-output paths
18.4. Conclusion
 18.5. Next pill
19. Fundamentals of Stdenv
19.1. What is stdenv
19.2. The setup file
19.3. How is the setup file built
19.4. The stdenv.mkDerivation builder
19.5. Conclusion
19.6. Next pill...
20. Basic Dependencies and Hooks
20.1. The buildInputs Attribute
20.2. The propagatedBuildInputs Attribute
20.3. Setup Hooks
20.4. Environment Hooks
20.5. Next pill...
Nix Pills
Version 1234-abcdef
Preface
This is a ported version of the Nix Pills, a series of blog posts written by
Luca Bruno (aka Lethalman) and orginally published in 2014 and 2015. It
provides a tutorial introduction into the Nix package manager and Nixpkgs
package collection, in the form of short chapters called 'pills'.

Since the Nix Pills are considered a classic introduction to Nix, an effort to
port them to the current format was led by Graham Christensen (aka grahamc
/ gchristensen) and other contributors in 2017.

If you encounter problems, please report them on the nixos/nix-pills issue

tracker.

Note

Commands prefixed with # have to be run as root, either requiring to login as

root user or temporarily switching to it using sudo for example.
Chapter 1. Why You Should Give it
a Try
1.1. Introduction
Welcome to the first post of the "Nix in pills" series. Nix is a purely
functional package manager and deployment system for POSIX.

There's a lot of documentation that describes what Nix, NixOS and related
projects are. But the purpose of this post is to convince you to give Nix a try.
Installing NixOS is not required, but sometimes I may refer to NixOS as a
real world example of Nix usage for building a whole operating system.

1.2. Rationale for this series

The Nix, Nixpkgs, and NixOS manuals; and wiki are excellent resources for
explaining how Nix/NixOS works, how you can use it, and the number of
cool things being done with it. However, at the beginning you may feel that
some of the magic which happens behind the scenes is hard to grasp.

This series aims to complement the existing explanations from the more
formal documents.

The following is a description of Nix. Just as with pills, I'll try to be as short
as possible.

1.3. Not being purely functional

Most, if not all, widely used package managers (dpkg, rpm, ...) mutate the
global state of the system. If a package foo-1.0 installs a program to
/usr/bin/foo, you cannot install foo-1.1 as well, unless you change the
installation paths or the binary name. But changing the binary names means
breaking users of that binary.
There are some attempts to mitigate this problem. Debian, for example,
partially solves the problem with the alternatives system.

So while in theory it's possible with some current systems to install multiple
versions of the same package, in practice it's very painful.

Let's say you need an nginx service and also an nginx-openresty service. You
have to create a new package that changes all the paths to have, for example,
an -openresty suffix.

Or suppose that you want to run two different instances of mysql: 5.2 and 5.5.
The same thing applies, plus you have to also make sure the two mysqlclient
libraries do not collide.

This is not impossible but it is very inconvenient. If you want to install two
whole stacks of software like GNOME 3.10 and GNOME 3.12, you can
imagine the amount of work.

From an administrator's point of view: you can use containers. The typical
solution nowadays is to create a container per service, especially when
different versions are needed. That somewhat solves the problem, but at a
different level and with other drawbacks. For example, needing orchestration
tools, setting up a shared cache of packages, and new machines to monitor
rather than simple services.

From a developer's point of view: you can use virtualenv for python, or
jhbuild for gnome, or whatever else. But then how do you mix the two
stacks? How do you avoid recompiling the same thing when it could instead
be shared? Also you need to set up your development tools to point to the
different directories where libraries are installed. Not only that, there's the
risk that some of the software incorrectly uses system libraries.

And so on. Nix solves all this at the packaging level and solves it well. A
single tool to rule them all.

1.4. Being purely functional

Nix makes no assumptions about the global state of the system. This has
many advantages, but also some drawbacks of course. The core of a Nix
system is the Nix store, usually installed under /nix/store, and some tools
to manipulate the store. In Nix there is the notion of a derivation rather than a
package. The difference can be subtle at the beginning, so I will often use the
words interchangeably.

Derivations/packages are stored in the Nix store as follows:

/nix/store/hash-name, where the hash uniquely identifies the derivation
(this isn't quite true, it's a little more complex), and the name is the name of
the derivation.

Let's take a bash derivation as an example:

/nix/store/s4zia7hhqkin1di0f187b79sa2srhv6k-bash-4.2-p45/. This is
a directory in the Nix store which contains bin/bash.

What that means is that there's no /bin/bash, there's only that self-contained
build output in the store. The same goes for coreutils and everything else. To
make them convenient to use from the shell, Nix will arrange for binaries to
appear in your PATH as appropriate.

What we have is basically a store of all packages (with different versions

occupying different locations), and everything in the Nix store is immutable.

In fact, there's no ldconfig cache either. So where does bash find libc?
$ ldd `which bash`
libc.so.6 => /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/libc

It turns out that when bash was built, it was built against that specific version
of glibc in the Nix store, and at runtime it will require exactly that glibc
version.

Don't be confused by the version in the derivation name: it's only a name for
us humans. You may end up having two derivations with the same name but
different hashes: it's the hash that really matters.

What does all this mean? It means that you could run mysql 5.2 with glibc-
2.18, and mysql 5.5 with glibc-2.19. You could use your python module with
python 2.7 compiled with gcc 4.6 and the same python module with python 3
compiled with gcc 4.8, all in the same system.

In other words: no dependency hell, not even a dependency resolution

algorithm. Straight dependencies from derivations to other derivations.

From an administrator's point of view: if you want an old PHP version for
one application, but want to upgrade the rest of the system, that's not painful
any more.

From a developer's point of view: if you want to develop webkit with llvm
3.4 and 3.3, that's not painful any more.

1.5. Mutable vs. immutable

When upgrading a library, most package managers replace it in-place. All
new applications run afterwards with the new library without being
recompiled. After all, they all refer dynamically to libc6.so.

Since Nix derivations are immutable, upgrading a library like glibc means
recompiling all applications, because the glibc path to the Nix store has been
hardcoded.

So how do we deal with security updates? In Nix we have some tricks (still
pure) to solve this problem, but that's another story.

Another problem is that unless software has in mind a pure functional model,
or can be adapted to it, it can be hard to compose applications at runtime.

Let's take Firefox for example. On most systems, you install flash, and it
starts working in Firefox because Firefox looks in a global path for plugins.

In Nix, there's no such global path for plugins. Firefox therefore must know
explicitly about the path to flash. The way we handle this problem is to wrap
the Firefox binary so that we can setup the necessary environment to make it
find flash in the nix store. That will produce a new Firefox derivation: be
aware that it takes a few seconds, and it makes composition harder at
runtime.
There are no upgrade/downgrade scripts for your data. It doesn't make sense
with this approach, because there's no real derivation to be upgraded. With
Nix you switch to using other software with its own stack of dependencies,
but there's no formal notion of upgrade or downgrade when doing so.

If there is a data format change, then migrating to the new data format
remains your own responsibility.

1.6. Conclusion
Nix lets you compose software at build time with maximum flexibility, and
with builds being as reproducible as possible. Not only that, due to its nature
deploying systems in the cloud is so easy, consistent, and reliable that in the
Nix world all existing self-containment and orchestration tools are deprecated
by NixOps.

It however currently falls short when working with dynamic composition at

runtime or replacing low level libraries, due to the need to rebuild
dependencies.

That may sound scary, however after running NixOS on both a server and a
laptop desktop, I'm very satisfied so far. Some of the architectural problems
just need some man-power, other design problems still need to be solved as a
community.

Considering Nixpkgs (github link) is a completely new repository of all the

existing software, with a completely fresh concept, and with few core
developers but overall year-over-year increasing contributions, the current
state is more than acceptable and beyond the experimental stage. In other
words, it's worth your investment.

1.7. Next pill...

...we will install Nix on top of your current system (I assume GNU/Linux, but
we also have OSX users) and start inspecting the installed software.
Chapter 2. Install on Your Running
System
Welcome to the second Nix pill. In the first pill we briefly described Nix.

Now we'll install Nix on our running system and understand what changed in
our system after the installation. If you're using NixOS, Nix is already
installed; you can skip to the next pill.

Installing Nix is as easy as installing any other package. It will not drastically
change our system, it will stay out of our way.

2.1. Installation
To install Nix, run curl https://nixos.org/nix/install | sh as a non-root user
and follow the instructions. Alternatively, you may prefer to download the
installation script and verify its integrity using GPG signatures. Instructions
for doing so can be found here: https://nixos.org/nix/download.html.

These articles are not a tutorial on using Nix. Instead, we're going to walk
through the Nix system to understand the fundamentals.

The first thing to note: derivations in the Nix store refer to other derivations
which are themselves in the Nix store. They don't use libc from our system
or anywhere else. It's a self-contained store of all the software we need to
bootstrap up to any particular package.

Note

In a multi-user installation, such as the one used in NixOS, the store is owned
by root and multiple users can install and build software through a Nix
daemon. You can read more about multi-user installations here:
https://nixos.org/nix/manual/#ssec-multi-user.
2.2. The beginnings of the Nix store
Start looking at the output of the install command:
copying Nix to /nix/store..........................

That's the /nix/store we were talking in the first article. We're copying in
the necessary software to bootstrap a Nix system. You can see bash, coreutils,
the C compiler toolchain, perl libraries, sqlite and Nix itself with its own
tools and libnix.

You may have noticed that /nix/store can contain not only directories, but
also files, still always in the form hash-name.

2.3. The Nix database

Right after copying the store, the installation process initializes a database:
initialising Nix database...

Yes, Nix also has a database. It's stored under /nix/var/nix/db. It is a sqlite
database that keeps track of the dependencies between derivations.

The schema is very simple: there's a table of valid paths, mapping from an
auto increment integer to a store path.

Then there's a dependency relation from path to paths upon which they
depend.

You can inspect the database by installing sqlite (nix-env -iA sqlite -f
'<nixpkgs>') and then running sqlite3 /nix/var/nix/db/db.sqlite.

Note

If this is the first time you're using Nix after the initial installation, remember
you must close and open your terminals first, so that your shell environment
will be updated.
Important

Never change /nix/store manually. If you do, then it will no longer be in

sync with the sqlite db, unless you really know what you are doing.

2.4. The first profile

Next in the installation, we encounter the concept of the profile:
creating /home/nix/.nix-profile
installing 'nix-2.1.3'
building path(s) `/nix/store/a7p1w3z2h8pl00ywvw6icr3g5l9vm5r7-user-environme
created 7 symlinks in user environment

A profile in Nix is a general and convenient concept for realizing rollbacks.

Profiles are used to compose components that are spread among multiple
paths under a new unified path. Not only that, but profiles are made up of
multiple "generations": they are versioned. Whenever you change a profile, a
new generation is created.

Generations can be switched and rolled back atomically, which makes them
convenient for managing changes to your system.

Let's take a closer look at our profile:

$ ls -l ~/.nix-profile/
bin -> /nix/store/ig31y9gfpp8pf3szdd7d4sf29zr7igbr-nix-2.1.3/bin
[...]
manifest.nix -> /nix/store/q8b5238akq07lj9gfb3qb5ycq4dxxiwm-env-manifest.nix
[...]
share -> /nix/store/ig31y9gfpp8pf3szdd7d4sf29zr7igbr-nix-2.1.3/share

That nix-2.1.3 derivation in the Nix store is Nix itself, with binaries and
libraries. The process of "installing" the derivation in the profile basically
reproduces the hierarchy of the nix-2.1.3 store derivation in the profile by
means of symbolic links.

The contents of this profile are special, because only one program has been
installed in our profile, therefore e.g. the bin directory points to the only
program which has been installed (Nix itself).
But that's only the contents of the latest generation of our profile. In fact,
~/.nix-profile itself is a symbolic link to
/nix/var/nix/profiles/default.

In turn, that's a symlink to default-1-link in the same directory. Yes, that

means it's the first generation of the default profile.

Finally, default-1-link is a symlink to the nix store "user-environment"

derivation that you saw printed during the installation process.

We'll talk about manifest.nix more in the next article.

2.5. Nixpkgs expressions

More output from the installer:
downloading Nix expressions from `http://releases.nixos.org/nixpkgs/nixpkgs-
unpacking channels...
created 2 symlinks in user environment
modifying /home/nix/.profile...

Nix expressions are used to describe packages and how to build them.
Nixpkgs is the repository containing all of the expressions:
https://github.com/NixOS/nixpkgs.

The installer downloaded the package descriptions from commit a1a2851.

The second profile we discover is the channels profile. ~/.nix-

defexpr/channels points to /nix/var/nix/profiles/per-
user/nix/channels which points to channels-1-link which points to a Nix
store directory containing the downloaded Nix expressions.

Channels are a set of packages and expressions available for download.

Similar to Debian stable and unstable, there's a stable and unstable channel.
In this installation, we're tracking nixpkgs-unstable.

Don't worry about Nix expressions yet, we'll get to them later.

Finally, for your convenience, the installer modified ~/.profile to

automatically enter the Nix environment. What ~/.nix-
profile/etc/profile.d/nix.sh really does is simply to add ~/.nix-
profile/bin to PATH and ~/.nix-defexpr/channels/nixpkgs to NIX_PATH.
We'll discuss NIX_PATH later.

Read nix.sh, it's short.

2.6. FAQ: Can I change /nix to something else?

You can, but there's a good reason to keep using /nix instead of a different
directory. All the derivations depend on other derivations by using absolute
paths. We saw in the first article that bash referenced a glibc under a specific
absolute path in /nix/store.

You can see for yourself, don't worry if you see multiple bash derivations:
$ ldd /nix/store/*bash*/bin/bash
[...]

Keeping the store in /nix means we can grab the binary cache from nixos.org
(just like you grab packages from debian mirrors) otherwise:

glibc would be installed under /foo/store

Thus bash would need to point to glibc under /foo/store, instead of

under /nix/store

So the binary cache can't help, because we need a different bash, and so
we'd have to recompile everything ourselves.

After all /nix is a sensible place for the store.

2.7. Conclusion
We've installed Nix on our system, fully isolated and owned by the nix user
as we're still coming to terms with this new system.

We learned some new concepts like profiles and channels. In particular, with
profiles we're able to manage multiple generations of a composition of
packages, while with channels we're able to download binaries from
nixos.org.

The installation put everything under /nix, and some symlinks in the Nix
user home. That's because every user is able to install and use software in her
own environment.

I hope I left nothing uncovered so that you think there's some kind of magic
going on behind the scenes. It's all about putting components in the store and
symlinking these components together.

2.8. Next pill...

...we will enter the Nix environment and learn how to interact with the store.
Chapter 3. Enter the Environment
Welcome to the third Nix pill. In the second pill we installed Nix on our
running system. Now we can finally play with it a little, these things also
apply to NixOS users.

3.1. Enter the environment

If you're using NixOS, you can skip to the next step.

In the previous article we created a Nix user, so let's start by switching to it

with su - nix. If your ~/.profile got evaluated, then you should now be able
to run commands like nix-env and nix-store.

If that's not the case:

$ source ~/.nix-profile/etc/profile.d/nix.sh

To remind you, ~/.nix-profile/etc points to the nix-2.1.3 derivation. At

this point, we are in our Nix user profile.

3.2. Install something

Finally something practical! Installation into the Nix environment is an
interesting process. Let's install hello, a simple CLI tool which prints Hello
world and is mainly used to test compilers and package installations.

Back to the installation:

$ nix-env -i hello
installing 'hello-2.10'
[...]
building '/nix/store/0vqw0ssmh6y5zj48yg34gc6macr883xk-user-environment.drv'.
created 36 symlinks in user environment

Now you can run hello. Things to notice:

 We installed software as a user, and only for the Nix user.

It created a new user environment. That's a new generation of our Nix

user profile.

The nix-env tool manages environments, profiles and their generations.

We installed hello by derivation name minus the version. I repeat: we

specified the derivation name (minus the version) to install it.

We can list generations without walking through the /nix hierarchy:

$ nix-env --list-generations
1 2014-07-24 09:23:30
2 2014-07-25 08:45:01 (current)

Listing installed derivations:

$ nix-env -q
nix-2.1.3
hello-2.10

So, where did hello really get installed? which hello is ~/.nix-
profile/bin/hello which points to the store. We can also list the derivation
paths with nix-env -q --out-path. So that's what those derivation paths are
called: the output of a build.

3.3. Path merging

At this point you probably want to run man to get some documentation. Even
if you already have man system-wide outside of the Nix environment, you
can install and use it within Nix with nix-env -i man. As usual, a new
generation will be created, and ~/.nix-profile will point to it.

Lets inspect the profile a bit:

$ ls -l ~/.nix-profile/
dr-xr-xr-x 2 nix nix 4096 Jan 1 1970 bin
lrwxrwxrwx 1 nix nix 55 Jan 1 1970 etc -> /nix/store/ig31y9gfpp8pf3szdd7
[...]
Now that's interesting. When only nix-2.1.3 was installed, bin was a
symlink to nix-2.1.3. Now that we've actually installed some things (man,
hello), it's a real directory, not a symlink.

$ ls -l ~/.nix-profile/bin/
[...]
man -> /nix/store/83cn9ing5sc6644h50dqzzfxcs07r2jn-man-1.6g/bin/man
[...]
nix-env -> /nix/store/ig31y9gfpp8pf3szdd7d4sf29zr7igbr-nix-2.1.3/bin/nix-env
[...]
hello -> /nix/store/58r35bqb4f3lxbnbabq718svq9i2pda3-hello-2.10/bin/hello
[...]

Okay, that's clearer now. nix-env merged the paths from the installed
derivations. which man points to the Nix profile, rather than the system man,
because ~/.nix-profile/bin is at the head of $PATH.

3.4. Rolling back and switching generation

The last command installed man. We should be at generation 3, unless you
changed something in the middle. Let's say we want to rollback to the old
generation:
$ nix-env --rollback
switching from generation 3 to 2

Now nix-env -q does not list man anymore. ls -l `which man` should now be
your system copy.

Enough with the rollback, let's go back to the most recent generation:
$ nix-env -G 3
switching from generation 2 to 3

I invite you to read the manpage of nix-env. nix-env requires an operation to

perform, then there are common options for all operations, as well as options
specific to each operation.

You can of course also uninstall and upgrade packages.

3.5. Querying the store
So far we learned how to query and manipulate the environment. But all of
the environment components point to the store.

To query and manipulate the store, there's the nix-store command. We can
do some interesting things, but we'll only see some queries for now.

To show the direct runtime dependencies of hello:

$ nix-store -q --references `which hello`
/nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27
/nix/store/58r35bqb4f3lxbnbabq718svq9i2pda3-hello-2.10

The argument to nix-store can be anything as long as it points to the Nix

store. It will follow symlinks.

It may not make sense to you right now, but let's print reverse dependencies
of hello:
$ nix-store -q --referrers `which hello`
/nix/store/58r35bqb4f3lxbnbabq718svq9i2pda3-hello-2.10
/nix/store/fhvy2550cpmjgcjcx5rzz328i0kfv3z3-env-manifest.nix
/nix/store/mp987abm20c70pl8p31ljw1r5by4xwfw-user-environment

Was it what you expected? It turns out that our environments depend upon
hello. Yes, that means that the environments are in the store, and since they
contain symlinks to hello, therefore the environment depends upon hello.

Two environments were listed, generation 2 and generation 3, since these are
the ones that had hello installed in them.

The manifest.nix file contains metadata about the environment, such as

which derivations are installed. So that nix-env can list, upgrade or remove
them. And yet again, the current manifest.nix can be found at ~/.nix-
profile/manifest.nix.

3.6. Closures
The closures of a derivation is a list of all its dependencies, recursively,
including absolutely everything necessary to use that derivation.
$ nix-store -qR `which man`
[...]

Copying all those derivations to the Nix store of another machine makes you
able to run man out of the box on that other machine. That's the base of
deployment using Nix, and you can already foresee the potential when
deploying software in the cloud (hint: nix-copy-closures and nix-store --
export).

A nicer view of the closure:

$ nix-store -q --tree `which man`
[...]

With the above command, you can find out exactly why a runtime
dependency, be it direct or indirect, exists for a given derivation.

The same applies to environments. As an exercise, run nix-store -q --tree

~/.nix-profile, and see that the first children are direct dependencies of the
user environment: the installed derivations, and the manifest.nix.

3.7. Dependency resolution

There isn't anything like apt which solves a SAT problem in order to satisfy
dependencies with lower and upper bounds on versions. There's no need for
this because all the dependencies are static: if a derivation X depends on a
derivation Y, then it always depends on it. A version of X which depended on
Z would be a different derivation.

3.8. Recovering the hard way

$ nix-env -e '*'
uninstalling 'hello-2.10'
uninstalling 'nix-2.1.3'
[...]
Oops, that uninstalled all derivations from the environment, including Nix.
That means we can't even run nix-env, what now?

Previously we got nix-env from the environment. Environments are a

convenience for the user, but Nix is still there in the store!

First, pick one nix-2.1.3 derivation: ls /nix/store/*nix-2.1.3, say

/nix/store/ig31y9gfpp8pf3szdd7d4sf29zr7igbr-nix-2.1.3.

The first option is to rollback:

$ /nix/store/ig31y9gfpp8pf3szdd7d4sf29zr7igbr-nix-2.1.3/bin/nix-env --rollba

The second option is to install Nix, thus creating a new generation:

$ /nix/store/ig31y9gfpp8pf3szdd7d4sf29zr7igbr-nix-2.1.3/bin/nix-env -i /nix/

3.9. Channels
So where are we getting packages from? We said something about this
already in the second article. There's a list of channels from which we get
packages, although usually we use a single channel. The tool to manage
channels is nix-channel.
$ nix-channel --list
nixpkgs http://nixos.org/channels/nixpkgs-unstable

If you're using NixOS, you may not see any output from the above command
(if you're using the default), or you may see a channel whose name begins
with "nixos-" instead of "nixpkgs".

That's essentially the contents of ~/.nix-channels.

Note
~/.nix-channels is not a symlink to the nix store!

To update the channel run nix-channel --update. That will download the
new Nix expressions (descriptions of the packages), create a new generation
of the channels profile and unpack it under ~/.nix-defexpr/channels.

This is quite similar to apt-get update. (See this table for a rough mapping
between Ubuntu and NixOS package management.)

3.10. Conclusion
We learned how to query the user environment and to manipulate it by
installing and uninstalling software. Upgrading software is also
straightforward, as you can read in the manual (nix-env -u will upgrade all
packages in the environment).

Everytime we change the environment, a new generation is created.

Switching between generations is easy and immediate.

Then we learned how to query the store. We inspected the dependencies and
reverse dependencies of store paths.

We saw how symlinks are used to compose paths from the Nix store, a useful
trick.

A quick analogy with programming languages: you have the heap with all the
objects, that corresponds to the Nix store. You have objects that point to other
objects, those correspond to derivations. This is a suggestive metaphor, but
will it be the right path?

3.11. Next pill

...we will learn the basics of the Nix language. The Nix language is used to
describe how to build derivations, and it's the basis for everything else,
including NixOS. Therefore it's very important to understand both the syntax
and the semantics of the language.
Chapter 4. The Basics of the
Language
Welcome to the fourth Nix pill. In the previous article we learned about Nix
environments. We installed software as a user, managed their profile,
switched between generations, and queried the Nix store. Those are the very
basics of system administration using Nix.

The Nix language is used to write expressions that produce derivations. The
nix-build tool is used to build a derivations from an expression. Even as a
system administrator that wants to customize the installation, it's necessary to
master Nix. Using Nix for your jobs means you get the features we saw in the
previous articles for free.

The syntax of Nix is quite unfamiliar, so looking at existing examples may

lead you to think that there's a lot of magic happening. In reality, it's mostly
about writing utility functions to make things convenient.

On the other hand, the same syntax is great for describing packages, so
learning the language itself will pay off when writing package expressions.

Important

In Nix, everything is an expression, there are no statements. This is common

in functional languages.

Important

Values in Nix are immutable.

4.1. Value types

Nix 2.0 contains a command named nix repl which is a simple command line
tool for playing with the Nix language. In fact, Nix is a pure, lazy, functional
language, not only a set of tools to manage derivations. The nix repl syntax
is slightly different to Nix syntax when it comes to assigning variables, but it
shouldn't be confusing so long as you bear it in mind. I prefer to start with
nix repl before cluttering your mind with more complex expressions.

Launch nix repl. First of all, Nix supports basic arithmetic operations: +, -,
* and /. (To exit nix repl, use the command :q. Help is available through
the :? command.)
nix-repl> 1+3
4

nix-repl> 7-4
3

nix-repl> 3*2
6

Attempting to perform division in Nix can lead to some surprises.

nix-repl> 6/3
/home/nix/6/3

What happened? Recall that Nix is not a general purpose language, it's a
domain-specific language for writing packages. Integer division isn't actually
that useful when writing package expressions. Nix parsed 6/3 as a relative
path to the current directory. To get Nix to perform division instead, leave a
space after the /. Alternatively, you can use builtins.div.
nix-repl> 6/ 3
2

nix-repl> builtins.div 6 3
2

Other operators are ||, && and ! for booleans, and relational operators such as
!=, ==, <, >, <=, >=. In Nix, <, >, <= and >= are not much used. There are also
other operators we will see in the course of this series.

Nix has integer, floating point, string, path, boolean and null simple types.
Then there are also lists, sets and functions. These types are enough to build
an operating system.

Nix is strongly typed, but it's not statically typed. That is, you cannot mix
strings and integers, you must first do the conversion.

As demonstrated above, expressions will be parsed as paths as long as there's

a slash not followed by a space. Therefore to specify the current directory,
use ./. In addition, Nix also parses urls specially.

Not all urls or paths can be parsed this way. If a syntax error occurs, it's still
possible to fallback to plain strings. Literal urls and paths are convenient for
additional safety.

4.2. Identifier
There's not much to say here, except that dash (-) is allowed in identifiers.
That's convenient since many packages use dash in their names. In fact:
nix-repl> a-b
error: undefined variable `a-b' at (string):1:1
nix-repl> a - b
error: undefined variable `a' at (string):1:1

As you can see, a-b is parsed as identifier, not as a subtraction.

4.3. Strings
It's important to understand the syntax for strings. When learning to read Nix
expressions, you may find dollars ($) ambiguous, but they are very important
. Strings are enclosed by double quotes ("), or two single quotes ('').
nix-repl> "foo"
"foo"
nix-repl> ''foo''
"foo"

In other languages like Python you can also use single quotes for strings (e.g.
'foo'), but not in Nix.
It's possible to interpolate whole Nix expressions inside strings with the
${...} syntax and only that syntax, not $foo or {$foo} or anything else.

nix-repl> foo = "strval"

nix-repl> "$foo"
"$foo"
nix-repl> "${foo}"
"strval"
nix-repl> "${2+3}"
error: cannot coerce an integer to a string, at (string):1:2

Note: ignore the foo = "strval" assignment, special syntax in nix repl.

As said previously, you cannot mix integers and strings. You need to
explicitly include conversions. We'll see this later: function calls are another
story.

Using the syntax with two single quotes is useful for writing double quotes
inside strings without needing to escape them:
nix-repl> ''test " test''
"test \" test"
nix-repl> ''${foo}''
"strval"

Escaping ${...} within double quoted strings is done with the backslash.
Within two single quotes, it's done with '':
nix-repl> "\${foo}"
"${foo}"
nix-repl> ''test ''${foo} test''
"test ${foo} test"

4.4. Lists
Lists are a sequence of expressions delimited by space (not comma):
nix-repl> [ 2 "foo" true (2+3) ]
[ 2 "foo" true 5 ]

Lists, like everything else in Nix, are immutable. Adding or removing

elements from a list is possible, but will return a new list.

4.5. Attribute sets

Attribute sets are an association between string keys and a Nix values. Keys
can only be strings. When writing attribute sets you can also use unquoted
identifiers as keys.
nix-repl> s = { foo = "bar"; a-b = "baz"; "123" = "num"; }
nix-repl> s
{ "123" = "num"; a-b = "baz"; foo = "bar"; }

For those reading Nix expressions from nixpkgs: do not confuse attribute sets
with argument sets used in functions.

To access elements in the attribute set:

nix-repl> s.a-b
"baz"
nix-repl> s."123"
"num"

Yes, you can use strings for to address keys which aren't valid identifiers.

Inside an attribute set you cannot normally refer to elements of the same
attribute set:
nix-repl> { a = 3; b = a+4; }
error: undefined variable `a' at (string):1:10

To do so, use recursive attribute sets:

nix-repl> rec { a = 3; b = a+4; }
{ a = 3; b = 7; }

This is very convenient when defining packages, which tend to be recursive

attribute sets.

4.6. If expressions
These are expressions, not statements.
nix-repl> a = 3
nix-repl> b = 4
nix-repl> if a > b then "yes" else "no"
"no"

You can't have only the then branch, you must specify also the else branch,
because an expression must have a value in all cases.

4.7. Let expressions

This kind of expression is used to define local variables for inner expressions.
nix-repl> let a = "foo"; in a
"foo"

The syntax is: first assign variables, then in, then an expression which can
use the defined variables. The value of the whole let expression will be the
value of the expression after the in.
nix-repl> let a = 3; b = 4; in a + b
7

Let's write two let expressions, one inside the other:

nix-repl> let a = 3; in let b = 4; in a + b
7

With let you cannot assign twice to the same variable. However, you can
shadow outer variables:
nix-repl> let a = 3; a = 8; in a
error: attribute `a' at (string):1:12 already defined at (string):1:5
nix-repl> let a = 3; in let a = 8; in a
8

You cannot refer to variables in a let expression outside of it:

nix-repl> let a = (let b = 3; in b); in b
error: undefined variable `b' at (string):1:31
You can refer to variables in the let expression when assigning variables,
like with recursive attribute sets:
nix-repl> let a = 4; b = a + 5; in b
9

So beware when you want to refer to a variable from the outer scope, but it's
also defined in the current let expression. The same applies to recursive
attribute sets.

4.8. With expression

This kind of expression is something you rarely see in other languages. You
can think of it like a more granular version of using from C++, or from
module import * from Python. You decide per-expression when to include
symbols into the scope.
nix-repl> longName = { a = 3; b = 4; }
nix-repl> longName.a + longName.b
7
nix-repl> with longName; a + b
7

That's it, it takes an attribute set and includes symbols from it in the scope of
the inner expression. Of course, only valid identifiers from the keys of the set
will be included. If a symbol exists in the outer scope and would also be
introduced by the with, it will not be shadowed. You can however still refer
to the attribute set:
nix-repl> let a = 10; in with longName; a + b
14
nix-repl> let a = 10; in with longName; longName.a + b
7

4.9. Laziness
Nix evaluates expression only when needed. This is a great feature when
working with packages.
nix-repl> let a = builtins.div 4 0; b = 6; in b
6

Since a is not needed, there's no error about division by zero, because the
expression is not in need to be evaluated. That's why we can have all the
packages defined on demand, yet have access to specific packages very
quickly.

4.10. Next pill

...we will talk about functions and imports. In this pill I've tried to avoid
function calls as much as possible, otherwise the post would have been too
long.
Chapter 5. Functions and Imports
Welcome to the fifth Nix pill. In the previous fourth pill we touched the Nix
language for a moment. We introduced basic types and values of the Nix
language, and basic expressions such as if, with and let. I invite you to re-
read about these expressions and play with them in the repl.

Functions help to build reusable components in a big repository like nixpkgs.

The Nix manual has a great explanation of functions. Let's go: pill on one
hand, Nix manual on the other hand.

I remind you how to enter the Nix environment: source ~/.nix-

profile/etc/profile.d/nix.sh

5.1. Nameless and single parameter

Functions are anonymous (lambdas), and only have a single parameter. The
syntax is extremely simple. Type the parameter name, then ":", then the body
of the function.
nix-repl> x: x*2
«lambda»

So here we defined a function that takes a parameter x, and returns x*2. The
problem is that we cannot use it in any way, because it's unnamed... joke!

We can store functions in variables.

nix-repl> double = x: x*2
nix-repl> double
«lambda»
nix-repl> double 3
6

As usual, please ignore the special syntax for assignments inside nix repl.
So, we defined a function x: x*2 that takes one parameter x, and returns x*2.
This function is then assigned to the variable double. Finally we did our first
function call: double 3.

Big note: it's not like many other programming languages where you write
double(3). It really is double 3.

In summary: to call a function, name the variable, then space, then the
argument. Nothing else to say, it's as easy as that.

5.2. More than one parameter

How do we create a function that accepts more than one parameter? For
people not used to functional programming, this may take a while to grasp.
Let's do it step by step.
nix-repl> mul = a: (b: a*b)
nix-repl> mul
«lambda»
nix-repl> mul 3
«lambda»
nix-repl> (mul 3) 4
12

We defined a function that takes the parameter a, the body returns another
function. This other function takes a parameter b and returns a*b. Therefore,
calling mul 3 returns this kind of function: b: 3*b. In turn, we call the
returned function with 4, and get the expected result.

You don't have to use parenthesis at all, Nix has sane priorities when parsing
the code:
nix-repl> mul = a: b: a*b
nix-repl> mul
«lambda»
nix-repl> mul 3
«lambda»
nix-repl> mul 3 4
12
nix-repl> mul (6+7) (8+9)
221

Much more readable, you don't even notice that functions only receive one
argument. Since the argument is separated by a space, to pass more complex
expressions you need parenthesis. In other common languages you would
write mul(6+7, 8+9).

Given that functions have only one parameter, it is straightforward to use

partial application:
nix-repl> foo = mul 3
nix-repl> foo 4
12
nix-repl> foo 5
15

We stored the function returned by mul 3 into a variable foo, then reused it.

5.3. Arguments set

Now this is a very cool feature of Nix. It is possible to pattern match over a
set in the parameter. We write an alternative version of mul = a: b: a*b
first by using a set as argument, then using pattern matching.
nix-repl> mul = s: s.a*s.b
nix-repl> mul { a = 3; b = 4; }
12
nix-repl> mul = { a, b }: a*b
nix-repl> mul { a = 3; b = 4; }
12

In the first case we defined a function that accepts a single parameter. We

then access attributes a and b from the given set. Note how the parenthesis-
less syntax for function calls is very elegant in this case, instead of doing
mul({ a=3; b=4; }) in other languages.

In the second case we defined an arguments set. It's like defining a set, except
without values. We require that the passed set contains the keys a and b. Then
we can use those a and b in the function body directly.
nix-repl> mul = { a, b }: a*b
nix-repl> mul { a = 3; b = 4; c = 6; }
error: anonymous function at (string):1:2 called with unexpected argument `c
nix-repl> mul { a = 3; }
error: anonymous function at (string):1:2 called without required argument `

Only a set with exactly the attributes required by the function is accepted,
nothing more, nothing less.

5.4. Default and variadic attributes

It is possible to specify default values of attributes in the arguments set:
nix-repl> mul = { a, b ? 2 }: a*b
nix-repl> mul { a = 3; }
6
nix-repl> mul { a = 3; b = 4; }
12

Also you can allow passing more attributes (variadic) than the expected
ones:
nix-repl> mul = { a, b, ... }: a*b
nix-repl> mul { a = 3; b = 4; c = 2; }

However, in the function body you cannot access the "c" attribute. The
solution is to give a name to the given set with the @-pattern:
nix-repl> mul = s@{ a, b, ... }: a*b*s.c
nix-repl> mul { a = 3; b = 4; c = 2; }
24

That's it, you give a name to the whole parameter with name@ before the set
pattern.

Advantages of using argument sets:

Named unordered arguments: you don't have to remember the order of

the arguments.

You can pass sets, that adds a whole new layer of flexibility and
convenience.

Disadvantages:
 Partial application does not work with argument sets. You have to
specify the whole attribute set, not part of it.

You may find similarities with Python **kwargs.

5.5. Imports
The import function is built-in and provides a way to parse a .nix file. The
natural approach is to define each component in a .nix file, then compose by
importing these files.

Let's start with the bare metal.

a.nix:

b.nix:

mul.nix:

a: b: a*b

nix-repl> a = import ./a.nix

nix-repl> b = import ./b.nix
nix-repl> mul = import ./mul.nix
nix-repl> mul a b
12

Yes it's really that simple. You import a file, and it gets parsed as expression.
Note that the scope of the imported file does not inherit the scope of the
importer.

test.nix:

nix-repl> let x = 5; in import ./test.nix

error: undefined variable `x' at /home/lethal/test.nix:1:1
So how do we pass information to the module? Use functions, like we did
with mul.nix. A more complex example:

test.nix:

{ a, b ? 3, trueMsg ? "yes", falseMsg ? "no" }:

if a > b
then builtins.trace trueMsg true
else builtins.trace falseMsg false

nix-repl> import ./test.nix { a = 5; trueMsg = "ok"; }

trace: ok
true

Explaining:

In test.nix we return a function. It accepts a set, with default attributes

b, trueMsg and falseMsg.

builtins.trace is a built-in function that takes two arguments. The

first is the message to display, the second is the value to return. It's
usually used for debugging purposes.

Then we import test.nix, and call the function with that set.

So when is the message shown? Only when it's in need to be evaluated.

5.6. Next pill

...we will finally write our first derivation.
Chapter 6. Our First Derivation
Welcome to the sixth Nix pill. In the previous fifth pill we introduced
functions and imports. Functions and imports are very simple concepts that
allows for building complex abstractions and composition of modules to
build a flexible Nix system.

In this post we finally arrived to writing a derivation. Derivations are the

building blocks of a Nix system, from a file system view point. The Nix
language is used to describe such derivations.

I remind you how to enter the Nix environment: source ~/.nix-

profile/etc/profile.d/nix.sh

6.1. The derivation function

The derivation built-in function is used to create derivations. I invite you to
read the link in the Nix manual about the derivation built-in. A derivation
from a Nix language view point is simply a set, with some attributes.
Therefore you can pass the derivation around with variables like anything
else.

That's where the real power comes in.

The derivation function receives a set as first argument. This set requires at
least the following three attributes:

name: the name of the derivation. In the nix store the format is hash-
name, that's the name.

system: is the name of the system in which the derivation can be built.
For example, x86_64-linux.

builder: it is the binary program that builds the derivation.

First of all, what's the name of our system as seen by nix?
nix-repl> builtins.currentSystem
"x86_64-linux"

Let's try to fake the name of the system:

nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system =
nix-repl> d
«derivation /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv»

Oh oh, what's that? Did it build the derivation? No it didn't, but it did create
the .drv file. nix repl does not build derivations unless you tell to do so.

6.2. Digression about .drv files

What's that .drv file? It is the specification of how to build the derivation,
without all the Nix language fuzz.

Before continuing, some analogies with the C language:

.nix files are like .c files

.drv files are intermediate files like .o files. The .drv describes how to
build a derivation, it's the bare minimum information.

out paths are then the product of the build

Both drv paths and out paths are stored in the nix store as you can see.

What's in that .drv file? You can read it, but it's better to pretty print it:
$ nix show-derivation /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-
{
"/nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv": {
"outputs": {
"out": {
"path": "/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname"
}
},
"inputSrcs": [],
 "inputDrvs": {},
"platform": "mysystem",
"builder": "mybuilder",
"args": [],
"env": {
"builder": "mybuilder",
"name": "myname",
"out": "/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname",
"system": "mysystem"
}
}
}

Ok we can see there's an out path, but it does not exist yet. We never told Nix
to build it, but we know beforehand where the build output will be. Why?

Think, if Nix ever built the derivation just because we accessed it in Nix, we
would have to wait a long time if it was, say, Firefox. That's why Nix let us
know the path beforehand and keep evaluating the Nix expressions, but it's
still empty because no build was ever made.

Important: the hash of the out path is based solely on the input derivations in
the current version of Nix, not on the contents of the build product. It's
possible however to have content-addressable derivations for e.g. tarballs as
we'll see later on.

Many things are empty in that .drv, however I write a summary of the .drv
format for you:

1. The output paths (there can be multiple ones). By default nix creates one
out path called "out".

2. The list of input derivations. It's empty because we are not referring to
any other derivation. Otherwise, there would a list of other .drv files.

3. The system and the builder executable (yes, it's a fake one).

4. Then a list of environment variables passed to the builder.

That's it, the minimum necessary information to build our derivation.

Important note: the environment variables passed to the builder are just those
you see in the .drv plus some other Nix related configuration (number of
cores, temp dir, ...). The builder will not inherit any variable from your
running shell, otherwise builds would suffer from non-determinism.

Back to our fake derivation.

Let's build our really fake derivation:

nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system =
nix-repl> :b d
[...]
these derivations will be built:
/nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv
building path(s) `/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname'
error: a `mysystem' is required to build `/nix/store/z3hhlxbckx4g3n9sw91nnvl

The :b is a nix repl specific command to build a derivation. You can see
more commands with :? . So in the output you can see that it takes the .drv
as information on how to build the derivation. Then it says it's trying to
produce our out path. Finally the error we were waiting for: that derivation
can't be built on our system.

We're doing the build inside nix repl, but what if we don't want to use nix
repl? You can realise a .drv with:

$ nix-store -r /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv

You will get the same output as before.

Let's fix the system attribute:

nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system =
nix-repl> :b d
[...]
build error: invalid file name `mybuilder'

A step forward: of course, that mybuilder executable does not really exist.
Stop for a moment.

6.3. What's in a derivation set

I find useful to inspect the returned value from the derivation function for
you. First of all, the returned value is a plain set:
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system =
nix-repl> builtins.isAttrs d
true
nix-repl> builtins.attrNames d
[ "all" "builder" "drvAttrs" "drvPath" "name" "out" "outPath" "outputName" "

You can guess what builtins.isAttrs does, it returns true if the argument is
a set. While builtins.attrNames returns a list of keys of the given set. Some
kind of reflection, you might say.

Start from drvAttrs:

nix-repl> d.drvAttrs
{ builder = "mybuilder"; name = "myname"; system = "mysystem"; }

That's basically the input we gave to the derivation function. Also d.name,
d.system and d.builder attributes are straight the ones we gave as input.

nix-repl> (d == d.out)
true

So out is just the derivation itself, it seems weird but the reason is that we
only have one output from the derivation. That's also the reason why d.all is
a singleton. We'll see multiple outputs later.

The d.drvPath is the path of the .drv file:

/nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv.

Something interesting is the type attribute. It's "derivation". Nix does add a
little of magic to sets with type derivation, but not that much. To let you
understand, you can create yourself a set with that type, it's a simple set:
nix-repl> { type = "derivation"; }
«derivation ???»

Of course it has no other information, so Nix doesn't know what to say :-) But
you get it, the type = "derivation" is just a convention for Nix and for us
to understand the set is a derivation.
When writing packages, we are interested in the outputs. The other metadata
is needed for Nix to know how to create the drv path and the out path.

The outPath attribute is the build path in the nix store:

/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname.

6.4. Referring to other derivations

Just like dependencies in other package managers, how do we refer to other
packages? How do we refer to other derivations in terms of files on the disk?
We use the outPath. The outPath tells where the files are of that derivation.
To make it more convenient, Nix is able to do a conversion from a derivation
set to a string.
nix-repl> d.outPath
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname"
nix-repl> builtins.toString d
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname"

Nix does the "set to string conversion" as long as there is the outPath
attribute (much like a toString method in other languages):
nix-repl> builtins.toString { outPath = "foo"; }
"foo"
nix-repl> builtins.toString { a = "b"; }
error: cannot coerce a set to a string, at (string):1:1

Say we want to use binaries from coreutils (ignore the nixpkgs etc.):
nix-repl> :l <nixpkgs>
Added 3950 variables.
nix-repl> coreutils
«derivation /nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-coreutils-8.21.drv»
nix-repl> builtins.toString coreutils
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21"

Apart the nixpkgs stuff, just think we added to the scope a series of variables.
One of them is coreutils. It is the derivation of the coreutils package you all
know of from other Linux distributions. It contains basic binaries for
GNU/Linux systems (you may have multiple derivations of coreutils in the
nix store, no worries):
$ ls /nix/store/*coreutils*/bin
[...]

I remind you, inside strings it's possible to interpolate Nix expressions with
${...}:

nix-repl> "${d}"
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname"
nix-repl> "${coreutils}"
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21"

That's very convenient, because then we could refer to e.g. the bin/true binary
like this:
nix-repl> "${coreutils}/bin/true"
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21/bin/true"

6.5. An almost working derivation

In the previous attempt we used a fake builder, mybuilder which obviously
does not exist. But we can use for example bin/true, which always exits with
0 (success).
nix-repl> :l <nixpkgs>
nix-repl> d = derivation { name = "myname"; builder = "${coreutils}/bin/true
nix-repl> :b d
[...]
builder for `/nix/store/qyfrcd53wmc0v22ymhhd5r6sz5xmdc8a-myname.drv' failed

Another step forward, it executed the builder (bin/true), but the builder did
not create the out path of course, it just exited with 0.

Obvious note: everytime we change the derivation, a new hash is created.

Let's examine the new .drv now that we referred to another derivation:
$ nix show-derivation /nix/store/qyfrcd53wmc0v22ymhhd5r6sz5xmdc8a-
{
"/nix/store/qyfrcd53wmc0v22ymhhd5r6sz5xmdc8a-myname.drv": {
"outputs": {
"out": {
"path": "/nix/store/ly2k1vswbfmswr33hw0kf0ccilrpisnk-myname"
 }
},
"inputSrcs": [],
"inputDrvs": {
"/nix/store/hixdnzz2wp75x1jy65cysq06yl74vx7q-coreutils-8.29.drv": [
"out"
]
},
"platform": "x86_64-linux",
"builder": "/nix/store/qrxs7sabhqcr3j9ai0j0cp58zfnny0jz-coreutils-8.29/b
"args": [],
"env": {
"builder": "/nix/store/qrxs7sabhqcr3j9ai0j0cp58zfnny0jz-coreutils-8.29
"name": "myname",
"out": "/nix/store/ly2k1vswbfmswr33hw0kf0ccilrpisnk-myname",
"system": "x86_64-linux"
}
}
}

Aha! Nix added a dependency to our myname.drv, it's the coreutils.drv.

Before doing our build, Nix should build the coreutils.drv. But since coreutils
is already in our nix store, no build is needed, it's already there with out path
/nix/store/dm20hrdk6s4jzfmk1197p2nya0p8fy3a-coreutils-8.29.

6.6. When is the derivation built

Nix does not build derivations during evaluation of Nix expressions. In fact,
that's why we have to do ":b drv" in nix repl, or use nix-store -r in the first
place.

An important separation is made in Nix:

Instantiate/Evaluation time: the Nix expression is parsed, interpreted

and finally returns a derivation set. During evaluation, you can refer to
other derivations because Nix will create .drv files and we will know out
paths beforehand. This is achieved with nix-instantiate.

Realise/Build time: the .drv from the derivation set is built, first
building .drv inputs (build dependencies). This is achieved with nix-
store -r.
Think of it as of compile time and link time like with C/C++ projects. You
first compile all source files to object files. Then link object files in a single
executable.

In Nix, first the Nix expression (usually in a .nix file) is compiled to .drv,
then each .drv is built and the product is installed in the relative out paths.

6.7. Conclusion
It's that complicated to create a package for Nix? No it's not.

We're walking through the fundamentals of Nix derivations, to understand

how they work, how they are represented. Packaging in Nix is certainly
easier than that, but we're not there yet in this post. More Nix pills are
needed.

With the derivation function we provide a set of information on how to build

a package, and we get back the information about where the package was
built. Nix converts a set to a string when there's an outPath, that's very
convenient. With that, it's easy to refer to other derivations.

When Nix builds a derivation, it first creates a .drv file from a derivation
expression, and uses it to build the output. It does so recursively for all the
dependencies (inputs). It "executes" the .drv files like a machine. Not much
magic after all.

6.8. Next pill

...we will finally write our first working derivation. Yes, this post is about
"our first derivation", but I never said it was a working one ;)
Chapter 7. Working Derivation
7.1. Introduction
Welcome to the seventh nix pill. In the previous sixth pill we introduced the
notion of derivation in the Nix language — how to define a raw derivation
and how to (try to) build it.

In this post we continue along the path, by creating a derivation that actually
builds something. Then, we try to package a real program: we compile a
simple C file and create a derivation out of it, given a blessed toolchain.

I remind you how to enter the Nix environment: source ~/.nix-

profile/etc/profile.d/nix.sh

7.2. Using a script as a builder

What's the easiest way to run a sequence of commands for building
something? A bash script. We write a custom bash script, and we want it to
be our builder. Given a builder.sh, we want the derivation to run bash
builder.sh.

We don't use hash bangs in builder.sh, because at the time we are writing it
we do not know the path to bash in the nix store. Yes, even bash is in the nix
store, everything is there.

We don't even use /usr/bin/env, because then we lose the cool stateless
property of Nix. Not to mention that PATH gets cleared when building, so it
wouldn't find bash anyway.

In summary, we want the builder to be bash, and pass it an argument,

builder.sh. Turns out the derivation function accepts an optional args
attribute which is used to pass arguments to the builder executable.
First of all, let's write our builder.sh in the current directory:
declare -xp
echo foo > $out

The command declare -xp lists exported variables. (declare is a builtin

bash function.) As we covered in the previous pill, Nix computes the output
path of the derivation. The resulting .drv file contains a list of environment
variables passed to the builder. One of these is $out.

What we have to do is create something in the path $out, be it a file or a

directory. In this case we are creating a file.

In addition, we print out the environment variables during the build process.
We cannot use env for this, because env is part of coreutils and we don't have
a dependency to it yet. We only have bash for now.

Like for coreutils in the previous pill, we get a blessed bash for free from our
magic nixpkgs stuff:
nix-repl> :l <nixpkgs>
Added 3950 variables.
nix-repl> "${bash}"
"/nix/store/ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6-bash-4.2-p45"

So with the usual trick, we can refer to bin/bash and create our derivation:
nix-repl> d = derivation { name = "foo"; builder = "${bash}/bin/bash"; args
nix-repl> :b d
these derivations will be built:
/nix/store/i76pr1cz0za3i9r6xq518bqqvd2raspw-foo.drv
building '/nix/store/i76pr1cz0za3i9r6xq518bqqvd2raspw-foo.drv'...
declare -x HOME="/homeless-shelter"
declare -x NIX_BUILD_CORES="4"
declare -x NIX_BUILD_TOP="/tmp/nix-build-foo.drv-0"
declare -x NIX_LOG_FD="2"
declare -x NIX_STORE="/nix/store"
declare -x OLDPWD
declare -x PATH="/path-not-set"
declare -x PWD="/tmp/nix-build-foo.drv-0"
declare -x SHLVL="1"
declare -x TEMP="/tmp/nix-build-foo.drv-0"
declare -x TEMPDIR="/tmp/nix-build-foo.drv-0"
declare -x TMP="/tmp/nix-build-foo.drv-0"
declare -x TMPDIR="/tmp/nix-build-foo.drv-0"
declare -x builder="/nix/store/q1g0rl8zfmz7r371fp5p42p4acmv297d-bash-4.4-p19
declare -x name="foo"
declare -x out="/nix/store/gczb4qrag22harvv693wwnflqy7lx5pb-foo"
declare -x system="x86_64-linux"
warning: you did not specify '--add-root'; the result might be removed by th
/nix/store/gczb4qrag22harvv693wwnflqy7lx5pb-foo

this derivation produced the following outputs:

out -> /nix/store/gczb4qrag22harvv693wwnflqy7lx5pb-foo

We did it! The contents of

/nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo is really foo. We've
built our first derivation.

Note that we used ./builder.sh and not "./builder.sh". This way, it is

parsed as a path, and Nix performs some magic which we will cover later.
Try using the string version and you will find that it cannot find builder.sh.
This is because it tries to find it relative to the temporary build directory.

7.3. The builder environment

Let's inspect those environment variables printed during the build process.

$HOME is not your home directory, and /homeless-shelter doesn't exist

at all. We force packages not to depend on $HOME during the build
process.

$PATH plays the same game as $HOME

$NIX_BUILD_CORES and $NIX_STORE are nix configuration options

$PWD and $TMP clearly show that nix created a temporary build directory

Then $builder, $name, $out, and $system are variables set due to the
.drv file's contents.

And that's how we were able to use $out in our derivation and put stuff in it.
It's like Nix reserved a slot in the nix store for us, and we must fill it.
In terms of autotools, $out will be the --prefix path. Yes, not the make
DESTDIR, but the --prefix. That's the essence of stateless packaging. You
don't install the package in a global common path under /, you install it in a
local isolated path under your nix store slot.

7.4. The .drv contents

We added something else to the derivation this time: the args attribute. Let's
see how this changed the .drv compared to the previous pill:
$ nix show-derivation /nix/store/i76pr1cz0za3i9r6xq518bqqvd2raspw-
{
"/nix/store/i76pr1cz0za3i9r6xq518bqqvd2raspw-foo.drv": {
"outputs": {
"out": {
"path": "/nix/store/gczb4qrag22harvv693wwnflqy7lx5pb-foo"
}
},
"inputSrcs": [
"/nix/store/lb0n38r2b20r8rl1k45a7s4pj6ny22f7-builder.sh"
],
"inputDrvs": {
"/nix/store/hcgwbx42mcxr7ksnv0i1fg7kw6jvxshb-bash-4.4-p19.drv": [
"out"
]
},
"platform": "x86_64-linux",
"builder": "/nix/store/q1g0rl8zfmz7r371fp5p42p4acmv297d-bash-4.4-p19/bin
"args": [
"/nix/store/lb0n38r2b20r8rl1k45a7s4pj6ny22f7-builder.sh"
],
"env": {
"builder": "/nix/store/q1g0rl8zfmz7r371fp5p42p4acmv297d-bash-4.4-p19/b
"name": "foo",
"out": "/nix/store/gczb4qrag22harvv693wwnflqy7lx5pb-foo",
"system": "x86_64-linux"
}
}
}

Much like the usual .drv, except that there's a list of arguments in there
passed to the builder (bash) with builder.sh… In the nix store..? Nix
automatically copies files or directories needed for the build into the store to
ensure that they are not changed during the build process and that the
deployment is stateless and independent of the building machine. builder.sh
is not only in the arguments passed to the builder, it's also in the input
derivations.

Given that builder.sh is a plain file, it has no .drv associated with it. The
store path is computed based on the filename and on the hash of its contents.
Store paths are covered in detail in a later pill.

7.5. Packaging a simple C program

Start off by writing a simple C program called simple.c:
void main() {
puts("Simple!");
}

And its simple_builder.sh:

export PATH="$coreutils/bin:$gcc/bin"
mkdir $out
gcc -o $out/simple $src

Don't worry too much about where those variables come from yet; let's write
the derivation and build it:
nix-repl> :l <nixpkgs>
nix-repl> simple = derivation { name = "simple"; builder = "${bash}/bin/bash
nix-repl> :b simple
this derivation produced the following outputs:

out -> /nix/store/ni66p4jfqksbmsl616llx3fbs1d232d4-simple

Now you can run /nix/store/ni66p4jfqksbmsl616llx3fbs1d232d4-

simple/simple in your shell.

7.6. Explanation
We added two new attributes to the derivation call, gcc and coreutils. In
gcc = gcc;, the name on the left is the name in the derivation set, and the
name on the right refers to the gcc derivation from nixpkgs. The same applies
for coreutils.

We also added the src attribute, nothing magical — it's just a name, to which
the path ./simple.c is assigned. Like simple-builder.sh, simple.c will be
added to the store.

The trick: every attribute in the set passed to derivation will be converted to
a string and passed to the builder as an environment variable. This is how the
builder gains access to coreutils and gcc: when converted to strings, the
derivations evaluate to their output paths, and appending /bin to these leads
us to their binaries.

The same goes for the src variable. $src is the path to simple.c in the nix
store. As an exercise, pretty print the .drv file. You'll see simple_builder.sh
and simple.c listed in the input derivations, along with bash, gcc and
coreutils .drv files. The newly added environment variables described above
will also appear.

In simple_builder.sh we set the PATH for gcc and coreutils binaries, so that
our build script can find the necessary utilities like mkdir and gcc.

We then create $out as a directory and place the binary inside it. Note that
gcc is found via the PATH environment variable, but it could equivalently be
referenced explicitly using $gcc/bin/gcc.

7.7. Enough of nix repl

Drop out of nix repl and write a file simple.nix:
with (import <nixpkgs> {});
derivation {
name = "simple";
builder = "${bash}/bin/bash";
args = [ ./simple_builder.sh ];
inherit gcc coreutils;
src = ./simple.c;
system = builtins.currentSystem;
}
Now you can build it with nix-build simple.nix. This will create a symlink
result in the current directory, pointing to the out path of the derivation.

nix-build does two jobs:

nix-instantiate : parse and evaluate simple.nix and return the .drv file
corresponding to the parsed derivation set

nix-store -r : realise the .drv file, which actually builds it.

Finally, it creates the symlink.

In the first line of simple.nix, we have an import function call nested in a

with statement. Recall that import accepts one argument, a nix file to load.
In this case, the contents of the file evaluated to a function.

Afterwards, we call the function with the empty set. We saw this already in
the fifth pill. To reiterate: import <nixpkgs> {} is calling two functions, not
one. Reading it as (import <nixpkgs>) {} makes this clearer.

The value returned by the nixpkgs function is a set. More specifically, it's a
set of derivations. Using the with expression we bring them into scope. This
is equivalent to the :l <nixpkgs> we used in nix repl; it allows us to easily
access derivations such as bash, gcc, and coreutils.

Then we meet the inherit keyword. inherit foo; is equivalent to foo =

foo;. Similarly, inherit foo bar; is equivalent to foo = foo; bar =
bar;.

This syntax only makes sense inside sets. There's no magic involved, it's
simply a convenience to avoid repeating the same name for both the attribute
name and the value in scope.

7.8. Next pill

We will generalize the builder. You may have noticed that we wrote two
separate builder.sh scripts in this post. We would like to have a generic
builder script instead, especially since each build script goes in the nix store:
a bit of a waste.

Is it really that hard to package stuff in Nix? No, here we're studying the
fundamentals of Nix.
Chapter 8. Generic Builders
Welcome to the 8th Nix pill. In the previous 7th pill we successfully built a
derivation. We wrote a builder script that compiled a C file and installed the
binary under the nix store.

In this post, we will generalize the builder script, write a Nix expression for
GNU hello world and create a wrapper around the derivation built-in
function.

8.1. Packaging GNU hello world

In the previous pill we packaged a simple .c file, which was being compiled
with a raw gcc call. That's not a good example of project. Many use
autotools, and since we're going to generalize our builder, better do it with the
most used build system.

GNU hello world, despite its name, is a simple yet complete project using
autotools. Fetch the latest tarball here: http://ftp.gnu.org/gnu/hello/hello-
2.10.tar.gz.

Let's create a builder script for GNU hello world, hello_builder.sh:

export PATH="$gnutar/bin:$gcc/bin:$gnumake/bin:$coreutils/bin:$gawk/bin:$gzi
tar -xzf $src
cd hello-2.10
./configure --prefix=$out
make
make install

And the derivation hello.nix:

with (import <nixpkgs> {});
derivation {
name = "hello";
builder = "${bash}/bin/bash";
args = [ ./hello_builder.sh ];
inherit gnutar gzip gnumake gcc coreutils gawk gnused gnugrep;
 binutils = binutils-unwrapped;
src = ./hello-2.10.tar.gz;
system = builtins.currentSystem;
}

Nix on darwin
The darwin (i.e. macOS) stdenv diverges from the Linux stdenv in several
ways. A main difference is that the darwin stdenv relies upon clang rather
than gcc as its C compiler. We can adapt this early example of how a stdenv
works for darwin by using this modified version of hello.nix:
with (import <nixpkgs> {});
derivation {
name = "hello";
builder = "${bash}/bin/bash";
args = [ ./hello_builder.sh ];
inherit gnutar gzip gnumake coreutils gawk gnused gnugrep;
gcc = clang;
binutils = clang.bintools.bintools_bin;
src = ./hello-2.10.tar.gz;
system = builtins.currentSystem;
}

Please be aware that similar changes may be needed in what follows.

Now build it with nix-build hello.nix and you can launch

result/bin/hello. Nothing easier, but do we have to create a builder.sh for
each package? Do we always have to pass the dependencies to the
derivation function?

Please note the --prefix=$out we were talking about in the previous pill.

8.2. A generic builder

Let's a create a generic builder.sh for autotools projects:
set -e
unset PATH
for p in $buildInputs; do
export PATH=$p/bin${PATH:+:}$PATH
done

tar -xf $src

for d in *; do
if [ -d "$d" ]; then
cd "$d"
break
fi
done

./configure --prefix=$out
make
make install

What do we do here?

1. Exit the build on any error with set -e.

2. First unset PATH, because it's initially set to a non-existant path.

3. We'll see this below in detail, however for each path in $buildInputs,
we append bin to PATH.

4. Unpack the source.

5. Find a directory where the source has been unpacked and cd into it.

6. Once we're set up, compile and install.

As you can see, there's no reference to "hello" in the builder anymore. It still
does several assumptions, but it's certainly more generic.

Now let's rewrite hello.nix:

with (import <nixpkgs> {});
derivation {
name = "hello";
builder = "${bash}/bin/bash";
args = [ ./builder.sh ];
buildInputs = [ gnutar gzip gnumake gcc binutils-unwrapped coreutils gawk
src = ./hello-2.10.tar.gz;
system = builtins.currentSystem;
}

All clear, except that buildInputs. However it's easier than any black magic
you are thinking in this moment.

Nix is able to convert a list to a string. It first converts the elements to strings,
and then concatenates them separated by a space:
nix-repl> builtins.toString 123
"123"
nix-repl> builtins.toString [ 123 456 ]
"123 456"

Recall that derivations can be converted to a string, hence:

nix-repl> :l <nixpkgs>
Added 3950 variables.
nix-repl> builtins.toString gnugrep
"/nix/store/g5gdylclfh6d224kqh9sja290pk186xd-gnugrep-2.14"
nix-repl> builtins.toString [ gnugrep gnused ]
"/nix/store/g5gdylclfh6d224kqh9sja290pk186xd-gnugrep-2.14 /nix/store/krgdc4s

Simple! The buildInputs variable is a string with out paths separated by

space, perfect for bash usage in a for loop.

8.3. A more convenient derivation function

We managed to write a builder that can be used for multiple autotools
projects. But in the hello.nix expression we are specifying tools that are
common to more projects; we don't want to pass them everytime.

A natural approach would be to create a function that accepts an attribute set,

similar to the one used by the derivation function, and merge it with another
attribute set containing values common to many projects.

Create autotools.nix:
pkgs: attrs:
with pkgs;
let defaultAttrs = {
builder = "${bash}/bin/bash";
 args = [ ./builder.sh ];
baseInputs = [ gnutar gzip gnumake gcc binutils-unwrapped coreutils gawk
buildInputs = [];
system = builtins.currentSystem;
};
in
derivation (defaultAttrs // attrs)

Ok now we have to remember a little about Nix functions. The whole nix
expression of this autotools.nix file will evaluate to a function. This
function accepts a parameter pkgs, then returns a function which accepts a
parameter attrs.

The body of the function is simple, yet at first sight it might be hard to grasp:

1. First drop in the scope the magic pkgs attribute set.

2. Within a let expression we define a helper variable, defaultAttrs,

which serves as a set of common attributes used in derivations.

3. Finally we create the derivation with that strange expression,

(defaultAttrs // attrs).

The // operator is an operator between two sets. The result is the union of the
two sets. In case of conflicts between attribute names, the value on the right
set is preferred.

So we use defaultAttrs as base set, and add (or override) the attributes from
attrs.

A couple of examples ought to be enough to clear out the behavior of the

operator:
nix-repl> { a = "b"; } // { c = "d"; }
{ a = "b"; c = "d"; }
nix-repl> { a = "b"; } // { a = "c"; }
{ a = "c"; }

Exercise: Complete the new builder.sh by adding $baseInputs in the for

loop together with $buildInputs. As you noticed, we passed that new
variable in the derivation. Instead of merging buildInputs with the base ones,
we prefer to preserve buildInputs as seen by the caller, so we keep them
separated. Just a matter of choice.

Then we rewrite hello.nix as follows:

let
pkgs = import <nixpkgs> {};
mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
name = "hello";
src = ./hello-2.10.tar.gz;
}

Finally! We got a very simple description of a package! A couple of remarks

that you may find useful to keep understanding the nix language:

We assigned to pkgs the import that we did in the previous expressions

in the "with", don't be afraid. It's that straightforward.

The mkDerivation variable is a nice example of partial application, look

at it as (import ./autotools.nix) pkgs. First we import the expression,
then we apply the pkgs parameter. That will give us a function that
accepts the attribute set attrs.

We create the derivation specifying only name and src. If the project
eventually needed other dependencies to be in PATH, then we would
simply add those to buildInputs (not specified in hello.nix because
empty).

Note we didn't use any other library. Special C flags may be needed to find
include files of other libraries at compile time, and ld flags at link time.

8.4. Conclusion
Nix gives us the bare metal tools for creating derivations, setting up a build
environment and storing the result in the nix store.

Out of this we managed to create a generic builder for autotools projects, and
a function mkDerivation that composes by default the common components
used in autotools projects instead of repeating them in all the packages we
would write.

We are feeling the way a Nix system grows up: it's about creating and
composing derivations with the Nix language.

Analogy: in C you create objects in the heap, and then you compose them
inside new objects. Pointers are used to refer to other objects.

In Nix you create derivations stored in the nix store, and then you compose
them by creating new derivations. Store paths are used to refer to other
derivations.

8.5. Next pill

...we will talk a little about runtime dependencies. Is the GNU hello world
package self-contained? What are its runtime dependencies? We only
specified build dependencies by means of using other derivations in the
"hello" derivation.
Chapter 9. Automatic Runtime
Dependencies
Welcome to the 9th Nix pill. In the previous 8th pill we wrote a generic
builder for autotools projects. We feed build dependencies, a source tarball,
and we get a Nix derivation as a result.

Today we stop by the GNU hello world program to analyze build and runtime
dependencies, and enhance the builder in order to avoid unnecessary runtime
dependencies.

9.1. Build dependencies

Let's start analyzing build dependencies for our GNU hello world package:
$ nix-instantiate hello.nix
/nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
$ nix-store -q --references /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hell
/nix/store/0q6pfasdma4as22kyaknk4kwx4h58480-hello-2.10.tar.gz
/nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-coreutils-8.21.drv
/nix/store/2h4b30hlfw4fhqx10wwi71mpim4wr877-gnused-4.2.2.drv
/nix/store/39bgdjissw9gyi4y5j9wanf4dbjpbl07-gnutar-1.27.1.drv
/nix/store/7qa70nay0if4x291rsjr7h9lfl6pl7b1-builder.sh
/nix/store/g6a0shr58qvx2vi6815acgp9lnfh9yy8-gnugrep-2.14.drv
/nix/store/jdggv3q1sb15140qdx0apvyrps41m4lr-bash-4.2-p45.drv
/nix/store/pglhiyp1zdbmax4cglkpz98nspfgbnwr-gnumake-3.82.drv
/nix/store/q9l257jn9lndbi3r9ksnvf4dr8cwxzk7-gawk-4.1.0.drv
/nix/store/rgyrqxz1ilv90r01zxl0sq5nq0cq7v3v-binutils-2.23.1.drv
/nix/store/qzxhby795niy6wlagfpbja27dgsz43xk-gcc-wrapper-4.8.3.drv
/nix/store/sk590g7fv53m3zp0ycnxsc41snc2kdhp-gzip-1.6.drv

It has exactly the derivations referenced in the derivation function, nothing

more, nothing less. Some of them might not be used at all, however given
that our generic mkDerivation function always pulls such dependencies
(think of it like build-essential of Debian), for every package you build from
now on, you will have these packages in the nix store.
Why are we looking at .drv files? Because the hello.drv file is the
representation of the build action to perform in order to build the hello out
path, and as such it also contains the input derivations needed to be built
before building hello.

9.2. Digression about NAR files

NAR is the Nix ARchive. First question: why not tar? Why another archiver?
Because commonly used archivers are not deterministic. They add padding,
they do not sort files, they add timestamps, etc.. Hence NAR, a very simple
deterministic archive format being used by Nix for deployment. NARs are
also used extensively within Nix itself as we'll see below.

For the rationale and implementation details you can find more in the
Dolstra's PhD Thesis.

To create NAR archives, it's possible to use nix-store --dump and nix-store -
-restore. Those two commands work regardless of /nix/store.

9.3. Runtime dependencies

Something is different for runtime dependencies however. Build
dependencies are automatically recognized by Nix once they are used in any
derivation call, but we never specify what are the runtime dependencies for
a derivation.

There's really black magic involved. It's something that at first glance makes
you think "no, this can't work in the long term", but at the same it works so
well that a whole operating system is built on top of this magic.

In other words, Nix automatically computes all the runtime dependencies of a

derivation, and it's possible thanks to the hash of the store paths.

Steps:

1. Dump the derivation as NAR, a serialization of the derivation output.

Works fine whether it's a single file or a directory.
 2. For each build dependency .drv and its relative out path, search the
contents of the NAR for this out path.

3. If found, then it's a runtime dependency.

You get really all the runtime dependencies, and that's why Nix deployments
are so easy.
$ nix-instantiate hello.nix
/nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
$ nix-store -r /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
/nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello
$ nix-store -q --references /nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hell
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19
/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-gcc-4.8.3
/nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello

Ok glibc and gcc. Well, gcc really should not be a runtime dependency!
$ strings result/bin/hello|grep gcc
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib:/nix/store/8jm0wk

Oh Nix added gcc because its out path is mentioned in the "hello" binary.
Why is that? That's the ld rpath. It's the list of directories where libraries can
be found at runtime. In other distributions, this is usually not abused. But in
Nix, we have to refer to particular versions of libraries, thus the rpath has an
important role.

The build process adds that gcc lib path thinking it may be useful at runtime,
but really it's not. How do we get rid of it? Nix authors have written another
magical tool called patchelf, which is able to reduce the rpath to the paths that
are really used by the binary.

Not only, even after reducing the rpath the hello binary would still depend
upon gcc. Because of debugging information. For that, the well known strip
can be used.

9.4. Another phase in the builder

We will add a new phase to our autotools builder. The builder has these
phases already:

1. First the environment is set up

2. Unpack phase: we unpack the sources in the current directory

(remember, Nix changes dir to a temporary directory first)

3. Change source root to the directory that has been unpacked

4. Configure phase: ./configure

5. Build phase: make

6. Install phase: make install

We add a new phase after the installation phase, which we call fixup phase.
At the end of the builder.sh follows:
find $out -type f -exec patchelf --shrink-rpath '{}' \; -exec strip '{}' \;

That is, for each file we run patchelf --shrink-rpath and strip. Note that we
used two new commands here, find and patchelf. Exercise: These two
deserve a place in baseInputs of autotools.nix as findutils and patchelf.

Rebuild hello.nix and...:

$ nix-build hello.nix
[...]
$ nix-store -q --references result
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19
/nix/store/md4a3zv0ipqzsybhjb8ndjhhga1dj88x-hello

...only glibc is the runtime dependency. Exactly what we wanted.

The package is self-contained, copy its closure on another machine and you
will be able to run it. I remind you the very few components under the
/nix/store necessary to run nix when we installed it. The hello binary will
use that exact version of glibc library and interpreter, not the system one:
$ ldd result/bin/hello
linux-vdso.so.1 (0x00007fff11294000)
 libc.so.6 => /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/lib
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/ld-linux-x86-64.

Of course, the executable runs fine as long as everything is under the

/nix/store path.

9.5. Conclusion
Short post compared to previous ones as I'm still on vacation, but I hope you
enjoyed it. Nix provides tools with cool features. In particular, Nix is able to
compute all runtime dependencies automatically for us. Not only shared
libraries, but also referenced executables, scripts, Python libraries etc..

This makes packages self-contained, because we're sure (apart data and
configuration) that copying the runtime closure on another machine is
sufficient to run the program. That's why Nix has one-click install, or reliable
deployment in the cloud. All with one tool.

9.6. Next pill

...we will introduce nix-shell. With nix-build we build derivations always
from scratch: the source gets unpacked, configured, built and installed. But
this may take a long time, think of WebKit. What if we want to apply some
small changes and compile incrementally instead, yet keeping a self-
contained environment similar to nix-build?
Chapter 10. Developing with nix-
shell
Welcome to the 10th Nix pill. In the previous 9th pill we have seen one of the
powerful features of nix, automatic discovery of runtime dependencies and
finalized the GNU hello world package.

In return from vacation, we want to hack a little the GNU hello world
program. The nix-build tool creates an isolated environment for building the
derivation, we want to do the same in order to modify some source files of
the project.

10.1. What's nix-shell

The nix-shell tool drops us in a shell by setting up the necessary environment
variables to hack a derivation. It does not build the derivation, it only serves
as a preparation so that we can run the build steps manually.

I remind you, in a nix environment you don't have access to libraries and
programs unless you install them with nix-env. However installing libraries
with nix-env is not good practice. We prefer to have isolated environments
for development.
$ nix-shell hello.nix
[nix-shell]$ make
bash: make: command not found
[nix-shell]$ echo $baseInputs
/nix/store/jff4a6zqi0yrladx3kwy4v6844s3swpc-gnutar-1.27.1 [...]

First thing to notice, we call nix-shell on a nix expression which returns a

derivation. We then enter a new bash shell, but it's really useless. We
expected to have the GNU hello world build inputs available in PATH,
including GNU make, but it's not the case.

But, we have the environment variables that we set in the derivation, like
$baseInputs, $buildInputs, $src and so on.

That means we can source our builder.sh, and it will build the derivation.
You may get an error in the installation phase, because the user may not have
the permission to write to /nix/store:
[nix-shell]$ source builder.sh
...

It didn't install, but it built. Things to notice:

We sourced builder.sh, therefore it ran all the steps including setting up

the PATH for us.

The working directory is no more a temp directory created by nix-build,

but the current directory. Therefore, hello-2.10 has been unpacked there.

We're able to cd into hello-2.10 and type make, because now it's available.

In other words, nix-shell drops us in a shell with the same (or almost)
environment used to run the builder!

10.2. A builder for nix-shell

The previous steps are a bit annoying of course, but we can improve our
builder to be more nix-shell friendly.

First of all, we were able to source builder.sh because it was in our current
directory, but that's not nice. We want the builder.sh that is stored in the nix
store, the one that would be used by nix-build. To do so, the right way is to
pass the usual environment variable through the derivation.

Note: $builder is already defined, but it's the bash executable, not our
builder.sh. Our builder.sh is an argument to bash.

Second, we don't want to run the whole builder, we only want it to setup the
necessary environment for manually building the project. So we'll write two
files, one for setting up the environment, and the real builder.sh that runs
with nix-build.

Additionally, we'll wrap the phases in functions, it may be useful, and move
the set -e to the builder instead of the setup. The set -e is annoying in nix-
shell.

Here is our modified autotools.nix. Noteworthy is the setup =

./setup.sh; attribute in the derivation, which adds setup.sh to the nix store
and as usual, adds a $setup environment variable in the builder.
pkgs: attrs:
with pkgs;
let defaultAttrs = {
builder = "${bash}/bin/bash";
args = [ ./builder.sh ];
setup = ./setup.sh;
baseInputs = [ gnutar gzip gnumake gcc binutils-unwrapped coreutils gawk
buildInputs = [];
system = builtins.currentSystem;
};
in
derivation (defaultAttrs // attrs)

Thanks to that, we can split builder.sh into setup.sh and builder.sh.

What builder.sh does is sourcing $setup and calling the genericBuild
function. Everything else is just some bash changes.

Here is the modified builder.sh.

set -e
source $setup
genericBuild

Here is the modified setup.sh.

unset PATH
for p in $baseInputs $buildInputs; do
export PATH=$p/bin${PATH:+:}$PATH
done

function unpackPhase() {
tar -xzf $src
 for d in *; do
if [ -d "$d" ]; then
cd "$d"
break
fi
done
}

function configurePhase() {
./configure --prefix=$out
}

function buildPhase() {
make
}

function installPhase() {
make install
}

function fixupPhase() {
find $out -type f -exec patchelf --shrink-rpath '{}' \; -exec strip '{}' \
}

function genericBuild() {
unpackPhase
configurePhase
buildPhase
installPhase
fixupPhase
}

Finally, here is hello.nix.

let
pkgs = import <nixpkgs> {};
mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
name = "hello";
src = ./hello-2.10.tar.gz;
}

Now back to nix-shell:

$ nix-shell hello.nix
[nix-shell]$ source $setup
[nix-shell]$
Now you can run, for example, unpackPhase which unpacks $src and enters
the directory. And you can run commands like ./configure, make etc.
manually, or run phases with their respective functions.

It's all that straight, nix-shell builds the .drv file and its input dependencies,
then drops into a shell by setting up the environment variables necessary to
build the .drv, in particular those passed to the derivation function.

10.3. Conclusion
With nix-shell we're able to drop into an isolated environment for developing
a project, with the necessary dependencies just like nix-build does, except we
can build and debug the project manually, step by step like you would do in
any other operating system. Note that we did never install gcc, make, etc.
system-wide. These tools and libraries are available per-build.

10.4. Next pill

...we will clean up the nix store. We wrote and built derivations, added stuff
to nix store, but until now we never worried about cleaning up the used space
in the store. It's time to collect some garbage.
Chapter 11. Garbage Collector
Welcome to the 11th Nix pill. In the previous 10th pill we managed to obtain
a self-contained environment for developing a project. The concept is that
whereas nix-build is able to build a derivation in isolation, nix-shell is able
to drop us in a shell with (almost) the same environment used by nix-build.
This allows us to debug, modify and manually build software.

Today we stop packaging and look at a mandatory nix component, the

garbage collector. When using nix tools, often derivations are built. This
include both .drv files and out paths. These artifacts go in the nix store, and
we never cared about deleting them until now.

11.1. How does it work

Other package managers, like dpkg, have somehow a way to remove unused
software. However, nix is much more precise compared to other systems.

I bet with dpkg, rpm or anything else, you end up with having some
unnecessary package installed or dangling files. With nix this does not
happen.

How do we determine whether a store path is still needed? The same way
programming languages with a garbage collector decide whether an object is
still alive.

Programming languages with a garbage collector have an important concept

in order to keep track of live objects: GC roots. A GC root is an object that is
always alive (unless explicitly removed as GC root). All objects recursively
referred to by a GC root are live.

Therefore, the garbage collection process starts from GC roots, and

recursively mark referenced objects as live. All other objects can be collected
and deleted.
In Nix there's this same concept. Instead of being objects, of course, GC roots
are store paths. The implementation is very simple and transparent to the
user. GC roots are stored under /nix/var/nix/gcroots. If there's a symlink
to a store path, then that store path is a GC root.

Nix allows this directory to have subdirectories: it will simply recurse

directories in search of symlinks to store paths.

So we have a list of GC roots. At this point, deleting dead store paths is as

easy as you can imagine. We have the list of all live store paths, hence the
rest of the store paths are dead.

In particular, Nix first moves dead store paths to /nix/store/trash which is

an atomic operation. Afterwards, the trash is emptied.

11.2. Playing with the GC

Before playing with the GC, first run the nix garbage collector once, so that
we have a cleaned up playground for our experiments:
$ nix-collect-garbage
finding garbage collector roots...
[...]
deleting unused links...
note: currently hard linking saves -0.00 MiB
1169 store paths deleted, 228.43 MiB freed

Perfect, if you run it again it won't find anything new to delete, as expected.

What's left in the nix store is everything being referenced from the GC roots.

Let's install for a moment bsd-games:

$ nix-env -iA nixpkgs.bsdgames
$ readlink -f `which fortune`
/nix/store/b3lxx3d3ggxcggvjw5n0m1ya1gcrmbyn-bsd-games-2.17/bin/fortune
$ nix-store -q --roots `which fortune`
/nix/var/nix/profiles/default-9-link
$ nix-env --list-generations
[...]
9 2014-08-20 12:44:14 (current)
The nix-store command can be used to query the GC roots that refer to a
given derivation. In this case, our current user environment does refer to bsd-
games.

Now remove it, collect garbage and note that bsd-games is still in the nix
store:
$ nix-env -e bsd-games
uninstalling `bsd-games-2.17'
$ nix-collect-garbage
[...]
$ ls /nix/store/b3lxx3d3ggxcggvjw5n0m1ya1gcrmbyn-bsd-games-2.17
bin share

That's because the old generation is still in the nix store because it's a GC
root. As we'll see below, all profiles and their generations are GC roots.

Removing a GC root is simple. Let's try deleting the generation that refers to
bsd-games, collect garbage, and note that now bsd-games is no longer in the
nix store:
$ rm /nix/var/nix/profiles/default-9-link
$ nix-env --list-generations
[...]
8 2014-07-28 10:23:24
10 2014-08-20 12:47:16 (current)
$ nix-collect-garbage
[...]
$ ls /nix/store/b3lxx3d3ggxcggvjw5n0m1ya1gcrmbyn-bsd-games-2.17
ls: cannot access /nix/store/b3lxx3d3ggxcggvjw5n0m1ya1gcrmbyn-bsd-games-2.17

Note: nix-env --list-generations does not rely on any particular metadata. It

is able to list generations based solely on the file names under the profiles
directory.

However we removed the link from /nix/var/nix/profiles, not from

/nix/var/nix/gcroots. Turns out, that /nix/var/nix/gcroots/profiles is
a symlink to /nix/var/nix/profiles. That is very handy. It means any
profile and its generations are GC roots.

It's as simple as that, anything under /nix/var/nix/gcroots is a GC root.

And anything not being garbage collected is because it's referred from one of
the GC roots.

11.3. Indirect roots

I remind you that building the GNU hello world package with nix-build
produces a result symlink in the current directory. Despite the collected
garbage done above, the hello program is still working: therefore it has not
been garbage collected. Clearly, since there's no other derivation that depends
upon the GNU hello world package, it must be a GC root.

In fact, nix-build automatically adds the result symlink as a GC root. Yes,

not the built derivation, but the symlink. These GC roots are added under
/nix/var/nix/gcroots/auto.

$ ls -l /nix/var/nix/gcroots/auto/
total 8
drwxr-xr-x 2 nix nix 4096 Aug 20 10:24 ./
drwxr-xr-x 3 nix nix 4096 Jul 24 10:38 ../
lrwxrwxrwx 1 nix nix 16 Jul 31 10:51 xlgz5x2ppa0m72z5qfc78b8wlciwvgiz -> /

Don't care about the name of the symlink. What's important is that a symlink
exists that point to /home/nix/result. This is called an indirect GC root.
That is, the GC root is effectively specified outside of
/nix/var/nix/gcroots. Whatever result points to, it will not be garbage
collected.

How do we remove the derivation then? There are two possibilities:

Remove the indirect GC root from /nix/var/nix/gcroots/auto.

Remove the result symlink.

In the first case, the derivation will be deleted from the nix store, and result
becomes a dangling symlink. In the second case, the derivation is removed as
well as the indirect root in /nix/var/nix/gcroots/auto.

Running nix-collect-garbage after deleting the GC root or the indirect GC

root, will remove the derivation from the store.
11.4. Cleanup everything
What's the main source of software duplication in the nix store? Clearly, GC
roots due to nix-build and profile generations. Doing a nix-build results in a
GC root for a build that somehow will refer to a specific version of glibc, and
other libraries. After an upgrade, if that build is not deleted by the user, it will
not be garbage collected. Thus the old dependencies referred to by the build
will not be deleted either.

Same goes for profiles. Manipulating the nix-env profile will create further
generations. Old generations refer to old software, thus increasing duplication
in the nix store after an upgrade.

What are the basic steps for upgrading and removing everything old,
including old generations? In other words, do an upgrade similar to other
systems, where they forget everything about the older state:
$ nix-channel --update
$ nix-env -u --always
$ rm /nix/var/nix/gcroots/auto/*
$ nix-collect-garbage -d

First, we download a new version of the nixpkgs channel, which holds the
description of all the software. Then we upgrade our installed packages with
nix-env -u. That will bring us into a fresh new generation with all updated
software.

Then we remove all the indirect roots generated by nix-build: beware, this
will result in dangling symlinks. You may be smarter and also remove the
target of those symlinks.

Finally, the -d option of nix-collect-garbage is used to delete old generations

of all profiles, then collect garbage. After this, you lose the ability to rollback
to any previous generation. So make sure the new generation is working well
before running the command.

11.5. Conclusion
Garbage collection in Nix is a powerful mechanism to cleanup your system.
The nix-store commands allows us to know why a certain derivation is in the
nix store.

Cleaning up everything down to the oldest bit of software after an upgrade

seems a bit contrived, but that's the price of having multiple generations,
multiple profiles, multiple versions of software, thus rollbacks etc.. The price
of having many possibilities.

11.6. Next pill

...we will package another project and introduce what I call the "inputs"
design pattern. We only played with a single derivation until now, however
we'd like to start organizing a small repository of software. The "inputs"
pattern is widely used in nixpkgs; it allows us to decouple derivations from
the repository itself and increase customization opportunities.
Chapter 12. Inputs Design Pattern
Welcome to the 12th Nix pill. In the previous 11th pill we stopped packaging
and cleaned up the system with the garbage collector.

We restart our packaging, but we will improve a different aspect. We only

packaged an hello world program so far, what if we want to create a
repository of multiple packages?

12.1. Repositories in Nix

Nix is a tool for build and deployment, it does not enforce any particular
repository format. A repository of packages is the main usage for Nix, but not
the only possibility. See it more like a consequence due to the need of
organizing packages.

Nix is a language, and it is powerful enough to let you choose the format of
your own repository. In this sense, it is not declarative, but functional.

There is no preset directory structure or preset packaging policy. It's all about
you and Nix.

The nixpkgs repository has a certain structure, which evolved and evolves
with the time. Like other languages, Nix has its own history and therefore I'd
like to say that it also has its own design patterns. Especially when
packaging, you often do the same task again and again except for different
software. It's inevitable to identify patterns during this process. Some of these
patterns get reused if the community thinks it's a good way to package the
software.

Some of the patterns I'm going to show do not apply only to Nix, but to other
systems of course.

12.2. The single repository pattern

Before introducing the "inputs" pattern, we can start talking about another
pattern first which I'd like to call "single repository" pattern.

Systems like Debian scatter packages in several small repositories.

Personally, this makes it hard to track interdependent changes and to
contribute to new packages.

Systems like Gentoo instead, put package descriptions all in a single

repository.

The nix reference for packages is nixpkgs, a single repository of all

descriptions of all packages. I find this approach very natural and attractive
for new contributions.

From now on, we will adopt this technique. The natural implementation in
Nix is to create a top-level Nix expression, and one expression for each
package. The top-level expression imports and combines all expressions in a
giant attribute set with name -> package pairs.

But isn't that heavy? It isn't, because Nix is a lazy language, it evaluates only
what's needed! And that's why nixpkgs is able to maintain such a big
software repository in a giant attribute set.

12.3. Packaging graphviz

We have packaged GNU hello world, I guess you would like to package
something else for creating at least a repository of two projects :-) . I chose
graphviz, which uses the standard autotools build system, requires no
patching and dependencies are optional.

Download graphviz from here. The graphviz.nix expression is

straightforward:
let
pkgs = import <nixpkgs> {};
mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
name = "graphviz";
src = ./graphviz-2.38.0.tar.gz;
}

Build with nix-build graphviz.nix and you will get runnable binaries under
result/bin. Notice how we reused the same autotools.nix of hello.nix.
Let's create a simple png:
$ echo 'graph test { a -- b }'|result/bin/dot -Tpng -o test.png
Format: "png" not recognized. Use one of: canon cmap [...]

Oh of course... graphviz can't know about png. It built only the output
formats it supports natively, without using any extra library.

I remind you, in autotools.nix there's a buildInputs variable which gets

concatenated to baseInputs. That would be the perfect place to add a build
dependency. We created that variable exactly for this reason to be overridable
from package expressions.

This 2.38 version of graphviz has several plugins to output png. For
simplicity, we will use libgd.

12.4. Digression about gcc and ld wrappers

The gd, jpeg, fontconfig and bzip2 libraries (dependencies of gd) don't use
pkg-config to specify which flags to pass to the compiler. Since there's no
global location for libraries, we need to tell gcc and ld where to find includes
and libs.

The nixpkgs provides gcc and binutils, and we are using them for our
packaging. Not only, it also provides wrappers for them which allow passing
extra arguments to gcc and ld, bypassing the project build systems:

NIX_CFLAGS_COMPILE: extra flags to gcc at compile time

NIX_LDFLAGS: extra flags to ld

What can we do about it? We can employ the same trick we did for PATH:
automatically filling the variables from buildInputs. This is the relevant
snippet of setup.sh:
for p in $baseInputs $buildInputs; do
if [ -d $p/bin ]; then
export PATH="$p/bin${PATH:+:}$PATH"
fi
if [ -d $p/include ]; then
export NIX_CFLAGS_COMPILE="-I $p/include${NIX_CFLAGS_COMPILE:+ }$NIX_CFL
fi
if [ -d $p/lib ]; then
export NIX_LDFLAGS="-rpath $p/lib -L $p/lib${NIX_LDFLAGS:+ }$NIX_LDFLAGS
fi
done

Now by adding derivations to buildInputs, will add the lib, include and
bin paths automatically in setup.sh.

The [-rpath] flag in ld is needed because at runtime, the executable must use
exactly that version of the library.

If unneeded paths are specified, the fixup phase will shrink the rpath for us!

12.5. Completing graphviz with gd

Finish the expression for graphviz with gd support (note the use of the with
expression in buildInputs to avoid repeating pkgs):
let
pkgs = import <nixpkgs> {};
mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
name = "graphviz";
src = ./graphviz-2.38.0.tar.gz;
buildInputs = with pkgs; [ gd fontconfig libjpeg bzip2 ];
}

Now you can create the png! Ignore any error from fontconfig, especially if
you are in a chroot.

12.6. The repository expression

Now that we have two packages, what's a good way to put them together in a
single repository? We do something like nixpkgs does. With nixpkgs, we
import it and then we peek derivations by accessing the giant attribute set.

For us nixers, this a good technique, because it abstracts from the file names.
We don't refer to a package by REPO/some/sub/dir/package.nix but by
importedRepo.package (or pkgs.package in our examples).

Create a default.nix in the current directory:

{
hello = import ./hello.nix;
graphviz = import ./graphviz.nix;
}

Ready to use! Try it with nix repl:

$ nix repl
nix-repl> :l default.nix
Added 2 variables.
nix-repl> hello
«derivation /nix/store/dkib02g54fpdqgpskswgp6m7bd7mgx89-hello.drv»
nix-repl> graphviz
«derivation /nix/store/zqv520v9mk13is0w980c91z7q1vkhhil-graphviz.drv»

With nix-build:
$ nix-build default.nix -A hello
[...]
$ result/bin/hello
Hello, world!

The [-A] argument is used to access an attribute of the set from the given .nix
expression.

Important: why did we choose the default.nix? Because when a directory

(by default the current directory) has a default.nix, that default.nix will
be used (see import here). In fact you can run nix-build -A hello without
specifying default.nix.

For pythoners, it is similar to __init__.py.

With nix-env, to install the package in your user environment:

$ nix-env -f . -iA graphviz
[...]
$ dot -V

The [-f] option is used to specify the expression to use, in this case the
current directory, therefore ./default.nix.

The [-i] stands for installation.

The [-A] is the same as above for nix-build.

We reproduced the very basic behavior of nixpkgs.

12.7. The inputs pattern

After a long preparation, we finally arrived. I know you have a big doubt in
this moment. It's about the hello.nix and graphviz.nix. They are very, very
dependent on nixpkgs:

First big problem: they import nixpkgs directly. In autotools.nix

instead we pass nixpkgs as an argument. That's a much better approach.

Second problem: what if we want a variant of graphviz without libgd

support?

Third problem: what if we want to test graphviz with a particular libgd

version?

The current answer to the above questions is: change the expression to match
your needs (or change the callee to match your needs).

With the inputs pattern, we choose to give another answer: let the user
change the inputs of the expression (or change the caller to pass different
inputs).

By inputs of an expression, we refer to the set of derivations needed to build

that expression. In this case:

mkDerivation from autotools. Recall that mkDerivation has an implicit

 dependency on the toolchain.

libgd and its dependencies.

The src is also an input but it's pointless to change the source from the caller.
For version bumps, in nixpkgs we prefer to write another expression (e.g.
because patches are needed or different inputs are needed).

Goal: make package expressions independent of the repository.

How do we achieve that? The answer is simple: use functions to declare

inputs for a derivation. Doing it for graphviz.nix, will make the derivation
independent of the repository and customizable:
{ mkDerivation, gdSupport ? true, gd, fontconfig, libjpeg, bzip2 }:

mkDerivation {
name = "graphviz";
src = ./graphviz-2.38.0.tar.gz;
buildInputs = if gdSupport then [ gd fontconfig libjpeg bzip2 ] else [];
}

I recall that "{...}: ..." is the syntax for defining functions accepting an
attribute set as argument.

We made gd and its dependencies optional. If gdSupport is true (by default),

we will fill buildInputs and thus graphviz will be built with gd support,
otherwise it won't.

Now back to default.nix:

let
pkgs = import <nixpkgs> {};
mkDerivation = import ./autotools.nix pkgs;
in with pkgs; {
hello = import ./hello.nix { inherit mkDerivation; };
graphviz = import ./graphviz.nix { inherit mkDerivation gd fontconfig libj
graphvizCore = import ./graphviz.nix {
inherit mkDerivation gd fontconfig libjpeg bzip2;
gdSupport = false;
};
}
So we factorized the import of nixpkgs and mkDerivation, and also added a
variant of graphviz with gd support disabled. The result is that both
hello.nix (exercise for the reader) and graphviz.nix are independent of the
repository and customizable by passing specific inputs.

If you wanted to build graphviz with a specific version of gd, it would suffice
to pass gd = ...;.

If you wanted to change the toolchain, you may pass a different

mkDerivation function.

Clearing up the syntax:

In the end we return an attribute set from default.nix. With "let" we

define some local variables.

We bring pkgs into the scope when defining the packages set, which is
very convenient instead of typing everytime "pkgs".

We import hello.nix and graphviz.nix, which will return a function,

and call it with a set of inputs to get back the derivation.

The "inherit x" syntax is equivalent to "x = x". So "inherit gd"

here, combined to the above "with pkgs;" is equivalent to "gd =
pkgs.gd".

You can find the whole repository at the pill 12 gist.

12.8. Conclusion
The "inputs" pattern allows our expressions to be easily customizable
through a set of arguments. These arguments could be flags, derivations, or
whatelse. Our package expressions are functions, don't think there's any
magic in there.

It also makes the expressions independent of the repository. Given that all the
needed information is passed through arguments, it is possible to use that
expression in any other context.
12.9. Next pill
...we will talk about the "callPackage" design pattern. It is tedious to specify
the names of the inputs twice, once in the top-level default.nix, and once in
the package expression. With callPackage, we will implicitly pass the
necessary inputs from the top-level expression.
Chapter 13. Callpackage Design
Pattern
Welcome to the 13th Nix pill. In the previous 12th pill we have introduced
the first basic design pattern for organizing a repository of software. In
addition we packaged graphviz to have at least another package for our little
repository.

The next design pattern worth noting is what I'd like to call the callPackage
pattern. This technique is extensively used in nixpkgs, it's the current
standard for importing packages in a repository.

13.1. The callPackage convenience

In the previous pill, we underlined the fact that the inputs pattern is great to
decouple packages from the repository, in that we can pass manually the
inputs to the derivation. The derivation declares its inputs, and the caller
passes the arguments.

However as with usual programming languages, we declare parameter names,

and then we have to pass arguments. We do the job twice. With package
management, we often see common patterns. In the case of nixpkgs it's the
following.

Some package derivation:

{ input1, input2, ... }:
...

Repository derivation:
rec {
lib1 = import package1.nix { inherit input1 input2 ...; };
program2 = import package1.nix { inherit inputX inputY lib1 ...; };
}
Where inputs may even be packages in the repository itself (note the rec
keyword). The pattern here is clear, often inputs have the same name of the
attributes in the repository itself. Our desire is to pass those inputs from the
repository automatically, and in case be able to specify a particular argument
(that is, override the automatically passed default argument).

To achieve this, we will define a callPackage function with the following

synopsis:
{
lib1 = callPackage package1.nix { };
program2 = callPackage package2.nix { someoverride = overriddenDerivation;
}

What should it do?

Import the given expression, which in turn returns a function.

Determine the name of its arguments.

Pass default arguments from the repository set, and let us override those
arguments.

13.2. Implementing callPackage

First of all, we need a way to introspect (reflection or whatever) at runtime
the argument names of a function. That's because we want to automatically
pass such arguments.

Then callPackage requires access to the whole packages set, because it

needs to find the packages to pass automatically.

We start off simple with :

nix-repl> add = { a ? 3, b }: a+b
nix-repl> builtins.functionArgs add
{ a = true; b = false; }

Nix provides a builtin function to introspect the names of the arguments of a

function. In addition, for each argument, it tells whether the argument has a
default value or not. We don't really care about default values in our case. We
are only interested in the argument names.

Now we need a set with all the values, let's call it values. And a way to
intersect the attributes of values with the function arguments:
nix-repl> values = { a = 3; b = 5; c = 10; }
nix-repl> builtins.intersectAttrs values (builtins.functionArgs add)
{ a = true; b = false; }
nix-repl> builtins.intersectAttrs (builtins.functionArgs add) values
{ a = 3; b = 5; }

Perfect, note from the example above that the intersectAttrs returns a set
whose names are the intersection, and the attribute values are taken from the
second set.

We're done, we have a way to get argument names from a function, and
match with an existing set of attributes. This is our simple implementation of
callPackage:

nix-repl> callPackage = set: f: f (builtins.intersectAttrs (builtins.functio

nix-repl> callPackage values add
8
nix-repl> with values; add { inherit a b; }
8

Clearing up the syntax:

We define a callPackage variable which is a function.

The second parameter is the function to "autocall".

We take the argument names of the function and intersect with the set of
all values.

Finally we call the passed function f with the resulting intersection.

In the code above, I've also shown that the callPackage call is equivalent to
directly calling add a b.
We achieved what we wanted. Automatically call functions given a set of
possible arguments. If an argument is not found in the set, that's nothing
special. It's a function call with a missing parameter, and that's an error
(unless the function has varargs ... as explained in the 5th pill).

Or not. We missed something. Being able to override some of the parameters.

We may not want to always call functions with values taken from the big set.
Then we add a further parameter, which takes a set of overrides:
nix-repl> callPackage = set: f: overrides: f ((builtins.intersectAttrs (buil
nix-repl> callPackage values add { }
8
nix-repl> callPackage values add { b = 12; }
15

Apart from the increasing number of parenthesis, it should be clear that we

simply do a set union between the default arguments, and the overriding set.

13.3. Use callPackage to simplify the repository

Given our brand new tool, we can simplify the repository expression
(default.nix).

Let me write it down first:

let
nixpkgs = import <nixpkgs> {};
allPkgs = nixpkgs // pkgs;
callPackage = path: overrides:
let f = import path;
in f ((builtins.intersectAttrs (builtins.functionArgs f) allPkgs) // ove
pkgs = with nixpkgs; {
mkDerivation = import ./autotools.nix nixpkgs;
hello = callPackage ./hello.nix { };
graphviz = callPackage ./graphviz.nix { };
graphvizCore = callPackage ./graphviz.nix { gdSupport = false; };
};
in pkgs

Wow, there's a lot to say here:

 We renamed the old pkgs of the previous pill to nixpkgs. Our package
set is now instead named pkgs. Sorry for the confusion.

We needed a way to pass pkgs to callPackage somehow. Instead of

returning the set of packages directly from default.nix, we first assign
it to a let variable and reuse it in callPackage.

For convenience, in callPackage we first import the file, instead of

calling it directly. Otherwise for each package we would have to write
the import.

Since our expressions use packages from nixpkgs, in callPackage we

use allPkgs, which is the union of nixpkgs and our packages.

We moved mkDerivation into pkgs itself, so that it also gets passed

automatically.

Note how easy is to override arguments in the case of graphviz without gd.
But most importantly, how easy it was to merge two repositories: nixpkgs
and our pkgs!

The reader should notice a magic thing happening. We're defining pkgs in
terms of callPackage, and callPackage in terms of pkgs. That magic is
possible thanks to lazy evaluation.

13.4. Conclusion
The "callPackage" pattern has simplified a lot our repository. We're able to
import packages that require some named arguments and call them
automatically, given the set of all packages.

We've also introduced some useful builtin functions that allows us to

introspect Nix functions and manipulate attributes. These builtin functions
are not usually used when packaging software, rather to provide tools for
packaging. That's why they are not documented in the nix manual.

Writing a repository in nix is an evolution of writing convenient functions for

combining the packages. This demonstrates even more how nix is a generic
tool to build and deploy something, and how suitable it is to create software
repositories with your own conventions.

13.5. Next pill

...we will talk about the "override" design pattern. The graphvizCore seems
straightforward. It starts from graphviz.nix and builds it without gd. Now I
want to give you another point of view: what if we instead wanted to start
from pkgs.graphviz and disable gd?
Chapter 14. Override Design
Pattern
Welcome to the 14th Nix pill. In the previous 13th pill we have introduced
the callPackage pattern, used to simplify the composition of software in a
repository.

The next design pattern is less necessary but useful in many cases and it's a
good exercise to learn more about Nix.

14.1. About composability

Functional languages are known for being able to compose functions. In
particular, you gain a lot from functions that are able to manipulate the
original value into a new value having the same structure. So that in the end
we're able to call multiple functions to have the desired modifications.

In Nix we mostly talk about functions that accept inputs in order to return
derivations. In our world we want nice utility functions that are able to
manipulate those structures. These utilities add some useful properties to the
original value, and we must be able to apply more utilities on top of it.

For example let's say we have an initial derivation drv and we want it to be a
drv with debugging information and also to apply some custom patches:
debugVersion (applyPatches [ ./patch1.patch ./patch2.patch ] drv)

The final result will be still the original derivation plus some changes. That's
both interesting and very different from other packaging approaches, which is
a consequence of using a functional language to describe packages.

Designing such utilities is not trivial in a functional language that is not

statically typed, because understanding what can or cannot be composed is
difficult. But we try to do the best.
14.2. The override pattern
In the pill 12 we introduced the inputs design pattern. We do not return a
derivation picking dependencies directly from the repository, rather we
declare the inputs and let the callers pass the necessary arguments.

In our repository we have a set of attributes that import the expressions of the
packages and pass these arguments, getting back a derivation. Let's take for
example the graphviz attribute:
graphviz = import ./graphviz.nix { inherit mkDerivation gd fontconfig libjpe

If we wanted to produce a derivation of graphviz with a customized gd

version, we would have to repeat most of the above plus specifying an
alternative gd:
mygraphviz = import ./graphviz.nix {
inherit mkDerivation fontconfig libjpeg bzip2;
gd = customgd;
};

That's hard to maintain. Using callPackage it would be easier:

mygraphviz = callPackage ./graphviz.nix { gd = customgd; };

But we may still be diverging from the original graphviz in the repository.

We would like to avoid specifying the nix expression again, instead reuse the
original graphviz attribute in the repository and add our overrides like this:
mygraphviz = graphviz.override { gd = customgd; };

The difference is obvious, as well as the advantages of this approach.

Note: that .override is not a "method" in the OO sense as you may think.
Nix is a functional language. That .override is simply an attribute of a set.

14.3. The override implementation

I remind you, the graphviz attribute in the repository is the derivation
returned by the function imported from graphviz.nix. We would like to add
a further attribute named "override" to the returned set.

Let's start simple by first creating a function "makeOverridable" that takes a

function and a set of original arguments to be passed to the function.

Contract: the wrapped function must return a set.

Let's write a lib.nix:

{
makeOverridable = f: origArgs:
let
origRes = f origArgs;
in
origRes // { override = newArgs: f (origArgs // newArgs); };
}

So makeOverridable takes a function and a set of original arguments. It

returns the original returned set, plus a new override attribute.

This override attribute is a function taking a set of new arguments, and

returns the result of the original function called with the original arguments
unified with the new arguments. What a mess.

Let's try it with nix repl:

$ nix repl
nix-repl> :l lib.nix
Added 1 variables.
nix-repl> f = { a, b }: { result = a+b; }
nix-repl> f { a = 3; b = 5; }
{ result = 8; }
nix-repl> res = makeOverridable f { a = 3; b = 5; }
nix-repl> res
{ override = «lambda»; result = 8; }
nix-repl> res.override { a = 10; }
{ result = 15; }

Note that the function f does not return the plain sum but a set, because of the
contract. You didn't forget already, did you? :-)
The variable res is the result of the function call without any override. It's
easy to see in the definition of makeOverridable. In addition you can see the
new override attribute being a function.

Calling that .override with a set will invoke the original function with the
overrides, as expected.

But: we can't override again! Because the returned set with result 15 does not
have an override attribute!

That's bad, it breaks further compositions.

The solution is simple, the .override function should make the result
overridable again:
rec {
makeOverridable = f: origArgs:
let
origRes = f origArgs;
in
origRes // { override = newArgs: makeOverridable f (origArgs // newArg
}

Please note the rec keyword. It's necessary so that we can refer to
makeOverridable from makeOverridable itself.

Now let's try overriding twice:

nix-repl> :l lib.nix
Added 1 variables.
nix-repl> f = { a, b }: { result = a+b; }
nix-repl> res = makeOverridable f { a = 3; b = 5; }
nix-repl> res2 = res.override { a = 10; }
nix-repl> res2
{ override = «lambda»; result = 15; }
nix-repl> res2.override { b = 20; }
{ override = «lambda»; result = 30; }

Success! The result is 30, as expected because a is overridden to 10 in the

first override, and b to 20.

Now it would be nice if callPackage made our derivations overridable. That

was the goal of this pill after all. This is an exercise for the reader.

14.4. Conclusion
The "override" pattern simplifies the way we customize packages starting
from an existing set of packages. This opens a world of possibilities about
using a central repository like nixpkgs, and defining overrides on our local
machine without even modifying the original package.

Dream of a custom isolated nix-shell environment for testing graphviz with a

custom gd:
debugVersion (graphviz.override { gd = customgd; })

Once a new version of the overridden package comes out in the repository,
the customized package will make use of it automatically.

The key in Nix is to find powerful yet simple abstractions in order to let the
user customize his environment with highest consistency and lowest
maintenance time, by using predefined composable components.

14.5. Next pill

...we will talk about Nix search paths. By search path I mean a place in the
file system where Nix looks for expressions. You may have wondered, where
does that holy <nixpkgs> come from?
Chapter 15. Nix Search Paths
Welcome to the 15th Nix pill. In the previous 14th pill we have introduced
the "override" pattern, useful for writing variants of derivations by passing
different inputs.

Assuming you followed the previous posts, I hope you are now ready to
understand nixpkgs. But we have to find nixpkgs in our system first! So this
is the step: introducing some options and environment variables used by nix
tools.

15.1. The NIX_PATH

The NIX_PATH environment variable is very important. It's very similar to
the PATH environment variable. The syntax is similar, several paths are
separated by a colon :. Nix will then search for something in those paths
from left to right.

Who uses NIX_PATH? The nix expressions! Yes, NIX_PATH is not of much use
by the nix tools themselves, rather it's used when writing nix expressions.

In the shell for example, when you execute the command ping, it's being
searched in the PATH directories. The first one found is the one being used.

In nix it's exactly the same, however the syntax is different. Instead of just
typing ping you have to type <ping>. Yes, I know... you are already thinking
of <nixpkgs>. However don't stop reading here, let's keep going.

What's NIX_PATH good for? Nix expressions may refer to an "abstract" path
such as <nixpkgs>, and it's possible to override it from the command line.

For ease we will use nix-instantiate --eval to do our tests. I remind you, nix-
instantiate is used to evaluate nix expressions and generate the .drv files.
Here we are not interested in building derivations, so evaluation is enough. It
can be used for one-shot expressions.
15.2. Fake it a little
It's useless from a nix view point, but I think it's useful for your own
understanding. Let's use PATH itself as NIX_PATH, and try to locate ping (or
another binary if you don't have it).
$ nix-instantiate --eval -E '<ping>'
error: file `ping' was not found in the Nix search path (add it using $NIX_P
$ NIX_PATH=$PATH nix-instantiate --eval -E '<ping>'
/bin/ping
$ nix-instantiate -I /bin --eval -E '<ping>'
/bin/ping

Great. At first attempt nix obviously said could not be found anywhere in the
search path. Note that the [-I] option accepts a single directory. Paths added
with [-I] take precedence over NIX_PATH.

The NIX_PATH also accepts a different yet very handy syntax:

"somename=somepath". That is, instead of searching inside a directory for a
name, we specify exactly the value of that name.
$ NIX_PATH="ping=/bin/ping" nix-instantiate --eval -E '<ping>'
/bin/ping
$ NIX_PATH="ping=/bin/foo" nix-instantiate --eval -E '<ping>'
error: file `ping' was not found in the Nix search path (add it using $N

Note in the second case how Nix checks whether the path exists or not.

15.3. The path to repository

You are out of curiosity, right?
$ nix-instantiate --eval -E '<nixpkgs>'
/home/nix/.nix-defexpr/channels/nixpkgs
$ echo $NIX_PATH
nixpkgs=/home/nix/.nix-defexpr/channels/nixpkgs

You may have a different path, depending on how you added channels etc..
Anyway that's the whole point. The <nixpkgs> stranger that we used in our
nix expressions, is referring to a path in the filesystem specified by NIX_PATH.
You can list that directory and realize it's simply a checkout of the nixpkgs
repository at a specific commit (hint: .version-suffix).

The NIX_PATH variable is exported by nix.sh, and that's the reason why I
always asked you to source nix.sh at the beginning of my posts.

You may wonder: then I can also specify a different nixpkgs path to, e.g., a
git checkout of nixpkgs? Yes, you can and I encourage doing that. We'll talk
about this in the next pill.

Let's define a path for our repository, then! Let's say all the default.nix,
graphviz.nix etc. are under /home/nix/mypkgs:

$ export NIX_PATH=mypkgs=/home/nix/mypkgs:$NIX_PATH
$ nix-instantiate --eval '<mypkgs>'
{ graphviz = <code>; graphvizCore = <code>; hello = <code>; mkDerivation = <

Yes, nix-build also accepts paths with angular brackets. We first evaluate the
whole repository (default.nix) and then peek the graphviz attribute.

15.4. A big word about nix-env

The nix-env command is a little different than nix-instantiate and nix-build.
Whereas nix-instantiate and nix-build require a starting nix expression, nix-
env does not.

You may be crippled by this concept at the beginning, you may think nix-env
uses NIX_PATH to find the nixpkgs repository. But that's not it.

The nix-env command uses ~/.nix-defexpr, which is also part of NIX_PATH

by default, but that's only a coincidence. If you empty NIX_PATH, nix-env will
still be able to find derivations because of ~/.nix-defexpr.

So if you run nix-env -i graphviz inside your repository, it will install the
nixpkgs one. Same if you set NIX_PATH to point to your repository.

In order to specify an alternative to ~/.nix-defexpr it's possible to use the [-

f] option:
$ nix-env -f '<mypkgs>' -i graphviz
warning: there are multiple derivations named `graphviz'; using the first on
replacing old `graphviz'
installing `graphviz'

Oh why did it say there's another derivation named graphviz? Because both
graphviz and graphvizCore attributes in our repository have the name
"graphviz" for the derivation:
$ nix-env -f '<mypkgs>' -qaP
graphviz graphviz
graphvizCore graphviz
hello hello

By default nix-env parses all derivations and use the derivation names to
interpret the command line. So in this case "graphviz" matched two
derivations. Alternatively, like for nix-build, one can use [-A] to specify an
attribute name instead of a derivation name:
$ nix-env -f '<mypkgs>' -i -A graphviz
replacing old `graphviz'
installing `graphviz'

This form, other than being more precise, it's also faster because nix-env
does not have to parse all the derivations.

For completeness: you must install graphvizCore with [-A,] since without
the [-A] switch it's ambiguous.

In summary, it may happen when playing with nix that nix-env peeks a
different derivation than nix-build. In such case you probably specified
NIX_PATH, but nix-env is instead looking into ~/.nix-defexpr.

Why is nix-env having this different behavior? I don't know specifically by

myself either, but the answers could be:

nix-env tries to be generic, thus it does not look for nixpkgs in

NIX_PATH, rather it looks in ~/.nix-defexpr.

nix-env is able to merge multiple trees in ~/.nix-defexpr by looking at

all the possible derivations
It may also happen to you that you cannot match a derivation name when
installing, because of the derivation name vs [-A] switch described above.
Maybe nix-env wanted to be more friendly in this case for default user
setups.

It may or may not make sense for you, or it's like that for historical reasons,
but that's how it works currently, unless somebody comes up with a better
idea.

15.5. Conclusion
The NIX_PATH variable is the search path used by nix when using the angular
brackets syntax. It's possible to refer to "abstract" paths inside nix expressions
and define the "concrete" path by means of NIX_PATH, or the usual [-I] flag in
nix tools.

We've also explained some of the uncommon nix-env behaviors for

newcomers. The nix-env tool does not use NIX_PATH to search for packages,
but rather for ~/.nix-defexpr. Beware of that!

In general do not abuse NIX_PATH, when possible use relative paths when
writing your own nix expressions. Of course, in the case of <nixpkgs> in our
repository, that's a perfectly fine usage of NIX_PATH. Instead, inside our
repository itself, refer to expressions with relative paths like ./hello.nix.

15.6. Next pill

...we will finally dive into nixpkgs. Most of the techniques we have
developed in this series are already in nixpkgs, like mkDerivation,
callPackage, override, etc., but of course better. With time, those base
utilities get enhanced by the community with more features in order to handle
more and more use cases and in a more general way.
Chapter 16. Nixpkgs Parameters
Welcome to the 16th Nix pill. In the previous 15th pill we've realized how
nix finds expressions with the angular brackets syntax, so that we finally
know where is <nixpkgs> located on our system.

We can start diving into the nixpkgs repository, through all the various tools
and design patterns. Please note that also nixpkgs has its own manual,
underlying the difference between the general nix language and the nixpkgs
repository.

16.1. The default.nix expression

We will not start inspecting packages at the beginning, rather the general
structure of nixpkgs.

In our custom repository we created a default.nix which composed the

expressions of the various packages.

Also nixpkgs has its own default.nix, which is the one being loaded when
referring to <nixpkgs>. It does a simple thing: check whether the nix version
is at least 1.7 (at the time of writing this blog post). Then import pkgs/top-
level/all-packages.nix. From now on, we will refer to this set of packages as
pkgs.

The all-packages.nix is then the file that composes all the packages. Note
the pkgs/ subdirectory, while nixos is in the nixos/ subdirectory.

The all-packages.nix is a bit contrived. First of all, it's a function. It

accepts a couple of interesting parameters:

system: defaults to the current system

config: defaults to null

 others...

The system parameter, as per comment in the expression, it's the system for
which the packages will be built. It allows for example to install i686
packages on amd64 machines.

The config parameter is a simple attribute set. Packages can read some of its
values and change the behavior of some derivations.

16.2. The system parameter

You will find this parameter in many other .nix expressions (e.g. release
expressions). The reason is that, given pkgs accepts a system parameter, then
whenever you want to import pkgs you also want to pass through the value of
system. E.g.:

myrelease.nix:

{ system ? builtins.currentSystem }:

let pkgs = import <nixpkgs> { inherit system; };

...

Why is it useful? With this parameter it's very easy to select a set of packages
for a particular system. For example:
nix-build -A psmisc --argstr system i686-linux

This will build the psmisc derivation for i686-linux instead of x86_64-linux.
This concept is very similar to multi-arch of Debian.

The setup for cross compiling is also in nixpkgs, however it's a little
contrived to talk about it and I don't know much of it either.

16.3. The config parameter

I'm sure on the wiki or other manuals you've read about
~/.nixpkgs/config.nix and I'm sure you've wondered whether that's
hardcoded in nix. It's not, it's in nixpkgs.

The all-packages.nix expression accepts the config parameter. If it's null,

then it reads the NIXPKGS_CONFIG environment variable. If not specified,
nixpkgs will peek $HOME/.nixpkgs/config.nix.

After determining config.nix, it will be imported as nix expression, and that

will be the value of config (in case it hasn't been passed as parameter to
import <nixpkgs>).

The config is available in the resulting repository:

$ nix repl
nix-repl> pkgs = import <nixpkgs> {}
nix-repl> pkgs.config
{ }
nix-repl> pkgs = import <nixpkgs> { config = { foo = "bar"; }; }
nix-repl> pkgs.config
{ foo = "bar"; }

What attributes go in config is a matter of convenience and conventions.

For example, config.allowUnfree is an attribute that forbids building

packages that have an unfree license by default. The config.pulseaudio
setting tells whether to build packages with pulseaudio support or not where
applicable and when the derivation obeys to the setting.

16.4. About .nix functions

A .nix file contains a nix expression. Thus it can also be a function. I remind
you that nix-build expects the expression to return a derivation. Therefore it's
natural to return straight a derivation from a .nix file. However, it's also very
natural for the .nix file to accept some parameters, in order to tweak the
derivation being returned.

In this case, nix does a trick:

If the expression is a derivation, well build it.

 If the expression is a function, call it and build the resulting derivation.

For example you can nix-build the .nix file below:

{ pkgs ? import <nixpkgs> {} }:

pkgs.psmisc

Nix is able to call the function because the pkgs parameter has a default
value. This allows you to pass a different value for pkgs using the --arg
option.

Does it work if you have a function returning a function that returns a

derivation? No, Nix only calls the function it encounters once.

16.5. Conclusion
We've unleashed the <nixpkgs> repository. It's a function that accepts some
parameters, and returns the set of all packages. Due to laziness, only the
accessed derivations will be built.

You can use this repository to build your own packages as we've seen in the
previous pill when creating our own repository.

Lately I'm a little busy with the NixOS 14.11 release and other stuff, and I'm
also looking toward migrating from blogger to a more coder-oriented
blogging platform. So sorry for the delayed and shorter pills :)

16.6. Next pill

...we will talk about overriding packages in the nixpkgs repository. What if
you want to change some options of a library and let all other packages pick
the new library? One possibility is to use, like described above, the config
parameter when applicable. The other possibility is to override derivations.
Chapter 17. Nixpkgs Overriding
Packages
Welcome to the 17th Nix pill. In the previous 16th pill we have started to
dive into the nixpkgs repository. Nixpkgs is a function, and we've looked at
some parameters like system and config.

Today we'll talk about a special attribute: config.packageOverrides.

Overriding packages in a set with fixed point can be considered another
design pattern in nixpkgs.

17.1. Overriding a package

Recall the override design pattern from the nix pill 14. Instead of calling a
function with parameters directly, we make the call (function + parameters)
overridable.

We put the override function in the returned attribute set of the original
function call.

Take for example graphviz. It has an input parameter xlibs. If it's null, then
graphviz will build without X support.
$ nix-repl
nix-repl> :l <nixpkgs>
Added 4360 variables.
nix-repl> :b graphviz.override { xlibs = null; }

This will build graphviz without X support, it's as simple as that.

However let's say a package P depends on graphviz, how do we make P

depend on the new graphviz without X support?

17.2. In an imperative world...

...you could do something like this:
pkgs = import <nixpkgs> {};
pkgs.graphviz = pkgs.graphviz.override { xlibs = null; };
build(pkgs.P)

Given pkgs.P depends on pkgs.graphviz, it's easy to build P with the

replaced graphviz. On a pure functional language it's not that easy because
you can assign to variables only once.

17.3. Fixed point

The fixed point with lazy evaluation is crippling but about necessary in a
language like Nix. It lets us achieve something similar to what we'd do
imperatively.

Follows the definition of fixed point in nixpkgs:

# Take a function and evaluate it with its own returned value.
fix = f: let result = f result; in result;

It's a function that accepts a function f, calls f result on the result just
returned by f result and returns it. In other words it's f(f(f(....

At first sight, it's an infinite loop. With lazy evaluation it isn't, because the
call is done only when needed.
nix-repl> fix = f: let result = f result; in result
nix-repl> pkgs = self: { a = 3; b = 4; c = self.a+self.b; }
nix-repl> fix pkgs
{ a = 3; b = 4; c = 7; }

Without the rec keyword, we were able to refer to a and b of the same set.

First pkgs gets called with an unevaluated thunk (pkgs(pkgs(...)

To set the value of c then self.a and self.b are evaluated.

The pkgs function gets called again to get the value of a and b.
The trick is that c is not needed to be evaluated in the inner call, thus it
doesn't go in an infinite loop.

Won't go further with the explanation here. A good post about fixed point and
Nix can be found here.

17.3.1. Overriding a set with fixed point

Given that self.a and self.b refer to the passed set and not to the literal set
in the function, we're able to override both a and b and get a new value for c:
nix-repl> overrides = { a = 1; b = 2; }
nix-repl> let newpkgs = pkgs (newpkgs // overrides); in newpkgs
{ a = 3; b = 4; c = 3; }
nix-repl> let newpkgs = pkgs (newpkgs // overrides); in newpkgs // overrides
{ a = 1; b = 2; c = 3; }

In the first case we computed pkgs with the overrides, in the second case we
also included the overriden attributes in the result.

17.4. Overriding nixpkgs packages

We've seen how to override attributes in a set such that they get recursively
picked by dependant attributes. This approach can be used for derivations
too, after all nixpkgs is a giant set of attributes that depend on each other.

To do this, nixpkgs offers config.packageOverrides. So nixpkgs returns a

fixed point of the package set, and packageOverrides is used to inject the
overrides.

Create a config.nix file like this somewhere:

{
packageOverrides = pkgs: {
graphviz = pkgs.graphviz.override { xlibs = null; };
};
}

Now we can build e.g. asciidocFull and it will automatically use the
overridden graphviz:
nix-repl> pkgs = import <nixpkgs> { config = import ./config.nix; }
nix-repl> :b pkgs.asciidocFull

Note how we pass the config with packageOverrides when importing

nixpkgs. Then pkgs.asciidocFull is a derivation that has graphviz input
(pkgs.asciidoc is the lighter version and doesn't use graphviz at all).

Since there's no version of asciidoc with graphviz without X support in the

binary cache, Nix will recompile the needed stuff for you.

17.5. The ~/.nixpkgs/config.nix file

In the previous pill we already talked about this file. The above config.nix
that we just wrote could be the content of ~/.nixpkgs/config.nix.

Instead of passing it explicitly whenever we import nixpkgs, it will be

automatically imported by nixpkgs.

17.6. Conclusion
We've learned about a new design pattern: using fixed point for overriding
packages in a package set.

Whereas in an imperative setting, like with other package managers, a library

is installed replacing the old version and applications will use it, in Nix it's
not that straight and simple. But it's more precise.

Nix applications will depend on specific versions of libraries, hence the

reason why we have to recompile asciidoc to use the new graphviz library.

The newly built asciidoc will depend on the new graphviz, and old asciidoc
will keep using the old graphviz undisturbed.

17.7. Next pill

...we will stop diving nixpkgs for a moment and talk about store paths. How
does Nix compute the path in the store where to place the result of builds?
How to add files to the store for which we have an integrity hash?
Chapter 18. Nix Store Paths
Welcome to the 18th Nix pill. In the previous 17th pill we have scratched the
surface of the nixpkgs repository structure. It is a set of packages, and it's
possible to override such packages so that all other packages will use the
overrides.

Before reading existing derivations, I'd like to talk about store paths and how
they are computed. In particular we are interested in fixed store paths that
depend on an integrity hash (e.g. a sha256), which is usually applied to
source tarballs.

The way store paths are computed is a little contrived, mostly due to
historical reasons. Our reference will be the Nix source code.

18.1. Source paths

Let's start simple. You know nix allows relative paths to be used, such that
the file or directory is stored in the nix store, that is ./myfile gets stored into
/nix/store/....... We want to understand how is the store path generated
for such a file:
$ echo mycontent > myfile

I remind you, the simplest derivation you can write has a name, a builder and
the system:
$ nix repl
nix-repl> derivation { system = "x86_64-linux"; builder = ./myfile; name = "
«derivation /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv»

Now inspect the .drv to see where is ./myfile being stored:

$ nix show-derivation /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-
{
"/nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv": {
"outputs": {
 "out": {
"path": "/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo"
}
},
"inputSrcs": [
"/nix/store/xv2iccirbrvklck36f1g7vldn5v58vck-myfile"
],
"inputDrvs": {},
"platform": "x86_64-linux",
"builder": "/nix/store/xv2iccirbrvklck36f1g7vldn5v58vck-myfile",
"args": [],
"env": {
"builder": "/nix/store/xv2iccirbrvklck36f1g7vldn5v58vck-myfile",
"name": "foo",
"out": "/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo",
"system": "x86_64-linux"
}
}
}

Great, how did nix decide to use xv2iccirbrvklck36f1g7vldn5v58vck ?

Keep looking at the nix comments.

Note: doing nix-store --add myfile will store the file in the same store path.

18.1.1. Step 1, compute the hash of the file

The comments tell us to first compute the sha256 of the NAR serialization of
the file. Can be done in two ways:
$ nix-hash --type sha256 myfile
2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3

Or:
$ nix-store --dump myfile|sha256sum
2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3

In general, Nix understands two contents: flat for regular files, or recursive
for NAR serializations which can be anything.

18.1.2. Step 2, build the string description

Then nix uses a special string which includes the hash, the path type and the
file name. We store this in another file:
$ echo -n "source:sha256:2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0

18.1.3. Step 3, compute the final hash

Finally the comments tell us to compute the base-32 representation of the
first 160 bits (truncation) of a sha256 of the above string:
$ nix-hash --type sha256 --truncate --base32 --flat myfile.str
xv2iccirbrvklck36f1g7vldn5v58vck

18.2. Output paths

Output paths are usually generated for derivations. We use the above
example because it's simple. Even if we didn't build the derivation, nix knows
the out path hs0yi5n5nw6micqhy8l1igkbhqdkzqa1. This is because the out
path only depends on inputs.

It's computed in a similar way to source paths, except that the .drv is hashed
and the type of derivation is output:out. In case of multiple outputs, we may
have different output:<id>.

At the time nix computes the out path, the .drv contains an empty string for
each out path. So what we do is getting our .drv and replacing the out path
with an empty string:
$ cp -f /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv myout.drv
$ sed -i 's,/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo,,g' myout.drv

The myout.drv is the .drv state in which nix is when computing the out path
for our derivation:
$ sha256sum myout.drv
1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5 myout.drv
$ echo -n "output:out:sha256:1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d94
$ nix-hash --type sha256 --truncate --base32 --flat myout.str
hs0yi5n5nw6micqhy8l1igkbhqdkzqa1
Then nix puts that out path in the .drv, and that's it.

In case the .drv has input derivations, that is it references other .drv, then
such .drv paths are replaced by this same algorithm which returns an hash.

In other words, you get a final .drv where every other .drv path is replaced by
its hash.

18.3. Fixed-output paths

Finally, the other most used kind of path is when we know beforehand an
integrity hash of a file. This is usual for tarballs.

A derivation can take three special attributes: outputHashMode, outputHash

and outputHashAlgo which are well documented in the nix manual.

The builder must create the out path and make sure its hash is the same as the
one declared with outputHash.

Let's say our builder should create a file whose contents is mycontent:
$ echo mycontent > myfile
$ sha256sum myfile
f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb myfile
nix-repl> derivation { name = "bar"; system = "x86_64-linux"; builder = "non
«derivation /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv»

Inspect the .drv and see that it also stored the fact that it's a fixed-output
derivation with sha256 algorithm, compared to the previous examples:
$ nix show-derivation /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-
{
"/nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv": {
"outputs": {
"out": {
"path": "/nix/store/a00d5f71k0vp5a6klkls0mvr1f7sx6ch-bar",
"hashAlgo": "sha256",
"hash": "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2c
}
},
[...]
}

It doesn't matter which input derivations are being used, the final out path
must only depend on the declared hash.

What nix does is to create an intermediate string representation of the fixed-

output content:
$ echo -n "fixed:out:sha256:f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500d
$ sha256sum mycontent.str
423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639 myfile.str

Then proceed as it was a normal derivation output path:

$ echo -n "output:out:sha256:423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5
$ nix-hash --type sha256 --truncate --base32 --flat myfile.str
a00d5f71k0vp5a6klkls0mvr1f7sx6ch

Hence, the store path only depends on the declared fixed-output hash.

18.4. Conclusion
There are other types of store paths, but you get the idea. Nix first hashes the
contents, then creates a string description, and the final store path is the hash
of this string.

Also we've introduced some fundamentals, in particular the fact that Nix
knows beforehand the out path of a derivation since it only depends on the
inputs. We've also introduced fixed-output derivations which are especially
used by the nixpkgs repository for downloading and verifying source tarballs.

18.5. Next pill

...we will introduce stdenv. In the previous pills we rolled our own
mkDerivation convenience function for wrapping the builtin derivation, but
the nixpkgs repository also has its own convenience functions for dealing
with autotools projects and other build systems.
Chapter 19. Fundamentals of
Stdenv
Welcome to the 19th Nix pill. In the previous 18th pill we did dive into the
algorithm used by Nix to compute the store paths, and also introduced fixed-
output store paths.

This time we will instead look into nixpkgs, in particular one of its core
derivation: stdenv.

The stdenv is not a special derivation to Nix, but it's very important for the
nixpkgs repository. It serves as base for packaging software. It is used to pull
in dependencies such as the GCC toolchain, GNU make, core utilities, patch
and diff utilities, and so on. Basic tools needed to compile a huge pile of
software currently present in nixpkgs.

19.1. What is stdenv

First of all stdenv is a derivation. And it's a very simple one:
$ nix-build '<nixpkgs>' -A stdenv
/nix/store/k4jklkcag4zq4xkqhkpy156mgfm34ipn-stdenv
$ ls -R result/
result/:
nix-support/ setup

result/nix-support:
propagated-user-env-packages

It has just two files: /setup and /nix-support/propagated-user-env-

packages. Don't care about the latter; it's empty, in fact. The important file is
/setup.

How can this simple derivation pull in all the toolchain and basic tools
needed to compile packages? Let's look at the runtime dependencies:
$ nix-store -q --references result
/nix/store/3a45nb37s0ndljp68228snsqr3qsyp96-bzip2-1.0.6
/nix/store/a457ywa1haa0sgr9g7a1pgldrg3s798d-coreutils-8.24
/nix/store/zmd4jk4db5lgxb8l93mhkvr3x92g2sx2-bash-4.3-p39
/nix/store/47sfpm2qclpqvrzijizimk4md1739b1b-gcc-wrapper-4.9.3
...

How can it be? The package must be referring to those package somehow. In
fact, they are hardcoded in the /setup file:
$ head result/setup
export SHELL=/nix/store/zmd4jk4db5lgxb8l93mhkvr3x92g2sx2-bash-4.3-p39/bin/ba
initialPath="/nix/store/a457ywa1haa0sgr9g7a1pgldrg3s798d-coreutils-8.24 ..."
defaultNativeBuildInputs="/nix/store/sgwq15xg00xnm435gjicspm048rqg9y6-patche

19.2. The setup file

Remember our generic builder.sh in Pill 8? It sets up a basic PATH, unpacks
the source and runs the usual autotools commands for us.

The stdenv setup file is exactly that. It sets up several environment variables
like PATH and creates some helper bash functions to build a package. I invite
you to read it, it's only 860 lines at the time of this writing.

The hardcoded toolchain and utilities are used to initially fill up the
environment variables so that it's more pleasant to run common commands,
similar to what we did with our builder with baseInputs and buildInputs.

The build with stdenv works in phases. Phases are like unpackPhase,
configurePhase, buildPhase, checkPhase, installPhase, fixupPhase. You
can see the default list in the genericBuild function.

What genericBuild does is just run these phases. Default phases are just
bash functions, you can easily read them.

Every phase has hooks to run commands before and after the phase has been
executed. Phases can be overwritten, reordered, whatever, it's just bash code.

How to use this file? Like our old builder. To test it, we enter a fake empty
derivation, source the stdenv setup, unpack the hello sources and build it:
$ nix-shell -E 'derivation { name = "fake"; builder = "fake"; system = "x86_
nix-shell$ unset PATH
nix-shell$ source /nix/store/k4jklkcag4zq4xkqhkpy156mgfm34ipn-stdenv/setup
nix-shell$ tar -xf hello-2.10.tar.gz
nix-shell$ cd hello-2.10
nix-shell$ configurePhase
...
nix-shell$ buildPhase
...

I unset PATH to further show that the stdenv is enough self-contained to build
autotools packages that have no other dependencies.

So we ran the configurePhase function and buildPhase function and they

worked. These bash functions should be self-explanatory, you can read the
code in the setup file.

19.3. How is the setup file built

Until now we worked with plain bash scripts. What about the Nix side? The
nixpkgs repository offers a useful function, like we did with our old builder.
It is a wrapper around the raw derivation function which pulls in the stdenv
for us, and runs genericBuild. It's stdenv.mkDerivation.

Note how stdenv is a derivation but it's also an attribute set which contains
some other attributes, like mkDerivation. Nothing fancy here, just
convenience.

Let's write a hello.nix expression using this new discovered stdenv:

with import <nixpkgs> {};
stdenv.mkDerivation {
name = "hello";
src = ./hello-2.10.tar.gz;
}

Don't be scared by the with expression. It pulls the nixpkgs repository into
scope, so we can directly use stdenv. It looks very similar to the hello
expression in Pill 8.
It builds, and runs fine:
$ nix-build hello.nix
...
/nix/store/6y0mzdarm5qxfafvn2zm9nr01d1j0a72-hello
$ result/bin/hello
Hello, world!

19.4. The stdenv.mkDerivation builder

Let's take a look at the builder used by mkDerivation. You can read the code
here in nixpkgs:
{
...
builder = attrs.realBuilder or shell;
args = attrs.args or ["-e" (attrs.builder or ./default-builder.sh)];
stdenv = result;
...
}

Also take a look at our old derivation wrapper in previous pills! The builder
is bash (that shell variable), the argument to the builder (bash) is default-
builder.sh, and then we add the environment variable $stdenv in the
derivation which is the stdenv derivation.

You can open default-builder.sh and see what it does:

source $stdenv/setup
genericBuild

It's what we did in Pill 10 to make the derivations nix-shell friendly. When
entering the shell, the setup file only sets up the environment without
building anything. When doing nix-build, it actually runs the build process.

To get a clear understanding of the environment variables, look at the .drv of

the hello derivation:
$ nix show-derivation $(nix-instantiate hello.nix)
warning: you did not specify '--add-root'; the result might be removed by th
{
"/nix/store/abwj50lycl0m515yblnrvwyydlhhqvj2-hello.drv": {
 "outputs": {
"out": {
"path": "/nix/store/6y0mzdarm5qxfafvn2zm9nr01d1j0a72-hello"
}
},
"inputSrcs": [
"/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh",
"/nix/store/svc70mmzrlgq42m9acs0prsmci7ksh6h-hello-2.10.tar.gz"
],
"inputDrvs": {
"/nix/store/hcgwbx42mcxr7ksnv0i1fg7kw6jvxshb-bash-4.4-p19.drv": [
"out"
],
"/nix/store/sfxh3ybqh97cgl4s59nrpi78kgcc8f3d-stdenv-linux.drv": [
"out"
]
},
"platform": "x86_64-linux",
"builder": "/nix/store/q1g0rl8zfmz7r371fp5p42p4acmv297d-bash-4.4-p19/bin
"args": [
"-e",
"/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
],
"env": {
"buildInputs": "",
"builder": "/nix/store/q1g0rl8zfmz7r371fp5p42p4acmv297d-bash-4.4-p19/b
"configureFlags": "",
"depsBuildBuild": "",
"depsBuildBuildPropagated": "",
"depsBuildTarget": "",
"depsBuildTargetPropagated": "",
"depsHostBuild": "",
"depsHostBuildPropagated": "",
"depsTargetTarget": "",
"depsTargetTargetPropagated": "",
"name": "hello",
"nativeBuildInputs": "",
"out": "/nix/store/6y0mzdarm5qxfafvn2zm9nr01d1j0a72-hello",
"propagatedBuildInputs": "",
"propagatedNativeBuildInputs": "",
"src": "/nix/store/svc70mmzrlgq42m9acs0prsmci7ksh6h-hello-2.10.tar.gz"
"stdenv": "/nix/store/6kz2vbh98s2r1pfshidkzhiy2s2qdw0a-stdenv-linux",
"system": "x86_64-linux"
}
}
}
So short I decided to paste it entirely above. The builder is bash, with -e
default-builder.sh arguments. Then you can see the src and stdenv
environment variables.

Last bit, the unpackPhase in the setup is used to unpack the sources and enter
the directory, again like we did in our old builder.

19.5. Conclusion
The stdenv is the core of the nixpkgs repository. All packages use the
stdenv.mkDerivation wrapper instead of the raw derivation. It does a bunch
of operations for us and also sets up a pleasant build environment.

The overall process is simple:

nix-build

bash -e default-builder.sh

source $stdenv/setup

genericBuild

That's it, everything you need to know about the stdenv phases is in the setup
file.

Really, take your time to read that file. Don't forget that juicy docs are also
available in the nixpkgs manual.

19.6. Next pill...

...we will talk about how to add dependencies to our packages with
buildInputs and propagatedBuildInputs, and influence downstream builds
with setup hooks and env hooks. These concepts are crucial to how nixpkgs
packages are composed.
Chapter 20. Basic Dependencies and
Hooks
Welcome to the 20th Nix pill. In the previous 19th pill we introduced
Nixpkgs' stdenv, including setup.sh script, default-builder.sh helper
script, and stdenv.mkDerivation builder. We focused on how stdenv is put
together, and how it's used, and a bit about the phases of genericBuild.

This time, we'll focus on the interaction of packages built with

stdenv.mkDerivation. Packages need to depend on each other, of course.
For this we have buildInputs and propagatedBuildInputs attributes. We've
also found that dependencies sometimes need to influence their dependents in
ways the dependents can't or shouldn't predict. For this we have setup hooks
and env hooks. Together, these 4 concepts support almost all build-time
package interactions.

Note

The complexity of the dependencies and hooks infrastructure has increased,

over time, to support cross compilation. Once you learn the core concepts,
you will be able to understand the extra complexity. As a starting point, you
might want to refer to nixpkgs commit 6675f0a5, the last version of stdenv
without cross-compilation complexity.

20.1. The buildInputs Attribute

For the simplest dependencies where the current package directly needs
another, we use the buildInputs attribute. This is exactly the pattern in
taught with our builder in Pill 8. To demo this, lets build GNU Hello, and
then another package which provides a shell script that execs it.
let

nixpkgs = import <nixpkgs> {};

 inherit (nixpkgs) stdenv fetchurl which;

actualHello = stdenv.mkDerivation {
name = "hello-2.3";

src = fetchurl {
url = mirror://gnu/hello/hello-2.3.tar.bz2;
sha256 = "0c7vijq8y68bpr7g6dh1gny0bff8qq81vnp4ch8pjzvg56wb3js1";
};
};

wrappedHello = stdenv.mkDerivation {
name = "hello-wrapper";

buildInputs = [ actualHello which ];

unpackPhase = "true";

installPhase = ''
mkdir -p "$out/bin"
echo "#! ${stdenv.shell}" >> "$out/bin/hello"
echo "exec $(which hello)" >> "$out/bin/hello"
'';
};

in wrappedHello

Notice that the wrappedHello derivation finds the hello binary from the PATH.
This works because stdenv contains something like:
pkgs=""
for i in $buildInputs; do
findInputs $i
done

where findInputs is defined like:

findInputs() {
local pkg=$1

## Don't need to repeat already processed package

case $pkgs in
*\ $pkg\ *)
return 0
;;
 esac

pkgs="$pkgs $pkg "

## More goes here in reality that we can ignore for now.

}

then after this is run:

for i in $pkgs; do
addToEnv $i
done

where addToEnv is defined like:

addToEnv() {
local pkg=$1

if test -d $1/bin; then

addToSearchPath _PATH $1/bin
fi

## More goes here in reality that we can ignore for now.

}

The addToSearchPath call adds $1/bin to _PATH if the former exists (code
here). Once all the packages in buildInputs have been processed, then
content of _PATH is added to PATH, as follows:
PATH="${_PATH-}${_PATH:+${PATH:+:}}$PATH"

With the real hello on the PATH, the installPhase should hopefully make
sense.

20.2. The propagatedBuildInputs Attribute

The buildInputs covers direct dependencies, but what about indirect
dependencies where one package needs a second package which needs a
third? Nix itself handles this just fine, understanding various dependency
closures as covered in previous builds. But what about the conveniences that
buildInputs provides, namely accumulating in pkgs environment variable
and inclusion of pkg/bin directories on the PATH? For this, stdenv provides
the propagatedBuildInputs:
let

nixpkgs = import <nixpkgs> {};

inherit (nixpkgs) stdenv fetchurl which;

actualHello = stdenv.mkDerivation {
name = "hello-2.3";

src = fetchurl {
url = mirror://gnu/hello/hello-2.3.tar.bz2;
sha256 = "0c7vijq8y68bpr7g6dh1gny0bff8qq81vnp4ch8pjzvg56wb3js1";
};
};

intermediary = stdenv.mkDerivation {
name = "middle-man";

propagatedBuildInputs = [ actualHello ];

unpackPhase = "true";

installPhase = ''
mkdir -p "$out"
'';
};

wrappedHello = stdenv.mkDerivation {
name = "hello-wrapper";

buildInputs = [ intermediary which ];

unpackPhase = "true";

installPhase = ''
mkdir -p "$out/bin"
echo "#! ${stdenv.shell}" >> "$out/bin/hello"
echo "exec $(which hello)" >> "$out/bin/hello"
'';
};

in wrappedHello

See how the intermediate package has a propagatedBuildInputs

dependency, but the wrapper only needs a buildInputs dependency on the
intermediary.

How does this work? You might think we do something in Nix, but actually
its done not at eval time but at build time in bash. lets look at part of the
fixupPhase of stdenv:

fixupPhase() {

## Elided

if test -n "$propagatedBuildInputs"; then

ensureDir "$out/nix-support"
echo "$propagatedBuildInputs" > "$out/nix-support/propagated-build-i
fi

## Elided

This dumps the propagated build inputs in a so-named file in $out/nix-

support/. Then, back in findInputs look at the lines at the bottom we elided
before:
findInputs() {
local pkg=$1

## More goes here in reality that we can ignore for now.

if test -f $pkg/nix-support/propagated-build-inputs; then

for i in $(cat $pkg/nix-support/propagated-build-inputs); do
findInputs $i
done
fi
}

See how findInputs is actually recursive, looking at the propagated build

inputs of each dependency, and those dependencies' propagated build inputs,
etc.

We actually simplified the findInputs call site from before;

propagatedBuildInputs is also looped over in reality:
pkgs=""
for i in $buildInputs $propagatedBuildInputs; do
findInputs $i
done

This demonstrates an important point. For the current package alone, it

doesn't matter whether a dependency is propagated or not. It will be
processed the same way: called with findInputs and addToEnv. (The
packages discovered by findInputs, which are also accumulated in pkgs and
passed to addToEnv, are also the same in both cases.) Downstream however,
it certainly does matter because only the propagated immediate dependencies
are put in the $out/nix-support/propagated-build-inputs.

20.3. Setup Hooks

As we mentioned above, sometimes dependencies need to influence the
packages that use them in ways other than just being a dependency. [1]
propagatedBuildInputs can actually be seen as an example of this:
packages using that are effectively "injecting" those dependencies as extra
buildInputs in their downstream dependents. But in general, a dependency
might affect the packages it depends on in arbitrary ways. Arbitrary is the
key word here. We could teach setup.sh things about upstream packages
like pkg/nix-support/propagated-build-inputs, but not arbitrary
interactions.

Setup hooks are the basic building block we have for this. In nixpkgs, a
"hook" is basically a bash callback, and a setup hook is no exception. Let's
look at the last part of findInputs we haven't covered:
findInputs() {
local pkg=$1

## More goes here in reality that we can ignore for now.

if test -f $pkg/nix-support/setup-hook; then

source $pkg/nix-support/setup-hook
fi

## More goes here in reality that we can ignore for now.

}

If a package includes the path pkg/nix-support/setup-hook, it will be

sourced by any stdenv-based build including that as a dependency.

This is strictly more general than any of the other mechanisms introduced in
this chapter. For example, try writing a setup hook that has the same effect as
a propagatedBuildInputs entry. One can almost think of this as an escape
hatch around Nix's normal isolation guarantees, and the principle that
dependencies are immutable and inert. We're not actually doing something
unsafe or modifying dependencies, but we are allowing arbitrary ad-hoc
behavior. For this reason, setup-hooks should only be used as a last resort.

20.4. Environment Hooks

As a final convenience, we have environment hooks. Recall in Pill 12 how
we created NIX_CFLAGS_COMPILE for -I flags and NIX_LDFLAGS for -L flags,
in a similar manner to how we prepared the PATH. One point of ugliness was
how anti-modular this was. It makes sense to build the PATH in generic
builder, because the PATH is used by the shell, and the generic builder is
intrinsically tied to the shell. But -I and -L flags are only relevant to the C
compiler. The stdenv isn't wedded to including a C compiler (though it does
by default), and there are other compilers too which may take completely
different flags.

As a first step, we can move that logic to a setup hook on the C compiler;
indeed that's just what we do in CC Wrapper. [2] But this pattern comes up
fairly often, so somebody decided to add some helper support to reduce
boilerplate.

The other half of addToEnv is:

addToEnv() {
local pkg=$1

## More goes here in reality that we can ignore for now.

# Run the package-specific hooks set by the setup-hook scripts.

for i in "${envHooks[@]}"; do
 $i $pkg
done
}

Functions listed in envHooks are applied to every package passed to

addToEnv. One can write a setup hook like:

anEnvHook() {
local pkg=$1

echo "I'm depending on \"$pkg\""

}

envHooks+=(anEnvHook)

and if one dependency has that setup hook then all of them will be so echoed.
Allowing dependencies to learn about their sibling dependencies is exactly
what compilers need.

20.5. Next pill...

...I'm not sure! We could talk about the additional dependency types and
hooks which cross compilation necessitates, building on our knowledge here
to cover stdenv as it works today. We could talk about how nixpkgs is
bootstrapped. Or we could talk about how localSystem and crossSystem are
elaborated into the buildPlatform, hostPlatform, and targetPlatform each
bootstrapping stage receives. Let us know which most interests you!

[1] Wecan now be precise and consider what addToEnv does alone the
minimal treatment of a dependency: i.e. a package that is just a dependency
would only have addToEnv applied to it.
[2] It
was called GCC Wrapper in the version of nixpkgs suggested for
following along in this pill; Darwin and Clang support hadn't yet motivated
the rename.