One post tagged with "basics"

This is the start of a series of blog posts aimed at simplifying Haskell development & packaging workflow using Nix. In this post, we begin the series by nixifying a Haskell application that uses PostgreSQL database and package it for end-users to run with one command.

Nix is a powerful package manager and build system that provides reproducible and declarative development environment. We will utilize Nix flakes to declaratively configure this environment.

If you're unfamiliar with Nix, we have a quick introduction available to help you get started quickly or you can take your time and explore it at Zero to Nix. A basic understanding of the Nix expression language is assumed. Throughout the series, we will utilize a simple Haskell app called todo-app to illustrate how to build a Haskell project and automatically manage runtime dependencies such as databases (ie., postgres) and and other services (here, we use postgREST), eliminating the need for any manual onboarding setup. This will allow us to highlight the advantages of using Nix.

Why Nixify?​

Why use Nix to develop a Haskell project rather than something like Stack or GHCup?

• Instant onboarding: Projects have READMEs that describe how to setup the development environment but these instructions do not work the same way for every developer and usually takes hours or days to setup. With Nix the setup is instantaneous and reproducible,¹ which means any new developer can get the development environment up and running with one command.
• Enhanced productivity: More time spent on writing Haskell as Nix gives a fully working development environment with nix develop.
• Multi-platform: Same configuration generally works on macOS, Linux and WSL.

The rest of this blog post will provide a step-by-step demonstration of how to Nixify the todo-app project.

Introduce Flake​

To begin, clone the todo-app repository and checkout the specified commit.

git clone https://github.com/juspay/todo-app.git
cd todo-app
git checkout 076185e34f70e903b992b597232bc622eadfcd51

Next, in the project's root directory, create a file named flake.nix and run git add flake.nix (Nix searches for flake.nix in git files). We will begin by setting up a basic template for the flake, which includes:

• Defining inputs and outputs
• Specifying the system corresponding to your machine.

Tl;dr This is how your flake.nix will look:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  };
  outputs = { self, nixpkgs }:
    let
      system = "aarch64-darwin";
      pkgs = nixpkgs.legacyPackages.${system};
    in
    {
      packages.${system}.default = pkgs.hello;
      apps.${system}.default = {
        type = "app";
        program = "${pkgs.hello}/bin/hello";
      };
      devShells.${system}.default = pkgs.mkShell {
        buildInputs = [
          pkgs.hello
        ];
      };
    };
}

A nix flake such as the one above consumes certain inputs and produces certain outputs. Let's break down each part of this flake.nix:

Inputs​

A flake can reference other flakes, which are specified in the inputs attribute. We will use the URL-like representation to specify our input flakes.

In this example, we will use GNU hello package from nixpkgs flake. Therefore, we'll specify the nixpkgs flake as an input, specifically using its nixpkgs-unstable branch.

Outputs​

The outputs attribute of a flake is essentially a Nix function that takes inputs and returns the outputs attribute.

The inputs argument is an attrset containing self as well as the flake inputs (in our flake, we reference the only input nixpkgs).

Refer here for a detailed schema of outputs. Note that the nixpkgs key within the inputs attrset refers to the outputs of the flake.nix located at nixpkgs.url. If nixpkgs.flake = false is set, then the parameter will represent the source code.

The body of the function defines the flake outputs. Within the let block we define two values -- system (set as "aarch64-darwin" in this example, assuming we are on an ARM mac) and pkgs (referring to nixpkgs packages for system). In our example, system is hardcoded to a single system, but forAllSystems can be used to define packages for an array of systems.

Here are some standard outputs a flake may produce:

Packages​

• A flake's packages.${system} output contains derivations that can be utilized to build the package.
• Executing nix build will build the packages.${system}.default output. Run nix build .#<packageName> to build the package named packages.${system}.<packageName>.

Apps​

• A flake's apps.${system}.<appName> output refers to a flake app that can be exeucted using nix run.
  • It is attribute set containing two keys type and program. The type attribute determines how the program should be executed. For instance, "shell" indicates a shell script, "python" indicates a Python script, and "app" indicates an executable. The program attribute is a string representing the path in the Nix store where the executable is located.
• Executing nix run will run the apps.${system}.default app. Run nix run .#<appName> to run the app named <appName>.

DevShells​

• By utilizing pkgs.mkShell, you can configure your development shell to include only the necessary packages.
• pkgs.mkShell generates a derivation that is evaluated when running the nix develop command.
• By default the derivation specified by devShells.${system}.default is evaluated. However, you also have the flexibility to define a custom development shell, such as devShells.${system}.mydevShell and execute it using nix develop .#mydevShell

Visualize the flake outputs​

• Run nix flake show

Here's how it will look:

├───apps
│   └───aarch64-darwin
│       └───default: app
├───devShells
│   └───aarch64-darwin
│       └───default: development environment 'nix-shell'
└───packages
    └───aarch64-darwin
        └───default: package 'hello-2.12.1'

See the flake in action​

Nixify Haskell package​

In the previous section we wrote a hello world flake that contains the "hello" package. Now, let's write a flake for our Haskell project todo-app.

Tl;dr Here is the flake.nix for this section:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  };
  outputs = { self, nixpkgs }:
    let
      system = "aarch64-darwin";
      pkgs = nixpkgs.legacyPackages.${system};
      overlay = final: prev: {
        todo-app = final.callCabal2nix "todo-app" ./. { };
      };
      myHaskellPackages = pkgs.haskellPackages.extend overlay;
    in
    {
      packages.${system}.default = myHaskellPackages.todo-app;
      apps.${system}.default = {
        type = "app";
        program = "${self.packages.${system}.default}/bin/todo-app";
      };
    };
}

Let's break it down!

haskellPackages​

Consult the official manual to learn more about the Haskell infrastructure in nixpkgs, but for the purpose of our blog post it is suffice to know that:

• pkgs.haskellPackages is an attribute set that contains all Haskell packages maintained within nixpkgs.
• Since our local package (todo-app) is not already included in pkgs.haskellPackages, we need to manually add it.
• Technically, you can use packages.${system}.default = pkgs.${system}.haskellPackages.callCabal2nix "todo-app" ./. { }; to include the package. However, adding it to haskellPackages consolidates every Haskell package in one place.

In summary, adding the local package to pkgs.haskellPackages centralizes the package management process and simplifies the usage of the package within other flakes.

Overlay​

Overlays are used to override an existing package set, such as pkgs.haskellPackages, and produce a new package set containing the changes. These changes could be either about overriding a single package in the package set (the second argument super references the original package set), or it could be about adding new packages to it.

callCabal2nix​

The callCabal2nix function generates a Haskell package derivation based on its source. This function internally utilizes "cabal2nix", which is a Haskell utility that generates Nix build instructions from a cabal file.

Time to run!​

Nixify DevShell​

Our existing flake enables us to build the todo-app. However, what if want to develop it, by adding a feature or fixing a bug? For Haskell development, we normally use cabal and tools like ghcid. These tools require a GHC environment that includes the packages specified in the build-depends of our cabal file. This is where devShell becomes useful as it provides an isolated environment with all packages required by the project, catering to our development needs.

Tl;dr Here is the flake.nix for this section:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  };
  outputs = { self, nixpkgs }:
    let
      system = "aarch64-darwin";
      pkgs = nixpkgs.legacyPackages.${system};
      overlay = final: prev: {
        todo-app = final.callCabal2nix "todo-app" ./. { };
      };
      myHaskellPackages = pkgs.haskellPackages.extend overlay;
    in
    {
      devShells.${system}.default = myHaskellPackages.shellFor {
        packages = p : [
          p.todo-app
        ];
        buildInputs = with myHaskellPackages; [
          ghcid
          cabal-install
        ];
      };
    };
}

shellFor​

• In the above flake, we utilize the shellFor function from the haskellPackages attribute set to set up the default shell for our project.
• shellFor is an abstraction over mkShell geared specifically for Haskell development shells. Generally, we only need to define two keys packages and nativeBuildInputs. packages marks which of the packages in the package set are local packages (to be compiled by cabal). nativeBuildInputs is used to ensure that the specified packages are present in the PATH of the isolated development environment.

Let's run!​

Nixify external dependencies​

Up until now, we have nixified the Haskell portion of our project. However, a project can also have non-Haskell dependencies, like Postgres, MySQL and Redis. In this section we will specifically look at how you can start a postgres server using Nix without relying or mutating global state (outside of project directory).

Tl;dr Here's the flake.nix:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  };
  outputs = { self, nixpkgs }:
  let
    system = "aarch64-darwin";
    pkgs = nixpkgs.legacyPackages.${system};
  in
  {
    apps.${system}.postgres = {
      type = "app";
      program = 
        let
          script = pkgs.writeShellApplication {
            name = "pg_start";
            runtimeInputs = [ pkgs.postgresql ];
            text = 
            ''
              # Initialize a database with data stored in current project dir
              [ ! -d "./data/db" ] && initdb --no-locale -D ./data/db

              postgres -D ./data/db -k "$PWD"/data
            '';
          };
        in "${script}/bin/pg_start";
    };
  };
}

writeShellApplication​

• The writeShellApplication function generates a derivation for a shell script specified as the value for text attribute.
• runtimeInputs: packages to be made available to the shell application's PATH.
• writeShellApplication uses shellcheck to statically analyze your bash script for issues.
• "${script}" provides the path in the nix/store where the application is located.

Run it!​

Nixify Combined​

Now it's time to consolidate all the previously discussed sections into a single flake.nix. Additionally, we should incorporate the necessary apps for postgrest and createdb. postgrest app will start the service and createdb will handle tasks such as loading the database dump, creating a database user, and configuring the database for postgREST.

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  };
  outputs = { self, nixpkgs }:
    let
      system = "aarch64-darwin";
      pkgs = nixpkgs.legacyPackages.${system};
      overlay = final: prev: {
        todo-app = final.callCabal2nix "todo-app" ./. { };
      };
      myHaskellPackages = pkgs.haskellPackages.extend overlay;
    in
    {
      packages.${system}.default = myHaskellPackages.todo-app;

      devShells.${system}.default = myHaskellPackages.shellFor {
        packages = p: [
          p.todo-app
        ];
        buildInputs = with myHaskellPackages; [
          ghcid
          cabal-install
          haskell-language-server
        ];
      };

      apps.${system} = {
        default = {
          type = "app";
          program = "${self.packages.${system}.default}/bin/todo-app";
        };
        postgres = {
          type = "app";
          program =
            let
              script = pkgs.writeShellApplication {
                name = "pg_start";
                runtimeInputs = [ pkgs.postgresql ];
                text =
                  ''
                    # Initialize a database with data stored in current project dir
                    [ ! -d "./data/db" ] && initdb --no-locale -D ./data/db

                    postgres -D ./data/db -k "$PWD"/data
                  '';
              };
            in
            "${script}/bin/pg_start";
        };
        createdb = {
          type = "app";
          program =
            let
              script = pkgs.writeShellApplication {
                name = "createDB";
                runtimeInputs = [ pkgs.postgresql ];
                text =
                  ''
                    # Create a database of your current user
                    if ! psql -h "$PWD"/data -lqt | cut -d \| -f 1 | grep -qw "$(whoami)"; then
                      createdb -h "$PWD"/data "$(whoami)"
                    fi

                    # Load DB dump
                    psql -h "$PWD"/data < db.sql

                    # Create configuration file for postgrest
                    echo "db-uri = \"postgres://authenticator:mysecretpassword@localhost:5432/$(whoami)\"
                    db-schemas = \"api\"
                    db-anon-role = \"todo_user\"" > data/db.conf
                  '';
              };
            in
            "${script}/bin/createDB";
        };
        postgrest = {
          type = "app";
          program =
            let
              script = pkgs.writeShellApplication {
                name = "pgREST";
                runtimeInputs = [ myHaskellPackages.postgrest ];
                text =
                  ''
                    postgrest ./data/db.conf
                  '';
              };
            in
            "${script}/bin/pgREST";
        };
      };
    };
}

For the complete souce code, visit here. It's worth noting that the source code uses forAllSystems, which was not included in the tutorial above to maintain simplicity.

Video Walkthrough​

Conclusion​

Let's see how the blog post addresses the points from the section Why Nixify?

• Instant onboarding: There is no confusion about how to setup the development environment. It is nix run .#postgres to start the postgres server, nix run .#createdb to setup the database and nix run .#postgrest to start the Postgrest web server. This happens in a reproducible way, ensuring every developer gets the same environment.
• Enhanced productivity: The commands mentioned in the previous points in conjunction with nix develop is all that is needed to make a quick change and see it in effect.
• Multi-platform: All the commands mentioned in the previous points will work in the same way across platforms.

In the next blog post, we will modularize this flake.nix using the flake-parts framework by Robert Hensing.

Credits​

• After going through numerous iterations, I am grateful to srid for his valuable assistance in bringing this blog to its current stage.
• I would also like to thank the following folks for their valuable feedback:
  • cdepillabout
  • Hao Liu