diff options
Diffstat (limited to 'nixpkgs/doc/languages-frameworks')
29 files changed, 7049 insertions, 0 deletions
diff --git a/nixpkgs/doc/languages-frameworks/agda.section.md b/nixpkgs/doc/languages-frameworks/agda.section.md new file mode 100644 index 000000000000..9ce046d05b6f --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/agda.section.md @@ -0,0 +1,106 @@ +--- +title: Agda +author: Alex Rice (alexarice) +date: 2020-01-06 +--- +# Agda + +## How to use Agda + +Agda can be installed from `agda`: +``` +$ nix-env -iA agda +``` + +To use agda with libraries, the `agda.withPackages` function can be used. This function either takes: ++ A list of packages, ++ or a function which returns a list of packages when given the `agdaPackages` attribute set, ++ or an attribute set containing a list of packages and a GHC derivation for compilation (see below). + +For example, suppose we wanted a version of agda which has access to the standard library. This can be obtained with the expressions: + +``` +agda.withPackages [ agdaPackages.standard-library ] +``` + +or + +``` +agda.withPackages (p: [ p.standard-library ]) +``` + +or can be called as in the [Compiling Agda](#compiling-agda) section. + +If you want to use a library in your home directory (for instance if it is a development version) then typecheck it manually (using `agda.withPackages` if necessary) and then override the `src` attribute of the package to point to your local repository. + +Agda will not by default use these libraries. To tell agda to use the library we have some options: +- Call `agda` with the library flag: +``` +$ agda -l standard-library -i . MyFile.agda +``` +- Write a `my-library.agda-lib` file for the project you are working on which may look like: +``` +name: my-library +include: . +depend: standard-library +``` +- Create the file `~/.agda/defaults` and add any libraries you want to use by default. + +More information can be found in the [official Agda documentation on library management](https://agda.readthedocs.io/en/v2.6.1/tools/package-system.html). + +## Compiling Agda +Agda modules can be compiled with the `--compile` flag. A version of `ghc` with `ieee` is made available to the Agda program via the `--with-compiler` flag. +This can be overridden by a different version of `ghc` as follows: + +``` +agda.withPackages { + pkgs = [ ... ]; + ghc = haskell.compiler.ghcHEAD; +} +``` + +## Writing Agda packages +To write a nix derivation for an agda library, first check that the library has a `*.agda-lib` file. + +A derivation can then be written using `agdaPackages.mkDerivation`. This has similar arguments to `stdenv.mkDerivation` with the following additions: ++ `everythingFile` can be used to specify the location of the `Everything.agda` file, defaulting to `./Everything.agda`. If this file does not exist then either it should be patched in or the `buildPhase` should be overridden (see below). ++ `libraryName` should be the name that appears in the `*.agda-lib` file, defaulting to `pname`. ++ `libraryFile` should be the file name of the `*.agda-lib` file, defaulting to `${libraryName}.agda-lib`. + +### Building Agda packages +The default build phase for `agdaPackages.mkDerivation` simply runs `agda` on the `Everything.agda` file. +If something else is needed to build the package (e.g. `make`) then the `buildPhase` should be overridden. +Additionally, a `preBuild` or `configurePhase` can be used if there are steps that need to be done prior to checking the `Everything.agda` file. +`agda` and the Agda libraries contained in `buildInputs` are made available during the build phase. + +### Installing Agda packages +The default install phase copies agda source files, agda interface files (`*.agdai`) and `*.agda-lib` files to the output directory. +This can be overridden. + +By default, agda sources are files ending on `.agda`, or literate agda files ending on `.lagda`, `.lagda.tex`, `.lagda.org`, `.lagda.md`, `.lagda.rst`. The list of recognised agda source extensions can be extended by setting the `extraExtensions` config variable. + +To add an agda package to `nixpkgs`, the derivation should be written to `pkgs/development/libraries/agda/${library-name}/` and an entry should be added to `pkgs/top-level/agda-packages.nix`. Here it is called in a scope with access to all other agda libraries, so the top line of the `default.nix` can look like: +``` +{ mkDerivation, standard-library, fetchFromGitHub }: +``` +and `mkDerivation` should be called instead of `agdaPackages.mkDerivation`. Here is an example skeleton derivation for iowa-stdlib: + +``` +mkDerivation { + version = "1.5.0"; + pname = "iowa-stdlib"; + + src = ... + + libraryFile = ""; + libraryName = "IAL-1.3"; + + buildPhase = '' + patchShebangs find-deps.sh + make + ''; +} +``` +This library has a file called `.agda-lib`, and so we give an empty string to `libraryFile` as nothing precedes `.agda-lib` in the filename. This file contains `name: IAL-1.3`, and so we let `libraryName = "IAL-1.3"`. This library does not use an `Everything.agda` file and instead has a Makefile, so there is no need to set `everythingFile` and we set a custom `buildPhase`. + +When writing an agda package it is essential to make sure that no `.agda-lib` file gets added to the store as a single file (for example by using `writeText`). This causes agda to think that the nix store is a agda library and it will attempt to write to it whenever it typechecks something. See [https://github.com/agda/agda/issues/4613](https://github.com/agda/agda/issues/4613). diff --git a/nixpkgs/doc/languages-frameworks/android.section.md b/nixpkgs/doc/languages-frameworks/android.section.md new file mode 100644 index 000000000000..1a8924082bfb --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/android.section.md @@ -0,0 +1,239 @@ +--- +title: Android +author: Sander van der Burg +date: 2018-11-18 +--- +# Android + +The Android build environment provides three major features and a number of +supporting features. + +Deploying an Android SDK installation with plugins +-------------------------------------------------- +The first use case is deploying the SDK with a desired set of plugins or subsets +of an SDK. + +```nix +with import <nixpkgs> {}; + +let + androidComposition = androidenv.composeAndroidPackages { + toolsVersion = "25.2.5"; + platformToolsVersion = "27.0.1"; + buildToolsVersions = [ "27.0.3" ]; + includeEmulator = false; + emulatorVersion = "27.2.0"; + platformVersions = [ "24" ]; + includeSources = false; + includeDocs = false; + includeSystemImages = false; + systemImageTypes = [ "default" ]; + abiVersions = [ "armeabi-v7a" ]; + lldbVersions = [ "2.0.2558144" ]; + cmakeVersions = [ "3.6.4111459" ]; + includeNDK = false; + ndkVersion = "16.1.4479499"; + useGoogleAPIs = false; + useGoogleTVAddOns = false; + includeExtras = [ + "extras;google;gcm" + ]; + }; +in +androidComposition.androidsdk +``` + +The above function invocation states that we want an Android SDK with the above +specified plugin versions. By default, most plugins are disabled. Notable +exceptions are the tools, platform-tools and build-tools sub packages. + +The following parameters are supported: + +* `toolsVersion`, specifies the version of the tools package to use +* `platformsToolsVersion` specifies the version of the `platform-tools` plugin +* `buildToolsVersion` specifies the versions of the `build-tools` plugins to + use. +* `includeEmulator` specifies whether to deploy the emulator package (`false` + by default). When enabled, the version of the emulator to deploy can be + specified by setting the `emulatorVersion` parameter. +* `includeDocs` specifies whether the documentation catalog should be included. +* `lldbVersions` specifies what LLDB versions should be deployed. +* `cmakeVersions` specifies which CMake versions should be deployed. +* `includeNDK` specifies that the Android NDK bundle should be included. + Defaults to: `false`. +* `ndkVersion` specifies the NDK version that we want to use. +* `includeExtras` is an array of identifier strings referring to arbitrary + add-on packages that should be installed. +* `platformVersions` specifies which platform SDK versions should be included. + +For each platform version that has been specified, we can apply the following +options: + +* `includeSystemImages` specifies whether a system image for each platform SDK + should be included. +* `includeSources` specifies whether the sources for each SDK version should be + included. +* `useGoogleAPIs` specifies that for each selected platform version the + Google API should be included. +* `useGoogleTVAddOns` specifies that for each selected platform version the + Google TV add-on should be included. + +For each requested system image we can specify the following options: + +* `systemImageTypes` specifies what kind of system images should be included. + Defaults to: `default`. +* `abiVersions` specifies what kind of ABI version of each system image should + be included. Defaults to: `armeabi-v7a`. + +Most of the function arguments have reasonable default settings. + +When building the above expression with: + +```bash +$ nix-build +``` + +The Android SDK gets deployed with all desired plugin versions. + +We can also deploy subsets of the Android SDK. For example, to only the +`platform-tools` package, you can evaluate the following expression: + +```nix +with import <nixpkgs> {}; + +let + androidComposition = androidenv.composeAndroidPackages { + # ... + }; +in +androidComposition.platform-tools +``` + +Using predefine Android package compositions +-------------------------------------------- +In addition to composing an Android package set manually, it is also possible +to use a predefined composition that contains all basic packages for a specific +Android version, such as version 9.0 (API-level 28). + +The following Nix expression can be used to deploy the entire SDK with all basic +plugins: + +```nix +with import <nixpkgs> {}; + +androidenv.androidPkgs_9_0.androidsdk +``` + +It is also possible to use one plugin only: + +```nix +with import <nixpkgs> {}; + +androidenv.androidPkgs_9_0.platform-tools +``` + +Building an Android application +------------------------------- +In addition to the SDK, it is also possible to build an Ant-based Android +project and automatically deploy all the Android plugins that a project +requires. + +```nix +with import <nixpkgs> {}; + +androidenv.buildApp { + name = "MyAndroidApp"; + src = ./myappsources; + release = true; + + # If release is set to true, you need to specify the following parameters + keyStore = ./keystore; + keyAlias = "myfirstapp"; + keyStorePassword = "mykeystore"; + keyAliasPassword = "myfirstapp"; + + # Any Android SDK parameters that install all the relevant plugins that a + # build requires + platformVersions = [ "24" ]; + + # When we include the NDK, then ndk-build is invoked before Ant gets invoked + includeNDK = true; +} +``` + +Aside from the app-specific build parameters (`name`, `src`, `release` and +keystore parameters), the `buildApp {}` function supports all the function +parameters that the SDK composition function (the function shown in the +previous section) supports. + +This build function is particularly useful when it is desired to use +[Hydra](https://nixos.org/hydra): the Nix-based continuous integration solution +to build Android apps. An Android APK gets exposed as a build product and can be +installed on any Android device with a web browser by navigating to the build +result page. + +Spawning emulator instances +--------------------------- +For testing purposes, it can also be quite convenient to automatically generate +scripts that spawn emulator instances with all desired configuration settings. + +An emulator spawn script can be configured by invoking the `emulateApp {}` +function: + +```nix +with import <nixpkgs> {}; + +androidenv.emulateApp { + name = "emulate-MyAndroidApp"; + platformVersion = "28"; + abiVersion = "x86"; # armeabi-v7a, mips, x86_64 + systemImageType = "google_apis_playstore"; +} +``` + +It is also possible to specify an APK to deploy inside the emulator +and the package and activity names to launch it: + +```nix +with import <nixpkgs> {}; + +androidenv.emulateApp { + name = "emulate-MyAndroidApp"; + platformVersion = "24"; + abiVersion = "armeabi-v7a"; # mips, x86, x86_64 + systemImageType = "default"; + useGoogleAPIs = false; + app = ./MyApp.apk; + package = "MyApp"; + activity = "MainActivity"; +} +``` + +In addition to prebuilt APKs, you can also bind the APK parameter to a +`buildApp {}` function invocation shown in the previous example. + +Querying the available versions of each plugin +---------------------------------------------- +When using any of the previously shown functions, it may be a bit inconvenient +to find out what options are supported, since the Android SDK provides many +plugins. + +A shell script in the `pkgs/development/mobile/androidenv/` sub directory can be used to retrieve all +possible options: + +```bash +sh ./querypackages.sh packages build-tools +``` + +The above command-line instruction queries all build-tools versions in the +generated `packages.nix` expression. + +Updating the generated expressions +---------------------------------- +Most of the Nix expressions are generated from XML files that the Android +package manager uses. To update the expressions run the `generate.sh` script +that is stored in the `pkgs/development/mobile/androidenv/` sub directory: + +```bash +./generate.sh +``` diff --git a/nixpkgs/doc/languages-frameworks/beam.xml b/nixpkgs/doc/languages-frameworks/beam.xml new file mode 100644 index 000000000000..278535237c2c --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/beam.xml @@ -0,0 +1,159 @@ +<section xmlns="http://docbook.org/ns/docbook" + xmlns:xlink="http://www.w3.org/1999/xlink" + xml:id="sec-beam"> + <title>BEAM Languages (Erlang, Elixir & LFE)</title> + + <section xml:id="beam-introduction"> + <title>Introduction</title> + + <para> + In this document and related Nix expressions, we use the term, <emphasis>BEAM</emphasis>, to describe the environment. BEAM is the name of the Erlang Virtual Machine and, as far as we're concerned, from a packaging perspective, all languages that run on the BEAM are interchangeable. That which varies, like the build system, is transparent to users of any given BEAM package, so we make no distinction. + </para> + </section> + + <section xml:id="beam-structure"> + <title>Structure</title> + + <para> + All BEAM-related expressions are available via the top-level <literal>beam</literal> attribute, which includes: + </para> + + <itemizedlist> + <listitem> + <para> + <literal>interpreters</literal>: a set of compilers running on the BEAM, including multiple Erlang/OTP versions (<literal>beam.interpreters.erlangR19</literal>, etc), Elixir (<literal>beam.interpreters.elixir</literal>) and LFE (<literal>beam.interpreters.lfe</literal>). + </para> + </listitem> + <listitem> + <para> + <literal>packages</literal>: a set of package builders (Mix and rebar3), each compiled with a specific Erlang/OTP version, e.g. <literal>beam.packages.erlangR19</literal>. + </para> + </listitem> + </itemizedlist> + + <para> + The default Erlang compiler, defined by <literal>beam.interpreters.erlang</literal>, is aliased as <literal>erlang</literal>. The default BEAM package set is defined by <literal>beam.packages.erlang</literal> and aliased at the top level as <literal>beamPackages</literal>. + </para> + + <para> + To create a package builder built with a custom Erlang version, use the lambda, <literal>beam.packagesWith</literal>, which accepts an Erlang/OTP derivation and produces a package builder similar to <literal>beam.packages.erlang</literal>. + </para> + + <para> + Many Erlang/OTP distributions available in <literal>beam.interpreters</literal> have versions with ODBC and/or Java enabled or without wx (no observer support). For example, there's <literal>beam.interpreters.erlangR22_odbc_javac</literal>, which corresponds to <literal>beam.interpreters.erlangR22</literal> and <literal>beam.interpreters.erlangR22_nox</literal>, which corresponds to <literal>beam.interpreters.erlangR22</literal>. + </para> + </section> + + <section xml:id="build-tools"> + <title>Build Tools</title> + + <section xml:id="build-tools-rebar3"> + <title>Rebar3</title> + + <para> + We provide a version of Rebar3, under <literal>rebar3</literal>. We also provide a helper to fetch Rebar3 dependencies from a lockfile under <literal>fetchRebar3Deps</literal>. + </para> + </section> + + <section xml:id="build-tools-other"> + <title>Mix & Erlang.mk</title> + + <para> + Both Mix and Erlang.mk work exactly as expected. There is a bootstrap process that needs to be run for both, however, which is supported by the <literal>buildMix</literal> and <literal>buildErlangMk</literal> derivations, respectively. + </para> + </section> + </section> + + <section xml:id="how-to-install-beam-packages"> + <title>How to Install BEAM Packages</title> + + <para> + BEAM builders are not registered at the top level, simply because they are not relevant to the vast majority of Nix users. + To install any of those builders into your profile, refer to them by their attribute path <literal>beamPackages.rebar3</literal>: + </para> + + <screen> + <prompt>$ </prompt>nix-env -f "<nixpkgs>" -iA beamPackages.rebar3 + </screen> +</section> + + <section xml:id="packaging-beam-applications"> + <title>Packaging BEAM Applications</title> + + <section xml:id="packaging-erlang-applications"> + <title>Erlang Applications</title> + + <section xml:id="rebar3-packages"> + <title>Rebar3 Packages</title> + + <para> + The Nix function, <literal>buildRebar3</literal>, defined in <literal>beam.packages.erlang.buildRebar3</literal> and aliased at the top level, can be used to build a derivation that understands how to build a Rebar3 project. + </para> + + <para> + If a package needs to compile native code via Rebar3's port compilation mechanism, add <literal>compilePort = true;</literal> to the derivation. + </para> + </section> + + <section xml:id="erlang-mk-packages"> + <title>Erlang.mk Packages</title> + + <para> + Erlang.mk functions similarly to Rebar3, except we use <literal>buildErlangMk</literal> instead of <literal>buildRebar3</literal>. + </para> + + </section> + + <section xml:id="mix-packages"> + <title>Mix Packages</title> + + <para> + Mix functions similarly to Rebar3, except we use <literal>buildMix</literal> instead of <literal>buildRebar3</literal>. + </para> + + <para> + Alternatively, we can use <literal>buildHex</literal> as a shortcut: + </para> + </section> + </section> + </section> + + <section xml:id="how-to-develop"> + <title>How to Develop</title> + + <section xml:id="creating-a-shell"> + <title>Creating a Shell</title> + + <para> + Usually, we need to create a <literal>shell.nix</literal> file and do our development inside of the environment specified therein. Just install your version of erlang and other interpreter, and then user your normal build tools. + As an example with elixir: + </para> + +<programlisting> +{ pkgs ? import "<nixpkgs"> {} }: + +with pkgs; + +let + + elixir = beam.packages.erlangR22.elixir_1_9; + +in +mkShell { + buildInputs = [ elixir ]; + + ERL_INCLUDE_PATH="${erlang}/lib/erlang/usr/include"; +} +</programlisting> + + <section xml:id="building-in-a-shell"> + <title>Building in a Shell (for Mix Projects)</title> + + <para> + Using a <literal>shell.nix</literal> as described (see <xref + linkend="creating-a-shell"/>) should just work. + </para> + </section> + </section> + </section> +</section> diff --git a/nixpkgs/doc/languages-frameworks/bower.xml b/nixpkgs/doc/languages-frameworks/bower.xml new file mode 100644 index 000000000000..b0738cad293b --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/bower.xml @@ -0,0 +1,196 @@ +<section xmlns="http://docbook.org/ns/docbook" + xmlns:xlink="http://www.w3.org/1999/xlink" + xml:id="sec-bower"> + <title>Bower</title> + + <para> + <link xlink:href="http://bower.io">Bower</link> is a package manager for web site front-end components. Bower packages (comprising of build artefacts and sometimes sources) are stored in <command>git</command> repositories, typically on Github. The package registry is run by the Bower team with package metadata coming from the <filename>bower.json</filename> file within each package. + </para> + + <para> + The end result of running Bower is a <filename>bower_components</filename> directory which can be included in the web app's build process. + </para> + + <para> + Bower can be run interactively, by installing <varname>nodePackages.bower</varname>. More interestingly, the Bower components can be declared in a Nix derivation, with the help of <varname>nodePackages.bower2nix</varname>. + </para> + + <section xml:id="ssec-bower2nix-usage"> + <title><command>bower2nix</command> usage</title> + + <para> + Suppose you have a <filename>bower.json</filename> with the following contents: + <example xml:id="ex-bowerJson"> + <title><filename>bower.json</filename></title> +<programlisting language="json"> +<![CDATA[{ + "name": "my-web-app", + "dependencies": { + "angular": "~1.5.0", + "bootstrap": "~3.3.6" + } +}]]> +</programlisting> + </example> + </para> + + <para> + Running <command>bower2nix</command> will produce something like the following output: +<programlisting language="nix"> +<![CDATA[{ fetchbower, buildEnv }: +buildEnv { name = "bower-env"; ignoreCollisions = true; paths = [ + (fetchbower "angular" "1.5.3" "~1.5.0" "1749xb0firxdra4rzadm4q9x90v6pzkbd7xmcyjk6qfza09ykk9y") + (fetchbower "bootstrap" "3.3.6" "~3.3.6" "1vvqlpbfcy0k5pncfjaiskj3y6scwifxygfqnw393sjfxiviwmbv") + (fetchbower "jquery" "2.2.2" "1.9.1 - 2" "10sp5h98sqwk90y4k6hbdviwqzvzwqf47r3r51pakch5ii2y7js1") +]; }]]> +</programlisting> + </para> + + <para> + Using the <command>bower2nix</command> command line arguments, the output can be redirected to a file. A name like <filename>bower-packages.nix</filename> would be fine. + </para> + + <para> + The resulting derivation is a union of all the downloaded Bower packages (and their dependencies). To use it, they still need to be linked together by Bower, which is where <varname>buildBowerComponents</varname> is useful. + </para> + </section> + + <section xml:id="ssec-build-bower-components"> + <title><varname>buildBowerComponents</varname> function</title> + + <para> + The function is implemented in <link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/bower-modules/generic/default.nix"> <filename>pkgs/development/bower-modules/generic/default.nix</filename></link>. Example usage: + <example xml:id="ex-buildBowerComponents"> + <title>buildBowerComponents</title> +<programlisting language="nix"> +bowerComponents = buildBowerComponents { + name = "my-web-app"; + generated = ./bower-packages.nix; <co xml:id="ex-buildBowerComponents-1" /> + src = myWebApp; <co xml:id="ex-buildBowerComponents-2" /> +}; +</programlisting> + </example> + </para> + + <para> + In <xref linkend="ex-buildBowerComponents" />, the following arguments are of special significance to the function: + <calloutlist> + <callout arearefs="ex-buildBowerComponents-1"> + <para> + <varname>generated</varname> specifies the file which was created by <command>bower2nix</command>. + </para> + </callout> + <callout arearefs="ex-buildBowerComponents-2"> + <para> + <varname>src</varname> is your project's sources. It needs to contain a <filename>bower.json</filename> file. + </para> + </callout> + </calloutlist> + </para> + + <para> + <varname>buildBowerComponents</varname> will run Bower to link together the output of <command>bower2nix</command>, resulting in a <filename>bower_components</filename> directory which can be used. + </para> + + <para> + Here is an example of a web frontend build process using <command>gulp</command>. You might use <command>grunt</command>, or anything else. + </para> + + <example xml:id="ex-bowerGulpFile"> + <title>Example build script (<filename>gulpfile.js</filename>)</title> +<programlisting language="javascript"> +<![CDATA[var gulp = require('gulp'); + +gulp.task('default', [], function () { + gulp.start('build'); +}); + +gulp.task('build', [], function () { + console.log("Just a dummy gulp build"); + gulp + .src(["./bower_components/**/*"]) + .pipe(gulp.dest("./gulpdist/")); +});]]> +</programlisting> + </example> + + <example xml:id="ex-buildBowerComponentsDefaultNix"> + <title>Full example — <filename>default.nix</filename></title> +<programlisting language="nix"> +{ myWebApp ? { outPath = ./.; name = "myWebApp"; } +, pkgs ? import <nixpkgs> {} +}: + +pkgs.stdenv.mkDerivation { + name = "my-web-app-frontend"; + src = myWebApp; + + buildInputs = [ pkgs.nodePackages.gulp ]; + + bowerComponents = pkgs.buildBowerComponents { <co xml:id="ex-buildBowerComponentsDefault-1" /> + name = "my-web-app"; + generated = ./bower-packages.nix; + src = myWebApp; + }; + + buildPhase = '' + cp --reflink=auto --no-preserve=mode -R $bowerComponents/bower_components . <co xml:id="ex-buildBowerComponentsDefault-2" /> + export HOME=$PWD <co xml:id="ex-buildBowerComponentsDefault-3" /> + ${pkgs.nodePackages.gulp}/bin/gulp build <co xml:id="ex-buildBowerComponentsDefault-4" /> + ''; + + installPhase = "mv gulpdist $out"; +} +</programlisting> + </example> + + <para> + A few notes about <xref linkend="ex-buildBowerComponentsDefaultNix" />: + <calloutlist> + <callout arearefs="ex-buildBowerComponentsDefault-1"> + <para> + The result of <varname>buildBowerComponents</varname> is an input to the frontend build. + </para> + </callout> + <callout arearefs="ex-buildBowerComponentsDefault-2"> + <para> + Whether to symlink or copy the <filename>bower_components</filename> directory depends on the build tool in use. In this case a copy is used to avoid <command>gulp</command> silliness with permissions. + </para> + </callout> + <callout arearefs="ex-buildBowerComponentsDefault-3"> + <para> + <command>gulp</command> requires <varname>HOME</varname> to refer to a writeable directory. + </para> + </callout> + <callout arearefs="ex-buildBowerComponentsDefault-4"> + <para> + The actual build command. Other tools could be used. + </para> + </callout> + </calloutlist> + </para> + </section> + + <section xml:id="ssec-bower2nix-troubleshooting"> + <title>Troubleshooting</title> + + <variablelist> + <varlistentry> + <term> + <literal>ENOCACHE</literal> errors from <varname>buildBowerComponents</varname> + </term> + <listitem> + <para> + This means that Bower was looking for a package version which doesn't exist in the generated <filename>bower-packages.nix</filename>. + </para> + <para> + If <filename>bower.json</filename> has been updated, then run <command>bower2nix</command> again. + </para> + <para> + It could also be a bug in <command>bower2nix</command> or <command>fetchbower</command>. If possible, try reformulating the version specification in <filename>bower.json</filename>. + </para> + </listitem> + </varlistentry> + </variablelist> + </section> +</section> diff --git a/nixpkgs/doc/languages-frameworks/coq.xml b/nixpkgs/doc/languages-frameworks/coq.xml new file mode 100644 index 000000000000..86d9226166f5 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/coq.xml @@ -0,0 +1,52 @@ +<section xmlns="http://docbook.org/ns/docbook" + xmlns:xlink="http://www.w3.org/1999/xlink" + xml:id="sec-language-coq"> + <title>Coq</title> + + <para> + Coq libraries should be installed in <literal>$(out)/lib/coq/${coq.coq-version}/user-contrib/</literal>. Such directories are automatically added to the <literal>$COQPATH</literal> environment variable by the hook defined in the Coq derivation. + </para> + + <para> + Some extensions (plugins) might require OCaml and sometimes other OCaml packages. The <literal>coq.ocamlPackages</literal> attribute can be used to depend on the same package set Coq was built against. + </para> + + <para> + Coq libraries may be compatible with some specific versions of Coq only. The <literal>compatibleCoqVersions</literal> attribute is used to precisely select those versions of Coq that are compatible with this derivation. + </para> + + <para> + Here is a simple package example. It is a pure Coq library, thus it depends on Coq. It builds on the Mathematical Components library, thus it also takes <literal>mathcomp</literal> as <literal>buildInputs</literal>. Its <literal>Makefile</literal> has been generated using <literal>coq_makefile</literal> so we only have to set the <literal>$COQLIB</literal> variable at install time. + </para> + +<programlisting> +{ stdenv, fetchFromGitHub, coq, mathcomp }: + +stdenv.mkDerivation rec { + name = "coq${coq.coq-version}-multinomials-${version}"; + version = "1.0"; + src = fetchFromGitHub { + owner = "math-comp"; + repo = "multinomials"; + rev = version; + sha256 = "1qmbxp1h81cy3imh627pznmng0kvv37k4hrwi2faa101s6bcx55m"; + }; + + buildInputs = [ coq ]; + propagatedBuildInputs = [ mathcomp ]; + + installFlags = "COQLIB=$(out)/lib/coq/${coq.coq-version}/"; + + meta = { + description = "A Coq/SSReflect Library for Monoidal Rings and Multinomials"; + inherit (src.meta) homepage; + license = stdenv.lib.licenses.cecill-b; + inherit (coq.meta) platforms; + }; + + passthru = { + compatibleCoqVersions = v: builtins.elem v [ "8.5" "8.6" "8.7" ]; + }; +} +</programlisting> +</section> diff --git a/nixpkgs/doc/languages-frameworks/crystal.section.md b/nixpkgs/doc/languages-frameworks/crystal.section.md new file mode 100644 index 000000000000..af0853dbf75b --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/crystal.section.md @@ -0,0 +1,71 @@ +# Crystal + +## Building a Crystal package + +This section uses [Mint](https://github.com/mint-lang/mint) as an example for how to build a Crystal package. + +If the Crystal project has any dependencies, the first step is to get a `shards.nix` file encoding those. Get a copy of the project and go to its root directory such that its `shard.lock` file is in the current directory, then run `crystal2nix` in it +```bash +$ git clone https://github.com/mint-lang/mint +$ cd mint +$ git checkout 0.5.0 +$ nix-shell -p crystal2nix --run crystal2nix +``` + +This should have generated a `shards.nix` file. + +Next create a Nix file for your derivation and use `pkgs.crystal.buildCrystalPackage` as follows: +```nix +with import <nixpkgs> {}; +crystal.buildCrystalPackage rec { + pname = "mint"; + version = "0.5.0"; + + src = fetchFromGitHub { + owner = "mint-lang"; + repo = "mint"; + rev = version; + sha256 = "0vxbx38c390rd2ysvbwgh89v2232sh5rbsp3nk9wzb70jybpslvl"; + }; + + # Insert the path to your shards.nix file here + shardsFile = ./shards.nix; + + ... +} +``` + +This won't build anything yet, because we haven't told it what files build. We can specify a mapping from binary names to source files with the `crystalBinaries` attribute. The project's compilation instructions should show this. For Mint, the binary is called "mint", which is compiled from the source file `src/mint.cr`, so we'll specify this as follows: + +```nix + crystalBinaries.mint.src = "src/mint.cr"; + + # ... +``` + +Additionally you can override the default `crystal build` options (which are currently `--release --progress --no-debug --verbose`) with + +```nix + crystalBinaries.mint.options = [ "--release" "--verbose" ]; +``` + +Depending on the project, you might need additional steps to get it to compile successfully. In Mint's case, we need to link against openssl, so in the end the Nix file looks as follows: + +```nix +with import <nixpkgs> {}; +crystal.buildCrystalPackage rec { + version = "0.5.0"; + pname = "mint"; + src = fetchFromGitHub { + owner = "mint-lang"; + repo = "mint"; + rev = version; + sha256 = "0vxbx38c390rd2ysvbwgh89v2232sh5rbsp3nk9wzb70jybpslvl"; + }; + + shardsFile = ./shards.nix; + crystalBinaries.mint.src = "src/mint.cr"; + + buildInputs = [ openssl ]; +} +``` diff --git a/nixpkgs/doc/languages-frameworks/dotnet.section.md b/nixpkgs/doc/languages-frameworks/dotnet.section.md new file mode 100644 index 000000000000..c56f4728bed8 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/dotnet.section.md @@ -0,0 +1,75 @@ +# Dotnet + +## Local Development Workflow + +For local development, it's recommended to use nix-shell to create a dotnet environment: + +``` +# shell.nix +with import <nixpkgs> {}; + +mkShell { + name = "dotnet-env"; + buildInputs = [ + dotnet-sdk_3 + ]; +} +``` + +### Using many sdks in a workflow + +It's very likely that more than one sdk will be needed on a given project. Dotnet provides several different frameworks (E.g dotnetcore, aspnetcore, etc.) as well as many versions for a given framework. Normally, dotnet is able to fetch a framework and install it relative to the executable. However, this would mean writing to the nix store in nixpkgs, which is read-only. To support the many-sdk use case, one can compose an environment using `dotnetCorePackages.combinePackages`: + +``` +with import <nixpkgs> {}; + +mkShell { + name = "dotnet-env"; + buildInputs = [ + (with dotnetCorePackages; combinePackages [ + sdk_3_1 + sdk_3_0 + sdk_2_1 + ]) + ]; +} +``` + +This will produce a dotnet installation that has the dotnet 3.1, 3.0, and 2.1 sdk. The first sdk listed will have it's cli utility present in the resulting environment. Example info output: + +``` +$ dotnet --info +.NET Core SDK (reflecting any global.json): + Version: 3.1.101 + Commit: b377529961 + +... + +.NET Core SDKs installed: + 2.1.803 [/nix/store/iiv98i2jdi226dgh4jzkkj2ww7f8jgpd-dotnet-core-combined/sdk] + 3.0.102 [/nix/store/iiv98i2jdi226dgh4jzkkj2ww7f8jgpd-dotnet-core-combined/sdk] + 3.1.101 [/nix/store/iiv98i2jdi226dgh4jzkkj2ww7f8jgpd-dotnet-core-combined/sdk] + +.NET Core runtimes installed: + Microsoft.AspNetCore.All 2.1.15 [/nix/store/iiv98i2jdi226dgh4jzkkj2ww7f8jgpd-dotnet-core-combined/shared/Microsoft.AspNetCore.All] + Microsoft.AspNetCore.App 2.1.15 [/nix/store/iiv98i2jdi226dgh4jzkkj2ww7f8jgpd-dotnet-core-combined/shared/Microsoft.AspNetCore.App] + Microsoft.AspNetCore.App 3.0.2 [/nix/store/iiv98i2jdi226dgh4jzkkj2ww7f8jgpd-dotnet-core-combined/shared/Microsoft.AspNetCore.App] + Microsoft.AspNetCore.App 3.1.1 [/nix/store/iiv98i2jdi226dgh4jzkkj2ww7f8jgpd-dotnet-core-combined/shared/Microsoft.AspNetCore.App] + Microsoft.NETCore.App 2.1.15 [/nix/store/iiv98i2jdi226dgh4jzkkj2ww7f8jgpd-dotnet-core-combined/shared/Microsoft.NETCore.App] + Microsoft.NETCore.App 3.0.2 [/nix/store/iiv98i2jdi226dgh4jzkkj2ww7f8jgpd-dotnet-core-combined/shared/Microsoft.NETCore.App] + Microsoft.NETCore.App 3.1.1 [/nix/store/iiv98i2jdi226dgh4jzkkj2ww7f8jgpd-dotnet-core-combined/shared/Microsoft.NETCore.App] +``` + +## dotnet-sdk vs dotnetCorePackages.sdk + +The `dotnetCorePackages.sdk_X_Y` is preferred over the old dotnet-sdk as both major and minor version are very important for a dotnet environment. If a given minor version isn't present (or was changed), then this will likely break your ability to build a project. + +## dotnetCorePackages.sdk vs dotnetCorePackages.netcore vs dotnetCorePackages.aspnetcore + +The `dotnetCorePackages.sdk` contains both a runtime and the full sdk of a given version. The `netcore` and `aspnetcore` packages are meant to serve as minimal runtimes to deploy alongside already built applications. + +## Packaging a Dotnet Application + +Ideally, we would like to build against the sdk, then only have the dotnet runtime available in the runtime closure. + +TODO: Create closure-friendly way to package dotnet applications diff --git a/nixpkgs/doc/languages-frameworks/emscripten.section.md b/nixpkgs/doc/languages-frameworks/emscripten.section.md new file mode 100644 index 000000000000..80e1094809ad --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/emscripten.section.md @@ -0,0 +1,185 @@ +# Emscripten + +[Emscripten](https://github.com/kripken/emscripten): An LLVM-to-JavaScript Compiler + +This section of the manual covers how to use `emscripten` in nixpkgs. + +Minimal requirements: + +* nix +* nixpkgs + +Modes of use of `emscripten`: + +* **Imperative usage** (on the command line): + + If you want to work with `emcc`, `emconfigure` and `emmake` as you are used to from Ubuntu and similar distributions you can use these commands: + + * `nix-env -i emscripten` + * `nix-shell -p emscripten` + +* **Declarative usage**: + + This mode is far more power full since this makes use of `nix` for dependency management of emscripten libraries and targets by using the `mkDerivation` which is implemented by `pkgs.emscriptenStdenv` and `pkgs.buildEmscriptenPackage`. The source for the packages is in `pkgs/top-level/emscripten-packages.nix` and the abstraction behind it in `pkgs/development/em-modules/generic/default.nix`. + * build and install all packages: + * `nix-env -iA emscriptenPackages` + + * dev-shell for zlib implementation hacking: + * `nix-shell -A emscriptenPackages.zlib` + + +## Imperative usage + +A few things to note: + +* `export EMCC_DEBUG=2` is nice for debugging +* `~/.emscripten`, the build artifact cache sometimes creates issues and needs to be removed from time to time + + +## Declarative usage + +Let's see two different examples from `pkgs/top-level/emscripten-packages.nix`: + +* `pkgs.zlib.override` +* `pkgs.buildEmscriptenPackage` + +Both are interesting concepts. + +A special requirement of the `pkgs.buildEmscriptenPackage` is the `doCheck = true` is a default meaning that each emscriptenPackage requires a `checkPhase` implemented. + +* Use `export EMCC_DEBUG=2` from within a emscriptenPackage's `phase` to get more detailed debug output what is going wrong. +* ~/.emscripten cache is requiring us to set `HOME=$TMPDIR` in individual phases. This makes compilation slower but also makes it more deterministic. + +### Usage 1: pkgs.zlib.override + +This example uses `zlib` from nixpkgs but instead of compiling **C** to **ELF** it compiles **C** to **JS** since we were using `pkgs.zlib.override` and changed stdenv to `pkgs.emscriptenStdenv`. A few adaptions and hacks were set in place to make it working. One advantage is that when `pkgs.zlib` is updated, it will automatically update this package as well. However, this can also be the downside... + +See the `zlib` example: + + zlib = (pkgs.zlib.override { + stdenv = pkgs.emscriptenStdenv; + }).overrideDerivation + (old: rec { + buildInputs = old.buildInputs ++ [ pkgconfig ]; + # we need to reset this setting! + NIX_CFLAGS_COMPILE=""; + configurePhase = '' + # FIXME: Some tests require writing at $HOME + HOME=$TMPDIR + runHook preConfigure + + #export EMCC_DEBUG=2 + emconfigure ./configure --prefix=$out --shared + + runHook postConfigure + ''; + dontStrip = true; + outputs = [ "out" ]; + buildPhase = '' + emmake make + ''; + installPhase = '' + emmake make install + ''; + checkPhase = '' + echo "================= testing zlib using node =================" + + echo "Compiling a custom test" + set -x + emcc -O2 -s EMULATE_FUNCTION_POINTER_CASTS=1 test/example.c -DZ_SOLO \ + libz.so.${old.version} -I . -o example.js + + echo "Using node to execute the test" + ${pkgs.nodejs}/bin/node ./example.js + + set +x + if [ $? -ne 0 ]; then + echo "test failed for some reason" + exit 1; + else + echo "it seems to work! very good." + fi + echo "================= /testing zlib using node =================" + ''; + + postPatch = pkgs.stdenv.lib.optionalString pkgs.stdenv.isDarwin '' + substituteInPlace configure \ + --replace '/usr/bin/libtool' 'ar' \ + --replace 'AR="libtool"' 'AR="ar"' \ + --replace 'ARFLAGS="-o"' 'ARFLAGS="-r"' + ''; + }); + +### Usage 2: pkgs.buildEmscriptenPackage + +This `xmlmirror` example features a emscriptenPackage which is defined completely from this context and no `pkgs.zlib.override` is used. + + xmlmirror = pkgs.buildEmscriptenPackage rec { + name = "xmlmirror"; + + buildInputs = [ pkgconfig autoconf automake libtool gnumake libxml2 nodejs openjdk json_c ]; + nativeBuildInputs = [ pkgconfig zlib ]; + + src = pkgs.fetchgit { + url = "https://gitlab.com/odfplugfest/xmlmirror.git"; + rev = "4fd7e86f7c9526b8f4c1733e5c8b45175860a8fd"; + sha256 = "1jasdqnbdnb83wbcnyrp32f36w3xwhwp0wq8lwwmhqagxrij1r4b"; + }; + + configurePhase = '' + rm -f fastXmlLint.js* + # a fix for ERROR:root:For asm.js, TOTAL_MEMORY must be a multiple of 16MB, was 234217728 + # https://gitlab.com/odfplugfest/xmlmirror/issues/8 + sed -e "s/TOTAL_MEMORY=234217728/TOTAL_MEMORY=268435456/g" -i Makefile.emEnv + # https://github.com/kripken/emscripten/issues/6344 + # https://gitlab.com/odfplugfest/xmlmirror/issues/9 + sed -e "s/\$(JSONC_LDFLAGS) \$(ZLIB_LDFLAGS) \$(LIBXML20_LDFLAGS)/\$(JSONC_LDFLAGS) \$(LIBXML20_LDFLAGS) \$(ZLIB_LDFLAGS) /g" -i Makefile.emEnv + # https://gitlab.com/odfplugfest/xmlmirror/issues/11 + sed -e "s/-o fastXmlLint.js/-s EXTRA_EXPORTED_RUNTIME_METHODS='[\"ccall\", \"cwrap\"]' -o fastXmlLint.js/g" -i Makefile.emEnv + ''; + + buildPhase = '' + HOME=$TMPDIR + make -f Makefile.emEnv + ''; + + outputs = [ "out" "doc" ]; + + installPhase = '' + mkdir -p $out/share + mkdir -p $doc/share/${name} + + cp Demo* $out/share + cp -R codemirror-5.12 $out/share + cp fastXmlLint.js* $out/share + cp *.xsd $out/share + cp *.js $out/share + cp *.xhtml $out/share + cp *.html $out/share + cp *.json $out/share + cp *.rng $out/share + cp README.md $doc/share/${name} + ''; + checkPhase = '' + + ''; + }; + +### Declarative debugging + +Use `nix-shell -I nixpkgs=/some/dir/nixpkgs -A emscriptenPackages.libz` and from there you can go trough the individual steps. This makes it easy to build a good `unit test` or list the files of the project. + +1. `nix-shell -I nixpkgs=/some/dir/nixpkgs -A emscriptenPackages.libz` +2. `cd /tmp/` +3. `unpackPhase` +4. cd libz-1.2.3 +5. `configurePhase` +6. `buildPhase` +7. ... happy hacking... + +## Summary + +Using this toolchain makes it easy to leverage `nix` from NixOS, MacOSX or even Windows (WSL+ubuntu+nix). This toolchain is reproducible, behaves like the rest of the packages from nixpkgs and contains a set of well working examples to learn and adapt from. + +If in trouble, ask the maintainers. + diff --git a/nixpkgs/doc/languages-frameworks/gnome.xml b/nixpkgs/doc/languages-frameworks/gnome.xml new file mode 100644 index 000000000000..159216ca981f --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/gnome.xml @@ -0,0 +1,299 @@ +<section xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="sec-language-gnome"> + <title>GNOME</title> + + <section xml:id="ssec-gnome-packaging"> + <title>Packaging GNOME applications</title> + + <para> + Programs in the GNOME universe are written in various languages but they all use GObject-based libraries like GLib, GTK or GStreamer. These libraries are often modular, relying on looking into certain directories to find their modules. However, due to Nix’s specific file system organization, this will fail without our intervention. Fortunately, the libraries usually allow overriding the directories through environment variables, either natively or thanks to a patch in nixpkgs. <link xlink:href="#fun-wrapProgram">Wrapping</link> the executables to ensure correct paths are available to the application constitutes a significant part of packaging a modern desktop application. In this section, we will describe various modules needed by such applications, environment variables needed to make the modules load, and finally a script that will do the work for us. + </para> + + <section xml:id="ssec-gnome-settings"> + <title>Settings</title> + + <para> + <link xlink:href="https://developer.gnome.org/gio/stable/GSettings.html">GSettings</link> API is often used for storing settings. GSettings schemas are required, to know the type and other metadata of the stored values. GLib looks for <filename>glib-2.0/schemas/gschemas.compiled</filename> files inside the directories of <envar>XDG_DATA_DIRS</envar>. + </para> + + <para> + On Linux, GSettings API is implemented using <link xlink:href="https://wiki.gnome.org/Projects/dconf">dconf</link> backend. You will need to add <literal>dconf</literal> GIO module to <envar>GIO_EXTRA_MODULES</envar> variable, otherwise the <literal>memory</literal> backend will be used and the saved settings will not be persistent. + </para> + + <para> + Last you will need the dconf database D-Bus service itself. You can enable it using <option>programs.dconf.enable</option>. + </para> + + <para> + Some applications will also require <package>gsettings-desktop-schemas</package> for things like reading proxy configuration or user interface customization. This dependency is often not mentioned by upstream, you should grep for <literal>org.gnome.desktop</literal> and <literal>org.gnome.system</literal> to see if the schemas are needed. + </para> + </section> + + <section xml:id="ssec-gnome-icons"> + <title>Icons</title> + + <para> + When an application uses icons, an icon theme should be available in <envar>XDG_DATA_DIRS</envar> during runtime. The package for the default, icon-less <link xlink:href="https://www.freedesktop.org/wiki/Software/icon-theme/">hicolor-icon-theme</link> (should be propagated by every icon theme) contains <link linkend="ssec-gnome-hooks-hicolor-icon-theme">a setup hook</link> that will pick up icon themes from <literal>buildInputs</literal> and pass it to our wrapper. Unfortunately, relying on that would mean every user has to download the theme included in the package expression no matter their preference. For that reason, we leave the installation of icon theme on the user. If you use one of the desktop environments, you probably already have an icon theme installed. + </para> + + <para> + To avoid costly file system access when locating icons, GTK, <link xlink:href="https://woboq.com/blog/qicon-reads-gtk-icon-cache-in-qt57.html">as well as Qt</link>, can rely on <filename>icon-theme.cache</filename> files from the themes’ top-level directories. These files are generated using <command>gtk-update-icon-cache</command>, which is expected to be run whenever an icon is added or removed to an icon theme (typically an application icon into <literal>hicolor</literal> theme) and some programs do indeed run this after icon installation. However, since packages are installed into their own prefix by Nix, this would lead to conflicts. For that reason, <package>gtk3</package> provides a <link xlink:href="#ssec-gnome-hooks-gtk-drop-icon-theme-cache">setup hook</link> that will clean the file from installation. Since most applications only ship their own icon that will be loaded on start-up, it should not affect them too much. On the other hand, icon themes are much larger and more widely used so we need to cache them. Because we recommend installing icon themes globally, we will generate the cache files from all packages in a profile using a NixOS module. You can enable the cache generation using <option>gtk.iconCache.enable</option> option if your desktop environment does not already do that. + </para> + </section> + + <section xml:id="ssec-icon-theme-packaging"> + <title>Packaging icon themes</title> + + <para> + Icon themes may inherit from other icon themes. The inheritance is specified using the <literal>Inherits</literal> key in the <filename>index.theme</filename> file distributed with the icon theme. According to the <link xlink:href="https://specifications.freedesktop.org/icon-theme-spec/icon-theme-spec-latest.html">icon theme specification</link>, icons not provided by the theme are looked for in its parent icon themes. Therefore the parent themes should be installed as dependencies for a more complete experience regarding the icon sets used. + </para> + + <para> + The package <package>hicolor-icon-theme</package> provides a setup hook which makes symbolic links for the parent themes into the directory <filename>share/icons</filename> of the current theme directory in the nix store, making sure they can be found at runtime. For that to work the packages providing parent icon themes should be listed as propagated build dependencies, together with <package>hicolor-icon-theme</package>. + </para> + + <para> + Also make sure that <filename>icon-theme.cache</filename> is installed for each theme provided by the package, and set <code>dontDropIconThemeCache</code> to <code>true</code> so that the cache file is not removed by the <package>gtk3</package> setup hook. + </para> + + </section> + + <section xml:id="ssec-gnome-themes"> + <title>GTK Themes</title> + + <para> + Previously, a GTK theme needed to be in <envar>XDG_DATA_DIRS</envar>. This is no longer necessary for most programs since GTK incorporated Adwaita theme. Some programs (for example, those designed for <link xlink:href="https://elementary.io/docs/human-interface-guidelines#human-interface-guidelines">elementary HIG</link>) might require a special theme like <package>pantheon.elementary-gtk-theme</package>. + </para> + </section> + + <section xml:id="ssec-gnome-typelibs"> + <title>GObject introspection typelibs</title> + + <para> + <link xlink:href="https://wiki.gnome.org/Projects/GObjectIntrospection">GObject introspection</link> allows applications to use C libraries in other languages easily. It does this through <literal>typelib</literal> files searched in <envar>GI_TYPELIB_PATH</envar>. + </para> + </section> + + <section xml:id="ssec-gnome-plugins"> + <title>Various plug-ins</title> + + <para> + If your application uses <link xlink:href="https://gstreamer.freedesktop.org/">GStreamer</link> or <link xlink:href="https://wiki.gnome.org/Projects/Grilo">Grilo</link>, you should set <envar>GST_PLUGIN_SYSTEM_PATH_1_0</envar> and <envar>GRL_PLUGIN_PATH</envar>, respectively. + </para> + </section> + </section> + + <section xml:id="ssec-gnome-hooks"> + <title>Onto <package>wrapGAppsHook</package></title> + + <para> + Given the requirements above, the package expression would become messy quickly: +<programlisting> +preFixup = '' + for f in $(find $out/bin/ $out/libexec/ -type f -executable); do + wrapProgram "$f" \ + --prefix GIO_EXTRA_MODULES : "${getLib dconf}/lib/gio/modules" \ + --prefix XDG_DATA_DIRS : "$out/share" \ + --prefix XDG_DATA_DIRS : "$out/share/gsettings-schemas/${name}" \ + --prefix XDG_DATA_DIRS : "${gsettings-desktop-schemas}/share/gsettings-schemas/${gsettings-desktop-schemas.name}" \ + --prefix XDG_DATA_DIRS : "${hicolor-icon-theme}/share" \ + --prefix GI_TYPELIB_PATH : "${lib.makeSearchPath "lib/girepository-1.0" [ pango json-glib ]}" + done +''; +</programlisting> + Fortunately, there is <package>wrapGAppsHook</package>, that does the wrapping for us. In particular, it works in conjunction with other setup hooks that will populate the variable: + <itemizedlist> + <listitem xml:id="ssec-gnome-hooks-wrapgappshook"> + <para> + <package>wrapGAppsHook</package> itself will add the package’s <filename>share</filename> directory to <envar>XDG_DATA_DIRS</envar>. + </para> + </listitem> + <listitem xml:id="ssec-gnome-hooks-glib"> + <para> + <package>glib</package> setup hook will populate <envar>GSETTINGS_SCHEMAS_PATH</envar> and then <package>wrapGAppsHook</package> will prepend it to <envar>XDG_DATA_DIRS</envar>. + </para> + </listitem> + <listitem xml:id="ssec-gnome-hooks-gtk-drop-icon-theme-cache"> + <para> + One of <package>gtk3</package>’s setup hooks will remove <filename>icon-theme.cache</filename> files from package’s icon theme directories to avoid conflicts. Icon theme packages should prevent this with <code>dontDropIconThemeCache = true;</code>. + </para> + </listitem> + <listitem xml:id="ssec-gnome-hooks-dconf"> + <para> + <package>dconf.lib</package> is a dependency of <package>wrapGAppsHook</package>, which then also adds it to the <envar>GIO_EXTRA_MODULES</envar> variable. + </para> + </listitem> + <listitem xml:id="ssec-gnome-hooks-hicolor-icon-theme"> + <para> + <package>hicolor-icon-theme</package>’s setup hook will add icon themes to <envar>XDG_ICON_DIRS</envar> which is prepended to <envar>XDG_DATA_DIRS</envar> by <package>wrapGAppsHook</package>. + </para> + </listitem> + <listitem xml:id="ssec-gnome-hooks-gobject-introspection"> + <para> + <package>gobject-introspection</package> setup hook populates <envar>GI_TYPELIB_PATH</envar> variable with <filename>lib/girepository-1.0</filename> directories of dependencies, which is then added to wrapper by <package>wrapGAppsHook</package>. It also adds <filename>share</filename> directories of dependencies to <envar>XDG_DATA_DIRS</envar>, which is intended to promote GIR files but it also <link xlink:href="https://github.com/NixOS/nixpkgs/issues/32790">pollutes the closures</link> of packages using <package>wrapGAppsHook</package>. + </para> + <warning> + <para> + The setup hook <link xlink:href="https://github.com/NixOS/nixpkgs/issues/56943">currently</link> does not work in expressions with <literal>strictDeps</literal> enabled, like Python packages. In those cases, you will need to disable it with <code>strictDeps = false;</code>. + </para> + </warning> + </listitem> + <listitem xml:id="ssec-gnome-hooks-gst-grl-plugins"> + <para> + Setup hooks of <package>gst_all_1.gstreamer</package> and <package>gnome3.grilo</package> will populate the <envar>GST_PLUGIN_SYSTEM_PATH_1_0</envar> and <envar>GRL_PLUGIN_PATH</envar> variables, respectively, which will then be added to the wrapper by <literal>wrapGAppsHook</literal>. + </para> + </listitem> + </itemizedlist> + </para> + + <para> + You can also pass additional arguments to <literal>makeWrapper</literal> using <literal>gappsWrapperArgs</literal> in <literal>preFixup</literal> hook: +<programlisting> +preFixup = '' + gappsWrapperArgs+=( + # Thumbnailers + --prefix XDG_DATA_DIRS : "${gdk-pixbuf}/share" + --prefix XDG_DATA_DIRS : "${librsvg}/share" + --prefix XDG_DATA_DIRS : "${shared-mime-info}/share" + ) +''; +</programlisting> + </para> + </section> + + <section xml:id="ssec-gnome-updating"> + <title>Updating GNOME packages</title> + + <para> + Most GNOME package offer <link linkend="var-passthru-updateScript"><literal>updateScript</literal></link>, it is therefore possible to update to latest source tarball by running <command>nix-shell maintainers/scripts/update.nix --argstr package gnome3.nautilus</command> or even en masse with <command>nix-shell maintainers/scripts/update.nix --argstr path gnome3</command>. Read the package’s <filename>NEWS</filename> file to see what changed. + </para> + </section> + + <section xml:id="ssec-gnome-common-issues"> + <title>Frequently encountered issues</title> + + <variablelist> + <varlistentry xml:id="ssec-gnome-common-issues-no-schemas"> + <term> + <computeroutput>GLib-GIO-ERROR **: <replaceable>06:04:50.903</replaceable>: No GSettings schemas are installed on the system</computeroutput> + </term> + <listitem> + <para> + There are no schemas avalable in <envar>XDG_DATA_DIRS</envar>. Temporarily add a random package containing schemas like <package>gsettings-desktop-schemas</package> to <literal>buildInputs</literal>. <link linkend="ssec-gnome-hooks-glib"><package>glib</package></link> and <link linkend="ssec-gnome-hooks-wrapgappshook"><package>wrapGAppsHook</package></link> setup hooks will take care of making the schemas available to application and you will see the actual missing schemas with the <link linkend="ssec-gnome-common-issues-missing-schema">next error</link>. Or you can try looking through the source code for the actual schemas used. + </para> + </listitem> + </varlistentry> + <varlistentry xml:id="ssec-gnome-common-issues-missing-schema"> + <term> + <computeroutput>GLib-GIO-ERROR **: <replaceable>06:04:50.903</replaceable>: Settings schema ‘<replaceable>org.gnome.foo</replaceable>’ is not installed</computeroutput> + </term> + <listitem> + <para> + Package is missing some GSettings schemas. You can find out the package containing the schema with <command>nix-locate <replaceable>org.gnome.foo</replaceable>.gschema.xml</command> and let the hooks handle the wrapping as <link linkend="ssec-gnome-common-issues-no-schemas">above</link>. + </para> + </listitem> + </varlistentry> + <varlistentry xml:id="ssec-gnome-common-issues-double-wrapped"> + <term> + When using <package>wrapGAppsHook</package> with special derivers you can end up with double wrapped binaries. + </term> + <listitem> + <para> + This is because derivers like <function>python.pkgs.buildPythonApplication</function> or <function>qt5.mkDerivation</function> have setup-hooks automatically added that produce wrappers with <package>makeWrapper</package>. The simplest way to workaround that is to disable the <package>wrapGAppsHook</package> automatic wrapping with <code>dontWrapGApps = true;</code> and pass the arguments it intended to pass to <package>makeWrapper</package> to another. + </para> + <para> + In the case of a Python application it could look like: +<programlisting> +python3.pkgs.buildPythonApplication { + pname = "gnome-music"; + version = "3.32.2"; + + nativeBuildInputs = [ + wrapGAppsHook + gobject-introspection + ... + ]; + + dontWrapGApps = true; + + # Arguments to be passed to `makeWrapper`, only used by buildPython* + preFixup = '' + makeWrapperArgs+=("''${gappsWrapperArgs[@]}") + ''; +} +</programlisting> + And for a QT app like: +<programlisting> +mkDerivation { + pname = "calibre"; + version = "3.47.0"; + + nativeBuildInputs = [ + wrapGAppsHook + qmake + ... + ]; + + dontWrapGApps = true; + + # Arguments to be passed to `makeWrapper`, only used by qt5’s mkDerivation + preFixup = '' + qtWrapperArgs+=("''${gappsWrapperArgs[@]}") + ''; +} +</programlisting> + </para> + </listitem> + </varlistentry> + <varlistentry xml:id="ssec-gnome-common-issues-unwrappable-package"> + <term> + I am packaging a project that cannot be wrapped, like a library or GNOME Shell extension. + </term> + <listitem> + <para> + You can rely on applications depending on the library setting the necessary environment variables but that is often easy to miss. Instead we recommend to patch the paths in the source code whenever possible. Here are some examples: + <itemizedlist> + <listitem xml:id="ssec-gnome-common-issues-unwrappable-package-gnome-shell-ext"> + <para> + <link xlink:href="https://github.com/NixOS/nixpkgs/blob/7bb8f05f12ca3cff9da72b56caa2f7472d5732bc/pkgs/desktops/gnome-3/core/gnome-shell-extensions/default.nix#L21-L24">Replacing a <envar>GI_TYPELIB_PATH</envar> in GNOME Shell extension</link> – we are using <function>substituteAll</function> to include the path to a typelib into a patch. + </para> + </listitem> + <listitem xml:id="ssec-gnome-common-issues-unwrappable-package-gsettings"> + <para> + The following examples are hardcoding GSettings schema paths. To get the schema paths we use the functions + <itemizedlist> + <listitem> + <para> + <function>glib.getSchemaPath</function> Takes a nix package attribute as an argument. + </para> + </listitem> + <listitem> + <para> + <function>glib.makeSchemaPath</function> Takes a package output like <literal>$out</literal> and a derivation name. You should use this if the schemas you need to hardcode are in the same derivation. + </para> + </listitem> + </itemizedlist> + </para> + <para xml:id="ssec-gnome-common-issues-unwrappable-package-gsettings-vala"> + <link xlink:href="https://github.com/NixOS/nixpkgs/blob/7bb8f05f12ca3cff9da72b56caa2f7472d5732bc/pkgs/desktops/pantheon/apps/elementary-files/default.nix#L78-L86">Hard-coding GSettings schema path in Vala plug-in (dynamically loaded library)</link> – here, <function>substituteAll</function> cannot be used since the schema comes from the same package preventing us from pass its path to the function, probably due to a <link xlink:href="https://github.com/NixOS/nix/issues/1846">Nix bug</link>. + </para> + <para xml:id="ssec-gnome-common-issues-unwrappable-package-gsettings-c"> + <link xlink:href="https://github.com/NixOS/nixpkgs/blob/29c120c065d03b000224872251bed93932d42412/pkgs/development/libraries/glib-networking/default.nix#L31-L34">Hard-coding GSettings schema path in C library</link> – nothing special other than using <link xlink:href="https://github.com/NixOS/nixpkgs/pull/67957#issuecomment-527717467">Coccinelle patch</link> to generate the patch itself. + </para> + </listitem> + </itemizedlist> + </para> + </listitem> + </varlistentry> + <varlistentry xml:id="ssec-gnome-common-issues-weird-location"> + <term> + I need to wrap a binary outside <filename>bin</filename> and <filename>libexec</filename> directories. + </term> + <listitem> + <para> + You can manually trigger the wrapping with <function>wrapGApp</function> in <literal>preFixup</literal> phase. It takes a path to a program as a first argument; the remaining arguments are passed directly to <function xlink:href="#fun-wrapProgram">wrapProgram</function> function. + </para> + </listitem> + </varlistentry> + </variablelist> + </section> +</section> diff --git a/nixpkgs/doc/languages-frameworks/go.xml b/nixpkgs/doc/languages-frameworks/go.xml new file mode 100644 index 000000000000..ff39276f640e --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/go.xml @@ -0,0 +1,203 @@ +<section xmlns="http://docbook.org/ns/docbook" + xmlns:xlink="http://www.w3.org/1999/xlink" + xml:id="sec-language-go"> + <title>Go</title> + + <section xml:id="ssec-go-modules"> + <title>Go modules</title> + + <para> + The function <varname> buildGoModule </varname> builds Go programs managed with Go modules. It builds a <link xlink:href="https://github.com/golang/go/wiki/Modules">Go modules</link> through a two phase build: + <itemizedlist> + <listitem> + <para> + An intermediate fetcher derivation. This derivation will be used to fetch all of the dependencies of the Go module. + </para> + </listitem> + <listitem> + <para> + A final derivation will use the output of the intermediate derivation to build the binaries and produce the final output. + </para> + </listitem> + </itemizedlist> + </para> + + <example xml:id='ex-buildGoModule'> + <title>buildGoModule</title> +<programlisting> +pet = buildGoModule rec { + pname = "pet"; + version = "0.3.4"; + + src = fetchFromGitHub { + owner = "knqyf263"; + repo = "pet"; + rev = "v${version}"; + sha256 = "0m2fzpqxk7hrbxsgqplkg7h2p7gv6s1miymv3gvw0cz039skag0s"; + }; + + vendorSha256 = "1879j77k96684wi554rkjxydrj8g3hpp0kvxz03sd8dmwr3lh83j"; <co xml:id='ex-buildGoModule-1' /> + + subPackages = [ "." ]; <co xml:id='ex-buildGoModule-2' /> + + meta = with lib; { + description = "Simple command-line snippet manager, written in Go"; + homepage = "https://github.com/knqyf263/pet"; + license = licenses.mit; + maintainers = with maintainers; [ kalbasit ]; + platforms = platforms.linux ++ platforms.darwin; + }; +} +</programlisting> + </example> + + <para> + <xref linkend='ex-buildGoModule'/> is an example expression using buildGoModule, the following arguments are of special significance to the function: + <calloutlist> + <callout arearefs='ex-buildGoModule-1'> + <para> + <varname>vendorSha256</varname> is the hash of the output of the intermediate fetcher derivation. + </para> + </callout> + <callout arearefs='ex-buildGoModule-2'> + <para> + <varname>subPackages</varname> limits the builder from building child packages that have not been listed. If <varname>subPackages</varname> is not specified, all child packages will be built. + </para> + </callout> + </calloutlist> + </para> + + <para> + <varname>vendorSha256</varname> can also take <varname>null</varname> as an input. + + When `null` is used as a value, rather than fetching the dependencies + and vendoring them, we use the vendoring included within the source repo. + If you'd like to not have to update this field on dependency changes, + run `go mod vendor` in your source repo and set 'vendorSha256 = null;' + </para> + </section> + + <section xml:id="ssec-go-legacy"> + <title>Go legacy</title> + + <para> + The function <varname> buildGoPackage </varname> builds legacy Go programs, not supporting Go modules. + </para> + + <example xml:id='ex-buildGoPackage'> + <title>buildGoPackage</title> +<programlisting> +deis = buildGoPackage rec { + pname = "deis"; + version = "1.13.0"; + + goPackagePath = "github.com/deis/deis"; <co xml:id='ex-buildGoPackage-1' /> + subPackages = [ "client" ]; <co xml:id='ex-buildGoPackage-2' /> + + src = fetchFromGitHub { + owner = "deis"; + repo = "deis"; + rev = "v${version}"; + sha256 = "1qv9lxqx7m18029lj8cw3k7jngvxs4iciwrypdy0gd2nnghc68sw"; + }; + + goDeps = ./deps.nix; <co xml:id='ex-buildGoPackage-3' /> + + buildFlags = [ "--tags" "release" ]; <co xml:id='ex-buildGoPackage-4' /> +} +</programlisting> + </example> + + <para> + <xref linkend='ex-buildGoPackage'/> is an example expression using buildGoPackage, the following arguments are of special significance to the function: + <calloutlist> + <callout arearefs='ex-buildGoPackage-1'> + <para> + <varname>goPackagePath</varname> specifies the package's canonical Go import path. + </para> + </callout> + <callout arearefs='ex-buildGoPackage-2'> + <para> + <varname>subPackages</varname> limits the builder from building child packages that have not been listed. If <varname>subPackages</varname> is not specified, all child packages will be built. + </para> + <para> + In this example only <literal>github.com/deis/deis/client</literal> will be built. + </para> + </callout> + <callout arearefs='ex-buildGoPackage-3'> + <para> + <varname>goDeps</varname> is where the Go dependencies of a Go program are listed as a list of package source identified by Go import path. It could be imported as a separate <varname>deps.nix</varname> file for readability. The dependency data structure is described below. + </para> + </callout> + <callout arearefs='ex-buildGoPackage-4'> + <para> + <varname>buildFlags</varname> is a list of flags passed to the go build command. + </para> + </callout> + </calloutlist> + </para> + + <para> + The <varname>goDeps</varname> attribute can be imported from a separate <varname>nix</varname> file that defines which Go libraries are needed and should be included in <varname>GOPATH</varname> for <varname>buildPhase</varname>. + </para> + + <example xml:id='ex-goDeps'> + <title>deps.nix</title> +<programlisting> +[ <co xml:id='ex-goDeps-1' /> + { + goPackagePath = "gopkg.in/yaml.v2"; <co xml:id='ex-goDeps-2' /> + fetch = { + type = "git"; <co xml:id='ex-goDeps-3' /> + url = "https://gopkg.in/yaml.v2"; + rev = "a83829b6f1293c91addabc89d0571c246397bbf4"; + sha256 = "1m4dsmk90sbi17571h6pld44zxz7jc4lrnl4f27dpd1l8g5xvjhh"; + }; + } + { + goPackagePath = "github.com/docopt/docopt-go"; + fetch = { + type = "git"; + url = "https://github.com/docopt/docopt-go"; + rev = "784ddc588536785e7299f7272f39101f7faccc3f"; + sha256 = "0wwz48jl9fvl1iknvn9dqr4gfy1qs03gxaikrxxp9gry6773v3sj"; + }; + } +] +</programlisting> + </example> + + <para> + <calloutlist> + <callout arearefs='ex-goDeps-1'> + <para> + <varname>goDeps</varname> is a list of Go dependencies. + </para> + </callout> + <callout arearefs='ex-goDeps-2'> + <para> + <varname>goPackagePath</varname> specifies Go package import path. + </para> + </callout> + <callout arearefs='ex-goDeps-3'> + <para> + <varname>fetch type</varname> that needs to be used to get package source. If <varname>git</varname> is used there should be <varname>url</varname>, <varname>rev</varname> and <varname>sha256</varname> defined next to it. + </para> + </callout> + </calloutlist> + </para> + + <para> + To extract dependency information from a Go package in automated way use <link xlink:href="https://github.com/kamilchm/go2nix">go2nix</link>. It can produce complete derivation and <varname>goDeps</varname> file for Go programs. + </para> + + <para> + You may use Go packages installed into the active Nix profiles by adding the following to your ~/.bashrc: +<screen> +for p in $NIX_PROFILES; do + GOPATH="$p/share/go:$GOPATH" +done +</screen> + </para> + </section> +</section> diff --git a/nixpkgs/doc/languages-frameworks/haskell.section.md b/nixpkgs/doc/languages-frameworks/haskell.section.md new file mode 100644 index 000000000000..cba4d0561b07 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/haskell.section.md @@ -0,0 +1,1094 @@ +--- +title: User's Guide for Haskell in Nixpkgs +author: Peter Simons +date: 2015-06-01 +--- +# Haskell + + +## How to install Haskell packages + +Nixpkgs distributes build instructions for all Haskell packages registered on +[Hackage](http://hackage.haskell.org/), but strangely enough normal Nix package +lookups don't seem to discover any of them, except for the default version of ghc, cabal-install, and stack: +``` +$ nix-env -i alex +error: selector ‘alex’ matches no derivations +$ nix-env -qa ghc +ghc-7.10.2 +``` + +The Haskell package set is not registered in the top-level namespace because it +is *huge*. If all Haskell packages were visible to these commands, then +name-based search/install operations would be much slower than they are now. We +avoided that by keeping all Haskell-related packages in a separate attribute +set called `haskellPackages`, which the following command will list: +``` +$ nix-env -f "<nixpkgs>" -qaP -A haskellPackages +haskellPackages.a50 a50-0.5 +haskellPackages.AAI AAI-0.2.0.1 +haskellPackages.abacate abacate-0.0.0.0 +haskellPackages.abc-puzzle abc-puzzle-0.2.1 +haskellPackages.abcBridge abcBridge-0.15 +haskellPackages.abcnotation abcnotation-1.9.0 +haskellPackages.abeson abeson-0.1.0.1 +[... some 14000 entries omitted ...] +``` + +To install any of those packages into your profile, refer to them by their +attribute path (first column): +```shell +nix-env -f "<nixpkgs>" -iA haskellPackages.Allure ... +``` + +The attribute path of any Haskell packages corresponds to the name of that +particular package on Hackage: the package `cabal-install` has the attribute +`haskellPackages.cabal-install`, and so on. (Actually, this convention causes +trouble with packages like `3dmodels` and `4Blocks`, because these names are +invalid identifiers in the Nix language. The issue of how to deal with these +rare corner cases is currently unresolved.) + +Haskell packages whose Nix name (second column) begins with a `haskell-` prefix +are packages that provide a library whereas packages without that prefix +provide just executables. Libraries may provide executables too, though: the +package `haskell-pandoc`, for example, installs both a library and an +application. You can install and use Haskell executables just like any other +program in Nixpkgs, but using Haskell libraries for development is a bit +trickier and we'll address that subject in great detail in section [How to +create a development environment](#how-to-create-a-development-environment). + +Attribute paths are deterministic inside of Nixpkgs, but the path necessary to +reach Nixpkgs varies from system to system. We dodged that problem by giving +`nix-env` an explicit `-f "<nixpkgs>"` parameter, but if you call `nix-env` +without that flag, then chances are the invocation fails: +``` +$ nix-env -iA haskellPackages.cabal-install +error: attribute ‘haskellPackages’ in selection path + ‘haskellPackages.cabal-install’ not found +``` + +On NixOS, for example, Nixpkgs does *not* exist in the top-level namespace by +default. To figure out the proper attribute path, it's easiest to query for the +path of a well-known Nixpkgs package, i.e.: +``` +$ nix-env -qaP coreutils +nixos.coreutils coreutils-8.23 +``` + +If your system responds like that (most NixOS installations will), then the +attribute path to `haskellPackages` is `nixos.haskellPackages`. Thus, if you +want to use `nix-env` without giving an explicit `-f` flag, then that's the way +to do it: +```shell +nix-env -qaP -A nixos.haskellPackages +nix-env -iA nixos.haskellPackages.cabal-install +``` + +Our current default compiler is GHC 8.8.x and the `haskellPackages` set +contains packages built with that particular version. Nixpkgs contains the last +three major releases of GHC and there is a whole family of package sets +available that defines Hackage packages built with each of those compilers, +too: +```shell +nix-env -f "<nixpkgs>" -qaP -A haskell.packages.ghc865 +nix-env -f "<nixpkgs>" -qaP -A haskell.packages.ghc8101 +``` + +The name `haskellPackages` is really just a synonym for +`haskell.packages.ghc882`, because we prefer that package set internally and +recommend it to our users as their default choice, but ultimately you are free +to compile your Haskell packages with any GHC version you please. The following +command displays the complete list of available compilers: +``` +$ nix-env -f "<nixpkgs>" -qaP -A haskell.compiler +haskell.compiler.ghc8101 ghc-8.10.1 +haskell.compiler.integer-simple.ghc8101 ghc-8.10.1 +haskell.compiler.ghcHEAD ghc-8.11.20200505 +haskell.compiler.integer-simple.ghcHEAD ghc-8.11.20200505 +haskell.compiler.ghc822Binary ghc-8.2.2-binary +haskell.compiler.ghc844 ghc-8.4.4 +haskell.compiler.ghc863Binary ghc-8.6.3-binary +haskell.compiler.ghc865 ghc-8.6.5 +haskell.compiler.integer-simple.ghc865 ghc-8.6.5 +haskell.compiler.ghc882 ghc-8.8.2 +haskell.compiler.integer-simple.ghc882 ghc-8.8.2 +haskell.compiler.ghc883 ghc-8.8.3 +haskell.compiler.integer-simple.ghc883 ghc-8.8.3 +haskell.compiler.ghcjs ghcjs-8.6.0.1 +``` + +We have no package sets for `jhc` or `uhc` yet, unfortunately, but for every +version of GHC listed above, there exists a package set based on that compiler. +Also, the attributes `haskell.compiler.ghcXYC` and +`haskell.packages.ghcXYC.ghc` are synonymous for the sake of convenience. + +## How to create a development environment + +### How to install a compiler + +A simple development environment consists of a Haskell compiler and one or both +of the tools `cabal-install` and `stack`. We saw in section +[How to install Haskell packages](#how-to-install-haskell-packages) how you can install those programs into your +user profile: +```shell +nix-env -f "<nixpkgs>" -iA haskellPackages.ghc haskellPackages.cabal-install +``` + +Instead of the default package set `haskellPackages`, you can also use the more +precise name `haskell.compiler.ghc7102`, which has the advantage that it refers +to the same GHC version regardless of what Nixpkgs considers "default" at any +given time. + +Once you've made those tools available in `$PATH`, it's possible to build +Hackage packages the same way people without access to Nix do it all the time: +```shell +cabal get lens-4.11 && cd lens-4.11 +cabal install -j --dependencies-only +cabal configure +cabal build +``` + +If you enjoy working with Cabal sandboxes, then that's entirely possible too: +just execute the command +```shell +cabal sandbox init +``` +before installing the required dependencies. + +The `nix-shell` utility makes it easy to switch to a different compiler +version; just enter the Nix shell environment with the command +```shell +nix-shell -p haskell.compiler.ghc784 +``` +to bring GHC 7.8.4 into `$PATH`. Alternatively, you can use Stack instead of +`nix-shell` directly to select compiler versions and other build tools +per-project. It uses `nix-shell` under the hood when Nix support is turned on. +See [How to build a Haskell project using Stack](#how-to-build-a-haskell-project-using-stack). + +If you're using `cabal-install`, re-running `cabal configure` inside the spawned +shell switches your build to use that compiler instead. If you're working on +a project that doesn't depend on any additional system libraries outside of GHC, +then it's even sufficient to just run the `cabal configure` command inside of +the shell: +```shell +nix-shell -p haskell.compiler.ghc784 --command "cabal configure" +``` + +Afterwards, all other commands like `cabal build` work just fine in any shell +environment, because the configure phase recorded the absolute paths to all +required tools like GHC in its build configuration inside of the `dist/` +directory. Please note, however, that `nix-collect-garbage` can break such an +environment because the Nix store paths created by `nix-shell` aren't "alive" +anymore once `nix-shell` has terminated. If you find that your Haskell builds +no longer work after garbage collection, then you'll have to re-run `cabal +configure` inside of a new `nix-shell` environment. + +### How to install a compiler with libraries + +GHC expects to find all installed libraries inside of its own `lib` directory. +This approach works fine on traditional Unix systems, but it doesn't work for +Nix, because GHC's store path is immutable once it's built. We cannot install +additional libraries into that location. As a consequence, our copies of GHC +don't know any packages except their own core libraries, like `base`, +`containers`, `Cabal`, etc. + +We can register additional libraries to GHC, however, using a special build +function called `ghcWithPackages`. That function expects one argument: a +function that maps from an attribute set of Haskell packages to a list of +packages, which determines the libraries known to that particular version of +GHC. For example, the Nix expression `ghcWithPackages (pkgs: [pkgs.mtl])` +generates a copy of GHC that has the `mtl` library registered in addition to +its normal core packages: +``` +$ nix-shell -p "haskellPackages.ghcWithPackages (pkgs: [pkgs.mtl])" + +[nix-shell:~]$ ghc-pkg list mtl +/nix/store/zy79...-ghc-7.10.2/lib/ghc-7.10.2/package.conf.d: + mtl-2.2.1 +``` + +This function allows users to define their own development environment by means +of an override. After adding the following snippet to `~/.config/nixpkgs/config.nix`, +```nix +{ + packageOverrides = super: let self = super.pkgs; in + { + myHaskellEnv = self.haskell.packages.ghc7102.ghcWithPackages + (haskellPackages: with haskellPackages; [ + # libraries + arrows async cgi criterion + # tools + cabal-install haskintex + ]); + }; +} +``` +it's possible to install that compiler with `nix-env -f "<nixpkgs>" -iA +myHaskellEnv`. If you'd like to switch that development environment to a +different version of GHC, just replace the `ghc7102` bit in the previous +definition with the appropriate name. Of course, it's also possible to define +any number of these development environments! (You can't install two of them +into the same profile at the same time, though, because that would result in +file conflicts.) + +The generated `ghc` program is a wrapper script that re-directs the real +GHC executable to use a new `lib` directory --- one that we specifically +constructed to contain all those packages the user requested: +``` +$ cat $(type -p ghc) +#! /nix/store/xlxj...-bash-4.3-p33/bin/bash -e +export NIX_GHC=/nix/store/19sm...-ghc-7.10.2/bin/ghc +export NIX_GHCPKG=/nix/store/19sm...-ghc-7.10.2/bin/ghc-pkg +export NIX_GHC_DOCDIR=/nix/store/19sm...-ghc-7.10.2/share/doc/ghc/html +export NIX_GHC_LIBDIR=/nix/store/19sm...-ghc-7.10.2/lib/ghc-7.10.2 +exec /nix/store/j50p...-ghc-7.10.2/bin/ghc "-B$NIX_GHC_LIBDIR" "$@" +``` + +The variables `$NIX_GHC`, `$NIX_GHCPKG`, etc. point to the *new* store path +`ghcWithPackages` constructed specifically for this environment. The last line +of the wrapper script then executes the real `ghc`, but passes the path to the +new `lib` directory using GHC's `-B` flag. + +The purpose of those environment variables is to work around an impurity in the +popular [ghc-paths](http://hackage.haskell.org/package/ghc-paths) library. That +library promises to give its users access to GHC's installation paths. Only, +the library can't possible know that path when it's compiled, because the path +GHC considers its own is determined only much later, when the user configures +it through `ghcWithPackages`. So we [patched +ghc-paths](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/patches/ghc-paths-nix.patch) +to return the paths found in those environment variables at run-time rather +than trying to guess them at compile-time. + +To make sure that mechanism works properly all the time, we recommend that you +set those variables to meaningful values in your shell environment, too, i.e. +by adding the following code to your `~/.bashrc`: +```bash +if type >/dev/null 2>&1 -p ghc; then + eval "$(egrep ^export "$(type -p ghc)")" +fi +``` + +If you are certain that you'll use only one GHC environment which is located in +your user profile, then you can use the following code, too, which has the +advantage that it doesn't contain any paths from the Nix store, i.e. those +settings always remain valid even if a `nix-env -u` operation updates the GHC +environment in your profile: +```bash +if [ -e ~/.nix-profile/bin/ghc ]; then + export NIX_GHC="$HOME/.nix-profile/bin/ghc" + export NIX_GHCPKG="$HOME/.nix-profile/bin/ghc-pkg" + export NIX_GHC_DOCDIR="$HOME/.nix-profile/share/doc/ghc/html" + export NIX_GHC_LIBDIR="$HOME/.nix-profile/lib/ghc-$($NIX_GHC --numeric-version)" +fi +``` + +### How to install a compiler with libraries, hoogle and documentation indexes + +If you plan to use your environment for interactive programming, not just +compiling random Haskell code, you might want to replace `ghcWithPackages` in +all the listings above with `ghcWithHoogle`. + +This environment generator not only produces an environment with GHC and all +the specified libraries, but also generates a `hoogle` and `haddock` indexes +for all the packages, and provides a wrapper script around `hoogle` binary that +uses all those things. A precise name for this thing would be +"`ghcWithPackagesAndHoogleAndDocumentationIndexes`", which is, regrettably, too +long and scary. + +For example, installing the following environment +```nix +{ + packageOverrides = super: let self = super.pkgs; in + { + myHaskellEnv = self.haskellPackages.ghcWithHoogle + (haskellPackages: with haskellPackages; [ + # libraries + arrows async cgi criterion + # tools + cabal-install haskintex + ]); + }; +} +``` +allows one to browse module documentation index [not too dissimilar to +this](https://downloads.haskell.org/~ghc/latest/docs/html/libraries/index.html) +for all the specified packages and their dependencies by directing a browser of +choice to `~/.nix-profile/share/doc/hoogle/index.html` (or +`/run/current-system/sw/share/doc/hoogle/index.html` in case you put it in +`environment.systemPackages` in NixOS). + +After you've marveled enough at that try adding the following to your +`~/.ghc/ghci.conf` +``` +:def hoogle \s -> return $ ":! hoogle search -cl --count=15 \"" ++ s ++ "\"" +:def doc \s -> return $ ":! hoogle search -cl --info \"" ++ s ++ "\"" +``` +and test it by typing into `ghci`: +``` +:hoogle a -> a +:doc a -> a +``` + +Be sure to note the links to `haddock` files in the output. With any modern and +properly configured terminal emulator you can just click those links to +navigate there. + +Finally, you can run +```shell +hoogle server --local -p 8080 +``` +and navigate to http://localhost:8080/ for your own local +[Hoogle](https://www.haskell.org/hoogle/). The `--local` flag makes the hoogle +server serve files from your nix store over http, without the flag it will use +`file://` URIs. Note, however, that Firefox and possibly other browsers +disallow navigation from `http://` to `file://` URIs for security reasons, +which might be quite an inconvenience. Versions before v5 did not have this +flag. See +[this page](http://kb.mozillazine.org/Links_to_local_pages_do_not_work) for +workarounds. + +For NixOS users there's a service which runs this exact command for you. +Specify the `packages` you want documentation for and the `haskellPackages` set +you want them to come from. Add the following to `configuration.nix`. + +```nix +services.hoogle = { + enable = true; + packages = (hpkgs: with hpkgs; [text cryptonite]); + haskellPackages = pkgs.haskellPackages; +}; +``` + +### How to build a Haskell project using Stack + +[Stack](http://haskellstack.org) is a popular build tool for Haskell projects. +It has first-class support for Nix. Stack can optionally use Nix to +automatically select the right version of GHC and other build tools to build, +test and execute apps in an existing project downloaded from somewhere on the +Internet. Pass the `--nix` flag to any `stack` command to do so, e.g. +```shell +git clone --recurse-submodules https://github.com/yesodweb/wai.git +cd wai +stack --nix build +``` + +If you want `stack` to use Nix by default, you can add a `nix` section to the +`stack.yaml` file, as explained in the [Stack documentation][stack-nix-doc]. For +example: +```yaml +nix: + enable: true + packages: [pkgconfig zeromq zlib] +``` + +The example configuration snippet above tells Stack to create an ad hoc +environment for `nix-shell` as in the below section, in which the `pkgconfig`, +`zeromq` and `zlib` packages from Nixpkgs are available. All `stack` commands +will implicitly be executed inside this ad hoc environment. + +Some projects have more sophisticated needs. For examples, some ad hoc +environments might need to expose Nixpkgs packages compiled in a certain way, or +with extra environment variables. In these cases, you'll need a `shell` field +instead of `packages`: +```yaml +nix: + enable: true + shell-file: shell.nix +``` + +For more on how to write a `shell.nix` file see the below section. You'll need +to express a derivation. Note that Nixpkgs ships with a convenience wrapper +function around `mkDerivation` called `haskell.lib.buildStackProject` to help you +create this derivation in exactly the way Stack expects. However for this to work +you need to disable the sandbox, which you can do by using `--option sandbox relaxed` +or `--option sandbox false` to the Nix command. All of the same inputs +as `mkDerivation` can be provided. For example, to build a Stack project that +including packages that link against a version of the R library compiled with +special options turned on: +```nix +with (import <nixpkgs> { }); + +let R = pkgs.R.override { enableStrictBarrier = true; }; +in +haskell.lib.buildStackProject { + name = "HaskellR"; + buildInputs = [ R zeromq zlib ]; +} +``` + +You can select a particular GHC version to compile with by setting the +`ghc` attribute as an argument to `buildStackProject`. Better yet, let +Stack choose what GHC version it wants based on the snapshot specified +in `stack.yaml` (only works with Stack >= 1.1.3): +```nix +{nixpkgs ? import <nixpkgs> { }, ghc ? nixpkgs.ghc}: + +with nixpkgs; + +let R = pkgs.R.override { enableStrictBarrier = true; }; +in +haskell.lib.buildStackProject { + name = "HaskellR"; + buildInputs = [ R zeromq zlib ]; + inherit ghc; +} +``` + +[stack-nix-doc]: http://docs.haskellstack.org/en/stable/nix_integration.html + +### How to create ad hoc environments for `nix-shell` + +The easiest way to create an ad hoc development environment is to run +`nix-shell` with the appropriate GHC environment given on the command-line: +```shell +nix-shell -p "haskellPackages.ghcWithPackages (pkgs: with pkgs; [mtl pandoc])" +``` + +For more sophisticated use-cases, however, it's more convenient to save the +desired configuration in a file called `shell.nix` that looks like this: +```nix +{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc7102" }: +let + inherit (nixpkgs) pkgs; + ghc = pkgs.haskell.packages.${compiler}.ghcWithPackages (ps: with ps; [ + monad-par mtl + ]); +in +pkgs.stdenv.mkDerivation { + name = "my-haskell-env-0"; + buildInputs = [ ghc ]; + shellHook = "eval $(egrep ^export ${ghc}/bin/ghc)"; +} +``` + +Now run `nix-shell` --- or even `nix-shell --pure` --- to enter a shell +environment that has the appropriate compiler in `$PATH`. If you use `--pure`, +then add all other packages that your development environment needs into the +`buildInputs` attribute. If you'd like to switch to a different compiler +version, then pass an appropriate `compiler` argument to the expression, i.e. +`nix-shell --argstr compiler ghc784`. + +If you need such an environment because you'd like to compile a Hackage package +outside of Nix --- i.e. because you're hacking on the latest version from Git +---, then the package set provides suitable nix-shell environments for you +already! Every Haskell package has an `env` attribute that provides a shell +environment suitable for compiling that particular package. If you'd like to +hack the `lens` library, for example, then you just have to check out the +source code and enter the appropriate environment: +``` +$ cabal get lens-4.11 && cd lens-4.11 +Downloading lens-4.11... +Unpacking to lens-4.11/ + +$ nix-shell "<nixpkgs>" -A haskellPackages.lens.env +[nix-shell:/tmp/lens-4.11]$ +``` + +At point, you can run `cabal configure`, `cabal build`, and all the other +development commands. Note that you need `cabal-install` installed in your +`$PATH` already to use it here --- the `nix-shell` environment does not provide +it. + +## How to create Nix builds for your own private Haskell packages + +If your own Haskell packages have build instructions for Cabal, then you can +convert those automatically into build instructions for Nix using the +`cabal2nix` utility, which you can install into your profile by running +`nix-env -i cabal2nix`. + +### How to build a stand-alone project + +For example, let's assume that you're working on a private project called +`foo`. To generate a Nix build expression for it, change into the project's +top-level directory and run the command: +```shell +cabal2nix . > foo.nix +``` +Then write the following snippet into a file called `default.nix`: +```nix +{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc7102" }: +nixpkgs.pkgs.haskell.packages.${compiler}.callPackage ./foo.nix { } +``` + +Finally, store the following code in a file called `shell.nix`: +```nix +{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc7102" }: +(import ./default.nix { inherit nixpkgs compiler; }).env +``` + +At this point, you can run `nix-build` to have Nix compile your project and +install it into a Nix store path. The local directory will contain a symlink +called `result` after `nix-build` returns that points into that location. Of +course, passing the flag `--argstr compiler ghc763` allows switching the build +to any version of GHC currently supported. + +Furthermore, you can call `nix-shell` to enter an interactive development +environment in which you can use `cabal configure` and `cabal build` to develop +your code. That environment will automatically contain a proper GHC derivation +with all the required libraries registered as well as all the system-level +libraries your package might need. + +If your package does not depend on any system-level libraries, then it's +sufficient to run +```shell +nix-shell --command "cabal configure" +``` +once to set up your build. `cabal-install` determines the absolute paths to all +resources required for the build and writes them into a config file in the +`dist/` directory. Once that's done, you can run `cabal build` and any other +command for that project even outside of the `nix-shell` environment. This +feature is particularly nice for those of us who like to edit their code with +an IDE, like Emacs' `haskell-mode`, because it's not necessary to start Emacs +inside of nix-shell just to make it find out the necessary settings for +building the project; `cabal-install` has already done that for us. + +If you want to do some quick-and-dirty hacking and don't want to bother setting +up a `default.nix` and `shell.nix` file manually, then you can use the +`--shell` flag offered by `cabal2nix` to have it generate a stand-alone +`nix-shell` environment for you. With that feature, running +```shell +cabal2nix --shell . > shell.nix +nix-shell --command "cabal configure" +``` +is usually enough to set up a build environment for any given Haskell package. +You can even use that generated file to run `nix-build`, too: +```shell +nix-build shell.nix +``` + +### How to build projects that depend on each other + +If you have multiple private Haskell packages that depend on each other, then +you'll have to register those packages in the Nixpkgs set to make them visible +for the dependency resolution performed by `callPackage`. First of all, change +into each of your projects top-level directories and generate a `default.nix` +file with `cabal2nix`: +```shell +cd ~/src/foo && cabal2nix . > default.nix +cd ~/src/bar && cabal2nix . > default.nix +``` +Then edit your `~/.config/nixpkgs/config.nix` file to register those builds in the +default Haskell package set: +```nix +{ + packageOverrides = super: let self = super.pkgs; in + { + haskellPackages = super.haskellPackages.override { + overrides = self: super: { + foo = self.callPackage ../src/foo {}; + bar = self.callPackage ../src/bar {}; + }; + }; + }; +} +``` +Once that's accomplished, `nix-env -f "<nixpkgs>" -qA haskellPackages` will +show your packages like any other package from Hackage, and you can build them +```shell +nix-build "<nixpkgs>" -A haskellPackages.foo +``` +or enter an interactive shell environment suitable for building them: +```shell +nix-shell "<nixpkgs>" -A haskellPackages.bar.env +``` + +## Miscellaneous Topics + +### How to build with profiling enabled + +Every Haskell package set takes a function called `overrides` that you can use +to manipulate the package as much as you please. One useful application of this +feature is to replace the default `mkDerivation` function with one that enables +library profiling for all packages. To accomplish that add the following +snippet to your `~/.config/nixpkgs/config.nix` file: +```nix +{ + packageOverrides = super: let self = super.pkgs; in + { + profiledHaskellPackages = self.haskellPackages.override { + overrides = self: super: { + mkDerivation = args: super.mkDerivation (args // { + enableLibraryProfiling = true; + }); + }; + }; + }; +} +``` +Then, replace instances of `haskellPackages` in the `cabal2nix`-generated +`default.nix` or `shell.nix` files with `profiledHaskellPackages`. + +### How to override package versions in a compiler-specific package set + +Nixpkgs provides the latest version of +[`ghc-events`](http://hackage.haskell.org/package/ghc-events), which is 0.4.4.0 +at the time of this writing. This is fine for users of GHC 7.10.x, but GHC +7.8.4 cannot compile that binary. Now, one way to solve that problem is to +register an older version of `ghc-events` in the 7.8.x-specific package set. +The first step is to generate Nix build instructions with `cabal2nix`: +```shell +cabal2nix cabal://ghc-events-0.4.3.0 > ~/.nixpkgs/ghc-events-0.4.3.0.nix +``` +Then add the override in `~/.config/nixpkgs/config.nix`: +```nix +{ + packageOverrides = super: let self = super.pkgs; in + { + haskell = super.haskell // { + packages = super.haskell.packages // { + ghc784 = super.haskell.packages.ghc784.override { + overrides = self: super: { + ghc-events = self.callPackage ./ghc-events-0.4.3.0.nix {}; + }; + }; + }; + }; + }; +} +``` + +This code is a little crazy, no doubt, but it's necessary because the intuitive +version +```nix +{ # ... + + haskell.packages.ghc784 = super.haskell.packages.ghc784.override { + overrides = self: super: { + ghc-events = self.callPackage ./ghc-events-0.4.3.0.nix {}; + }; + }; +} +``` +doesn't do what we want it to: that code replaces the `haskell` package set in +Nixpkgs with one that contains only one entry,`packages`, which contains only +one entry `ghc784`. This override loses the `haskell.compiler` set, and it +loses the `haskell.packages.ghcXYZ` sets for all compilers but GHC 7.8.4. To +avoid that problem, we have to perform the convoluted little dance from above, +iterating over each step in hierarchy. + +Once it's accomplished, however, we can install a variant of `ghc-events` +that's compiled with GHC 7.8.4: +```shell +nix-env -f "<nixpkgs>" -iA haskell.packages.ghc784.ghc-events +``` +Unfortunately, it turns out that this build fails again while executing the +test suite! Apparently, the release archive on Hackage is missing some data +files that the test suite requires, so we cannot run it. We accomplish that by +re-generating the Nix expression with the `--no-check` flag: +```shell +cabal2nix --no-check cabal://ghc-events-0.4.3.0 > ~/.nixpkgs/ghc-events-0.4.3.0.nix +``` +Now the builds succeeds. + +Of course, in the concrete example of `ghc-events` this whole exercise is not +an ideal solution, because `ghc-events` can analyze the output emitted by any +version of GHC later than 6.12 regardless of the compiler version that was used +to build the `ghc-events` executable, so strictly speaking there's no reason to +prefer one built with GHC 7.8.x in the first place. However, for users who +cannot use GHC 7.10.x at all for some reason, the approach of downgrading to an +older version might be useful. + +### How to override packages in all compiler-specific package sets + +In the previous section we learned how to override a package in a single +compiler-specific package set. You may have some overrides defined that you want +to use across multiple package sets. To accomplish this you could use the +technique that we learned in the previous section by repeating the overrides for +all the compiler-specific package sets. For example: + +```nix +{ + packageOverrides = super: let self = super.pkgs; in + { + haskell = super.haskell // { + packages = super.haskell.packages // { + ghc784 = super.haskell.packages.ghc784.override { + overrides = self: super: { + my-package = ...; + my-other-package = ...; + }; + }; + ghc822 = super.haskell.packages.ghc784.override { + overrides = self: super: { + my-package = ...; + my-other-package = ...; + }; + }; + ... + }; + }; + }; +} +``` + +However there's a more convenient way to override all compiler-specific package +sets at once: + +```nix +{ + packageOverrides = super: let self = super.pkgs; in + { + haskell = super.haskell // { + packageOverrides = self: super: { + my-package = ...; + my-other-package = ...; + }; + }; + }; +} +``` + +### How to specify source overrides for your Haskell package + +When starting a Haskell project you can use `developPackage` +to define a derivation for your package at the `root` path +as well as source override versions for Hackage packages, like so: + +```nix +# default.nix +{ compilerVersion ? "ghc842" }: +let + # pinning nixpkgs using new Nix 2.0 builtin `fetchGit` + pkgs = import (fetchGit (import ./version.nix)) { }; + compiler = pkgs.haskell.packages."${compilerVersion}"; + pkg = compiler.developPackage { + root = ./.; + source-overrides = { + # Let's say the GHC 8.4.2 haskellPackages uses 1.6.0.0 and your test suite is incompatible with >= 1.6.0.0 + HUnit = "1.5.0.0"; + }; + }; +in pkg +``` + +This could be used in place of a simplified `stack.yaml` defining a Nix +derivation for your Haskell package. + +As you can see this allows you to specify only the source version found on +Hackage and nixpkgs will take care of the rest. + +You can also specify `buildInputs` for your Haskell derivation for packages +that directly depend on external libraries like so: + +```nix +# default.nix +{ compilerVersion ? "ghc842" }: +let + # pinning nixpkgs using new Nix 2.0 builtin `fetchGit` + pkgs = import (fetchGit (import ./version.nix)) { }; + compiler = pkgs.haskell.packages."${compilerVersion}"; + pkg = compiler.developPackage { + root = ./.; + source-overrides = { + HUnit = "1.5.0.0"; # Let's say the GHC 8.4.2 haskellPackages uses 1.6.0.0 and your test suite is incompatible with >= 1.6.0.0 + }; + }; + # in case your package source depends on any libraries directly, not just transitively. + buildInputs = [ zlib ]; +in pkg.overrideAttrs(attrs: { + buildInputs = attrs.buildInputs ++ buildInputs; +}) +``` + +Notice that you will need to override (via `overrideAttrs` or similar) the +derivation returned by the `developPackage` Nix lambda as there is no `buildInputs` +named argument you can pass directly into the `developPackage` lambda. + +### How to recover from GHC's infamous non-deterministic library ID bug + +GHC and distributed build farms don't get along well: + + - https://ghc.haskell.org/trac/ghc/ticket/4012 + +When you see an error like this one +``` +package foo-0.7.1.0 is broken due to missing package +text-1.2.0.4-98506efb1b9ada233bb5c2b2db516d91 +``` +then you have to download and re-install `foo` and all its dependents from +scratch: +```shell +nix-store -q --referrers /nix/store/*-haskell-text-1.2.0.4 \ + | xargs -L 1 nix-store --repair-path +``` + +If you're using additional Hydra servers other than `hydra.nixos.org`, then it +might be necessary to purge the local caches that store data from those +machines to disable these binary channels for the duration of the previous +command, i.e. by running: +```shell +rm ~/.cache/nix/binary-cache*.sqlite +``` + +### Builds on Darwin fail with `math.h` not found + +Users of GHC on Darwin have occasionally reported that builds fail, because the +compiler complains about a missing include file: +``` +fatal error: 'math.h' file not found +``` +The issue has been discussed at length in [ticket +6390](https://github.com/NixOS/nixpkgs/issues/6390), and so far no good +solution has been proposed. As a work-around, users who run into this problem +can configure the environment variables +```shell +export NIX_CFLAGS_COMPILE="-idirafter /usr/include" +export NIX_CFLAGS_LINK="-L/usr/lib" +``` +in their `~/.bashrc` file to avoid the compiler error. + +### Builds using Stack complain about missing system libraries + +``` +-- While building package zlib-0.5.4.2 using: + runhaskell -package=Cabal-1.22.4.0 -clear-package-db [... lots of flags ...] +Process exited with code: ExitFailure 1 +Logs have been written to: /home/foo/src/stack-ide/.stack-work/logs/zlib-0.5.4.2.log + +Configuring zlib-0.5.4.2... +Setup.hs: Missing dependency on a foreign library: +* Missing (or bad) header file: zlib.h +This problem can usually be solved by installing the system package that +provides this library (you may need the "-dev" version). If the library is +already installed but in a non-standard location then you can use the flags +--extra-include-dirs= and --extra-lib-dirs= to specify where it is. +If the header file does exist, it may contain errors that are caught by the C +compiler at the preprocessing stage. In this case you can re-run configure +with the verbosity flag -v3 to see the error messages. +``` + +When you run the build inside of the nix-shell environment, the system +is configured to find `libz.so` without any special flags -- the compiler +and linker "just know" how to find it. Consequently, Cabal won't record +any search paths for `libz.so` in the package description, which means +that the package works fine inside of nix-shell, but once you leave the +shell the shared object can no longer be found. That issue is by no +means specific to Stack: you'll have that problem with any other +Haskell package that's built inside of nix-shell but run outside of that +environment. + +You can remedy this issue in several ways. The easiest is to add a `nix` section +to the `stack.yaml` like the following: +```yaml +nix: + enable: true + packages: [ zlib ] +``` + +Stack's Nix support knows to add `${zlib.out}/lib` and `${zlib.dev}/include` +as an `--extra-lib-dirs` and `extra-include-dirs`, respectively. +Alternatively, you can achieve the same effect by hand. First of all, run +``` +$ nix-build --no-out-link "<nixpkgs>" -A zlib +/nix/store/alsvwzkiw4b7ip38l4nlfjijdvg3fvzn-zlib-1.2.8 +``` +to find out the store path of the system's zlib library. Now, you can + + 1. add that path (plus a "/lib" suffix) to your `$LD_LIBRARY_PATH` + environment variable to make sure your system linker finds `libz.so` + automatically. It's no pretty solution, but it will work. + + 2. As a variant of (1), you can also install any number of system + libraries into your user's profile (or some other profile) and point + `$LD_LIBRARY_PATH` to that profile instead, so that you don't have to + list dozens of those store paths all over the place. + + 3. The solution I prefer is to call stack with an appropriate + --extra-lib-dirs flag like so: + ```shell + stack --extra-lib-dirs=/nix/store/alsvwzkiw4b7ip38l4nlfjijdvg3fvzn-zlib-1.2.8/lib build + ``` + +Typically, you'll need `--extra-include-dirs` as well. It's possible +to add those flag to the project's `stack.yaml` or your user's +global `~/.stack/global/stack.yaml` file so that you don't have to +specify them manually every time. But again, you're likely better off +using Stack's Nix support instead. + +The same thing applies to `cabal configure`, of course, if you're +building with `cabal-install` instead of Stack. + +### Creating statically linked binaries + +There are two levels of static linking. The first option is to configure the +build with the Cabal flag `--disable-executable-dynamic`. In Nix expressions, +this can be achieved by setting the attribute: +``` +enableSharedExecutables = false; +``` +That gives you a binary with statically linked Haskell libraries and +dynamically linked system libraries. + +To link both Haskell libraries and system libraries statically, the additional +flags `--ghc-option=-optl=-static --ghc-option=-optl=-pthread` need to be used. +In Nix, this is accomplished with: +``` +configureFlags = [ "--ghc-option=-optl=-static" "--ghc-option=-optl=-pthread" ]; +``` + +It's important to realize, however, that most system libraries in Nix are +built as shared libraries only, i.e. there is just no static library +available that Cabal could link! + +### Building GHC with integer-simple + +By default GHC implements the Integer type using the +[GNU Multiple Precision Arithmetic (GMP) library](https://gmplib.org/). +The implementation can be found in the +[integer-gmp](http://hackage.haskell.org/package/integer-gmp) package. + +A potential problem with this is that GMP is licensed under the +[GNU Lesser General Public License (LGPL)](https://www.gnu.org/copyleft/lesser.html), +a kind of "copyleft" license. According to the terms of the LGPL, paragraph 5, +you may distribute a program that is designed to be compiled and dynamically +linked with the library under the terms of your choice (i.e., commercially) but +if your program incorporates portions of the library, if it is linked +statically, then your program is a "derivative"--a "work based on the +library"--and according to paragraph 2, section c, you "must cause the whole of +the work to be licensed" under the terms of the LGPL (including for free). + +The LGPL licensing for GMP is a problem for the overall licensing of binary +programs compiled with GHC because most distributions (and builds) of GHC use +static libraries. (Dynamic libraries are currently distributed only for macOS.) +The LGPL licensing situation may be worse: even though +[The Glasgow Haskell Compiler License](https://www.haskell.org/ghc/license) +is essentially a "free software" license (BSD3), according to +paragraph 2 of the LGPL, GHC must be distributed under the terms of the LGPL! + +To work around these problems GHC can be build with a slower but LGPL-free +alternative implementation for Integer called +[integer-simple](http://hackage.haskell.org/package/integer-simple). + +To get a GHC compiler build with `integer-simple` instead of `integer-gmp` use +the attribute: `haskell.compiler.integer-simple."${ghcVersion}"`. +For example: +``` +$ nix-build -E '(import <nixpkgs> {}).haskell.compiler.integer-simple.ghc802' +... +$ result/bin/ghc-pkg list | grep integer + integer-simple-0.1.1.1 +``` +The following command displays the complete list of GHC compilers build with `integer-simple`: +``` +$ nix-env -f "<nixpkgs>" -qaP -A haskell.compiler.integer-simple +haskell.compiler.integer-simple.ghc7102 ghc-7.10.2 +haskell.compiler.integer-simple.ghc7103 ghc-7.10.3 +haskell.compiler.integer-simple.ghc722 ghc-7.2.2 +haskell.compiler.integer-simple.ghc742 ghc-7.4.2 +haskell.compiler.integer-simple.ghc783 ghc-7.8.3 +haskell.compiler.integer-simple.ghc784 ghc-7.8.4 +haskell.compiler.integer-simple.ghc801 ghc-8.0.1 +haskell.compiler.integer-simple.ghc802 ghc-8.0.2 +haskell.compiler.integer-simple.ghcHEAD ghc-8.1.20170106 +``` + +To get a package set supporting `integer-simple` use the attribute: +`haskell.packages.integer-simple."${ghcVersion}"`. For example +use the following to get the `scientific` package build with `integer-simple`: +```shell +nix-build -A haskell.packages.integer-simple.ghc802.scientific +``` + +### Quality assurance + +The `haskell.lib` library includes a number of functions for checking for +various imperfections in Haskell packages. It's useful to apply these functions +to your own Haskell packages and integrate that in a Continuous Integration +server like [hydra](https://nixos.org/hydra/) to assure your packages maintain a +minimum level of quality. This section discusses some of these functions. + +#### failOnAllWarnings + +Applying `haskell.lib.failOnAllWarnings` to a Haskell package enables the +`-Wall` and `-Werror` GHC options to turn all warnings into build failures. + +#### buildStrictly + +Applying `haskell.lib.buildStrictly` to a Haskell package calls +`failOnAllWarnings` on the given package to turn all warnings into build +failures. Additionally the source of your package is gotten from first invoking +`cabal sdist` to ensure all needed files are listed in the Cabal file. + +#### checkUnusedPackages + +Applying `haskell.lib.checkUnusedPackages` to a Haskell package invokes +the [packunused](http://hackage.haskell.org/package/packunused) tool on the +package. `packunused` complains when it finds packages listed as build-depends +in the Cabal file which are redundant. For example: + +``` +$ nix-build -E 'let pkgs = import <nixpkgs> {}; in pkgs.haskell.lib.checkUnusedPackages {} pkgs.haskellPackages.scientific' +these derivations will be built: + /nix/store/3lc51cxj2j57y3zfpq5i69qbzjpvyci1-scientific-0.3.5.1.drv +... +detected package components +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + - library + - testsuite(s): test-scientific + - benchmark(s): bench-scientific* + +(component names suffixed with '*' are not configured to be built) + +library +~~~~~~~ + +The following package dependencies seem redundant: + + - ghc-prim-0.5.0.0 + +testsuite(test-scientific) +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +no redundant packages dependencies found + +builder for ‘/nix/store/3lc51cxj2j57y3zfpq5i69qbzjpvyci1-scientific-0.3.5.1.drv’ failed with exit code 1 +error: build of ‘/nix/store/3lc51cxj2j57y3zfpq5i69qbzjpvyci1-scientific-0.3.5.1.drv’ failed +``` + +As you can see, `packunused` finds out that although the testsuite component has +no redundant dependencies the library component of `scientific-0.3.5.1` depends +on `ghc-prim` which is unused in the library. + +### Using hackage2nix with nixpkgs + +Hackage package derivations are found in the +[`hackage-packages.nix`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/hackage-packages.nix) +file within `nixpkgs` and are used as the initial package set for +`haskellPackages`. The `hackage-packages.nix` file is not meant to be edited +by hand, but rather autogenerated by [`hackage2nix`](https://github.com/NixOS/cabal2nix/tree/master/hackage2nix), +which by default uses the [`configuration-hackage2nix.yaml`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/configuration-hackage2nix.yaml) +file to generate all the derivations. + +To modify the contents `configuration-hackage2nix.yaml`, follow the +instructions on [`hackage2nix`](https://github.com/NixOS/cabal2nix/tree/master/hackage2nix). + +## Other resources + + - The Youtube video [Nix Loves Haskell](https://www.youtube.com/watch?v=BsBhi_r-OeE) + provides an introduction into Haskell NG aimed at beginners. The slides are + available at http://cryp.to/nixos-meetup-3-slides.pdf and also -- in a form + ready for cut & paste -- at + https://github.com/NixOS/cabal2nix/blob/master/doc/nixos-meetup-3-slides.md. + + - Another Youtube video is [Escaping Cabal Hell with Nix](https://www.youtube.com/watch?v=mQd3s57n_2Y), + which discusses the subject of Haskell development with Nix but also provides + a basic introduction to Nix as well, i.e. it's suitable for viewers with + almost no prior Nix experience. + + - Oliver Charles wrote a very nice [Tutorial how to develop Haskell packages with Nix](http://wiki.ocharles.org.uk/Nix). + + - The *Journey into the Haskell NG infrastructure* series of postings + describe the new Haskell infrastructure in great detail: + + - [Part 1](https://nixos.org/nix-dev/2015-January/015591.html) + explains the differences between the old and the new code and gives + instructions how to migrate to the new setup. + + - [Part 2](https://nixos.org/nix-dev/2015-January/015608.html) + looks in-depth at how to tweak and configure your setup by means of + overrides. + + - [Part 3](https://nixos.org/nix-dev/2015-April/016912.html) + describes the infrastructure that keeps the Haskell package set in Nixpkgs + up-to-date. diff --git a/nixpkgs/doc/languages-frameworks/idris.section.md b/nixpkgs/doc/languages-frameworks/idris.section.md new file mode 100644 index 000000000000..f071b9ce1785 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/idris.section.md @@ -0,0 +1,144 @@ +# Idris + +## Installing Idris + +The easiest way to get a working idris version is to install the `idris` attribute: + +``` +$ # On NixOS +$ nix-env -i nixos.idris +$ # On non-NixOS +$ nix-env -i nixpkgs.idris +``` + +This however only provides the `prelude` and `base` libraries. To install idris with additional libraries, you can use the `idrisPackages.with-packages` function, e.g. in an overlay in `~/.config/nixpkgs/overlays/my-idris.nix`: + +```nix +self: super: { + myIdris = with self.idrisPackages; with-packages [ contrib pruviloj ]; +} +``` + +And then: + +``` +$ # On NixOS +$ nix-env -iA nixos.myIdris +$ # On non-NixOS +$ nix-env -iA nixpkgs.myIdris +``` + +To see all available Idris packages: +``` +$ # On NixOS +$ nix-env -qaPA nixos.idrisPackages +$ # On non-NixOS +$ nix-env -qaPA nixpkgs.idrisPackages +``` + +Similarly, entering a `nix-shell`: +``` +$ nix-shell -p 'idrisPackages.with-packages (with idrisPackages; [ contrib pruviloj ])' +``` + +## Starting Idris with library support + +To have access to these libraries in idris, call it with an argument `-p <library name>` for each library: + +``` +$ nix-shell -p 'idrisPackages.with-packages (with idrisPackages; [ contrib pruviloj ])' +[nix-shell:~]$ idris -p contrib -p pruviloj +``` + +A listing of all available packages the Idris binary has access to is available via `--listlibs`: + +``` +$ idris --listlibs +00prelude-idx.ibc +pruviloj +base +contrib +prelude +00pruviloj-idx.ibc +00base-idx.ibc +00contrib-idx.ibc +``` + +## Building an Idris project with Nix + +As an example of how a Nix expression for an Idris package can be created, here is the one for `idrisPackages.yaml`: + +```nix +{ build-idris-package +, fetchFromGitHub +, contrib +, lightyear +, lib +}: +build-idris-package { + name = "yaml"; + version = "2018-01-25"; + + # This is the .ipkg file that should be built, defaults to the package name + # In this case it should build `Yaml.ipkg` instead of `yaml.ipkg` + # This is only necessary because the yaml packages ipkg file is + # different from its package name here. + ipkgName = "Yaml"; + # Idris dependencies to provide for the build + idrisDeps = [ contrib lightyear ]; + + src = fetchFromGitHub { + owner = "Heather"; + repo = "Idris.Yaml"; + rev = "5afa51ffc839844862b8316faba3bafa15656db4"; + sha256 = "1g4pi0swmg214kndj85hj50ccmckni7piprsxfdzdfhg87s0avw7"; + }; + + meta = { + description = "Idris YAML lib"; + homepage = "https://github.com/Heather/Idris.Yaml"; + license = lib.licenses.mit; + maintainers = [ lib.maintainers.brainrape ]; + }; +} +``` + +Assuming this file is saved as `yaml.nix`, it's buildable using + +``` +$ nix-build -E '(import <nixpkgs> {}).idrisPackages.callPackage ./yaml.nix {}' +``` + +Or it's possible to use + +```nix +with import <nixpkgs> {}; + +{ + yaml = idrisPackages.callPackage ./yaml.nix {}; +} +``` + +in another file (say `default.nix`) to be able to build it with + +``` +$ nix-build -A yaml +``` + +## Passing options to `idris` commands + +The `build-idris-package` function provides also optional input values to set additional options for the used `idris` commands. + +Specifically, you can set `idrisBuildOptions`, `idrisTestOptions`, `idrisInstallOptions` and `idrisDocOptions` to provide additional options to the `idris` command respectively when building, testing, installing and generating docs for your package. + +For example you could set + +``` +build-idris-package { + idrisBuildOptions = [ "--log" "1" "--verbose" ] + + ... +} +``` + +to require verbose output during `idris` build phase. diff --git a/nixpkgs/doc/languages-frameworks/index.xml b/nixpkgs/doc/languages-frameworks/index.xml new file mode 100644 index 000000000000..728a38c264a3 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/index.xml @@ -0,0 +1,34 @@ +<chapter xmlns="http://docbook.org/ns/docbook" + xmlns:xi="http://www.w3.org/2001/XInclude" + xml:id="chap-language-support"> + <title>Languages and frameworks</title> + <para> + The <link linkend="chap-stdenv">standard build environment</link> makes it easy to build typical Autotools-based packages with very little code. Any other kind of package can be accomodated by overriding the appropriate phases of <literal>stdenv</literal>. However, there are specialised functions in Nixpkgs to easily build packages for other programming languages, such as Perl or Haskell. These are described in this chapter. + </para> + <xi:include href="agda.section.xml" /> + <xi:include href="android.section.xml" /> + <xi:include href="beam.xml" /> + <xi:include href="bower.xml" /> + <xi:include href="coq.xml" /> + <xi:include href="crystal.section.xml" /> + <xi:include href="emscripten.section.xml" /> + <xi:include href="gnome.xml" /> + <xi:include href="go.xml" /> + <xi:include href="haskell.section.xml" /> + <xi:include href="idris.section.xml" /> + <xi:include href="ios.section.xml" /> + <xi:include href="java.xml" /> + <xi:include href="lua.section.xml" /> + <xi:include href="node.section.xml" /> + <xi:include href="ocaml.xml" /> + <xi:include href="perl.xml" /> + <xi:include href="php.section.xml" /> + <xi:include href="python.section.xml" /> + <xi:include href="qt.xml" /> + <xi:include href="r.section.xml" /> + <xi:include href="ruby.xml" /> + <xi:include href="rust.section.xml" /> + <xi:include href="texlive.xml" /> + <xi:include href="titanium.section.xml" /> + <xi:include href="vim.section.xml" /> +</chapter> diff --git a/nixpkgs/doc/languages-frameworks/ios.section.md b/nixpkgs/doc/languages-frameworks/ios.section.md new file mode 100644 index 000000000000..768e0690b962 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/ios.section.md @@ -0,0 +1,229 @@ +--- +title: iOS +author: Sander van der Burg +date: 2019-11-10 +--- +# iOS + +This component is basically a wrapper/workaround that makes it possible to +expose an Xcode installation as a Nix package by means of symlinking to the +relevant executables on the host system. + +Since Xcode can't be packaged with Nix, nor we can publish it as a Nix package +(because of its license) this is basically the only integration strategy +making it possible to do iOS application builds that integrate with other +components of the Nix ecosystem + +The primary objective of this project is to use the Nix expression language to +specify how iOS apps can be built from source code, and to automatically spawn +iOS simulator instances for testing. + +This component also makes it possible to use [Hydra](https://nixos.org/hydra), +the Nix-based continuous integration server to regularly build iOS apps and to +do wireless ad-hoc installations of enterprise IPAs on iOS devices through +Hydra. + +The Xcode build environment implements a number of features. + +Deploying a proxy component wrapper exposing Xcode +-------------------------------------------------- +The first use case is deploying a Nix package that provides symlinks to the Xcode +installation on the host system. This package can be used as a build input to +any build function implemented in the Nix expression language that requires +Xcode. + +```nix +let + pkgs = import <nixpkgs> {}; + + xcodeenv = import ./xcodeenv { + inherit (pkgs) stdenv; + }; +in +xcodeenv.composeXcodeWrapper { + version = "9.2"; + xcodeBaseDir = "/Applications/Xcode.app"; +} +``` + +By deploying the above expression with `nix-build` and inspecting its content +you will notice that several Xcode-related executables are exposed as a Nix +package: + +```bash +$ ls result/bin +lrwxr-xr-x 1 sander staff 94 1 jan 1970 Simulator -> /Applications/Xcode.app/Contents/Developer/Applications/Simulator.app/Contents/MacOS/Simulator +lrwxr-xr-x 1 sander staff 17 1 jan 1970 codesign -> /usr/bin/codesign +lrwxr-xr-x 1 sander staff 17 1 jan 1970 security -> /usr/bin/security +lrwxr-xr-x 1 sander staff 21 1 jan 1970 xcode-select -> /usr/bin/xcode-select +lrwxr-xr-x 1 sander staff 61 1 jan 1970 xcodebuild -> /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild +lrwxr-xr-x 1 sander staff 14 1 jan 1970 xcrun -> /usr/bin/xcrun +``` + +Building an iOS application +--------------------------- +We can build an iOS app executable for the simulator, or an IPA/xcarchive file +for release purposes, e.g. ad-hoc, enterprise or store installations, by +executing the `xcodeenv.buildApp {}` function: + +```nix +let + pkgs = import <nixpkgs> {}; + + xcodeenv = import ./xcodeenv { + inherit (pkgs) stdenv; + }; +in +xcodeenv.buildApp { + name = "MyApp"; + src = ./myappsources; + sdkVersion = "11.2"; + + target = null; # Corresponds to the name of the app by default + configuration = null; # Release for release builds, Debug for debug builds + scheme = null; # -scheme will correspond to the app name by default + sdk = null; # null will set it to 'iphonesimulator` for simulator builds or `iphoneos` to real builds + xcodeFlags = ""; + + release = true; + certificateFile = ./mycertificate.p12; + certificatePassword = "secret"; + provisioningProfile = ./myprovisioning.profile; + signMethod = "ad-hoc"; # 'enterprise' or 'store' + generateIPA = true; + generateXCArchive = false; + + enableWirelessDistribution = true; + installURL = "/installipa.php"; + bundleId = "mycompany.myapp"; + appVersion = "1.0"; + + # Supports all xcodewrapper parameters as well + xcodeBaseDir = "/Applications/Xcode.app"; +} +``` + +The above function takes a variety of parameters: +* The `name` and `src` parameters are mandatory and specify the name of the app + and the location where the source code resides +* `sdkVersion` specifies which version of the iOS SDK to use. + +It also possile to adjust the `xcodebuild` parameters. This is only needed in +rare circumstances. In most cases the default values should suffice: + +* Specifies which `xcodebuild` target to build. By default it takes the target + that has the same name as the app. +* The `configuration` parameter can be overridden if desired. By default, it + will do a debug build for the simulator and a release build for real devices. +* The `scheme` parameter specifies which `-scheme` parameter to propagate to + `xcodebuild`. By default, it corresponds to the app name. +* The `sdk` parameter specifies which SDK to use. By default, it picks + `iphonesimulator` for simulator builds and `iphoneos` for release builds. +* The `xcodeFlags` parameter specifies arbitrary command line parameters that + should be propagated to `xcodebuild`. + +By default, builds are carried out for the iOS simulator. To do release builds +(builds for real iOS devices), you must set the `release` parameter to `true`. +In addition, you need to set the following parameters: + +* `certificateFile` refers to a P12 certificate file. +* `certificatePassword` specifies the password of the P12 certificate. +* `provisioningProfile` refers to the provision profile needed to sign the app +* `signMethod` should refer to `ad-hoc` for signing the app with an ad-hoc + certificate, `enterprise` for enterprise certificates and `app-store` for App + store certificates. +* `generateIPA` specifies that we want to produce an IPA file (this is probably + what you want) +* `generateXCArchive` specifies thet we want to produce an xcarchive file. + +When building IPA files on Hydra and when it is desired to allow iOS devices to +install IPAs by browsing to the Hydra build products page, you can enable the +`enableWirelessDistribution` parameter. + +When enabled, you need to configure the following options: + +* The `installURL` parameter refers to the URL of a PHP script that composes the + `itms-services://` URL allowing iOS devices to install the IPA file. +* `bundleId` refers to the bundle ID value of the app +* `appVersion` refers to the app's version number + +To use wireless adhoc distributions, you must also install the corresponding +PHP script on a web server (see section: 'Installing the PHP script for wireless +ad hoc installations from Hydra' for more information). + +In addition to the build parameters, you can also specify any parameters that +the `xcodeenv.composeXcodeWrapper {}` function takes. For example, the +`xcodeBaseDir` parameter can be overridden to refer to a different Xcode +version. + +Spawning simulator instances +---------------------------- +In addition to building iOS apps, we can also automatically spawn simulator +instances: + +```nix +let + pkgs = import <nixpkgs> {}; + + xcodeenv = import ./xcodeenv { + inherit (pkgs) stdenv; + }; +in +xcode.simulateApp { + name = "simulate"; + + # Supports all xcodewrapper parameters as well + xcodeBaseDir = "/Applications/Xcode.app"; +} +``` + +The above expression produces a script that starts the simulator from the +provided Xcode installation. The script can be started as follows: + +```bash +./result/bin/run-test-simulator +``` + +By default, the script will show an overview of UDID for all available simulator +instances and asks you to pick one. You can also provide a UDID as a +command-line parameter to launch an instance automatically: + +```bash +./result/bin/run-test-simulator 5C93129D-CF39-4B1A-955F-15180C3BD4B8 +``` + +You can also extend the simulator script to automatically deploy and launch an +app in the requested simulator instance: + +```nix +let + pkgs = import <nixpkgs> {}; + + xcodeenv = import ./xcodeenv { + inherit (pkgs) stdenv; + }; +in +xcode.simulateApp { + name = "simulate"; + bundleId = "mycompany.myapp"; + app = xcode.buildApp { + # ... + }; + + # Supports all xcodewrapper parameters as well + xcodeBaseDir = "/Applications/Xcode.app"; +} +``` + +By providing the result of an `xcode.buildApp {}` function and configuring the +app bundle id, the app gets deployed automatically and started. + +Troubleshooting +--------------- +In some rare cases, it may happen that after a failure, changes are not picked +up. Most likely, this is caused by a derived data cache that Xcode maintains. +To wipe it you can run: + +```bash +$ rm -rf ~/Library/Developer/Xcode/DerivedData +``` diff --git a/nixpkgs/doc/languages-frameworks/java.xml b/nixpkgs/doc/languages-frameworks/java.xml new file mode 100644 index 000000000000..bf0fc4883922 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/java.xml @@ -0,0 +1,63 @@ +<section xmlns="http://docbook.org/ns/docbook" + xmlns:xlink="http://www.w3.org/1999/xlink" + xml:id="sec-language-java"> + <title>Java</title> + + <para> + Ant-based Java packages are typically built from source as follows: +<programlisting> +stdenv.mkDerivation { + name = "..."; + src = fetchurl { ... }; + + nativeBuildInputs = [ jdk ant ]; + + buildPhase = "ant"; +} +</programlisting> + Note that <varname>jdk</varname> is an alias for the OpenJDK (self-built where available, or pre-built via Zulu). Platforms with OpenJDK not (yet) in Nixpkgs (<literal>Aarch32</literal>, <literal>Aarch64</literal>) point to the (unfree) <literal>oraclejdk</literal>. + </para> + + <para> + JAR files that are intended to be used by other packages should be installed in <filename>$out/share/java</filename>. JDKs have a stdenv setup hook that add any JARs in the <filename>share/java</filename> directories of the build inputs to the <envar>CLASSPATH</envar> environment variable. For instance, if the package <literal>libfoo</literal> installs a JAR named <filename>foo.jar</filename> in its <filename>share/java</filename> directory, and another package declares the attribute +<programlisting> +buildInputs = [ libfoo ]; +nativeBuildInputs = [ jdk ]; +</programlisting> + then <envar>CLASSPATH</envar> will be set to <filename>/nix/store/...-libfoo/share/java/foo.jar</filename>. + </para> + + <para> + Private JARs should be installed in a location like <filename>$out/share/<replaceable>package-name</replaceable></filename>. + </para> + + <para> + If your Java package provides a program, you need to generate a wrapper script to run it using the OpenJRE. You can use <literal>makeWrapper</literal> for this: +<programlisting> +nativeBuildInputs = [ makeWrapper ]; + +installPhase = + '' + mkdir -p $out/bin + makeWrapper ${jre}/bin/java $out/bin/foo \ + --add-flags "-cp $out/share/java/foo.jar org.foo.Main" + ''; +</programlisting> + Note the use of <literal>jre</literal>, which is the part of the OpenJDK package that contains the Java Runtime Environment. By using <literal>${jre}/bin/java</literal> instead of <literal>${jdk}/bin/java</literal>, you prevent your package from depending on the JDK at runtime. + </para> + + <para> + Note all JDKs passthru <literal>home</literal>, so if your application requires environment variables like <envar>JAVA_HOME</envar> being set, that can be done in a generic fashion with the <literal>--set</literal> argument of <literal>makeWrapper</literal>: +<programlisting> +--set JAVA_HOME ${jdk.home} +</programlisting> + </para> + + <para> + It is possible to use a different Java compiler than <command>javac</command> from the OpenJDK. For instance, to use the GNU Java Compiler: +<programlisting> +nativeBuildInputs = [ gcj ant ]; +</programlisting> + Here, Ant will automatically use <command>gij</command> (the GNU Java Runtime) instead of the OpenJRE. + </para> +</section> diff --git a/nixpkgs/doc/languages-frameworks/lua.section.md b/nixpkgs/doc/languages-frameworks/lua.section.md new file mode 100644 index 000000000000..a0e9917b8ec5 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/lua.section.md @@ -0,0 +1,252 @@ +--- +title: Lua +author: Matthieu Coudron +date: 2019-02-05 +--- + +# User's Guide to Lua Infrastructure + +## Using Lua + +### Overview of Lua + +Several versions of the Lua interpreter are available: luajit, lua 5.1, 5.2, 5.3. +The attribute `lua` refers to the default interpreter, it is also possible to refer to specific versions, e.g. `lua5_2` refers to Lua 5.2. + +Lua libraries are in separate sets, with one set per interpreter version. + +The interpreters have several common attributes. One of these attributes is +`pkgs`, which is a package set of Lua libraries for this specific +interpreter. E.g., the `busted` package corresponding to the default interpreter +is `lua.pkgs.busted`, and the lua 5.2 version is `lua5_2.pkgs.busted`. +The main package set contains aliases to these package sets, e.g. +`luaPackages` refers to `lua5_1.pkgs` and `lua52Packages` to +`lua5_2.pkgs`. + +### Installing Lua and packages + +#### Lua environment defined in separate `.nix` file + +Create a file, e.g. `build.nix`, with the following expression +```nix +with import <nixpkgs> {}; + +lua5_2.withPackages (ps: with ps; [ busted luafilesystem ]) +``` +and install it in your profile with +```shell +nix-env -if build.nix +``` +Now you can use the Lua interpreter, as well as the extra packages (`busted`, +`luafilesystem`) that you added to the environment. + +#### Lua environment defined in `~/.config/nixpkgs/config.nix` + +If you prefer to, you could also add the environment as a package override to the Nixpkgs set, e.g. +using `config.nix`, +```nix +{ # ... + + packageOverrides = pkgs: with pkgs; { + myLuaEnv = lua5_2.withPackages (ps: with ps; [ busted luafilesystem ]); + }; +} +``` +and install it in your profile with +```shell +nix-env -iA nixpkgs.myLuaEnv +``` +The environment is is installed by referring to the attribute, and considering +the `nixpkgs` channel was used. + +#### Lua environment defined in `/etc/nixos/configuration.nix` + +For the sake of completeness, here's another example how to install the environment system-wide. + +```nix +{ # ... + + environment.systemPackages = with pkgs; [ + (lua.withPackages(ps: with ps; [ busted luafilesystem ])) + ]; +} +``` + +### How to override a Lua package using overlays? + +Use the following overlay template: + +```nix +final: prev: +{ + + lua = prev.lua.override { + packageOverrides = luaself: luaprev: { + + luarocks-nix = luaprev.luarocks-nix.overrideAttrs(oa: { + pname = "luarocks-nix"; + src = /home/my_luarocks/repository; + }); + }; + + luaPackages = lua.pkgs; +} +``` + +### Temporary Lua environment with `nix-shell` + + +There are two methods for loading a shell with Lua packages. The first and recommended method +is to create an environment with `lua.buildEnv` or `lua.withPackages` and load that. E.g. +```sh +$ nix-shell -p 'lua.withPackages(ps: with ps; [ busted luafilesystem ])' +``` +opens a shell from which you can launch the interpreter +```sh +[nix-shell:~] lua +``` +The other method, which is not recommended, does not create an environment and requires you to list the packages directly, + +```sh +$ nix-shell -p lua.pkgs.busted lua.pkgs.luafilesystem +``` +Again, it is possible to launch the interpreter from the shell. +The Lua interpreter has the attribute `pkgs` which contains all Lua libraries for that specific interpreter. + + +## Developing with Lua + +Now that you know how to get a working Lua environment with Nix, it is time +to go forward and start actually developing with Lua. There are two ways to +package lua software, either it is on luarocks and most of it can be taken care +of by the luarocks2nix converter or the packaging has to be done manually. +Let's present the luarocks way first and the manual one in a second time. + +### Packaging a library on luarocks + +[Luarocks.org](www.luarocks.org) is the main repository of lua packages. +The site proposes two types of packages, the rockspec and the src.rock +(equivalent of a [rockspec](https://github.com/luarocks/luarocks/wiki/Rockspec-format) but with the source). +These packages can have different build types such as `cmake`, `builtin` etc . + +Luarocks-based packages are generated in pkgs/development/lua-modules/generated-packages.nix from +the whitelist maintainers/scripts/luarocks-packages.csv and updated by running maintainers/scripts/update-luarocks-packages. + +[luarocks2nix](https://github.com/nix-community/luarocks) is a tool capable of generating nix derivations from both rockspec and src.rock (and favors the src.rock). +The automation only goes so far though and some packages need to be customized. +These customizations go in `pkgs/development/lua-modules/overrides.nix`. +For instance if the rockspec defines `external_dependencies`, these need to be manually added in in its rockspec file then it won't work. + +You can try converting luarocks packages to nix packages with the command `nix-shell -p luarocks-nix` and then `luarocks nix PKG_NAME`. +Nix rely on luarocks to install lua packages, basically it runs: +`luarocks make --deps-mode=none --tree $out` + +#### Packaging a library manually + +You can develop your package as you usually would, just don't forget to wrap it +within a `toLuaModule` call, for instance +```nix +mynewlib = toLuaModule ( stdenv.mkDerivation { ... }); +``` + +There is also the `buildLuaPackage` function that can be used when lua modules +are not packaged for luarocks. You can see a few examples at `pkgs/top-level/lua-packages.nix`. + +## Lua Reference + +### Lua interpreters + +Versions 5.1, 5.2 and 5.3 of the lua interpreter are available as +respectively `lua5_1`, `lua5_2` and `lua5_3`. Luajit is available too. +The Nix expressions for the interpreters can be found in `pkgs/development/interpreters/lua-5`. + + +#### Attributes on lua interpreters packages + +Each interpreter has the following attributes: + +- `interpreter`. Alias for `${pkgs.lua}/bin/lua`. +- `buildEnv`. Function to build lua interpreter environments with extra packages bundled together. See section *lua.buildEnv function* for usage and documentation. +- `withPackages`. Simpler interface to `buildEnv`. +- `pkgs`. Set of Lua packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`. + + +#### `buildLuarocksPackage` function + +The `buildLuarocksPackage` function is implemented in `pkgs/development/interpreters/lua-5/build-lua-package.nix` +The following is an example: +```nix +luaposix = buildLuarocksPackage { + pname = "luaposix"; + version = "34.0.4-1"; + + src = fetchurl { + url = "https://raw.githubusercontent.com/rocks-moonscript-org/moonrocks-mirror/master/luaposix-34.0.4-1.src.rock"; + sha256 = "0yrm5cn2iyd0zjd4liyj27srphvy0gjrjx572swar6zqr4dwjqp2"; + }; + disabled = (luaOlder "5.1") || (luaAtLeast "5.4"); + propagatedBuildInputs = [ bit32 lua std_normalize ]; + + meta = with stdenv.lib; { + homepage = "https://github.com/luaposix/luaposix/"; + description = "Lua bindings for POSIX"; + maintainers = with maintainers; [ vyp lblasc ]; + license.fullName = "MIT/X11"; + }; +}; +``` + +The `buildLuarocksPackage` delegates most tasks to luarocks: + +* it adds `luarocks` as an unpacker for `src.rock` files (zip files really). +* configurePhase` writes a temporary luarocks configuration file which location +is exported via the environment variable `LUAROCKS_CONFIG`. +* the `buildPhase` does nothing. +* `installPhase` calls `luarocks make --deps-mode=none --tree $out` to build and +install the package +* In the `postFixup` phase, the `wrapLuaPrograms` bash function is called to + wrap all programs in the `$out/bin/*` directory to include `$PATH` + environment variable and add dependent libraries to script's `LUA_PATH` and + `LUA_CPATH`. + +By default `meta.platforms` is set to the same value as the interpreter unless overridden otherwise. + +#### `buildLuaApplication` function + +The `buildLuaApplication` function is practically the same as `buildLuaPackage`. +The difference is that `buildLuaPackage` by default prefixes the names of the packages with the version of the interpreter. +Because with an application we're not interested in multiple version the prefix is dropped. + +#### lua.withPackages function + +The `lua.withPackages` takes a function as an argument that is passed the set of lua packages and returns the list of packages to be included in the environment. +Using the `withPackages` function, the previous example for the luafilesystem environment can be written like this: +```nix +with import <nixpkgs> {}; + +lua.withPackages (ps: [ps.luafilesystem]) +``` + +`withPackages` passes the correct package set for the specific interpreter version as an argument to the function. In the above example, `ps` equals `luaPackages`. +But you can also easily switch to using `lua5_2`: +```nix +with import <nixpkgs> {}; + +lua5_2.withPackages (ps: [ps.lua]) +``` + +Now, `ps` is set to `lua52Packages`, matching the version of the interpreter. + + +### Possible Todos + +* export/use version specific variables such as `LUA_PATH_5_2`/`LUAROCKS_CONFIG_5_2` +* let luarocks check for dependencies via exporting the different rocktrees in temporary config + +### Lua Contributing guidelines + +Following rules should be respected: + +* Make sure libraries build for all Lua interpreters. +* Commit names of Lua libraries should reflect that they are Lua libraries, so write for example `luaPackages.luafilesystem: 1.11 -> 1.12`. + diff --git a/nixpkgs/doc/languages-frameworks/node.section.md b/nixpkgs/doc/languages-frameworks/node.section.md new file mode 100644 index 000000000000..c1f4294711a1 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/node.section.md @@ -0,0 +1,50 @@ +Node.js +======= +The `pkgs/development/node-packages` folder contains a generated collection of +[NPM packages](https://npmjs.com/) that can be installed with the Nix package +manager. + +As a rule of thumb, the package set should only provide *end user* software +packages, such as command-line utilities. Libraries should only be added to the +package set if there is a non-NPM package that requires it. + +When it is desired to use NPM libraries in a development project, use the +`node2nix` generator directly on the `package.json` configuration file of the +project. + +The package set provides support for the official stable Node.js versions. +The latest stable LTS release in `nodePackages`, as well as the latest stable +Current release in `nodePackages_latest`. + +If your package uses native addons, you need to examine what kind of native +build system it uses. Here are some examples: + +* `node-gyp` +* `node-gyp-builder` +* `node-pre-gyp` + +After you have identified the correct system, you need to override your package +expression while adding in build system as a build input. For example, `dat` +requires `node-gyp-build`, so we override its expression in `default.nix`: + +```nix +dat = nodePackages.dat.override (oldAttrs: { + buildInputs = oldAttrs.buildInputs ++ [ nodePackages.node-gyp-build ]; +}); +``` + +To add a package from NPM to nixpkgs: + + 1. Modify `pkgs/development/node-packages/node-packages.json` to add, update + or remove package entries to have it included in `nodePackages` and + `nodePackages_latest`. + 2. Run the script: `(cd pkgs/development/node-packages && ./generate.sh)`. + 3. Build your new package to test your changes: + `cd /path/to/nixpkgs && nix-build -A nodePackages.<new-or-updated-package>`. + To build against the latest stable Current Node.js version (e.g. 14.x): + `nix-build -A nodePackages_latest.<new-or-updated-package>` + 4. Add and commit all modified and generated files. + +For more information about the generation process, consult the +[README.md](https://github.com/svanderburg/node2nix) file of the `node2nix` +tool. diff --git a/nixpkgs/doc/languages-frameworks/ocaml.xml b/nixpkgs/doc/languages-frameworks/ocaml.xml new file mode 100644 index 000000000000..3f72092ec150 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/ocaml.xml @@ -0,0 +1,73 @@ +<section xmlns="http://docbook.org/ns/docbook" + xmlns:xlink="http://www.w3.org/1999/xlink" + xml:id="sec-language-ocaml"> + <title>OCaml</title> + + <para> + OCaml libraries should be installed in <literal>$(out)/lib/ocaml/${ocaml.version}/site-lib/</literal>. Such directories are automatically added to the <literal>$OCAMLPATH</literal> environment variable when building another package that depends on them or when opening a <literal>nix-shell</literal>. + </para> + + <para> + Given that most of the OCaml ecosystem is now built with dune, nixpkgs includes a convenience build support function called <literal>buildDunePackage</literal> that will build an OCaml package using dune, OCaml and findlib and any additional dependencies provided as <literal>buildInputs</literal> or <literal>propagatedBuildInputs</literal>. + </para> + + <para> + Here is a simple package example. It defines an (optional) attribute <literal>minimumOCamlVersion</literal> that will be used to throw a descriptive evaluation error if building with an older OCaml is attempted. It uses the <literal>fetchFromGitHub</literal> fetcher to get its source. It sets the <literal>doCheck</literal> (optional) attribute to <literal>true</literal> which means that tests will be run with <literal>dune runtest -p angstrom</literal> after the build (<literal>dune build -p angstrom</literal>) is complete. It uses <literal>alcotest</literal> as a build input (because it is needed to run the tests) and <literal>bigstringaf</literal> and <literal>result</literal> as propagated build inputs (thus they will also be available to libraries depending on this library). The library will be installed using the <literal>angstrom.install</literal> file that dune generates. + </para> + +<programlisting> +{ stdenv, fetchFromGitHub, buildDunePackage, alcotest, result, bigstringaf }: + +buildDunePackage rec { + pname = "angstrom"; + version = "0.10.0"; + + minimumOCamlVersion = "4.03"; + + src = fetchFromGitHub { + owner = "inhabitedtype"; + repo = pname; + rev = version; + sha256 = "0lh6024yf9ds0nh9i93r9m6p5psi8nvrqxl5x7jwl13zb0r9xfpw"; + }; + + buildInputs = [ alcotest ]; + propagatedBuildInputs = [ bigstringaf result ]; + doCheck = true; + + meta = { + homepage = "https://github.com/inhabitedtype/angstrom"; + description = "OCaml parser combinators built for speed and memory efficiency"; + license = stdenv.lib.licenses.bsd3; + maintainers = with stdenv.lib.maintainers; [ sternenseemann ]; + }; +} +</programlisting> + + <para> + Here is a second example, this time using a source archive generated with <literal>dune-release</literal>. It is a good idea to use this archive when it is available as it will usually contain substituted variables such as a <literal>%%VERSION%%</literal> field. This library does not depend on any other OCaml library and no tests are run after building it. + </para> + +<programlisting> +{ stdenv, fetchurl, buildDunePackage }: + +buildDunePackage rec { + pname = "wtf8"; + version = "1.0.1"; + + minimumOCamlVersion = "4.01"; + + src = fetchurl { + url = "https://github.com/flowtype/ocaml-${pname}/releases/download/v${version}/${pname}-${version}.tbz"; + sha256 = "1msg3vycd3k8qqj61sc23qks541cxpb97vrnrvrhjnqxsqnh6ygq"; + }; + + meta = with stdenv.lib; { + homepage = "https://github.com/flowtype/ocaml-wtf8"; + description = "WTF-8 is a superset of UTF-8 that allows unpaired surrogates."; + license = licenses.mit; + maintainers = [ maintainers.eqyiel ]; + }; +} +</programlisting> +</section> diff --git a/nixpkgs/doc/languages-frameworks/perl.xml b/nixpkgs/doc/languages-frameworks/perl.xml new file mode 100644 index 000000000000..d9b6b2721c67 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/perl.xml @@ -0,0 +1,161 @@ +<section xmlns="http://docbook.org/ns/docbook" + xmlns:xlink="http://www.w3.org/1999/xlink" + xml:id="sec-language-perl"> + <title>Perl</title> + + <para> + Nixpkgs provides a function <varname>buildPerlPackage</varname>, a generic package builder function for any Perl package that has a standard <varname>Makefile.PL</varname>. It’s implemented in <link +xlink:href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/perl-modules/generic"><filename>pkgs/development/perl-modules/generic</filename></link>. + </para> + + <para> + Perl packages from CPAN are defined in <link +xlink:href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/perl-packages.nix"><filename>pkgs/top-level/perl-packages.nix</filename></link>, rather than <filename>pkgs/all-packages.nix</filename>. Most Perl packages are so straight-forward to build that they are defined here directly, rather than having a separate function for each package called from <filename>perl-packages.nix</filename>. However, more complicated packages should be put in a separate file, typically in <filename>pkgs/development/perl-modules</filename>. Here is an example of the former: +<programlisting> +ClassC3 = buildPerlPackage rec { + name = "Class-C3-0.21"; + src = fetchurl { + url = "mirror://cpan/authors/id/F/FL/FLORA/${name}.tar.gz"; + sha256 = "1bl8z095y4js66pwxnm7s853pi9czala4sqc743fdlnk27kq94gz"; + }; +}; +</programlisting> + Note the use of <literal>mirror://cpan/</literal>, and the <literal>${name}</literal> in the URL definition to ensure that the name attribute is consistent with the source that we’re actually downloading. Perl packages are made available in <filename>all-packages.nix</filename> through the variable <varname>perlPackages</varname>. For instance, if you have a package that needs <varname>ClassC3</varname>, you would typically write +<programlisting> +foo = import ../path/to/foo.nix { + inherit stdenv fetchurl ...; + inherit (perlPackages) ClassC3; +}; +</programlisting> + in <filename>all-packages.nix</filename>. You can test building a Perl package as follows: +<screen> +<prompt>$ </prompt>nix-build -A perlPackages.ClassC3 +</screen> + <varname>buildPerlPackage</varname> adds <literal>perl-</literal> to the start of the name attribute, so the package above is actually called <literal>perl-Class-C3-0.21</literal>. So to install it, you can say: +<screen> +<prompt>$ </prompt>nix-env -i perl-Class-C3 +</screen> + (Of course you can also install using the attribute name: <literal>nix-env -i -A perlPackages.ClassC3</literal>.) + </para> + + <para> + So what does <varname>buildPerlPackage</varname> do? It does the following: + <orderedlist> + <listitem> + <para> + In the configure phase, it calls <literal>perl Makefile.PL</literal> to generate a Makefile. You can set the variable <varname>makeMakerFlags</varname> to pass flags to <filename>Makefile.PL</filename> + </para> + </listitem> + <listitem> + <para> + It adds the contents of the <envar>PERL5LIB</envar> environment variable to <literal>#! .../bin/perl</literal> line of Perl scripts as <literal>-I<replaceable>dir</replaceable></literal> flags. This ensures that a script can find its dependencies. (This can cause this shebang line to become too long for Darwin to handle; see the note below.) + </para> + </listitem> + <listitem> + <para> + In the fixup phase, it writes the propagated build inputs (<varname>propagatedBuildInputs</varname>) to the file <filename>$out/nix-support/propagated-user-env-packages</filename>. <command>nix-env</command> recursively installs all packages listed in this file when you install a package that has it. This ensures that a Perl package can find its dependencies. + </para> + </listitem> + </orderedlist> + </para> + + <para> + <varname>buildPerlPackage</varname> is built on top of <varname>stdenv</varname>, so everything can be customised in the usual way. For instance, the <literal>BerkeleyDB</literal> module has a <varname>preConfigure</varname> hook to generate a configuration file used by <filename>Makefile.PL</filename>: +<programlisting> +{ buildPerlPackage, fetchurl, db }: + +buildPerlPackage rec { + name = "BerkeleyDB-0.36"; + + src = fetchurl { + url = "mirror://cpan/authors/id/P/PM/PMQS/${name}.tar.gz"; + sha256 = "07xf50riarb60l1h6m2dqmql8q5dij619712fsgw7ach04d8g3z1"; + }; + + preConfigure = '' + echo "LIB = ${db.out}/lib" > config.in + echo "INCLUDE = ${db.dev}/include" >> config.in + ''; +} +</programlisting> + </para> + + <para> + Dependencies on other Perl packages can be specified in the <varname>buildInputs</varname> and <varname>propagatedBuildInputs</varname> attributes. If something is exclusively a build-time dependency, use <varname>buildInputs</varname>; if it’s (also) a runtime dependency, use <varname>propagatedBuildInputs</varname>. For instance, this builds a Perl module that has runtime dependencies on a bunch of other modules: +<programlisting> +ClassC3Componentised = buildPerlPackage rec { + name = "Class-C3-Componentised-1.0004"; + src = fetchurl { + url = "mirror://cpan/authors/id/A/AS/ASH/${name}.tar.gz"; + sha256 = "0xql73jkcdbq4q9m0b0rnca6nrlvf5hyzy8is0crdk65bynvs8q1"; + }; + propagatedBuildInputs = [ + ClassC3 ClassInspector TestException MROCompat + ]; +}; +</programlisting> + </para> + + <para> + On Darwin, if a script has too many <literal>-I<replaceable>dir</replaceable></literal> flags in its first line (its “shebang line”), it will not run. This can be worked around by calling the <literal>shortenPerlShebang</literal> function from the <literal>postInstall</literal> phase: +<programlisting> +{ stdenv, buildPerlPackage, fetchurl, shortenPerlShebang }: + +ImageExifTool = buildPerlPackage { + pname = "Image-ExifTool"; + version = "11.50"; + + src = fetchurl { + url = "https://www.sno.phy.queensu.ca/~phil/exiftool/Image-ExifTool-11.50.tar.gz"; + sha256 = "0d8v48y94z8maxkmw1rv7v9m0jg2dc8xbp581njb6yhr7abwqdv3"; + }; + + buildInputs = stdenv.lib.optional stdenv.isDarwin shortenPerlShebang; + postInstall = stdenv.lib.optional stdenv.isDarwin '' + shortenPerlShebang $out/bin/exiftool + ''; +}; +</programlisting> + This will remove the <literal>-I</literal> flags from the shebang line, rewrite them in the <literal>use lib</literal> form, and put them on the next line instead. This function can be given any number of Perl scripts as arguments; it will modify them in-place. + </para> + + <section xml:id="ssec-generation-from-CPAN"> + <title>Generation from CPAN</title> + + <para> + Nix expressions for Perl packages can be generated (almost) automatically from CPAN. This is done by the program <command>nix-generate-from-cpan</command>, which can be installed as follows: + </para> + +<screen> +<prompt>$ </prompt>nix-env -i nix-generate-from-cpan +</screen> + + <para> + This program takes a Perl module name, looks it up on CPAN, fetches and unpacks the corresponding package, and prints a Nix expression on standard output. For example: +<screen> +<prompt>$ </prompt>nix-generate-from-cpan XML::Simple + XMLSimple = buildPerlPackage rec { + name = "XML-Simple-2.22"; + src = fetchurl { + url = "mirror://cpan/authors/id/G/GR/GRANTM/${name}.tar.gz"; + sha256 = "b9450ef22ea9644ae5d6ada086dc4300fa105be050a2030ebd4efd28c198eb49"; + }; + propagatedBuildInputs = [ XMLNamespaceSupport XMLSAX XMLSAXExpat ]; + meta = { + description = "An API for simple XML files"; + license = with stdenv.lib.licenses; [ artistic1 gpl1Plus ]; + }; + }; +</screen> + The output can be pasted into <filename>pkgs/top-level/perl-packages.nix</filename> or wherever else you need it. + </para> + </section> + + <section xml:id="ssec-perl-cross-compilation"> + <title>Cross-compiling modules</title> + + <para> + Nixpkgs has experimental support for cross-compiling Perl modules. In many cases, it will just work out of the box, even for modules with native extensions. Sometimes, however, the Makefile.PL for a module may (indirectly) import a native module. In that case, you will need to make a stub for that module that will satisfy the Makefile.PL and install it into <filename>lib/perl5/site_perl/cross_perl/${perl.version}</filename>. See the <varname>postInstall</varname> for <varname>DBI</varname> for an example. + </para> + </section> +</section> diff --git a/nixpkgs/doc/languages-frameworks/php.section.md b/nixpkgs/doc/languages-frameworks/php.section.md new file mode 100644 index 000000000000..763beeb59358 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/php.section.md @@ -0,0 +1,137 @@ +# PHP {#sec-php} + +## User Guide {#ssec-php-user-guide} + +### Overview {#ssec-php-user-guide-overview} + +Several versions of PHP are available on Nix, each of which having a +wide variety of extensions and libraries available. + +The different versions of PHP that nixpkgs provides are located under +attributes named based on major and minor version number; e.g., +`php74` is PHP 7.4. + +Only versions of PHP that are supported by upstream for the entirety +of a given NixOS release will be included in that release of +NixOS. See [PHP Supported +Versions](https://www.php.net/supported-versions.php). + +The attribute `php` refers to the version of PHP considered most +stable and thoroughly tested in nixpkgs for any given release of +NixOS - not necessarily the latest major release from upstream. + +All available PHP attributes are wrappers around their respective +binary PHP package and provide commonly used extensions this way. The +real PHP 7.4 package, i.e. the unwrapped one, is available as +`php74.unwrapped`; see the next section for more details. + +Interactive tools built on PHP are put in `php.packages`; composer is +for example available at `php.packages.composer`. + +Most extensions that come with PHP, as well as some popular +third-party ones, are available in `php.extensions`; for example, the +opcache extension shipped with PHP is available at +`php.extensions.opcache` and the third-party ImageMagick extension at +`php.extensions.imagick`. + +### Installing PHP with extensions {#ssec-php-user-guide-installing-with-extensions} + +A PHP package with specific extensions enabled can be built using +`php.withExtensions`. This is a function which accepts an anonymous +function as its only argument; the function should accept two named +parameters: `enabled` - a list of currently enabled extensions and +`all` - the set of all extensions, and return a list of wanted +extensions. For example, a PHP package with all default extensions and +ImageMagick enabled: + +```nix +php.withExtensions ({ enabled, all }: + enabled ++ [ all.imagick ]) +``` + +To exclude some, but not all, of the default extensions, you can +filter the `enabled` list like this: + +```nix +php.withExtensions ({ enabled, all }: + (lib.filter (e: e != php.extensions.opcache) enabled) + ++ [ all.imagick ]) +``` + +To build your list of extensions from the ground up, you can simply +ignore `enabled`: + +```nix +php.withExtensions ({ all, ... }: with all; [ imagick opcache ]) +``` + +`php.withExtensions` provides extensions by wrapping a minimal php +base package, providing a `php.ini` file listing all extensions to be +loaded. You can access this package through the `php.unwrapped` +attribute; useful if you, for example, need access to the `dev` +output. The generated `php.ini` file can be accessed through the +`php.phpIni` attribute. + +If you want a PHP build with extra configuration in the `php.ini` +file, you can use `php.buildEnv`. This function takes two named and +optional parameters: `extensions` and `extraConfig`. `extensions` +takes an extension specification equivalent to that of +`php.withExtensions`, `extraConfig` a string of additional `php.ini` +configuration parameters. For example, a PHP package with the opcache +and ImageMagick extensions enabled, and `memory_limit` set to `256M`: + +```nix +php.buildEnv { + extensions = { all, ... }: with all; [ imagick opcache ]; + extraConfig = "memory_limit=256M"; +} +``` + +#### Example setup for `phpfpm` {#ssec-php-user-guide-installing-with-extensions-phpfpm} + +You can use the previous examples in a `phpfpm` pool called `foo` as +follows: + +```nix +let + myPhp = php.withExtensions ({ all, ... }: with all; [ imagick opcache ]); +in { + services.phpfpm.pools."foo".phpPackage = myPhp; +}; +``` + +```nix +let + myPhp = php.buildEnv { + extensions = { all, ... }: with all; [ imagick opcache ]; + extraConfig = "memory_limit=256M"; + }; +in { + services.phpfpm.pools."foo".phpPackage = myPhp; +}; +``` + +#### Example usage with `nix-shell` {#ssec-php-user-guide-installing-with-extensions-nix-shell} + +This brings up a temporary environment that contains a PHP interpreter +with the extensions `imagick` and `opcache` enabled: + +```sh +nix-shell -p 'php.withExtensions ({ all, ... }: with all; [ imagick opcache ])' +``` + +### Installing PHP packages with extensions {#ssec-php-user-guide-installing-packages-with-extensions} + +All interactive tools use the PHP package you get them from, so all +packages at `php.packages.*` use the `php` package with its default +extensions. Sometimes this default set of extensions isn't enough and +you may want to extend it. A common case of this is the `composer` +package: a project may depend on certain extensions and `composer` +won't work with that project unless those extensions are loaded. + +Example of building `composer` with additional extensions: +```nix +(php.withExtensions ({ all, enabled }: + enabled ++ (with all; [ imagick redis ])) +).packages.composer +``` diff --git a/nixpkgs/doc/languages-frameworks/python.section.md b/nixpkgs/doc/languages-frameworks/python.section.md new file mode 100644 index 000000000000..dc10483ce694 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/python.section.md @@ -0,0 +1,1452 @@ +# Python + +## User Guide + +### Using Python + +#### Overview + +Several versions of the Python interpreter are available on Nix, as well as a +high amount of packages. The attribute `python` refers to the default +interpreter, which is currently CPython 2.7. It is also possible to refer to +specific versions, e.g. `python38` refers to CPython 3.8, and `pypy` refers to +the default PyPy interpreter. + +Python is used a lot, and in different ways. This affects also how it is +packaged. In the case of Python on Nix, an important distinction is made between +whether the package is considered primarily an application, or whether it should +be used as a library, i.e., of primary interest are the modules in +`site-packages` that should be importable. + +In the Nixpkgs tree Python applications can be found throughout, depending on +what they do, and are called from the main package set. Python libraries, +however, are in separate sets, with one set per interpreter version. + +The interpreters have several common attributes. One of these attributes is +`pkgs`, which is a package set of Python libraries for this specific +interpreter. E.g., the `toolz` package corresponding to the default interpreter +is `python.pkgs.toolz`, and the CPython 3.8 version is `python38.pkgs.toolz`. +The main package set contains aliases to these package sets, e.g. +`pythonPackages` refers to `python.pkgs` and `python38Packages` to +`python38.pkgs`. + +#### Installing Python and packages + +The Nix and NixOS manuals explain how packages are generally installed. In the +case of Python and Nix, it is important to make a distinction between whether the +package is considered an application or a library. + +Applications on Nix are typically installed into your user profile imperatively +using `nix-env -i`, and on NixOS declaratively by adding the package name to +`environment.systemPackages` in `/etc/nixos/configuration.nix`. Dependencies +such as libraries are automatically installed and should not be installed +explicitly. + +The same goes for Python applications. Python applications can be installed in +your profile, and will be wrapped to find their exact library dependencies, +without impacting other applications or polluting your user environment. + +But Python libraries you would like to use for development cannot be installed, +at least not individually, because they won't be able to find each other +resulting in import errors. Instead, it is possible to create an environment +with `python.buildEnv` or `python.withPackages` where the interpreter and other +executables are wrapped to be able to find each other and all of the modules. + +In the following examples we will start by creating a simple, ad-hoc environment +with a nix-shell that has `numpy` and `toolz` in Python 3.8; then we will create +a re-usable environment in a single-file Python script; then we will create a +full Python environment for development with this same environment. + +Philosphically, this should be familiar to users who are used to a `venv` style +of development: individual projects create their own Python environments without +impacting the global environment or each other. + +#### Ad-hoc temporary Python environment with `nix-shell` + +The simplest way to start playing with the way nix wraps and sets up Python +environments is with `nix-shell` at the cmdline. These environments create a +temporary shell session with a Python and a *precise* list of packages (plus +their runtime dependencies), with no other Python packages in the Python +interpreter's scope. + +To create a Python 3.8 session with `numpy` and `toolz` available, run: + +```sh +$ nix-shell -p 'python38.withPackages(ps: with ps; [ numpy toolz ])' +``` + +By default `nix-shell` will start a `bash` session with this interpreter in our +`PATH`, so if we then run: + +``` +[nix-shell:~/src/nixpkgs]$ python3 +Python 3.8.1 (default, Dec 18 2019, 19:06:26) +[GCC 9.2.0] on linux +Type "help", "copyright", "credits" or "license" for more information. +>>> import numpy; import toolz +``` + +Note that no other modules are in scope, even if they were imperatively +installed into our user environment as a dependency of a Python application: + +``` +>>> import requests +Traceback (most recent call last): + File "<stdin>", line 1, in <module> +ModuleNotFoundError: No module named 'requests' +``` + +We can add as many additional modules onto the `nix-shell` as we need, and we +will still get 1 wrapped Python interpreter. We can start the interpreter +directly like so: + +```sh +$ nix-shell -p 'python38.withPackages(ps: with ps; [ numpy toolz requests ])' --run python3 +these derivations will be built: + /nix/store/xbdsrqrsfa1yva5s7pzsra8k08gxlbz1-python3-3.8.1-env.drv +building '/nix/store/xbdsrqrsfa1yva5s7pzsra8k08gxlbz1-python3-3.8.1-env.drv'... +created 277 symlinks in user environment +Python 3.8.1 (default, Dec 18 2019, 19:06:26) +[GCC 9.2.0] on linux +Type "help", "copyright", "credits" or "license" for more information. +>>> import requests +>>> +``` + +Notice that this time it built a new Python environment, which now includes +`requests`. Building an environment just creates wrapper scripts that expose the +selected dependencies to the interpreter while re-using the actual modules. This +means if any other env has installed `requests` or `numpy` in a different +context, we don't need to recompile them -- we just recompile the wrapper script +that sets up an interpreter pointing to them. This matters much more for "big" +modules like `pytorch` or `tensorflow`. + +Module names usually match their names on [pypi.org](https://pypi.org/), but +you can use the [Nixpkgs search website](https://nixos.org/nixos/packages.html) +to find them as well (along with non-python packages). + +At this point we can create throwaway experimental Python environments with +arbitrary dependencies. This is a good way to get a feel for how the Python +interpreter and dependencies work in Nix and NixOS, but to do some actual +development, we'll want to make it a bit more persistent. + +##### Running Python scripts and using `nix-shell` as shebang + +Sometimes, we have a script whose header looks like this: + +```python +#!/usr/bin/env python3 +import numpy as np +a = np.array([1,2]) +b = np.array([3,4]) +print(f"The dot product of {a} and {b} is: {np.dot(a, b)}") +``` + +Executing this script requires a `python3` that has `numpy`. Using what we learned +in the previous section, we could startup a shell and just run it like so: + +``` +nix-shell -p 'python38.withPackages(ps: with ps; [ numpy ])' --run 'python3 foo.py' +The dot product of [1 2] and [3 4] is: 11 +``` + +But if we maintain the script ourselves, and if there are more dependencies, it +may be nice to encode those dependencies in source to make the script re-usable +without that bit of knowledge. That can be done by using `nix-shell` as a +[shebang](https://en.wikipedia.org/wiki/Shebang_(Unix), like so: + +```python +#!/usr/bin/env nix-shell +#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])" +import numpy as np +a = np.array([1,2]) +b = np.array([3,4]) +print(f"The dot product of {a} and {b} is: {np.dot(a, b)}") +``` + +Then we simply execute it, without requiring any environment setup at all! + +```sh +$ ./foo.py +The dot product of [1 2] and [3 4] is: 11 +``` + +If the dependencies are not available on the host where `foo.py` is executed, it +will build or download them from a Nix binary cache prior to starting up, prior +that it is executed on a machine with a multi-user nix installation. + +This provides a way to ship a self bootstrapping Python script, akin to a +statically linked binary, where it can be run on any machine (provided nix is +installed) without having to assume that `numpy` is installed globally on the +system. + +By default it is pulling the import checkout of Nixpkgs itself from our nix +channel, which is nice as it cache aligns with our other package builds, but we +can make it fully reproducible by pinning the `nixpkgs` import: + +```python +#!/usr/bin/env nix-shell +#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])" +#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/d373d80b1207d52621961b16aa4a3438e4f98167.tar.gz +import numpy as np +a = np.array([1,2]) +b = np.array([3,4]) +print(f"The dot product of {a} and {b} is: {np.dot(a, b)}") +``` + +This will execute with the exact same versions of Python 3.8, numpy, and system +dependencies a year from now as it does today, because it will always use +exactly git commit `d373d80b1207d52621961b16aa4a3438e4f98167` of Nixpkgs for all +of the package versions. + +This is also a great way to ensure the script executes identically on different +servers. + +##### Load environment from `.nix` expression + +We've now seen how to create an ad-hoc temporary shell session, and how to +create a single script with Python dependencies, but in the course of normal +development we're usually working in an entire package repository. + +As explained in the Nix manual, `nix-shell` can also load an expression from a +`.nix` file. Say we want to have Python 3.8, `numpy` and `toolz`, like before, +in an environment. We can add a `shell.nix` file describing our dependencies: + +```nix +with import <nixpkgs> {}; +(python38.withPackages (ps: [ps.numpy ps.toolz])).env +``` + +And then at the command line, just typing `nix-shell` produces the same +environment as before. In a normal project, we'll likely have many more +dependencies; this can provide a way for developers to share the environments +with each other and with CI builders. + +What's happening here? + +1. We begin with importing the Nix Packages collections. `import <nixpkgs>` + imports the `<nixpkgs>` function, `{}` calls it and the `with` statement + brings all attributes of `nixpkgs` in the local scope. These attributes form + the main package set. +2. Then we create a Python 3.8 environment with the `withPackages` function, as before. +3. The `withPackages` function expects us to provide a function as an argument + that takes the set of all Python packages and returns a list of packages to + include in the environment. Here, we select the packages `numpy` and `toolz` + from the package set. + +To combine this with `mkShell` you can: + +```nix +with import <nixpkgs> {}; +let + pythonEnv = python38.withPackages (ps: [ + ps.numpy + ps.toolz + ]); +in mkShell { + buildInputs = [ + pythonEnv + + black + mypy + + libffi + openssl + ]; +} +``` + +This will create a unified environment that has not just our Python interpreter +and its Python dependencies, but also tools like `black` or `mypy` and libraries +like `libffi` the `openssl` in scope. This is generic and can span any number of +tools or languages across the Nixpkgs ecosystem. + +##### Installing environments globally on the system + +Up to now, we've been creating environments scoped to an ad-hoc shell session, +or a single script, or a single project. This is generally advisable, as it +avoids pollution across contexts. + +However, sometimes we know we will often want a Python with some basic packages, +and want this available without having to enter into a shell or build context. +This can be useful to have things like vim/emacs editors and plugins or shell +tools "just work" without having to set them up, or when running other software +that expects packages to be installed globally. + +To create your own custom environment, create a file in `~/.config/nixpkgs/overlays/` +that looks like this: + +```nix +# ~/.config/nixpkgs/overlays/myEnv.nix +self: super: { + myEnv = super.buildEnv { + name = "myEnv"; + paths = [ + # A Python 3 interpreter with some packages + (self.python3.withPackages ( + ps: with ps; [ + pyflakes + pytest + python-language-server + ] + )) + + # Some other packages we'd like as part of this env + self.mypy + self.black + self.ripgrep + self.tmux + ]; + }; +} +``` + +You can then build and install this to your profile with: + +```sh +nix-env -iA myEnv +``` + +One limitation of this is that you can only have 1 Python env installed +globally, since they conflict on the `python` to load out of your `PATH`. + +If you get a conflict or prefer to keep the setup clean, you can have `nix-env` +atomically *uninstall* all other imperatively installed packages and replace +your profile with just `myEnv` by using the `--replace` flag. + +##### Environment defined in `/etc/nixos/configuration.nix` + +For the sake of completeness, here's how to install the environment system-wide +on NixOS. + +```nix +{ # ... + + environment.systemPackages = with pkgs; [ + (python38.withPackages(ps: with ps; [ numpy toolz ])) + ]; +} +``` + +### Developing with Python + +Above, we were mostly just focused on use cases and what to do to get started +creating working Python environments in nix. + +Now that you know the basics to be up and running, it is time to take a step +back and take a deeper look at at how Python packages are packaged on Nix. Then, +we will look at how you can use development mode with your code. + +#### Python library packages in Nixpkgs + +With Nix all packages are built by functions. The main function in Nix for +building Python libraries is `buildPythonPackage`. Let's see how we can build the +`toolz` package. + +```nix +{ lib, buildPythonPackage, fetchPypi }: + +buildPythonPackage rec { + pname = "toolz"; + version = "0.10.0"; + + src = fetchPypi { + inherit pname version; + sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560"; + }; + + doCheck = false; + + meta = with lib; { + homepage = "https://github.com/pytoolz/toolz"; + description = "List processing tools and functional utilities"; + license = licenses.bsd3; + maintainers = with maintainers; [ fridh ]; + }; +} +``` + +What happens here? The function `buildPythonPackage` is called and as argument +it accepts a set. In this case the set is a recursive set, `rec`. One of the +arguments is the name of the package, which consists of a basename (generally +following the name on PyPi) and a version. Another argument, `src` specifies the +source, which in this case is fetched from PyPI using the helper function +`fetchPypi`. The argument `doCheck` is used to set whether tests should be run +when building the package. Furthermore, we specify some (optional) meta +information. The output of the function is a derivation. + +An expression for `toolz` can be found in the Nixpkgs repository. As explained +in the introduction of this Python section, a derivation of `toolz` is available +for each interpreter version, e.g. `python38.pkgs.toolz` refers to the `toolz` +derivation corresponding to the CPython 3.8 interpreter. + +The above example works when you're directly working on +`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though, +you will want to test a Nix expression outside of the Nixpkgs tree. + +The following expression creates a derivation for the `toolz` package, +and adds it along with a `numpy` package to a Python environment. + +```nix +with import <nixpkgs> {}; + +( let + my_toolz = python38.pkgs.buildPythonPackage rec { + pname = "toolz"; + version = "0.10.0"; + + src = python38.pkgs.fetchPypi { + inherit pname version; + sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560"; + }; + + doCheck = false; + + meta = { + homepage = "https://github.com/pytoolz/toolz/"; + description = "List processing tools and functional utilities"; + }; + }; + + in python38.withPackages (ps: [ps.numpy my_toolz]) +).env +``` + +Executing `nix-shell` will result in an environment in which you can use +Python 3.8 and the `toolz` package. As you can see we had to explicitly mention +for which Python version we want to build a package. + +So, what did we do here? Well, we took the Nix expression that we used earlier +to build a Python environment, and said that we wanted to include our own +version of `toolz`, named `my_toolz`. To introduce our own package in the scope +of `withPackages` we used a `let` expression. You can see that we used +`ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take +`toolz` from the Nixpkgs package set this time, but instead took our own version +that we introduced with the `let` expression. + +#### Handling dependencies + +Our example, `toolz`, does not have any dependencies on other Python packages or +system libraries. According to the manual, `buildPythonPackage` uses the +arguments `buildInputs` and `propagatedBuildInputs` to specify dependencies. If +something is exclusively a build-time dependency, then the dependency should be +included in `buildInputs`, but if it is (also) a runtime dependency, then it +should be added to `propagatedBuildInputs`. Test dependencies are considered +build-time dependencies and passed to `checkInputs`. + +The following example shows which arguments are given to `buildPythonPackage` in +order to build [`datashape`](https://github.com/blaze/datashape). + +```nix +{ lib, buildPythonPackage, fetchPypi, numpy, multipledispatch, dateutil, pytest }: + +buildPythonPackage rec { + pname = "datashape"; + version = "0.4.7"; + + src = fetchPypi { + inherit pname version; + sha256 = "14b2ef766d4c9652ab813182e866f493475e65e558bed0822e38bf07bba1a278"; + }; + + checkInputs = [ pytest ]; + propagatedBuildInputs = [ numpy multipledispatch dateutil ]; + + meta = with lib; { + homepage = "https://github.com/ContinuumIO/datashape"; + description = "A data description language"; + license = licenses.bsd2; + maintainers = with maintainers; [ fridh ]; + }; +} +``` + +We can see several runtime dependencies, `numpy`, `multipledispatch`, and +`dateutil`. Furthermore, we have one `checkInputs`, i.e. `pytest`. `pytest` is a +test runner and is only used during the `checkPhase` and is therefore not added +to `propagatedBuildInputs`. + +In the previous case we had only dependencies on other Python packages to consider. +Occasionally you have also system libraries to consider. E.g., `lxml` provides +Python bindings to `libxml2` and `libxslt`. These libraries are only required +when building the bindings and are therefore added as `buildInputs`. + +```nix +{ lib, pkgs, buildPythonPackage, fetchPypi }: + +buildPythonPackage rec { + pname = "lxml"; + version = "3.4.4"; + + src = fetchPypi { + inherit pname version; + sha256 = "16a0fa97hym9ysdk3rmqz32xdjqmy4w34ld3rm3jf5viqjx65lxk"; + }; + + buildInputs = [ pkgs.libxml2 pkgs.libxslt ]; + + meta = with lib; { + description = "Pythonic binding for the libxml2 and libxslt libraries"; + homepage = "https://lxml.de"; + license = licenses.bsd3; + maintainers = with maintainers; [ sjourdois ]; + }; +} +``` + +In this example `lxml` and Nix are able to work out exactly where the relevant +files of the dependencies are. This is not always the case. + +The example below shows bindings to The Fastest Fourier Transform in the West, +commonly known as FFTW. On Nix we have separate packages of FFTW for the +different types of floats (`"single"`, `"double"`, `"long-double"`). The +bindings need all three types, and therefore we add all three as `buildInputs`. +The bindings don't expect to find each of them in a different folder, and +therefore we have to set `LDFLAGS` and `CFLAGS`. + +```nix +{ lib, pkgs, buildPythonPackage, fetchPypi, numpy, scipy }: + +buildPythonPackage rec { + pname = "pyFFTW"; + version = "0.9.2"; + + src = fetchPypi { + inherit pname version; + sha256 = "f6bbb6afa93085409ab24885a1a3cdb8909f095a142f4d49e346f2bd1b789074"; + }; + + buildInputs = [ pkgs.fftw pkgs.fftwFloat pkgs.fftwLongDouble]; + + propagatedBuildInputs = [ numpy scipy ]; + + # Tests cannot import pyfftw. pyfftw works fine though. + doCheck = false; + + preConfigure = '' + export LDFLAGS="-L${pkgs.fftw.dev}/lib -L${pkgs.fftwFloat.out}/lib -L${pkgs.fftwLongDouble.out}/lib" + export CFLAGS="-I${pkgs.fftw.dev}/include -I${pkgs.fftwFloat.dev}/include -I${pkgs.fftwLongDouble.dev}/include" + ''; + + meta = with lib; { + description = "A pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms"; + homepage = "http://hgomersall.github.com/pyFFTW"; + license = with licenses; [ bsd2 bsd3 ]; + maintainers = with maintainers; [ fridh ]; + }; +} +``` +Note also the line `doCheck = false;`, we explicitly disabled running the test-suite. + + +#### Develop local package + +As a Python developer you're likely aware of [development mode](http://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode) +(`python setup.py develop`); instead of installing the package this command +creates a special link to the project code. That way, you can run updated code +without having to reinstall after each and every change you make. Development +mode is also available. Let's see how you can use it. + +In the previous Nix expression the source was fetched from an url. We can also +refer to a local source instead using `src = ./path/to/source/tree;` + +If we create a `shell.nix` file which calls `buildPythonPackage`, and if `src` +is a local source, and if the local source has a `setup.py`, then development +mode is activated. + +In the following example we create a simple environment that has a Python 3.8 +version of our package in it, as well as its dependencies and other packages we +like to have in the environment, all specified with `propagatedBuildInputs`. +Indeed, we can just add any package we like to have in our environment to +`propagatedBuildInputs`. + +```nix +with import <nixpkgs> {}; +with python38Packages; + +buildPythonPackage rec { + name = "mypackage"; + src = ./path/to/package/source; + propagatedBuildInputs = [ pytest numpy pkgs.libsndfile ]; +} +``` + +It is important to note that due to how development mode is implemented on Nix +it is not possible to have multiple packages simultaneously in development mode. + +### Organising your packages + +So far we discussed how you can use Python on Nix, and how you can develop with +it. We've looked at how you write expressions to package Python packages, and we +looked at how you can create environments in which specified packages are +available. + +At some point you'll likely have multiple packages which you would +like to be able to use in different projects. In order to minimise unnecessary +duplication we now look at how you can maintain a repository with your +own packages. The important functions here are `import` and `callPackage`. + +### Including a derivation using `callPackage` + +Earlier we created a Python environment using `withPackages`, and included the +`toolz` package via a `let` expression. +Let's split the package definition from the environment definition. + +We first create a function that builds `toolz` in `~/path/to/toolz/release.nix` + +```nix +{ lib, buildPythonPackage }: + +buildPythonPackage rec { + pname = "toolz"; + version = "0.10.0"; + + src = fetchPypi { + inherit pname version; + sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560"; + }; + + meta = with lib; { + homepage = "https://github.com/pytoolz/toolz/"; + description = "List processing tools and functional utilities"; + license = licenses.bsd3; + maintainers = with maintainers; [ fridh ]; + }; +} +``` + +It takes an argument `buildPythonPackage`. We now call this function using +`callPackage` in the definition of our environment + +```nix +with import <nixpkgs> {}; + +( let + toolz = callPackage /path/to/toolz/release.nix { + buildPythonPackage = python38Packages.buildPythonPackage; + }; + in python38.withPackages (ps: [ ps.numpy toolz ]) +).env +``` + +Important to remember is that the Python version for which the package is made +depends on the `python` derivation that is passed to `buildPythonPackage`. Nix +tries to automatically pass arguments when possible, which is why generally you +don't explicitly define which `python` derivation should be used. In the above +example we use `buildPythonPackage` that is part of the set `python38Packages`, +and in this case the `python38` interpreter is automatically used. + +## Reference + +### Interpreters + +Versions 2.7, 3.5, 3.6, 3.7 and 3.8 of the CPython interpreter are available as +respectively `python27`, `python35`, `python36`, `python37` and `python38`. The +aliases `python2` and `python3` correspond to respectively `python27` and +`python38`. The default interpreter, `python`, maps to `python2`. The PyPy +interpreters compatible with Python 2.7 and 3 are available as `pypy27` and +`pypy3`, with aliases `pypy2` mapping to `pypy27` and `pypy` mapping to `pypy2`. +The Nix expressions for the interpreters can be found in +`pkgs/development/interpreters/python`. + +All packages depending on any Python interpreter get appended +`out/{python.sitePackages}` to `$PYTHONPATH` if such directory +exists. + +#### Missing `tkinter` module standard library + +To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`. + +#### Attributes on interpreters packages + +Each interpreter has the following attributes: + +- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter. +- `interpreter`. Alias for `${python}/bin/${executable}`. +- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See section *python.buildEnv function* for usage and documentation. +- `withPackages`. Simpler interface to `buildEnv`. See section *python.withPackages function* for usage and documentation. +- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`. +- `executable`. Name of the interpreter executable, e.g. `python3.8`. +- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`. + +### Building packages and applications + +Python libraries and applications that use `setuptools` or +`distutils` are typically built with respectively the `buildPythonPackage` and +`buildPythonApplication` functions. These two functions also support installing a `wheel`. + +All Python packages reside in `pkgs/top-level/python-packages.nix` and all +applications elsewhere. In case a package is used as both a library and an +application, then the package should be in `pkgs/top-level/python-packages.nix` +since only those packages are made available for all interpreter versions. The +preferred location for library expressions is in +`pkgs/development/python-modules`. It is important that these packages are +called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee +the right version of the package is built. + +Based on the packages defined in `pkgs/top-level/python-packages.nix` an +attribute set is created for each available Python interpreter. The available +sets are + +* `pkgs.python27Packages` +* `pkgs.python35Packages` +* `pkgs.python36Packages` +* `pkgs.python37Packages` +* `pkgs.pypyPackages` + +and the aliases + +* `pkgs.python2Packages` pointing to `pkgs.python27Packages` +* `pkgs.python3Packages` pointing to `pkgs.python37Packages` +* `pkgs.pythonPackages` pointing to `pkgs.python2Packages` + +#### `buildPythonPackage` function + +The `buildPythonPackage` function is implemented in +`pkgs/development/interpreters/python/mk-python-derivation` +using setup hooks. + +The following is an example: + +```nix +{ lib, buildPythonPackage, fetchPypi, hypothesis, setuptools_scm, attrs, py, setuptools, six, pluggy }: + +buildPythonPackage rec { + pname = "pytest"; + version = "3.3.1"; + + src = fetchPypi { + inherit pname version; + sha256 = "cf8436dc59d8695346fcd3ab296de46425ecab00d64096cebe79fb51ecb2eb93"; + }; + + postPatch = '' + # don't test bash builtins + rm testing/test_argcomplete.py + ''; + + checkInputs = [ hypothesis ]; + nativeBuildInputs = [ setuptools_scm ]; + propagatedBuildInputs = [ attrs py setuptools six pluggy ]; + + meta = with lib; { + maintainers = with maintainers; [ domenkozar lovek323 madjar lsix ]; + description = "Framework for writing tests"; + }; +} +``` + +The `buildPythonPackage` mainly does four things: + +* In the `buildPhase`, it calls `${python.interpreter} setup.py bdist_wheel` to + build a wheel binary zipfile. +* In the `installPhase`, it installs the wheel file using `pip install *.whl`. +* In the `postFixup` phase, the `wrapPythonPrograms` bash function is called to + wrap all programs in the `$out/bin/*` directory to include `$PATH` + environment variable and add dependent libraries to script's `sys.path`. +* In the `installCheck` phase, `${python.interpreter} setup.py test` is ran. + +By default tests are run because `doCheck = true`. Test dependencies, like +e.g. the test runner, should be added to `checkInputs`. + +By default `meta.platforms` is set to the same value +as the interpreter unless overridden otherwise. + +##### `buildPythonPackage` parameters + +All parameters from `stdenv.mkDerivation` function are still supported. The +following are specific to `buildPythonPackage`: + +* `catchConflicts ? true`: If `true`, abort package build if a package name + appears more than once in dependency tree. Default is `true`. +* `disabled` ? false: If `true`, package is not built for the particular Python + interpreter version. +* `dontWrapPythonPrograms ? false`: Skip wrapping of Python programs. +* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment + variable in wrapped programs. +* `format ? "setuptools"`: Format of the source. Valid options are + `"setuptools"`, `"pyproject"`, `"flit"`, `"wheel"`, and `"other"`. + `"setuptools"` is for when the source has a `setup.py` and `setuptools` is + used to build a wheel, `flit`, in case `flit` should be used to build a wheel, + and `wheel` in case a wheel is provided. Use `other` when a custom + `buildPhase` and/or `installPhase` is needed. +* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to + `makeWrapper`, which wraps generated binaries. By default, the arguments to + `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling + the binary. Additional arguments here can allow a developer to set environment + variables which will be available when the binary is run. For example, + `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`. +* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this + defaults to `"python3.8-"` for Python 3.8, etc., and in case of applications + to `""`. +* `pipInstallFlags ? []`: A list of strings. Arguments to be passed to `pip + install`. To pass options to `python setup.py install`, use + `--install-option`. E.g., `pipInstallFlags=["--install-option='--cpp_implementation'"]`. +* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages + in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`). +* `preShellHook`: Hook to execute commands before `shellHook`. +* `postShellHook`: Hook to execute commands after `shellHook`. +* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only + created when the filenames end with `.py`. +* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command. +* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command. + +The `stdenv.mkDerivation` function accepts various parameters for describing +build inputs (see "Specifying dependencies"). The following are of special +interest for Python packages, either because these are primarily used, or +because their behaviour is different: + +* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables + as well as the items listed in `setup_requires`. +* `buildInputs ? []`: Build and/or run-time dependencies that need to be be + compiled for the host machine. Typically non-Python libraries which are being + linked. +* `checkInputs ? []`: Dependencies needed for running the `checkPhase`. These + are added to `nativeBuildInputs` when `doCheck = true`. Items listed in + `tests_require` go here. +* `propagatedBuildInputs ? []`: Aside from propagating dependencies, + `buildPythonPackage` also injects code into and wraps executables with the + paths included in this list. Items listed in `install_requires` go here. + +##### Overriding Python packages + +The `buildPythonPackage` function has a `overridePythonAttrs` method that can be +used to override the package. In the following example we create an environment +where we have the `blaze` package using an older version of `pandas`. We +override first the Python interpreter and pass `packageOverrides` which contains +the overrides for packages in the package set. + +```nix +with import <nixpkgs> {}; + +(let + python = let + packageOverrides = self: super: { + pandas = super.pandas.overridePythonAttrs(old: rec { + version = "0.19.1"; + src = super.fetchPypi { + pname = "pandas"; + inherit version; + sha256 = "08blshqj9zj1wyjhhw3kl2vas75vhhicvv72flvf1z3jvapgw295"; + }; + }); + }; + in pkgs.python3.override {inherit packageOverrides; self = python;}; + +in python.withPackages(ps: [ps.blaze])).env +``` + +#### `buildPythonApplication` function + +The `buildPythonApplication` function is practically the same as +`buildPythonPackage`. The main purpose of this function is to build a Python +package where one is interested only in the executables, and not importable +modules. For that reason, when adding this package to a `python.buildEnv`, the +modules won't be made available. + +Another difference is that `buildPythonPackage` by default prefixes the names of +the packages with the version of the interpreter. Because this is irrelevant for +applications, the prefix is omitted. + +When packaging a Python application with `buildPythonApplication`, it should be +called with `callPackage` and passed `python` or `pythonPackages` (possibly +specifying an interpreter version), like this: + +```nix +{ lib, python3Packages }: + +python3Packages.buildPythonApplication rec { + pname = "luigi"; + version = "2.7.9"; + + src = python3Packages.fetchPypi { + inherit pname version; + sha256 = "035w8gqql36zlan0xjrzz9j4lh9hs0qrsgnbyw07qs7lnkvbdv9x"; + }; + + propagatedBuildInputs = with python3Packages; [ tornado_4 python-daemon ]; + + meta = with lib; { + ... + }; +} +``` + +This is then added to `all-packages.nix` just as any other application would be. + +```nix +luigi = callPackage ../applications/networking/cluster/luigi { }; +``` + +Since the package is an application, a consumer doesn't need to care about +Python versions or modules, which is why they don't go in `pythonPackages`. + +#### `toPythonApplication` function + +A distinction is made between applications and libraries, however, sometimes a +package is used as both. In this case the package is added as a library to +`python-packages.nix` and as an application to `all-packages.nix`. To reduce +duplication the `toPythonApplication` can be used to convert a library to an +application. + +The Nix expression shall use `buildPythonPackage` and be called from +`python-packages.nix`. A reference shall be created from `all-packages.nix` to +the attribute in `python-packages.nix`, and the `toPythonApplication` shall be +applied to the reference: +```nix +youtube-dl = with pythonPackages; toPythonApplication youtube-dl; +``` + +#### `toPythonModule` function + +In some cases, such as bindings, a package is created using +`stdenv.mkDerivation` and added as attribute in `all-packages.nix`. The Python +bindings should be made available from `python-packages.nix`. The +`toPythonModule` function takes a derivation and makes certain Python-specific +modifications. + +```nix +opencv = toPythonModule (pkgs.opencv.override { + enablePython = true; + pythonPackages = self; +}); +``` + +Do pay attention to passing in the right Python version! + +#### `python.buildEnv` function + +Python environments can be created using the low-level `pkgs.buildEnv` function. +This example shows how to create an environment that has the Pyramid Web Framework. +Saving the following as `default.nix` + +```nix +with import <nixpkgs> {}; + +python.buildEnv.override { + extraLibs = [ pythonPackages.pyramid ]; + ignoreCollisions = true; +} +``` + +and running `nix-build` will create + +``` +/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env +``` + +with wrapped binaries in `bin/`. + +You can also use the `env` attribute to create local environments with needed +packages installed. This is somewhat comparable to `virtualenv`. For example, +running `nix-shell` with the following `shell.nix` + +```nix +with import <nixpkgs> {}; + +(python3.buildEnv.override { + extraLibs = with python3Packages; [ numpy requests ]; +}).env +``` + +will drop you into a shell where Python will have the +specified packages in its path. + + +##### `python.buildEnv` arguments + +* `extraLibs`: List of packages installed inside the environment. +* `postBuild`: Shell command executed after the build of environment. +* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`). +* `permitUserSite`: Skip setting the `PYTHONNOUSERSITE` environment variable in + wrapped binaries in the environment. + +#### `python.withPackages` function + +The `python.withPackages` function provides a simpler interface to the `python.buildEnv` functionality. +It takes a function as an argument that is passed the set of python packages and returns the list +of the packages to be included in the environment. Using the `withPackages` function, the previous +example for the Pyramid Web Framework environment can be written like this: + +```nix +with import <nixpkgs> {}; + +python.withPackages (ps: [ps.pyramid]) +``` + +`withPackages` passes the correct package set for the specific interpreter +version as an argument to the function. In the above example, `ps` equals +`pythonPackages`. But you can also easily switch to using python3: + +```nix +with import <nixpkgs> {}; + +python3.withPackages (ps: [ps.pyramid]) +``` + +Now, `ps` is set to `python3Packages`, matching the version of the interpreter. + +As `python.withPackages` simply uses `python.buildEnv` under the hood, it also +supports the `env` attribute. The `shell.nix` file from the previous section can +thus be also written like this: + +```nix +with import <nixpkgs> {}; + +(python38.withPackages (ps: [ps.numpy ps.requests])).env +``` + +In contrast to `python.buildEnv`, `python.withPackages` does not support the +more advanced options such as `ignoreCollisions = true` or `postBuild`. If you +need them, you have to use `python.buildEnv`. + +Python 2 namespace packages may provide `__init__.py` that collide. In that case +`python.buildEnv` should be used with `ignoreCollisions = true`. + +#### Setup hooks + +The following are setup hooks specifically for Python packages. Most of these +are used in `buildPythonPackage`. + +- `eggUnpackhook` to move an egg to the correct folder so it can be installed + with the `eggInstallHook` +- `eggBuildHook` to skip building for eggs. +- `eggInstallHook` to install eggs. +- `flitBuildHook` to build a wheel using `flit`. +- `pipBuildHook` to build a wheel using `pip` and PEP 517. Note a build system + (e.g. `setuptools` or `flit`) should still be added as `nativeBuildInput`. +- `pipInstallHook` to install wheels. +- `pytestCheckHook` to run tests with `pytest`. +- `pythonCatchConflictsHook` to check whether a Python package is not already existing. +- `pythonImportsCheckHook` to check whether importing the listed modules works. +- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder. +- `setuptoolsBuildHook` to build a wheel using `setuptools`. +- `setuptoolsCheckHook` to run tests with `python setup.py test`. +- `venvShellHook` to source a Python 3 `venv` at the `venvDir` location. A + `venv` is created if it does not yet exist. `postVenvCreation` can be used to + to run commands only after venv is first created. +- `wheelUnpackHook` to move a wheel to the correct folder so it can be installed + with the `pipInstallHook`. + +### Development mode + +Development or editable mode is supported. To develop Python packages +`buildPythonPackage` has additional logic inside `shellPhase` to run `pip +install -e . --prefix $TMPDIR/`for the package. + +Warning: `shellPhase` is executed only if `setup.py` exists. + +Given a `default.nix`: +```nix +with import <nixpkgs> {}; + +pythonPackages.buildPythonPackage { + name = "myproject"; + buildInputs = with pythonPackages; [ pyramid ]; + + src = ./.; +} +``` + +Running `nix-shell` with no arguments should give you the environment in which +the package would be built with `nix-build`. + +Shortcut to setup environments with C headers/libraries and Python packages: + +```shell +nix-shell -p pythonPackages.pyramid zlib libjpeg git +``` + +Note: There is a boolean value `lib.inNixShell` set to `true` if nix-shell is invoked. + +### Tools + +Packages inside nixpkgs are written by hand. However many tools exist in +community to help save time. No tool is preferred at the moment. + +- [pypi2nix](https://github.com/nix-community/pypi2nix): Generate Nix + expressions for your Python project. Note that [sharing derivations from + pypi2nix with nixpkgs is possible but not + encouraged](https://github.com/nix-community/pypi2nix/issues/222#issuecomment-443497376). +- [python2nix](https://github.com/proger/python2nix) by Vladimir Kirillov. + +### Deterministic builds + +The Python interpreters are now built deterministically. Minor modifications had +to be made to the interpreters in order to generate deterministic bytecode. This +has security implications and is relevant for those using Python in a +`nix-shell`. + +When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will +have timestamp 1. The `buildPythonPackage` function sets `DETERMINISTIC_BUILD=1` +and [PYTHONHASHSEED=0](https://docs.python.org/3.8/using/cmdline.html#envvar-PYTHONHASHSEED). +Both are also exported in `nix-shell`. + +### Automatic tests + +It is recommended to test packages as part of the build process. +Source distributions (`sdist`) often include test files, but not always. + +By default the command `python setup.py test` is run as part of the +`checkPhase`, but often it is necessary to pass a custom `checkPhase`. An +example of such a situation is when `py.test` is used. + +#### Common issues + +* Non-working tests can often be deselected. By default `buildPythonPackage` + runs `python setup.py test`. Most Python modules follows the standard test + protocol where the pytest runner can be used instead. `py.test` supports a + `-k` parameter to ignore test methods or classes: + + ```nix + buildPythonPackage { + # ... + # assumes the tests are located in tests + checkInputs = [ pytest ]; + checkPhase = '' + py.test -k 'not function_name and not other_function' tests + ''; + } + ``` +* Tests that attempt to access `$HOME` can be fixed by using the following + work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)` + +## FAQ + +### How to solve circular dependencies? + +Consider the packages `A` and `B` that depend on each other. When packaging `B`, +a solution is to override package `A` not to depend on `B` as an input. The same +should also be done when packaging `A`. + +### How to override a Python package? + +We can override the interpreter and pass `packageOverrides`. In the following +example we rename the `pandas` package and build it. + +```nix +with import <nixpkgs> {}; + +(let + python = let + packageOverrides = self: super: { + pandas = super.pandas.overridePythonAttrs(old: {name="foo";}); + }; + in pkgs.python38.override {inherit packageOverrides;}; + +in python.withPackages(ps: [ps.pandas])).env +``` + +Using `nix-build` on this expression will build an environment that contains the +package `pandas` but with the new name `foo`. + +All packages in the package set will use the renamed package. A typical use case +is to switch to another version of a certain package. For example, in the +Nixpkgs repository we have multiple versions of `django` and `scipy`. In the +following example we use a different version of `scipy` and create an +environment that uses it. All packages in the Python package set will now use +the updated `scipy` version. + +```nix +with import <nixpkgs> {}; + +( let + packageOverrides = self: super: { + scipy = super.scipy_0_17; + }; + in (pkgs.python38.override {inherit packageOverrides;}).withPackages (ps: [ps.blaze]) +).env +``` + +The requested package `blaze` depends on `pandas` which itself depends on `scipy`. + +If you want the whole of Nixpkgs to use your modifications, then you can use +`overlays` as explained in this manual. In the following example we build a +`inkscape` using a different version of `numpy`. + +```nix +let + pkgs = import <nixpkgs> {}; + newpkgs = import pkgs.path { overlays = [ (self: super: { + python38 = let + packageOverrides = python-self: python-super: { + numpy = python-super.numpy_1_18; + }; + in super.python38.override {inherit packageOverrides;}; + } ) ]; }; +in newpkgs.inkscape +``` + +### `python setup.py bdist_wheel` cannot create .whl + +Executing `python setup.py bdist_wheel` in a `nix-shell `fails with +``` +ValueError: ZIP does not support timestamps before 1980 +``` + +This is because files from the Nix store (which have a timestamp of the UNIX +epoch of January 1, 1970) are included in the .ZIP, but .ZIP archives follow the +DOS convention of counting timestamps from 1980. + +The command `bdist_wheel` reads the `SOURCE_DATE_EPOCH` environment variable, +which `nix-shell` sets to 1. Unsetting this variable or giving it a value +corresponding to 1980 or later enables building wheels. + +Use 1980 as timestamp: + +```shell +nix-shell --run "SOURCE_DATE_EPOCH=315532800 python3 setup.py bdist_wheel" +``` + +or the current time: + +```shell +nix-shell --run "SOURCE_DATE_EPOCH=$(date +%s) python3 setup.py bdist_wheel" +``` + +or unset `SOURCE_DATE_EPOCH`: + +```shell +nix-shell --run "unset SOURCE_DATE_EPOCH; python3 setup.py bdist_wheel" +``` + +### `install_data` / `data_files` problems + +If you get the following error: + +``` +could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc': +Permission denied +``` + +This is a [known bug](https://github.com/pypa/setuptools/issues/130) in +`setuptools`. Setuptools `install_data` does not respect `--prefix`. An example +of such package using the feature is `pkgs/tools/X11/xpra/default.nix`. + +As workaround install it as an extra `preInstall` step: + +```shell +${python.interpreter} setup.py install_data --install-dir=$out --root=$out +sed -i '/ = data\_files/d' setup.py +``` + +### Rationale of non-existent global site-packages + +On most operating systems a global `site-packages` is maintained. This however +becomes problematic if you want to run multiple Python versions or have multiple +versions of certain libraries for your projects. Generally, you would solve such +issues by creating virtual environments using `virtualenv`. + +On Nix each package has an isolated dependency tree which, in the case of +Python, guarantees the right versions of the interpreter and libraries or +packages are available. There is therefore no need to maintain a global `site-packages`. + +If you want to create a Python environment for development, then the recommended +method is to use `nix-shell`, either with or without the `python.buildEnv` +function. + +### How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems? + +While this approach is not very idiomatic from Nix perspective, it can still be +useful when dealing with pre-existing projects or in situations where it's not +feasible or desired to write derivations for all required dependencies. + +This is an example of a `default.nix` for a `nix-shell`, which allows to consume +a virtual environment created by `venv`, and install Python modules through +`pip` the traditional way. + +Create this `default.nix` file, together with a `requirements.txt` and simply +execute `nix-shell`. + +```nix +with import <nixpkgs> { }; + +let + pythonPackages = python3Packages; +in pkgs.mkShell rec { + name = "impurePythonEnv"; + venvDir = "./.venv"; + buildInputs = [ + # A Python interpreter including the 'venv' module is required to bootstrap + # the environment. + pythonPackages.python + + # This execute some shell code to initialize a venv in $venvDir before + # dropping into the shell + pythonPackages.venvShellHook + + # Those are dependencies that we would like to use from nixpkgs, which will + # add them to PYTHONPATH and thus make them accessible from within the venv. + pythonPackages.numpy + pythonPackages.requests + + # In this particular example, in order to compile any binary extensions they may + # require, the Python modules listed in the hypothetical requirements.txt need + # the following packages to be installed locally: + taglib + openssl + git + libxml2 + libxslt + libzip + zlib + ]; + + # Run this command, only after creating the virtual environment + postVenvCreation = '' + unset SOURCE_DATE_EPOCH + pip install -r requirements.txt + ''; + + # Now we can execute any commands within the virtual environment. + # This is optional and can be left out to run pip manually. + postShellHook = '' + # allow pip to install wheels + unset SOURCE_DATE_EPOCH + ''; + +} +``` + +In case the supplied venvShellHook is insufficient, or when Python 2 support is +needed, you can define your own shell hook and adapt to your needs like in the +following example: + +```nix +with import <nixpkgs> { }; + +let + venvDir = "./.venv"; + pythonPackages = python3Packages; +in pkgs.mkShell rec { + name = "impurePythonEnv"; + buildInputs = [ + pythonPackages.python + # Needed when using python 2.7 + # pythonPackages.virtualenv + # ... + ]; + + # This is very close to how venvShellHook is implemented, but + # adapted to use 'virtualenv' + shellHook = '' + SOURCE_DATE_EPOCH=$(date +%s) + + if [ -d "${venvDir}" ]; then + echo "Skipping venv creation, '${venvDir}' already exists" + else + echo "Creating new venv environment in path: '${venvDir}'" + # Note that the module venv was only introduced in python 3, so for 2.7 + # this needs to be replaced with a call to virtualenv + ${pythonPackages.python.interpreter} -m venv "${venvDir}" + fi + + # Under some circumstances it might be necessary to add your virtual + # environment to PYTHONPATH, which you can do here too; + # PYTHONPATH=$PWD/${venvDir}/${pythonPackages.python.sitePackages}/:$PYTHONPATH + + source "${venvDir}/bin/activate" + + # As in the previous example, this is optional. + pip install -r requirements.txt + ''; +} +``` + +Note that the `pip install` is an imperative action. So every time `nix-shell` +is executed it will attempt to download the Python modules listed in +requirements.txt. However these will be cached locally within the `virtualenv` +folder and not downloaded again. + +### How to override a Python package from `configuration.nix`? + +If you need to change a package's attribute(s) from `configuration.nix` you could do: + +```nix + nixpkgs.config.packageOverrides = super: { + python = super.python.override { + packageOverrides = python-self: python-super: { + zerobin = python-super.zerobin.overrideAttrs (oldAttrs: { + src = super.fetchgit { + url = "https://github.com/sametmax/0bin"; + rev = "a344dbb18fe7a855d0742b9a1cede7ce423b34ec"; + sha256 = "16d769kmnrpbdr0ph0whyf4yff5df6zi4kmwx7sz1d3r6c8p6xji"; + }; + }); + }; + }; + }; +``` + +`pythonPackages.zerobin` is now globally overridden. All packages and also the +`zerobin` NixOS service use the new definition. Note that `python-super` refers +to the old package set and `python-self` to the new, overridden version. + +To modify only a Python package set instead of a whole Python derivation, use +this snippet: + +```nix + myPythonPackages = pythonPackages.override { + overrides = self: super: { + zerobin = ...; + }; + } +``` + +### How to override a Python package using overlays? + +Use the following overlay template: + +```nix +self: super: { + python = super.python.override { + packageOverrides = python-self: python-super: { + zerobin = python-super.zerobin.overrideAttrs (oldAttrs: { + src = super.fetchgit { + url = "https://github.com/sametmax/0bin"; + rev = "a344dbb18fe7a855d0742b9a1cede7ce423b34ec"; + sha256 = "16d769kmnrpbdr0ph0whyf4yff5df6zi4kmwx7sz1d3r6c8p6xji"; + }; + }); + }; + }; +} +``` + +### How to use Intel's MKL with numpy and scipy? + +MKL can be configured using an overlay. See the section "[Using overlays to +configure alternatives](#sec-overlays-alternatives-blas-lapack)". + +### What inputs do `setup_requires`, `install_requires` and `tests_require` map to? + +In a `setup.py` or `setup.cfg` it is common to declare dependencies: + +* `setup_requires` corresponds to `nativeBuildInputs` +* `install_requires` corresponds to `propagatedBuildInputs` +* `tests_require` corresponds to `checkInputs` + +## Contributing + +### Contributing guidelines + +Following rules are desired to be respected: + +* Python libraries are called from `python-packages.nix` and packaged with + `buildPythonPackage`. The expression of a library should be in + `pkgs/development/python-modules/<name>/default.nix`. Libraries in + `pkgs/top-level/python-packages.nix` are sorted quasi-alphabetically to avoid + merge conflicts. +* Python applications live outside of `python-packages.nix` and are packaged + with `buildPythonApplication`. +* Make sure libraries build for all Python interpreters. +* By default we enable tests. Make sure the tests are found and, in the case of + libraries, are passing for all interpreters. If certain tests fail they can be + disabled individually. Try to avoid disabling the tests altogether. In any + case, when you disable tests, leave a comment explaining why. +* Commit names of Python libraries should reflect that they are Python + libraries, so write for example `pythonPackages.numpy: 1.11 -> 1.12`. +* Attribute names in `python-packages.nix` should be normalized according to + [PEP 0503](https://www.python.org/dev/peps/pep-0503/#normalized-names). This + means that characters should be converted to lowercase and `.` and `_` should + be replaced by a single `-` (foo-bar-baz instead of Foo__Bar.baz ) diff --git a/nixpkgs/doc/languages-frameworks/qt.xml b/nixpkgs/doc/languages-frameworks/qt.xml new file mode 100644 index 000000000000..8d97de504ad3 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/qt.xml @@ -0,0 +1,149 @@ +<section xmlns="http://docbook.org/ns/docbook" + xmlns:xlink="http://www.w3.org/1999/xlink" + xml:id="sec-language-qt"> + <title>Qt</title> + + <para> + This section describes the differences between Nix expressions for Qt libraries and applications and Nix expressions for other C++ software. Some knowledge of the latter is assumed. There are primarily two problems which the Qt infrastructure is designed to address: ensuring consistent versioning of all dependencies and finding dependencies at runtime. + </para> + + <example xml:id='qt-default-nix'> + <title>Nix expression for a Qt package (<filename>default.nix</filename>)</title> +<programlisting> +{ mkDerivation, lib, qtbase }: <co xml:id='qt-default-nix-co-1' /> + +mkDerivation { <co xml:id='qt-default-nix-co-2' /> + pname = "myapp"; + version = "1.0"; + + buildInputs = [ qtbase ]; <co xml:id='qt-default-nix-co-3' /> +} + </programlisting> + </example> + + <calloutlist> + <callout arearefs='qt-default-nix-co-1'> + <para> + Import <literal>mkDerivation</literal> and Qt (such as <literal>qtbase</literal> modules directly. <emphasis>Do not</emphasis> import Qt package sets; the Qt versions of dependencies may not be coherent, causing build and runtime failures. + </para> + </callout> + <callout arearefs='qt-default-nix-co-2'> + <para> + Use <literal>mkDerivation</literal> instead of <literal>stdenv.mkDerivation</literal>. <literal>mkDerivation</literal> is a wrapper around <literal>stdenv.mkDerivation</literal> which applies some Qt-specific settings. This deriver accepts the same arguments as <literal>stdenv.mkDerivation</literal>; refer to <xref linkend='chap-stdenv' /> for details. + </para> + <para> + To use another deriver instead of <literal>stdenv.mkDerivation</literal>, use <literal>mkDerivationWith</literal>: +<programlisting> +mkDerivationWith myDeriver { + # ... +} +</programlisting> + If you cannot use <literal>mkDerivationWith</literal>, please refer to <xref linkend='qt-runtime-dependencies' />. + </para> + </callout> + <callout arearefs='qt-default-nix-co-3'> + <para> + <literal>mkDerivation</literal> accepts the same arguments as <literal>stdenv.mkDerivation</literal>, such as <literal>buildInputs</literal>. + </para> + </callout> + </calloutlist> + + <formalpara xml:id='qt-runtime-dependencies'> + <title>Locating runtime dependencies</title> + <para> + Qt applications need to be wrapped to find runtime dependencies. If you cannot use <literal>mkDerivation</literal> or <literal>mkDerivationWith</literal> above, include <literal>wrapQtAppsHook</literal> in <literal>nativeBuildInputs</literal>: +<programlisting> +stdenv.mkDerivation { + # ... + + nativeBuildInputs = [ wrapQtAppsHook ]; +} +</programlisting> + </para> + </formalpara> + + <para> + Entries added to <literal>qtWrapperArgs</literal> are used to modify the wrappers created by <literal>wrapQtAppsHook</literal>. The entries are passed as arguments to <xref linkend='fun-wrapProgram' />. +<programlisting> +mkDerivation { + # ... + + qtWrapperArgs = [ ''--prefix PATH : /path/to/bin'' ]; +} +</programlisting> + </para> + + <para> + Set <literal>dontWrapQtApps</literal> to stop applications from being wrapped automatically. It is required to wrap applications manually with <literal>wrapQtApp</literal>, using the syntax of <xref linkend='fun-wrapProgram' />: +<programlisting> +mkDerivation { + # ... + + dontWrapQtApps = true; + preFixup = '' + wrapQtApp "$out/bin/myapp" --prefix PATH : /path/to/bin + ''; +} +</programlisting> + </para> + + <note> + <para> + <literal>wrapQtAppsHook</literal> ignores files that are non-ELF executables. This means that scripts won't be automatically wrapped so you'll need to manually wrap them as previously mentioned. An example of when you'd always need to do this is with Python applications that use PyQT. + </para> + </note> + + <para> + Libraries are built with every available version of Qt. Use the <literal>meta.broken</literal> attribute to disable the package for unsupported Qt versions: +<programlisting> +mkDerivation { + # ... + + # Disable this library with Qt < 5.9.0 + meta.broken = builtins.compareVersions qtbase.version "5.9.0" < 0; +} +</programlisting> + </para> + + <formalpara> + <title>Adding a library to Nixpkgs</title> + <para> + Add a Qt library to <filename>all-packages.nix</filename> by adding it to the collection inside <literal>mkLibsForQt5</literal>. This ensures that the library is built with every available version of Qt as needed. + <example xml:id='qt-library-all-packages-nix'> + <title>Adding a Qt library to <filename>all-packages.nix</filename></title> +<programlisting> +{ + # ... + + mkLibsForQt5 = self: with self; { + # ... + + mylib = callPackage ../path/to/mylib {}; + }; + + # ... +} +</programlisting> + </example> + </para> + </formalpara> + + <formalpara> + <title>Adding an application to Nixpkgs</title> + <para> + Add a Qt application to <filename>all-packages.nix</filename> using <literal>libsForQt5.callPackage</literal> instead of the usual <literal>callPackage</literal>. The former ensures that all dependencies are built with the same version of Qt. + <example xml:id='qt-application-all-packages-nix'> + <title>Adding a Qt application to <filename>all-packages.nix</filename></title> +<programlisting> +{ + # ... + + myapp = libsForQt5.callPackage ../path/to/myapp/ {}; + + # ... +} +</programlisting> + </example> + </para> + </formalpara> +</section> diff --git a/nixpkgs/doc/languages-frameworks/r.section.md b/nixpkgs/doc/languages-frameworks/r.section.md new file mode 100644 index 000000000000..d4e1617779ce --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/r.section.md @@ -0,0 +1,120 @@ +R += + +## Installation + +Define an environment for R that contains all the libraries that you'd like to +use by adding the following snippet to your $HOME/.config/nixpkgs/config.nix file: + +```nix +{ + packageOverrides = super: let self = super.pkgs; in + { + + rEnv = super.rWrapper.override { + packages = with self.rPackages; [ + devtools + ggplot2 + reshape2 + yaml + optparse + ]; + }; + }; +} +``` + +Then you can use `nix-env -f "<nixpkgs>" -iA rEnv` to install it into your user +profile. The set of available libraries can be discovered by running the +command `nix-env -f "<nixpkgs>" -qaP -A rPackages`. The first column from that +output is the name that has to be passed to rWrapper in the code snipped above. + +However, if you'd like to add a file to your project source to make the +environment available for other contributors, you can create a `default.nix` +file like so: +```nix +let + pkgs = import <nixpkgs> {}; + stdenv = pkgs.stdenv; +in with pkgs; { + myProject = stdenv.mkDerivation { + name = "myProject"; + version = "1"; + src = if pkgs.lib.inNixShell then null else nix; + + buildInputs = with rPackages; [ + R + ggplot2 + knitr + ]; + }; +} +``` +and then run `nix-shell .` to be dropped into a shell with those packages +available. + +## RStudio + +RStudio uses a standard set of packages and ignores any custom R +environments or installed packages you may have. To create a custom +environment, see `rstudioWrapper`, which functions similarly to +`rWrapper`: + +```nix +{ + packageOverrides = super: let self = super.pkgs; in + { + + rstudioEnv = super.rstudioWrapper.override { + packages = with self.rPackages; [ + dplyr + ggplot2 + reshape2 + ]; + }; + }; +} +``` + +Then like above, `nix-env -f "<nixpkgs>" -iA rstudioEnv` will install +this into your user profile. + +Alternatively, you can create a self-contained `shell.nix` without the need to +modify any configuration files: + +```nix +{ pkgs ? import <nixpkgs> {} +}: + +pkgs.rstudioWrapper.override { + packages = with pkgs.rPackages; [ dplyr ggplot2 reshape2 ]; +} + +``` + +Executing `nix-shell` will then drop you into an environment equivalent to the +one above. If you need additional packages just add them to the list and +re-enter the shell. + +## Updating the package set + +```bash +nix-shell generate-shell.nix + +Rscript generate-r-packages.R cran > cran-packages.nix.new +mv cran-packages.nix.new cran-packages.nix + +Rscript generate-r-packages.R bioc > bioc-packages.nix.new +mv bioc-packages.nix.new bioc-packages.nix +``` + +`generate-r-packages.R <repo>` reads `<repo>-packages.nix`, therefor the renaming. + + +## Testing if the Nix-expression could be evaluated + +```bash +nix-build test-evaluation.nix --dry-run +``` + +If this exits fine, the expression is ok. If not, you have to edit `default.nix` diff --git a/nixpkgs/doc/languages-frameworks/ruby.section.md b/nixpkgs/doc/languages-frameworks/ruby.section.md new file mode 100644 index 000000000000..e4c4ffce0432 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/ruby.section.md @@ -0,0 +1,365 @@ +--- +title: Ruby +author: Michael Fellinger +date: 2019-05-23 +--- + +# Ruby + +## User Guide + +### Using Ruby + +#### Overview + +Several versions of Ruby interpreters are available on Nix, as well as over 250 gems and many applications written in Ruby. +The attribute `ruby` refers to the default Ruby interpreter, which is currently +MRI 2.5. It's also possible to refer to specific versions, e.g. `ruby_2_6`, `jruby`, or `mruby`. + +In the nixpkgs tree, Ruby packages can be found throughout, depending on what +they do, and are called from the main package set. Ruby gems, however are +separate sets, and there's one default set for each interpreter (currently MRI +only). + +There are two main approaches for using Ruby with gems. +One is to use a specifically locked `Gemfile` for an application that has very strict dependencies. +The other is to depend on the common gems, which we'll explain further down, and +rely on them being updated regularly. + +The interpreters have common attributes, namely `gems`, and `withPackages`. So +you can refer to `ruby.gems.nokogiri`, or `ruby_2_5.gems.nokogiri` to get the +Nokogiri gem already compiled and ready to use. + +Since not all gems have executables like `nokogiri`, it's usually more +convenient to use the `withPackages` function like this: +`ruby.withPackages (p: with p; [ nokogiri ])`. This will also make sure that the +Ruby in your environment will be able to find the gem and it can be used in your +Ruby code (for example via `ruby` or `irb` executables) via `require "nokogiri"` +as usual. + +#### Temporary Ruby environment with `nix-shell` + +Rather than having a single Ruby environment shared by all Ruby +development projects on a system, Nix allows you to create separate +environments per project. `nix-shell` gives you the possibility to +temporarily load another environment akin to a combined `chruby` or +`rvm` and `bundle exec`. + +There are two methods for loading a shell with Ruby packages. The first and +recommended method is to create an environment with `ruby.withPackages` and load +that. + +```shell +nix-shell -p "ruby.withPackages (ps: with ps; [ nokogiri pry ])" +``` + +The other method, which is not recommended, is to create an environment and list +all the packages directly. + +```shell +nix-shell -p ruby.gems.nokogiri ruby.gems.pry +``` + +Again, it's possible to launch the interpreter from the shell. The Ruby +interpreter has the attribute `gems` which contains all Ruby gems for that +specific interpreter. + +##### Load environment from `.nix` expression + +As explained in the Nix manual, `nix-shell` can also load an expression from a +`.nix` file. Say we want to have Ruby 2.5, `nokogori`, and `pry`. Consider a +`shell.nix` file with: + +```nix +with import <nixpkgs> {}; +ruby.withPackages (ps: with ps; [ nokogiri pry ]) +``` + +What's happening here? + +1. We begin with importing the Nix Packages collections. `import <nixpkgs>` + imports the `<nixpkgs>` function, `{}` calls it and the `with` statement + brings all attributes of `nixpkgs` in the local scope. These attributes form + the main package set. +2. Then we create a Ruby environment with the `withPackages` function. +3. The `withPackages` function expects us to provide a function as an argument + that takes the set of all ruby gems and returns a list of packages to include + in the environment. Here, we select the packages `nokogiri` and `pry` from + the package set. + +##### Execute command with `--run` + +A convenient flag for `nix-shell` is `--run`. It executes a command in the +`nix-shell`. We can e.g. directly open a `pry` REPL: + +```shell +nix-shell -p "ruby.withPackages (ps: with ps; [ nokogiri pry ])" --run "pry" +``` + +Or immediately require `nokogiri` in pry: + +```shell +nix-shell -p "ruby.withPackages (ps: with ps; [ nokogiri pry ])" --run "pry -rnokogiri" +``` + +Or run a script using this environment: + +```shell +nix-shell -p "ruby.withPackages (ps: with ps; [ nokogiri pry ])" --run "ruby example.rb" +``` + +##### Using `nix-shell` as shebang + +In fact, for the last case, there is a more convenient method. You can add a +[shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) to your script +specifying which dependencies `nix-shell` needs. With the following shebang, you +can just execute `./example.rb`, and it will run with all dependencies. + +```ruby +#! /usr/bin/env nix-shell +#! nix-shell -i ruby -p "ruby.withPackages (ps: with ps; [ nokogiri rest-client ])" + +require 'nokogiri' +require 'rest-client' + +body = RestClient.get('http://example.com').body +puts Nokogiri::HTML(body).at('h1').text +``` + +### Developing with Ruby + +#### Using an existing Gemfile + +In most cases, you'll already have a `Gemfile.lock` listing all your dependencies. +This can be used to generate a `gemset.nix` which is used to fetch the gems and +combine them into a single environment. +The reason why you need to have a separate file for this, is that Nix requires +you to have a checksum for each input to your build. +Since the `Gemfile.lock` that `bundler` generates doesn't provide us with +checksums, we have to first download each gem, calculate its SHA256, and store +it in this separate file. + +So the steps from having just a `Gemfile` to a `gemset.nix` are: + +```shell +bundle lock +bundix +``` + +If you already have a `Gemfile.lock`, you can simply run `bundix` and it will +work the same. + +To update the gems in your `Gemfile.lock`, you may use the `bundix -l` flag, +which will create a new `Gemfile.lock` in case the `Gemfile` has a more recent +time of modification. + +Once the `gemset.nix` is generated, it can be used in a +`bundlerEnv` derivation. Here is an example you could use for your `shell.nix`: + +```nix +# ... +let + gems = bundlerEnv { + name = "gems-for-some-project"; + gemdir = ./.; + }; +in mkShell { buildInputs = [ gems gems.wrappedRuby ]; } +``` + +With this file in your directory, you can run `nix-shell` to build and use the gems. +The important parts here are `bundlerEnv` and `wrappedRuby`. + +The `bundlerEnv` is a wrapper over all the gems in your gemset. This means that +all the `/lib` and `/bin` directories will be available, and the executables of +all gems (even of indirect dependencies) will end up in your `$PATH`. +The `wrappedRuby` provides you with all executables that come with Ruby itself, +but wrapped so they can easily find the gems in your gemset. + +One common issue that you might have is that you have Ruby 2.6, but also +`bundler` in your gemset. That leads to a conflict for `/bin/bundle` and +`/bin/bundler`. You can resolve this by wrapping either your Ruby or your gems +in a `lowPrio` call. So in order to give the `bundler` from your gemset +priority, it would be used like this: + +```nix +# ... +mkShell { buildInputs = [ gems (lowPrio gems.wrappedRuby) ]; } +``` + + +#### Gem-specific configurations and workarounds + +In some cases, especially if the gem has native extensions, you might need to +modify the way the gem is built. + +This is done via a common configuration file that includes all of the +workarounds for each gem. + +This file lives at `/pkgs/development/ruby-modules/gem-config/default.nix`, +since it already contains a lot of entries, it should be pretty easy to add the +modifications you need for your needs. + +In the meanwhile, or if the modification is for a private gem, you can also add +the configuration to only your own environment. + +Two places that allow this modification are the `ruby` derivation, or `bundlerEnv`. + +Here's the `ruby` one: + +```nix +{ pg_version ? "10", pkgs ? import <nixpkgs> { } }: +let + myRuby = pkgs.ruby.override { + defaultGemConfig = pkgs.defaultGemConfig // { + pg = attrs: { + buildFlags = + [ "--with-pg-config=${pkgs."postgresql_${pg_version}"}/bin/pg_config" ]; + }; + }; + }; +in myRuby.withPackages (ps: with ps; [ pg ]) +``` + +And an example with `bundlerEnv`: + +```nix +{ pg_version ? "10", pkgs ? import <nixpkgs> { } }: +let + gems = pkgs.bundlerEnv { + name = "gems-for-some-project"; + gemdir = ./.; + gemConfig = pkgs.defaultGemConfig // { + pg = attrs: { + buildFlags = + [ "--with-pg-config=${pkgs."postgresql_${pg_version}"}/bin/pg_config" ]; + }; + }; + }; +in mkShell { buildInputs = [ gems gems.wrappedRuby ]; } +``` + +And finally via overlays: + +```nix +{ pg_version ? "10" }: +let + pkgs = import <nixpkgs> { + overlays = [ + (self: super: { + defaultGemConfig = super.defaultGemConfig // { + pg = attrs: { + buildFlags = [ + "--with-pg-config=${ + pkgs."postgresql_${pg_version}" + }/bin/pg_config" + ]; + }; + }; + }) + ]; + }; +in pkgs.ruby.withPackages (ps: with ps; [ pg ]) +``` + +Then we can get whichever postgresql version we desire and the `pg` gem will +always reference it correctly: + +```shell +$ nix-shell --argstr pg_version 9_4 --run 'ruby -rpg -e "puts PG.library_version"' +90421 + +$ nix-shell --run 'ruby -rpg -e "puts PG.library_version"' +100007 +``` + +Of course for this use-case one could also use overlays since the configuration +for `pg` depends on the `postgresql` alias, but for demonstration purposes this +has to suffice. + +#### Adding a gem to the default gemset + +Now that you know how to get a working Ruby environment with Nix, it's time to +go forward and start actually developing with Ruby. +We will first have a look at how Ruby gems are packaged on Nix. Then, we will +look at how you can use development mode with your code. + +All gems in the standard set are automatically generated from a single +`Gemfile`. The dependency resolution is done with `bundler` and makes it more +likely that all gems are compatible to each other. + +In order to add a new gem to nixpkgs, you can put it into the +`/pkgs/development/ruby-modules/with-packages/Gemfile` and run +`./maintainers/scripts/update-ruby-packages`. + +To test that it works, you can then try using the gem with: + +```shell +NIX_PATH=nixpkgs=$PWD nix-shell -p "ruby.withPackages (ps: with ps; [ name-of-your-gem ])" +``` + +#### Packaging applications + +A common task is to add a ruby executable to nixpkgs, popular examples would be +`chef`, `jekyll`, or `sass`. A good way to do that is to use the `bundlerApp` +function, that allows you to make a package that only exposes the listed +executables, otherwise the package may cause conflicts through common paths like +`bin/rake` or `bin/bundler` that aren't meant to be used. + +The absolute easiest way to do that is to write a +`Gemfile` along these lines: + +```ruby +source 'https://rubygems.org' do + gem 'mdl' +end +``` + +If you want to package a specific version, you can use the standard Gemfile +syntax for that, e.g. `gem 'mdl', '0.5.0'`, but if you want the latest stable +version anyway, it's easier to update by simply running the `bundle lock` and +`bundix` steps again. + +Now you can also also make a `default.nix` that looks like this: + +```nix +{ lib, bundlerApp }: + +bundlerApp { + pname = "mdl"; + gemdir = ./.; + exes = [ "mdl" ]; +} +``` + +All that's left to do is to generate the corresponding `Gemfile.lock` and +`gemset.nix` as described above in the `Using an existing Gemfile` section. + +##### Packaging executables that require wrapping + +Sometimes your app will depend on other executables at runtime, and tries to +find it through the `PATH` environment variable. + +In this case, you can provide a `postBuild` hook to `bundlerApp` that wraps the +gem in another script that prefixes the `PATH`. + +Of course you could also make a custom `gemConfig` if you know exactly how to +patch it, but it's usually much easier to maintain with a simple wrapper so the +patch doesn't have to be adjusted for each version. + +Here's another example: + +```nix +{ lib, bundlerApp, makeWrapper, git, gnutar, gzip }: + +bundlerApp { + pname = "r10k"; + gemdir = ./.; + exes = [ "r10k" ]; + + buildInputs = [ makeWrapper ]; + + postBuild = '' + wrapProgram $out/bin/r10k --prefix PATH : ${lib.makeBinPath [ git gnutar gzip ]} + ''; +} +``` diff --git a/nixpkgs/doc/languages-frameworks/ruby.xml b/nixpkgs/doc/languages-frameworks/ruby.xml new file mode 100644 index 000000000000..9b36801fb966 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/ruby.xml @@ -0,0 +1,108 @@ +<section xmlns="http://docbook.org/ns/docbook" + xmlns:xlink="http://www.w3.org/1999/xlink" + xml:id="sec-language-ruby"> + <title>Ruby</title> + + <para> + There currently is support to bundle applications that are packaged as Ruby gems. The utility "bundix" allows you to write a <filename>Gemfile</filename>, let bundler create a <filename>Gemfile.lock</filename>, and then convert this into a nix expression that contains all Gem dependencies automatically. + </para> + + <para> + For example, to package sensu, we did: + </para> + +<screen> +<![CDATA[$ cd pkgs/servers/monitoring +$ mkdir sensu +$ cd sensu +$ cat > Gemfile +source 'https://rubygems.org' +gem 'sensu' +$ $(nix-build '<nixpkgs>' -A bundix --no-out-link)/bin/bundix --magic +$ cat > default.nix +{ lib, bundlerEnv, ruby }: + +bundlerEnv rec { + name = "sensu-${version}"; + + version = (import gemset).sensu.version; + inherit ruby; + # expects Gemfile, Gemfile.lock and gemset.nix in the same directory + gemdir = ./.; + + meta = with lib; { + description = "A monitoring framework that aims to be simple, malleable, and scalable"; + homepage = "http://sensuapp.org/"; + license = with licenses; mit; + maintainers = with maintainers; [ theuni ]; + platforms = platforms.unix; + }; +}]]> +</screen> + + <para> + Please check in the <filename>Gemfile</filename>, <filename>Gemfile.lock</filename> and the <filename>gemset.nix</filename> so future updates can be run easily. + </para> + + <para> + Updating Ruby packages can then be done like this: + </para> + +<screen> +<![CDATA[$ cd pkgs/servers/monitoring/sensu +$ nix-shell -p bundler --run 'bundle lock --update' +$ nix-shell -p bundix --run 'bundix' +]]> +</screen> + + <para> + For tools written in Ruby - i.e. where the desire is to install a package and then execute e.g. <command>rake</command> at the command line, there is an alternative builder called <literal>bundlerApp</literal>. Set up the <filename>gemset.nix</filename> the same way, and then, for example: + </para> + +<screen> +<![CDATA[{ lib, bundlerApp }: + +bundlerApp { + pname = "corundum"; + gemdir = ./.; + exes = [ "corundum-skel" ]; + + meta = with lib; { + description = "Tool and libraries for maintaining Ruby gems."; + homepage = "https://github.com/nyarly/corundum"; + license = licenses.mit; + maintainers = [ maintainers.nyarly ]; + platforms = platforms.unix; + }; +}]]> +</screen> + + <para> + The chief advantage of <literal>bundlerApp</literal> over <literal>bundlerEnv</literal> is the executables introduced in the environment are precisely those selected in the <literal>exes</literal> list, as opposed to <literal>bundlerEnv</literal> which adds all the executables made available by gems in the gemset, which can mean e.g. <command>rspec</command> or <command>rake</command> in unpredictable versions available from various packages. + </para> + + <para> + Resulting derivations for both builders also have two helpful attributes, <literal>env</literal> and <literal>wrappedRuby</literal>. The first one allows one to quickly drop into <command>nix-shell</command> with the specified environment present. E.g. <command>nix-shell -A sensu.env</command> would give you an environment with Ruby preset so it has all the libraries necessary for <literal>sensu</literal> in its paths. The second one can be used to make derivations from custom Ruby scripts which have <filename>Gemfile</filename>s with their dependencies specified. It is a derivation with <command>ruby</command> wrapped so it can find all the needed dependencies. For example, to make a derivation <literal>my-script</literal> for a <filename>my-script.rb</filename> (which should be placed in <filename>bin</filename>) you should run <command>bundix</command> as specified above and then use <literal>bundlerEnv</literal> like this: + </para> + +<programlisting> +<![CDATA[let env = bundlerEnv { + name = "my-script-env"; + + inherit ruby; + gemfile = ./Gemfile; + lockfile = ./Gemfile.lock; + gemset = ./gemset.nix; +}; + +in stdenv.mkDerivation { + name = "my-script"; + buildInputs = [ env.wrappedRuby ]; + script = ./my-script.rb; + buildCommand = '' + install -D -m755 $script $out/bin/my-script + patchShebangs $out/bin/my-script + ''; +}]]> +</programlisting> +</section> diff --git a/nixpkgs/doc/languages-frameworks/rust.section.md b/nixpkgs/doc/languages-frameworks/rust.section.md new file mode 100644 index 000000000000..ec418c563359 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/rust.section.md @@ -0,0 +1,494 @@ +--- +title: Rust +author: Matthias Beyer +date: 2017-03-05 +--- + +# Rust + +To install the rust compiler and cargo put + +``` +rustc +cargo +``` + +into the `environment.systemPackages` or bring them into +scope with `nix-shell -p rustc cargo`. + +For daily builds (beta and nightly) use either rustup from +nixpkgs or use the [Rust nightlies +overlay](#using-the-rust-nightlies-overlay). + +## Compiling Rust applications with Cargo + +Rust applications are packaged by using the `buildRustPackage` helper from `rustPlatform`: + +``` +rustPlatform.buildRustPackage rec { + pname = "ripgrep"; + version = "11.0.2"; + + src = fetchFromGitHub { + owner = "BurntSushi"; + repo = pname; + rev = version; + sha256 = "1iga3320mgi7m853la55xip514a3chqsdi1a1rwv25lr9b1p7vd3"; + }; + + cargoSha256 = "17ldqr3asrdcsh4l29m3b5r37r5d0b3npq1lrgjmxb6vlx6a36qh"; + + meta = with stdenv.lib; { + description = "A fast line-oriented regex search tool, similar to ag and ack"; + homepage = "https://github.com/BurntSushi/ripgrep"; + license = licenses.unlicense; + maintainers = [ maintainers.tailhook ]; + platforms = platforms.all; + }; +} +``` + +`buildRustPackage` requires a `cargoSha256` attribute which is computed over +all crate sources of this package. Currently it is obtained by inserting a +fake checksum into the expression and building the package once. The correct +checksum can be then take from the failed build. + +Per the instructions in the [Cargo Book](https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html) +best practices guide, Rust applications should always commit the `Cargo.lock` +file in git to ensure a reproducible build. However, a few packages do not, and +Nix depends on this file, so if it missing you can use `cargoPatches` to apply +it in the `patchPhase`. Consider sending a PR upstream with a note to the +maintainer describing why it's important to include in the application. + +The fetcher will verify that the `Cargo.lock` file is in sync with the `src` +attribute, and fail the build if not. It will also will compress the vendor +directory into a tar.gz archive. + +### Building a crate for a different target + +To build your crate with a different cargo `--target` simply specify the `target` attribute: + +```nix +pkgs.rustPlatform.buildRustPackage { + (...) + target = "x86_64-fortanix-unknown-sgx"; +} +``` + +### Running package tests + +When using `buildRustPackage`, the `checkPhase` is enabled by default and runs +`cargo test` on the package to build. To make sure that we don't compile the +sources twice and to actually test the artifacts that will be used at runtime, +the tests will be ran in the `release` mode by default. + +However, in some cases the test-suite of a package doesn't work properly in the +`release` mode. For these situations, the mode for `checkPhase` can be changed like +so: + +```nix +rustPlatform.buildRustPackage { + /* ... */ + checkType = "debug"; +} +``` + +Please note that the code will be compiled twice here: once in `release` mode +for the `buildPhase`, and again in `debug` mode for the `checkPhase`. + +#### Tests relying on the structure of the `target/` directory + +Some tests may rely on the structure of the `target/` directory. Those tests +are likely to fail because we use `cargo --target` during the build. This means that +the artifacts +[are stored in `target/<architecture>/release/`](https://doc.rust-lang.org/cargo/guide/build-cache.html), +rather than in `target/release/`. + +This can only be worked around by patching the affected tests accordingly. + +#### Disabling package-tests + +In some instances, it may be necessary to disable testing altogether (with `doCheck = false;`): + +* If no tests exist -- the `checkPhase` should be explicitly disabled to skip + unnecessary build steps to speed up the build. +* If tests are highly impure (e.g. due to network usage). + +There will obviously be some corner-cases not listed above where it's sensible to disable tests. +The above are just guidelines, and exceptions may be granted on a case-by-case basis. + +However, please check if it's possible to disable a problematic subset of the +test suite and leave a comment explaining your reasoning. + +### Building a package in `debug` mode + +By default, `buildRustPackage` will use `release` mode for builds. If a package +should be built in `debug` mode, it can be configured like so: + +```nix +rustPlatform.buildRustPackage { + /* ... */ + buildType = "debug"; +} +``` + +In this scenario, the `checkPhase` will be ran in `debug` mode as well. + +### Custom `build`/`install`-procedures + +Some packages may use custom scripts for building/installing, e.g. with a `Makefile`. +In these cases, it's recommended to override the `buildPhase`/`installPhase`/`checkPhase`. + +Otherwise, some steps may fail because of the modified directory structure of `target/`. + +### Building a crate with an absent or out-of-date Cargo.lock file + +`buildRustPackage` needs a `Cargo.lock` file to get all dependencies in the +source code in a reproducible way. If it is missing or out-of-date one can use +the `cargoPatches` attribute to update or add it. + +``` +{ lib, rustPlatform, fetchFromGitHub }: + +rustPlatform.buildRustPackage rec { + (...) + cargoPatches = [ + # a patch file to add/update Cargo.lock in the source code + ./add-Cargo.lock.patch + ]; +} +``` + +## Compiling Rust crates using Nix instead of Cargo + +### Simple operation + +When run, `cargo build` produces a file called `Cargo.lock`, +containing pinned versions of all dependencies. Nixpkgs contains a +tool called `carnix` (`nix-env -iA nixos.carnix`), which can be used +to turn a `Cargo.lock` into a Nix expression. + +That Nix expression calls `rustc` directly (hence bypassing Cargo), +and can be used to compile a crate and all its dependencies. Here is +an example for a minimal `hello` crate: + + + $ cargo new hello + $ cd hello + $ cargo build + Compiling hello v0.1.0 (file:///tmp/hello) + Finished dev [unoptimized + debuginfo] target(s) in 0.20 secs + $ carnix -o hello.nix --src ./. Cargo.lock --standalone + $ nix-build hello.nix -A hello_0_1_0 + +Now, the file produced by the call to `carnix`, called `hello.nix`, looks like: + +``` +# Generated by carnix 0.6.5: carnix -o hello.nix --src ./. Cargo.lock --standalone +{ lib, stdenv, buildRustCrate, fetchgit }: +let kernel = stdenv.buildPlatform.parsed.kernel.name; + # ... (content skipped) +in +rec { + hello = f: hello_0_1_0 { features = hello_0_1_0_features { hello_0_1_0 = f; }; }; + hello_0_1_0_ = { dependencies?[], buildDependencies?[], features?[] }: buildRustCrate { + crateName = "hello"; + version = "0.1.0"; + authors = [ "pe@pijul.org <pe@pijul.org>" ]; + src = ./.; + inherit dependencies buildDependencies features; + }; + hello_0_1_0 = { features?(hello_0_1_0_features {}) }: hello_0_1_0_ {}; + hello_0_1_0_features = f: updateFeatures f (rec { + hello_0_1_0.default = (f.hello_0_1_0.default or true); + }) [ ]; +} +``` + +In particular, note that the argument given as `--src` is copied +verbatim to the source. If we look at a more complicated +dependencies, for instance by adding a single line `libc="*"` to our +`Cargo.toml`, we first need to run `cargo build` to update the +`Cargo.lock`. Then, `carnix` needs to be run again, and produces the +following nix file: + +``` +# Generated by carnix 0.6.5: carnix -o hello.nix --src ./. Cargo.lock --standalone +{ lib, stdenv, buildRustCrate, fetchgit }: +let kernel = stdenv.buildPlatform.parsed.kernel.name; + # ... (content skipped) +in +rec { + hello = f: hello_0_1_0 { features = hello_0_1_0_features { hello_0_1_0 = f; }; }; + hello_0_1_0_ = { dependencies?[], buildDependencies?[], features?[] }: buildRustCrate { + crateName = "hello"; + version = "0.1.0"; + authors = [ "pe@pijul.org <pe@pijul.org>" ]; + src = ./.; + inherit dependencies buildDependencies features; + }; + libc_0_2_36_ = { dependencies?[], buildDependencies?[], features?[] }: buildRustCrate { + crateName = "libc"; + version = "0.2.36"; + authors = [ "The Rust Project Developers" ]; + sha256 = "01633h4yfqm0s302fm0dlba469bx8y6cs4nqc8bqrmjqxfxn515l"; + inherit dependencies buildDependencies features; + }; + hello_0_1_0 = { features?(hello_0_1_0_features {}) }: hello_0_1_0_ { + dependencies = mapFeatures features ([ libc_0_2_36 ]); + }; + hello_0_1_0_features = f: updateFeatures f (rec { + hello_0_1_0.default = (f.hello_0_1_0.default or true); + libc_0_2_36.default = true; + }) [ libc_0_2_36_features ]; + libc_0_2_36 = { features?(libc_0_2_36_features {}) }: libc_0_2_36_ { + features = mkFeatures (features.libc_0_2_36 or {}); + }; + libc_0_2_36_features = f: updateFeatures f (rec { + libc_0_2_36.default = (f.libc_0_2_36.default or true); + libc_0_2_36.use_std = + (f.libc_0_2_36.use_std or false) || + (f.libc_0_2_36.default or false) || + (libc_0_2_36.default or false); + }) []; +} +``` + +Here, the `libc` crate has no `src` attribute, so `buildRustCrate` +will fetch it from [crates.io](https://crates.io). A `sha256` +attribute is still needed for Nix purity. + +### Handling external dependencies + +Some crates require external libraries. For crates from +[crates.io](https://crates.io), such libraries can be specified in +`defaultCrateOverrides` package in nixpkgs itself. + +Starting from that file, one can add more overrides, to add features +or build inputs by overriding the hello crate in a seperate file. + +``` +with import <nixpkgs> {}; +((import ./hello.nix).hello {}).override { + crateOverrides = defaultCrateOverrides // { + hello = attrs: { buildInputs = [ openssl ]; }; + }; +} +``` + +Here, `crateOverrides` is expected to be a attribute set, where the +key is the crate name without version number and the value a function. +The function gets all attributes passed to `buildRustCrate` as first +argument and returns a set that contains all attribute that should be +overwritten. + +For more complicated cases, such as when parts of the crate's +derivation depend on the crate's version, the `attrs` argument of +the override above can be read, as in the following example, which +patches the derivation: + +``` +with import <nixpkgs> {}; +((import ./hello.nix).hello {}).override { + crateOverrides = defaultCrateOverrides // { + hello = attrs: lib.optionalAttrs (lib.versionAtLeast attrs.version "1.0") { + postPatch = '' + substituteInPlace lib/zoneinfo.rs \ + --replace "/usr/share/zoneinfo" "${tzdata}/share/zoneinfo" + ''; + }; + }; +} +``` + +Another situation is when we want to override a nested +dependency. This actually works in the exact same way, since the +`crateOverrides` parameter is forwarded to the crate's +dependencies. For instance, to override the build inputs for crate +`libc` in the example above, where `libc` is a dependency of the main +crate, we could do: + +``` +with import <nixpkgs> {}; +((import hello.nix).hello {}).override { + crateOverrides = defaultCrateOverrides // { + libc = attrs: { buildInputs = []; }; + }; +} +``` + +### Options and phases configuration + +Actually, the overrides introduced in the previous section are more +general. A number of other parameters can be overridden: + +- The version of rustc used to compile the crate: + + ``` + (hello {}).override { rust = pkgs.rust; }; + ``` + +- Whether to build in release mode or debug mode (release mode by + default): + + ``` + (hello {}).override { release = false; }; + ``` + +- Whether to print the commands sent to rustc when building + (equivalent to `--verbose` in cargo: + + ``` + (hello {}).override { verbose = false; }; + ``` + +- Extra arguments to be passed to `rustc`: + + ``` + (hello {}).override { extraRustcOpts = "-Z debuginfo=2"; }; + ``` + +- Phases, just like in any other derivation, can be specified using + the following attributes: `preUnpack`, `postUnpack`, `prePatch`, + `patches`, `postPatch`, `preConfigure` (in the case of a Rust crate, + this is run before calling the "build" script), `postConfigure` + (after the "build" script),`preBuild`, `postBuild`, `preInstall` and + `postInstall`. As an example, here is how to create a new module + before running the build script: + + ``` + (hello {}).override { + preConfigure = '' + echo "pub const PATH=\"${hi.out}\";" >> src/path.rs" + ''; + }; + ``` + +### Features + +One can also supply features switches. For example, if we want to +compile `diesel_cli` only with the `postgres` feature, and no default +features, we would write: + +``` +(callPackage ./diesel.nix {}).diesel { + default = false; + postgres = true; +} +``` + +Where `diesel.nix` is the file generated by Carnix, as explained above. + + +## Setting Up `nix-shell` +Oftentimes you want to develop code from within `nix-shell`. Unfortunately +`buildRustCrate` does not support common `nix-shell` operations directly +(see [this issue](https://github.com/NixOS/nixpkgs/issues/37945)) +so we will use `stdenv.mkDerivation` instead. + +Using the example `hello` project above, we want to do the following: +- Have access to `cargo` and `rustc` +- Have the `openssl` library available to a crate through it's _normal_ + compilation mechanism (`pkg-config`). + +A typical `shell.nix` might look like: + +``` +with import <nixpkgs> {}; + +stdenv.mkDerivation { + name = "rust-env"; + nativeBuildInputs = [ + rustc cargo + + # Example Build-time Additional Dependencies + pkgconfig + ]; + buildInputs = [ + # Example Run-time Additional Dependencies + openssl + ]; + + # Set Environment Variables + RUST_BACKTRACE = 1; +} +``` + +You should now be able to run the following: +``` +$ nix-shell --pure +$ cargo build +$ cargo test +``` + +### Controlling Rust Version Inside `nix-shell` +To control your rust version (i.e. use nightly) from within `shell.nix` (or +other nix expressions) you can use the following `shell.nix` + +``` +# Latest Nightly +with import <nixpkgs> {}; +let src = fetchFromGitHub { + owner = "mozilla"; + repo = "nixpkgs-mozilla"; + # commit from: 2019-05-15 + rev = "9f35c4b09fd44a77227e79ff0c1b4b6a69dff533"; + sha256 = "18h0nvh55b5an4gmlgfbvwbyqj91bklf1zymis6lbdh75571qaz0"; + }; +in +with import "${src.out}/rust-overlay.nix" pkgs pkgs; +stdenv.mkDerivation { + name = "rust-env"; + buildInputs = [ + # Note: to use use stable, just replace `nightly` with `stable` + latest.rustChannels.nightly.rust + + # Add some extra dependencies from `pkgs` + pkgconfig openssl + ]; + + # Set Environment Variables + RUST_BACKTRACE = 1; +} +``` + +Now run: +``` +$ rustc --version +rustc 1.26.0-nightly (188e693b3 2018-03-26) +``` + +To see that you are using nightly. + + +## Using the Rust nightlies overlay + +Mozilla provides an overlay for nixpkgs to bring a nightly version of Rust into scope. +This overlay can _also_ be used to install recent unstable or stable versions +of Rust, if desired. + +To use this overlay, clone +[nixpkgs-mozilla](https://github.com/mozilla/nixpkgs-mozilla), +and create a symbolic link to the file +[rust-overlay.nix](https://github.com/mozilla/nixpkgs-mozilla/blob/master/rust-overlay.nix) +in the `~/.config/nixpkgs/overlays` directory. + + $ git clone https://github.com/mozilla/nixpkgs-mozilla.git + $ mkdir -p ~/.config/nixpkgs/overlays + $ ln -s $(pwd)/nixpkgs-mozilla/rust-overlay.nix ~/.config/nixpkgs/overlays/rust-overlay.nix + +The latest version can be installed with the following command: + + $ nix-env -Ai nixos.latest.rustChannels.stable.rust + +Or using the attribute with nix-shell: + + $ nix-shell -p nixos.latest.rustChannels.stable.rust + +To install the beta or nightly channel, "stable" should be substituted by +"nightly" or "beta", or +use the function provided by this overlay to pull a version based on a +build date. + +The overlay automatically updates itself as it uses the same source as +[rustup](https://www.rustup.rs/). diff --git a/nixpkgs/doc/languages-frameworks/texlive.xml b/nixpkgs/doc/languages-frameworks/texlive.xml new file mode 100644 index 000000000000..a581ec5911cb --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/texlive.xml @@ -0,0 +1,152 @@ +<section xmlns="http://docbook.org/ns/docbook" + xmlns:xlink="http://www.w3.org/1999/xlink" + xml:id="sec-language-texlive"> + <title>TeX Live</title> + + <para> + Since release 15.09 there is a new TeX Live packaging that lives entirely under attribute <varname>texlive</varname>. + </para> + + <section xml:id="sec-language-texlive-users-guide"> + <title>User's guide</title> + + <itemizedlist> + <listitem> + <para> + For basic usage just pull <varname>texlive.combined.scheme-basic</varname> for an environment with basic LaTeX support. + </para> + </listitem> + <listitem> + <para> + It typically won't work to use separately installed packages together. Instead, you can build a custom set of packages like this: +<programlisting> +texlive.combine { + inherit (texlive) scheme-small collection-langkorean algorithms cm-super; +} +</programlisting> + There are all the schemes, collections and a few thousand packages, as defined upstream (perhaps with tiny differences). + </para> + </listitem> + <listitem> + <para> + By default you only get executables and files needed during runtime, and a little documentation for the core packages. To change that, you need to add <varname>pkgFilter</varname> function to <varname>combine</varname>. +<programlisting> +texlive.combine { + # inherit (texlive) whatever-you-want; + pkgFilter = pkg: + pkg.tlType == "run" || pkg.tlType == "bin" || pkg.pname == "cm-super"; + # elem tlType [ "run" "bin" "doc" "source" ] + # there are also other attributes: version, name +} +</programlisting> + </para> + </listitem> + <listitem> + <para> + You can list packages e.g. by <command>nix repl</command>. +<programlisting><![CDATA[ +$ nix repl +nix-repl> :l <nixpkgs> +nix-repl> texlive.collection-<TAB> +]]></programlisting> + </para> + </listitem> + <listitem> + <para> + Note that the wrapper assumes that the result has a chance to be useful. For example, the core executables should be present, as well as some core data files. The supported way of ensuring this is by including some scheme, for example <varname>scheme-basic</varname>, into the combination. + </para> + </listitem> + </itemizedlist> + </section> + + <section xml:id="sec-language-texlive-custom-packages"> + <title>Custom packages</title> + <para> + You may find that you need to use an external TeX package. A derivation for such package has to provide contents of the "texmf" directory in its output and provide the <varname>tlType</varname> attribute. Here is a (very verbose) example: +<programlisting><![CDATA[ +with import <nixpkgs> {}; + +let + foiltex_run = stdenvNoCC.mkDerivation { + pname = "latex-foiltex"; + version = "2.1.4b"; + passthru.tlType = "run"; + + srcs = [ + (fetchurl { + url = "http://mirrors.ctan.org/macros/latex/contrib/foiltex/foiltex.dtx"; + sha256 = "07frz0krpz7kkcwlayrwrj2a2pixmv0icbngyw92srp9fp23cqpz"; + }) + (fetchurl { + url = "http://mirrors.ctan.org/macros/latex/contrib/foiltex/foiltex.ins"; + sha256 = "09wkyidxk3n3zvqxfs61wlypmbhi1pxmjdi1kns9n2ky8ykbff99"; + }) + ]; + + unpackPhase = '' + runHook preUnpack + + for _src in $srcs; do + cp "$_src" $(stripHash "$_src") + done + + runHook postUnpack + ''; + + nativeBuildInputs = [ texlive.combined.scheme-small ]; + + dontConfigure = true; + + buildPhase = '' + runHook preBuild + + # Generate the style files + latex foiltex.ins + + runHook postBuild + ''; + + installPhase = '' + runHook preInstall + + path="$out/tex/latex/foiltex" + mkdir -p "$path" + cp *.{cls,def,clo} "$path/" + + runHook postInstall + ''; + + meta = with lib; { + description = "A LaTeX2e class for overhead transparencies"; + license = licenses.unfreeRedistributable; + maintainers = with maintainers; [ veprbl ]; + platforms = platforms.all; + }; + }; + foiltex = { pkgs = [ foiltex_run ]; }; + + latex_with_foiltex = texlive.combine { + inherit (texlive) scheme-small; + inherit foiltex; + }; +in + runCommand "test.pdf" { + nativeBuildInputs = [ latex_with_foiltex ]; + } '' +cat >test.tex <<EOF +\documentclass{foils} + +\title{Presentation title} +\date{} + +\begin{document} +\maketitle +\end{document} +EOF + pdflatex test.tex + cp test.pdf $out +'' +]]></programlisting> + </para> + </section> +</section> diff --git a/nixpkgs/doc/languages-frameworks/titanium.section.md b/nixpkgs/doc/languages-frameworks/titanium.section.md new file mode 100644 index 000000000000..7a97664ec598 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/titanium.section.md @@ -0,0 +1,115 @@ +--- +title: Titanium +author: Sander van der Burg +date: 2018-11-18 +--- +# Titanium + +The Nixpkgs repository contains facilities to deploy a variety of versions of +the [Titanium SDK](https://www.appcelerator.com) versions, a cross-platform +mobile app development framework using JavaScript as an implementation language, +and includes a function abstraction making it possible to build Titanium +applications for Android and iOS devices from source code. + +Not all Titanium features supported -- currently, it can only be used to build +Android and iOS apps. + +Building a Titanium app +----------------------- +We can build a Titanium app from source for Android or iOS and for debugging or +release purposes by invoking the `titaniumenv.buildApp {}` function: + +```nix +titaniumenv.buildApp { + name = "myapp"; + src = ./myappsource; + + preBuild = ""; + target = "android"; # or 'iphone' + tiVersion = "7.1.0.GA"; + release = true; + + androidsdkArgs = { + platformVersions = [ "25" "26" ]; + }; + androidKeyStore = ./keystore; + androidKeyAlias = "myfirstapp"; + androidKeyStorePassword = "secret"; + + xcodeBaseDir = "/Applications/Xcode.app"; + xcodewrapperArgs = { + version = "9.3"; + }; + iosMobileProvisioningProfile = ./myprovisioning.profile; + iosCertificateName = "My Company"; + iosCertificate = ./mycertificate.p12; + iosCertificatePassword = "secret"; + iosVersion = "11.3"; + iosBuildStore = false; + + enableWirelessDistribution = true; + installURL = "/installipa.php"; +} +``` + +The `titaniumenv.buildApp {}` function takes the following parameters: + +* The `name` parameter refers to the name in the Nix store. +* The `src` parameter refers to the source code location of the app that needs + to be built. +* `preRebuild` contains optional build instructions that are carried out before + the build starts. +* `target` indicates for which device the app must be built. Currently only + 'android' and 'iphone' (for iOS) are supported. +* `tiVersion` can be used to optionally override the requested Titanium version + in `tiapp.xml`. If not specified, it will use the version in `tiapp.xml`. +* `release` should be set to true when building an app for submission to the + Google Playstore or Apple Appstore. Otherwise, it should be false. + +When the `target` has been set to `android`, we can configure the following +parameters: + +* The `androidSdkArgs` parameter refers to an attribute set that propagates all + parameters to the `androidenv.composeAndroidPackages {}` function. This can + be used to install all relevant Android plugins that may be needed to perform + the Android build. If no parameters are given, it will deploy the platform + SDKs for API-levels 25 and 26 by default. + +When the `release` parameter has been set to true, you need to provide +parameters to sign the app: + +* `androidKeyStore` is the path to the keystore file +* `androidKeyAlias` is the key alias +* `androidKeyStorePassword` refers to the password to open the keystore file. + +When the `target` has been set to `iphone`, we can configure the following +parameters: + +* The `xcodeBaseDir` parameter refers to the location where Xcode has been + installed. When none value is given, the above value is the default. +* The `xcodewrapperArgs` parameter passes arbitrary parameters to the + `xcodeenv.composeXcodeWrapper {}` function. This can, for example, be used + to adjust the default version of Xcode. + +When `release` has been set to true, you also need to provide the following +parameters: + +* `iosMobileProvisioningProfile` refers to a mobile provisioning profile needed + for signing. +* `iosCertificateName` refers to the company name in the P12 certificate. +* `iosCertificate` refers to the path to the P12 file. +* `iosCertificatePassword` contains the password to open the P12 file. +* `iosVersion` refers to the iOS SDK version to use. It defaults to the latest + version. +* `iosBuildStore` should be set to `true` when building for the Apple Appstore + submission. For enterprise or ad-hoc builds it should be set to `false`. + +When `enableWirelessDistribution` has been enabled, you must also provide the +path of the PHP script (`installURL`) (that is included with the iOS build +environment) to enable wireless ad-hoc installations. + +Emulating or simulating the app +------------------------------- +It is also possible to simulate the correspond iOS simulator build by using +`xcodeenv.simulateApp {}` and emulate an Android APK by using +`androidenv.emulateApp {}`. diff --git a/nixpkgs/doc/languages-frameworks/vim.section.md b/nixpkgs/doc/languages-frameworks/vim.section.md new file mode 100644 index 000000000000..4911509212e6 --- /dev/null +++ b/nixpkgs/doc/languages-frameworks/vim.section.md @@ -0,0 +1,272 @@ +--- +title: User's Guide for Vim in Nixpkgs +author: Marc Weber +date: 2016-06-25 +--- +# Vim + +Both Neovim and Vim can be configured to include your favorite plugins +and additional libraries. + +Loading can be deferred; see examples. + +At the moment we support three different methods for managing plugins: + +- Vim packages (*recommend*) +- VAM (=vim-addon-manager) +- Pathogen +- vim-plug + +## Custom configuration + +Adding custom .vimrc lines can be done using the following code: + +```nix +vim_configurable.customize { + # `name` specifies the name of the executable and package + name = "vim-with-plugins"; + + vimrcConfig.customRC = '' + set hidden + ''; +} +``` + +This configuration is used when Vim is invoked with the command specified as name, in this case `vim-with-plugins`. + +For Neovim the `configure` argument can be overridden to achieve the same: + +```nix +neovim.override { + configure = { + customRC = '' + # here your custom configuration goes! + ''; + }; +} +``` + +If you want to use `neovim-qt` as a graphical editor, you can configure it by overriding Neovim in an overlay +or passing it an overridden Neovimn: + +```nix +neovim-qt.override { + neovim = neovim.override { + configure = { + customRC = '' + # your custom configuration + ''; + }; + }; +} +``` + +## Managing plugins with Vim packages + +To store you plugins in Vim packages (the native Vim plugin manager, see `:help packages`) the following example can be used: + +```nix +vim_configurable.customize { + vimrcConfig.packages.myVimPackage = with pkgs.vimPlugins; { + # loaded on launch + start = [ youcompleteme fugitive ]; + # manually loadable by calling `:packadd $plugin-name` + # however, if a Vim plugin has a dependency that is not explicitly listed in + # opt that dependency will always be added to start to avoid confusion. + opt = [ phpCompletion elm-vim ]; + # To automatically load a plugin when opening a filetype, add vimrc lines like: + # autocmd FileType php :packadd phpCompletion + }; +} +``` + +`myVimPackage` is an arbitrary name for the generated package. You can choose any name you like. +For Neovim the syntax is: + +```nix +neovim.override { + configure = { + customRC = '' + # here your custom configuration goes! + ''; + packages.myVimPackage = with pkgs.vimPlugins; { + # see examples below how to use custom packages + start = [ ]; + # If a Vim plugin has a dependency that is not explicitly listed in + # opt that dependency will always be added to start to avoid confusion. + opt = [ ]; + }; + }; +} +``` + +The resulting package can be added to `packageOverrides` in `~/.nixpkgs/config.nix` to make it installable: + +```nix +{ + packageOverrides = pkgs: with pkgs; { + myVim = vim_configurable.customize { + # `name` specifies the name of the executable and package + name = "vim-with-plugins"; + # add here code from the example section + }; + myNeovim = neovim.override { + configure = { + # add here code from the example section + }; + }; + }; +} +``` + +After that you can install your special grafted `myVim` or `myNeovim` packages. + +## Managing plugins with vim-plug + +To use [vim-plug](https://github.com/junegunn/vim-plug) to manage your Vim +plugins the following example can be used: + +```nix +vim_configurable.customize { + vimrcConfig.packages.myVimPackage = with pkgs.vimPlugins; { + # loaded on launch + plug.plugins = [ youcompleteme fugitive phpCompletion elm-vim ]; + }; +} +``` + +For Neovim the syntax is: + +```nix +neovim.override { + configure = { + customRC = '' + # here your custom configuration goes! + ''; + plug.plugins = with pkgs.vimPlugins; [ + vim-go + ]; + }; +} +``` + +## Managing plugins with VAM + +### Handling dependencies of Vim plugins + +VAM introduced .json files supporting dependencies without versioning +assuming that "using latest version" is ok most of the time. + +### Example + +First create a vim-scripts file having one plugin name per line. Example: + +``` +"tlib" +{'name': 'vim-addon-sql'} +{'filetype_regex': '\%(vim)$', 'names': ['reload', 'vim-dev-plugin']} +``` + +Such vim-scripts file can be read by VAM as well like this: + +```vim +call vam#Scripts(expand('~/.vim-scripts'), {}) +``` + +Create a default.nix file: + +```nix +{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc7102" }: +nixpkgs.vim_configurable.customize { name = "vim"; vimrcConfig.vam.pluginDictionaries = [ "vim-addon-vim2nix" ]; } +``` + +Create a generate.vim file: + +```vim +ActivateAddons vim-addon-vim2nix +let vim_scripts = "vim-scripts" +call nix#ExportPluginsForNix({ +\ 'path_to_nixpkgs': eval('{"'.substitute(substitute(substitute($NIX_PATH, ':', ',', 'g'), '=',':', 'g'), '\([:,]\)', '"\1"',"g").'"}')["nixpkgs"], +\ 'cache_file': '/tmp/vim2nix-cache', +\ 'try_catch': 0, +\ 'plugin_dictionaries': ["vim-addon-manager"]+map(readfile(vim_scripts), 'eval(v:val)') +\ }) +``` + +Then run + +```bash +nix-shell -p vimUtils.vim_with_vim2nix --command "vim -c 'source generate.vim'" +``` + +You should get a Vim buffer with the nix derivations (output1) and vam.pluginDictionaries (output2). +You can add your Vim to your system's configuration file like this and start it by "vim-my": + +``` +my-vim = + let plugins = let inherit (vimUtils) buildVimPluginFrom2Nix; in { + copy paste output1 here + }; in vim_configurable.customize { + name = "vim-my"; + + vimrcConfig.vam.knownPlugins = plugins; # optional + vimrcConfig.vam.pluginDictionaries = [ + copy paste output2 here + ]; + + # Pathogen would be + # vimrcConfig.pathogen.knownPlugins = plugins; # plugins + # vimrcConfig.pathogen.pluginNames = ["tlib"]; + }; +``` + +Sample output1: + +``` +"reload" = buildVimPluginFrom2Nix { # created by nix#NixDerivation + name = "reload"; + src = fetchgit { + url = "git://github.com/xolox/vim-reload"; + rev = "0a601a668727f5b675cb1ddc19f6861f3f7ab9e1"; + sha256 = "0vb832l9yxj919f5hfg6qj6bn9ni57gnjd3bj7zpq7d4iv2s4wdh"; + }; + dependencies = ["nim-misc"]; + +}; +[...] +``` + +Sample output2: + +```nix +[ + ''vim-addon-manager'' + ''tlib'' + { "name" = ''vim-addon-sql''; } + { "filetype_regex" = ''\%(vim)$$''; "names" = [ ''reload'' ''vim-dev-plugin'' ]; } +] +``` + +## Adding new plugins to nixpkgs + +Nix expressions for Vim plugins are stored in [pkgs/misc/vim-plugins](/pkgs/misc/vim-plugins). For the vast majority of plugins, Nix expressions are automatically generated by running [`./update.py`](/pkgs/misc/vim-plugins/update.py). This creates a [generated.nix](/pkgs/misc/vim-plugins/generated.nix) file based on the plugins listed in [vim-plugin-names](/pkgs/misc/vim-plugins/vim-plugin-names). Plugins are listed in alphabetical order in `vim-plugin-names` using the format `[github username]/[repository]`. For example https://github.com/scrooloose/nerdtree becomes `scrooloose/nerdtree`. + +Some plugins require overrides in order to function properly. Overrides are placed in [overrides.nix](/pkgs/misc/vim-plugins/overrides.nix). Overrides are most often required when a plugin requires some dependencies, or extra steps are required during the build process. For example `deoplete-fish` requires both `deoplete-nvim` and `vim-fish`, and so the following override was added: + +``` +deoplete-fish = super.deoplete-fish.overrideAttrs(old: { + dependencies = with super; [ deoplete-nvim vim-fish ]; +}); +``` + +Sometimes plugins require an override that must be changed when the plugin is updated. This can cause issues when Vim plugins are auto-updated but the associated override isn't updated. For these plugins, the override should be written so that it specifies all information required to install the plugin, and running `./update.py` doesn't change the derivation for the plugin. Manually updating the override is required to update these types of plugins. An example of such a plugin is `LanguageClient-neovim`. + +To add a new plugin, run `./update.py --add "[owner]/[name]"`. **NOTE**: This script automatically commits to your git repository. Be sure to check out a fresh branch before running. + +## Important repositories + +- [vim-pi](https://bitbucket.org/vimcommunity/vim-pi) is a plugin repository + from VAM plugin manager meant to be used by others as well used by + +- [vim2nix](https://github.com/MarcWeber/vim-addon-vim2nix) which generates the + .nix code |