diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/build-helpers/fetchers.chapter.md | 529 | ||||
| -rw-r--r-- | doc/languages-frameworks/dlang.section.md | 69 | ||||
| -rw-r--r-- | doc/languages-frameworks/index.md | 1 | ||||
| -rw-r--r-- | doc/manpage-urls.json | 4 |
4 files changed, 564 insertions, 39 deletions
diff --git a/doc/build-helpers/fetchers.chapter.md b/doc/build-helpers/fetchers.chapter.md index 5c7c3257e6d4..6efe64281e82 100644 --- a/doc/build-helpers/fetchers.chapter.md +++ b/doc/build-helpers/fetchers.chapter.md @@ -1,88 +1,541 @@ # Fetchers {#chap-pkgs-fetchers} Building software with Nix often requires downloading source code and other files from the internet. -To this end, Nixpkgs provides *fetchers*: functions to obtain remote sources via various protocols and services. +To this end, we use functions that we call _fetchers_, which obtain remote sources via various protocols and services. + +Nix provides built-in fetchers such as [`builtins.fetchTarball`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-fetchTarball). +Nixpkgs provides its own fetchers, which work differently: -Nixpkgs fetchers differ from built-in fetchers such as [`builtins.fetchTarball`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-fetchTarball): - A built-in fetcher will download and cache files at evaluation time and produce a [store path](https://nixos.org/manual/nix/stable/glossary#gloss-store-path). - A Nixpkgs fetcher will create a ([fixed-output](https://nixos.org/manual/nix/stable/glossary#gloss-fixed-output-derivation)) [derivation](https://nixos.org/manual/nix/stable/language/derivations), and files are downloaded at build time. + A Nixpkgs fetcher will create a ([fixed-output](https://nixos.org/manual/nix/stable/glossary#gloss-fixed-output-derivation)) [derivation](https://nixos.org/manual/nix/stable/glossary#gloss-derivation), and files are downloaded at build time. - Built-in fetchers will invalidate their cache after [`tarball-ttl`](https://nixos.org/manual/nix/stable/command-ref/conf-file#conf-tarball-ttl) expires, and will require network activity to check if the cache entry is up to date. - Nixpkgs fetchers only re-download if the specified hash changes or the store object is not otherwise available. + Nixpkgs fetchers only re-download if the specified hash changes or the store object is not available. - Built-in fetchers do not use [substituters](https://nixos.org/manual/nix/stable/command-ref/conf-file#conf-substituters). Derivations produced by Nixpkgs fetchers will use any configured binary cache transparently. -This significantly reduces the time needed to evaluate the entirety of Nixpkgs, and allows [Hydra](https://nixos.org/hydra) to retain and re-distribute sources used by Nixpkgs in the [public binary cache](https://cache.nixos.org). -For these reasons, built-in fetchers are not allowed in Nixpkgs source code. +This significantly reduces the time needed to evaluate Nixpkgs, and allows [Hydra](https://nixos.org/hydra) to retain and re-distribute sources used by Nixpkgs in the [public binary cache](https://cache.nixos.org). +For these reasons, Nix's built-in fetchers are not allowed in Nixpkgs. -The following table shows an overview of the differences: +The following table summarises the differences: | Fetchers | Download | Output | Cache | Re-download when | |-|-|-|-|-| | `builtins.fetch*` | evaluation time | store path | `/nix/store`, `~/.cache/nix` | `tarball-ttl` expires, cache miss in `~/.cache/nix`, output store object not in local store | | `pkgs.fetch*` | build time | derivation | `/nix/store`, substituters | output store object not available | +:::{.tip} +`pkgs.fetchFrom*` helpers retrieve _snapshots_ of version-controlled sources, as opposed to the entire version history, which is more efficient. +`pkgs.fetchgit` by default also has the same behaviour, but can be changed through specific attributes given to it. +::: + ## Caveats {#chap-pkgs-fetchers-caveats} -The fact that the hash belongs to the Nix derivation output and not the file itself can lead to confusion. -For example, consider the following fetcher: +Because Nixpkgs fetchers are fixed-output derivations, an [output hash](https://nixos.org/manual/nix/stable/language/advanced-attributes#adv-attr-outputHash) has to be specified, usually indirectly through a `hash` attribute. +This hash refers to the derivation output, which can be different from the remote source itself! + +This has the following implications that you should be aware of: + +- Use Nix (or Nix-aware) tooling to produce the output hash. + +- When changing any fetcher parameters, always update the output hash. + Use one of the methods from [](#sec-pkgs-fetchers-updating-source-hashes). + Otherwise, existing store objects that match the output hash will be re-used rather than fetching new content. + + :::{.note} + A similar problem arises while testing changes to a fetcher's implementation. + If the output of the derivation already exists in the Nix store, test failures can go undetected. + The [`invalidateFetcherByDrvHash`](#tester-invalidateFetcherByDrvHash) function helps prevent reusing cached derivations. + ::: + +## Updating source hashes {#sec-pkgs-fetchers-updating-source-hashes} + +There are several ways to obtain the hash corresponding to a remote source. +Unless you understand how the fetcher you're using calculates the hash from the downloaded contents, you should use [the fake hash method](#sec-pkgs-fetchers-updating-source-hashes-fakehash-method). + +1. []{#sec-pkgs-fetchers-updating-source-hashes-fakehash-method} The fake hash method: In your package recipe, set the hash to one of + + - `""` + - `lib.fakeHash` + - `lib.fakeSha256` + - `lib.fakeSha512` + + Attempt to build, extract the calculated hashes from error messages, and put them into the recipe. + + :::{.warning} + You must use one of these four fake hashes and not some arbitrarily-chosen hash. + See [](#sec-pkgs-fetchers-secure-hashes) for details. + ::: + + :::{.example #ex-fetchers-update-fod-hash} + # Update source hash with the fake hash method + + Consider the following recipe that produces a plain file: + + ```nix + { fetchurl }: + fetchurl { + url = "https://raw.githubusercontent.com/NixOS/nixpkgs/23.05/.version"; + hash = "sha256-ZHl1emidXVojm83LCVrwULpwIzKE/mYwfztVkvpruOM="; + } + ``` + + A common mistake is to update a fetcher parameter, such as `url`, without updating the hash: + + ```nix + { fetchurl }: + fetchurl { + url = "https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/.version"; + hash = "sha256-ZHl1emidXVojm83LCVrwULpwIzKE/mYwfztVkvpruOM="; + } + ``` + + **This will produce the same output as before!** + Set the hash to an empty string: + + ```nix + { fetchurl }: + fetchurl { + url = "https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/.version"; + hash = ""; + } + ``` + + When building the package, use the error message to determine the correct hash: + + ```shell + $ nix-build + (some output removed for clarity) + error: hash mismatch in fixed-output derivation '/nix/store/7yynn53jpc93l76z9zdjj4xdxgynawcw-version.drv': + specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= + got: sha256-BZqI7r0MNP29yGH5+yW2tjU9OOpOCEvwWKrWCv5CQ0I= + error: build of '/nix/store/bqdjcw5ij5ymfbm41dq230chk9hdhqff-version.drv' failed + ``` + ::: + +2. Prefetch the source with [`nix-prefetch-<type> <URL>`](https://search.nixos.org/packages?buckets={%22package_attr_set%22%3A[%22No%20package%20set%22]%2C%22package_license_set%22%3A[]%2C%22package_maintainers_set%22%3A[]%2C%22package_platforms%22%3A[]}&query=nix-prefetch), where `<type>` is one of + + - `url` + - `git` + - `hg` + - `cvs` + - `bzr` + - `svn` + + The hash is printed to stdout. + +3. Prefetch by package source (with `nix-prefetch-url '<nixpkgs>' -A <package>.src`, where `<package>` is package attribute name). + The hash is printed to stdout. + + This works well when you've upgraded the existing package version and want to find out new hash, but is useless if the package can't be accessed by attribute or the package has multiple sources (`.srcs`, architecture-dependent sources, etc). + +4. Upstream hash: use it when upstream provides `sha256` or `sha512`. + Don't use it when upstream provides `md5`, compute `sha256` instead. + + A little nuance is that `nix-prefetch-*` tools produce hashes with the `nix32` encoding (a Nix-specific base32 adaptation), but upstream usually provides hexadecimal (`base16`) encoding. + Fetchers understand both formats. + Nixpkgs does not standardise on any one format. + + You can convert between hash formats with [`nix-hash`](https://nixos.org/manual/nix/stable/command-ref/nix-hash). + +5. Extract the hash from a local source archive with `sha256sum`. + Use `nix-prefetch-url file:///path/to/archive` if you want the custom Nix `base32` hash. + +## Obtaining hashes securely {#sec-pkgs-fetchers-secure-hashes} + +It's always a good idea to avoid Man-in-the-Middle (MITM) attacks when downloading source contents. +Otherwise, you could unknowingly download malware instead of the intended source, and instead of the actual source hash, you'll end up using the hash of malware. +Here are security considerations for this scenario: + +- `http://` URLs are not secure to prefetch hashes. + +- Upstream hashes should be obtained via a secure protocol. + +- `https://` URLs give you more protections when using `nix-prefetch-*` or for upstream hashes. + +- `https://` URLs are secure when using the [fake hash method](#sec-pkgs-fetchers-updating-source-hashes-fakehash-method) *only if* you use one of the listed fake hashes. + If you use any other hash, the download will be exposed to MITM attacks even if you use HTTPS URLs. + + In more concrete terms, if you use any other hash, the [`--insecure` flag](https://curl.se/docs/manpage.html#-k) will be passed to the underlying call to `curl` when downloading content. + +[]{#fetchurl} +## `fetchurl` {#sec-pkgs-fetchers-fetchurl} + +`fetchurl` returns a [fixed-output derivation](https://nixos.org/manual/nix/stable/glossary.html#gloss-fixed-output-derivation) which downloads content from a given URL and stores the unaltered contents within the Nix store. + +It uses {manpage}`curl(1)` internally, and allows its behaviour to be modified by specifying a few attributes in the argument to `fetchurl` (see the documentation for attributes `curlOpts`, `curlOptsList`, and `netrcPhase`). + +The resulting [store path](https://nixos.org/manual/nix/stable/store/store-path) is determined by the hash given to `fetchurl`, and also the `name` (or `pname` and `version`) values. + +If neither `name` nor `pname` and `version` are specified when calling `fetchurl`, it will default to using the [basename](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-baseNameOf) of `url` or the first element of `urls`. +If `pname` and `version` are specified, `fetchurl` will use those values and will ignore `name`, even if it is also specified. + +### Inputs {#sec-pkgs-fetchers-fetchurl-inputs} + +`fetchurl` requires an attribute set with the following attributes: + +`url` (String; _optional_) +: The URL to download from. + + :::{.note} + Either `url` or `urls` must be specified, but not both. + ::: + + All URLs of the format [specified here](https://curl.se/docs/url-syntax.html#rfc-3986-plus) are supported. + + _Default value:_ `""`. + +`urls` (List of String; _optional_) +: A list of URLs, specifying download locations for the same content. + Each URL will be tried in order until one of them succeeds with some content or all of them fail. + See [](#ex-fetchers-fetchurl-nixpkgs-version-multiple-urls) to understand how this attribute affects the behaviour of `fetchurl`. + + :::{.note} + Either `url` or `urls` must be specified, but not both. + ::: + + _Default value:_ `[]`. + +`hash` (String; _optional_) +: Hash of the derivation output of `fetchurl`, following the format for integrity metadata as defined by [SRI](https://www.w3.org/TR/SRI/). + For more information, see [](#chap-pkgs-fetchers-caveats). + + :::{.note} + It is recommended that you use the `hash` attribute instead of the other hash-specific attributes that exist for backwards compatibility. + + If `hash` is not specified, you must specify `outputHash` and `outputHashAlgo`, or one of `sha512`, `sha256`, or `sha1`. + ::: + + _Default value:_ `""`. + +`outputHash` (String; _optional_) +: Hash of the derivation output of `fetchurl` in the format expected by Nix. + See [the documentation on the Nix manual](https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-outputHash) for more information about its format. + + :::{.note} + It is recommended that you use the `hash` attribute instead. + + If `outputHash` is specified, you must also specify `outputHashAlgo`. + ::: + + _Default value:_ `""`. + +`outputHashAlgo` (String; _optional_) +: Algorithm used to generate the value specified in `outputHash`. + See [the documentation on the Nix manual](https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-outputHashAlgo) for more information about the values it supports. + + :::{.note} + It is recommended that you use the `hash` attribute instead. + + The value specified in `outputHashAlgo` will be ignored if `outputHash` isn't also specified. + ::: + + _Default value:_ `""`. + +`sha1` (String; _optional_) +: SHA-1 hash of the derivation output of `fetchurl` in the format expected by Nix. + See [the documentation on the Nix manual](https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-outputHash) for more information about its format. + + :::{.note} + It is recommended that you use the `hash` attribute instead. + ::: + + _Default value:_ `""`. + +`sha256` (String; _optional_) +: SHA-256 hash of the derivation output of `fetchurl` in the format expected by Nix. + See [the documentation on the Nix manual](https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-outputHash) for more information about its format. + + :::{.note} + It is recommended that you use the `hash` attribute instead. + ::: + + _Default value:_ `""`. + +`sha512` (String; _optional_) +: SHA-512 hash of the derivation output of `fetchurl` in the format expected by Nix. + See [the documentation on the Nix manual](https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-outputHash) for more information about its format. + + :::{.note} + It is recommended that you use the `hash` attribute instead. + ::: + + _Default value:_ `""`. + +`name` (String; _optional_) +: The symbolic name of the downloaded file when saved in the Nix store. + See [the `fetchurl` overview](#sec-pkgs-fetchers-fetchurl) for details on how the name of the file is decided. + + _Default value:_ `""`. + +`pname` (String; _optional_) +: A base name, which will be combined with `version` to form the symbolic name of the downloaded file when saved in the Nix store. + See [the `fetchurl` overview](#sec-pkgs-fetchers-fetchurl) for details on how the name of the file is decided. + + :::{.note} + If `pname` is specified, you must also specify `version`, otherwise `fetchurl` will ignore the value of `pname`. + ::: + + _Default value:_ `""`. + +`version` (String; _optional_) +: A version, which will be combined with `pname` to form the symbolic name of the downloaded file when saved in the Nix store. + See [the `fetchurl` overview](#sec-pkgs-fetchers-fetchurl) for details on how the name of the file is decided. + + _Default value:_ `""`. + +`recursiveHash` (Boolean; _optional_) +: If set to `true`, will signal to Nix that the hash given to `fetchurl` was calculated using the `"recursive"` mode. + See [the documentation on the Nix manual](https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-outputHashMode) for more information about the existing modes. + + By default, `fetchurl` uses `"recursive"` mode when the `executable` attribute is set to `true`, so you don't need to specify `recursiveHash` in this case. + + _Default value:_ `false`. + +`executable` (Boolean; _optional_) +: If `true`, sets the executable bit on the downloaded file. + + _Default value_: `false`. + +`downloadToTemp` (Boolean; _optional_) +: If `true`, saves the downloaded file to a temporary location instead of the expected Nix store location. + This is useful when used in conjunction with `postFetch` attribute, otherwise `fetchurl` will not produce any meaningful output. + + The location of the downloaded file will be set in the `$downloadedFile` variable, which should be used by the script in the `postFetch` attribute. + See [](#ex-fetchers-fetchurl-nixpkgs-version-postfetch) to understand how to work with this attribute. + + _Default value:_ `false`. + +`postFetch` (String; _optional_) +: Script executed after the file has been downloaded successfully, and before `fetchurl` finishes running. + Useful for post-processing, to check or transform the file in some way. + See [](#ex-fetchers-fetchurl-nixpkgs-version-postfetch) to understand how to work with this attribute. + + _Default value:_ `""`. + +`netrcPhase` (String or Null; _optional_) +: Script executed to create a {manpage}`netrc(5)` file to be used with {manpage}`curl(1)`. + The script should create the `netrc` file (note that it does not begin with a ".") in the directory it's currently running in (`$PWD`). + + The script is executed during the setup done by `fetchurl` before it runs any of its code to download the specified content. + + :::{.note} + If specified, `fetchurl` will automatically alter its invocation of {manpage}`curl(1)` to use the `netrc` file, so you don't need to add anything to `curlOpts` or `curlOptsList`. + ::: + + :::{.caution} + Since `netrcPhase` needs to be specified in your source Nix code, any secrets that you put directly in it will be world-readable by design (both in your source code, and when the derivation gets created in the Nix store). + + If you want to avoid this behaviour, see the documentation of `netrcImpureEnvVars` for an alternative way of dealing with these secrets. + ::: + + _Default value_: `null`. + +`netrcImpureEnvVars` (List of String; _optional_) +: If specified, `fetchurl` will add these environment variable names to the list of [impure environment variables](https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-impureEnvVars), which will be passed from the environment of the calling user to the builder running the `fetchurl` code. + + This is useful when used with `netrcPhase` to hide any secrets that are used in it, because the script in `netrcPhase` only needs to reference the environment variables with the secrets in them instead. + However, note that these are called _impure_ variables for a reason: + the environment that starts the build needs to have these variables declared for everything to work properly, which means that additional setup is required outside what Nix controls. + + _Default value:_ `[]`. + +`curlOpts` (String; _optional_) +: If specified, this value will be appended to the invocation of {manpage}`curl(1)` when downloading the URL(s) given to `fetchurl`. + Multiple arguments can be separated by spaces normally, but values with whitespaces will be interpreted as multiple arguments (instead of a single value), even if the value is escaped. + See `curlOptsList` for a way to pass values with whitespaces in them. + + _Default value:_ `""`. + +`curlOptsList` (List of String; _optional_) +: If specified, each element of this list will be passed as an argument to the invocation of {manpage}`curl(1)` when downloading the URL(s) given to `fetchurl`. + This allows passing values that contain spaces, with no escaping needed. + + _Default value:_ `[]`. + +`showURLs` (Boolean; _optional_) +: If set to `true`, this will stop `fetchurl` from downloading anything at all. + Instead, it will output a list of all the URLs it would've used to download the content (after resolving `mirror://` URLs, for example). + This is useful for debugging. + + _Default value:_ `false`. + +`meta` (Attribute Set; _optional_) +: Specifies any [meta-attributes](#chap-meta) for the derivation returned by `fetchurl`. + + _Default value:_ `{}`. + +`passthru` (Attribute Set; _optional_) +: Specifies any extra [passthru](#var-stdenv-passthru) attributes for the derivation returned by `fetchurl`. + Note that `fetchurl` defines [passthru attributes of its own](#ssec-pkgs-fetchers-fetchurl-passthru-outputs). + Attributes specified in `passthru` can override the default attributes returned by `fetchurl`. + + _Default value:_ `{}`. + +`preferLocalBuild` (Boolean; _optional_) +: This is the same attribute as [defined in the Nix manual](https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-preferLocalBuild). + It is `true` by default because making a remote machine download the content just duplicates network traffic (since the local machine might download the results from the derivation anyway), but this could be useful in cases where network access is restricted on local machines. + + _Default value:_ `true`. + +`nativeBuildInputs` (List of Attribute Set; _optional_) +: Additional packages needed to download the content. + This is useful if you need extra packages for `postFetch` or `netrcPhase`, for example. + Has the same semantics as in [](#var-stdenv-nativeBuildInputs). + See [](#ex-fetchers-fetchurl-nixpkgs-version-postfetch) to understand how this can be used with `postFetch`. + + _Default value:_ `[]`. + +### Passthru outputs {#ssec-pkgs-fetchers-fetchurl-passthru-outputs} + +`fetchurl` also defines its own [`passthru`](#var-stdenv-passthru) attributes: + +`url` (String) + +: The same `url` attribute passed in the argument to `fetchurl`. + +### Examples {#ssec-pkgs-fetchers-fetchurl-examples} + +:::{.example #ex-fetchers-fetchurl-nixpkgs-version} +# Using `fetchurl` to download a file + +The following package downloads a small file from a URL and shows the most common way to use `fetchurl`: ```nix +{ fetchurl }: fetchurl { - url = "http://www.example.org/hello-1.0.tar.gz"; - hash = "sha256-lTeyxzJNQeMdu1IVdovNMtgn77jRIhSybLdMbTkf2Ww="; + url = "https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/.version"; + hash = "sha256-BZqI7r0MNP29yGH5+yW2tjU9OOpOCEvwWKrWCv5CQ0I="; } ``` -A common mistake is to update a fetcher’s URL, or a version parameter, without updating the hash. +After building the package, the file will be downloaded and place into the Nix store: + +```shell +$ nix-build +(output removed for clarity) +/nix/store/4g9y3x851wqrvim4zcz5x2v3zivmsq8n-version + +$ cat /nix/store/4g9y3x851wqrvim4zcz5x2v3zivmsq8n-version +23.11 +``` +::: + +:::{.example #ex-fetchers-fetchurl-nixpkgs-version-multiple-urls} +# Using `fetchurl` to download a file with multiple possible URLs + +The following package adapts [](#ex-fetchers-fetchurl-nixpkgs-version) to use multiple URLs. +The first URL was crafted to intentionally return an error to illustrate how `fetchurl` will try multiple URLs until it finds one that works (or all URLs fail). ```nix +{ fetchurl }: fetchurl { - url = "http://www.example.org/hello-1.1.tar.gz"; - hash = "sha256-lTeyxzJNQeMdu1IVdovNMtgn77jRIhSybLdMbTkf2Ww="; + urls = [ + "https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/does-not-exist" + "https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/.version" + ]; + hash = "sha256-BZqI7r0MNP29yGH5+yW2tjU9OOpOCEvwWKrWCv5CQ0I="; } ``` -**This will reuse the old contents**. -Remember to invalidate the hash argument, in this case by setting the `hash` attribute to an empty string. +After building the package, both URLs will be used to download the file: + +```shell +$ nix-build +(some output removed for clarity) +trying https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/does-not-exist +(some output removed for clarity) +curl: (22) The requested URL returned error: 404 + +trying https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/.version +(some output removed for clarity) +/nix/store/n9asny31z32q7sdw6a8r1gllrsfy53kl-does-not-exist + +$ cat /nix/store/n9asny31z32q7sdw6a8r1gllrsfy53kl-does-not-exist +23.11 +``` + +However, note that the name of the file was derived from the first URL (this is further explained in [the `fetchurl` overview](#sec-pkgs-fetchers-fetchurl)). +To ensure the result will have the same name regardless of which URLs are used, we can modify the package: ```nix +{ fetchurl }: fetchurl { - url = "http://www.example.org/hello-1.1.tar.gz"; - hash = ""; + name = "nixpkgs-version"; + urls = [ + "https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/does-not-exist" + "https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/.version" + ]; + hash = "sha256-BZqI7r0MNP29yGH5+yW2tjU9OOpOCEvwWKrWCv5CQ0I="; } ``` -Use the resulting error message to determine the correct hash. +After building the package, the result will have the name we specified: +```shell +$ nix-build +(output removed for clarity) +/nix/store/zczb6wl3al6jm9sm5h3pr6nqn0i5ji9z-nixpkgs-version ``` -error: hash mismatch in fixed-output derivation '/path/to/my.drv': - specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= - got: sha256-lTeyxzJNQeMdu1IVdovNMtgn77jRIhSybLdMbTkf2Ww= -``` - -A similar problem arises while testing changes to a fetcher's implementation. If the output of the derivation already exists in the Nix store, test failures can go undetected. The [`invalidateFetcherByDrvHash`](#tester-invalidateFetcherByDrvHash) function helps prevent reusing cached derivations. +::: -## `fetchurl` and `fetchzip` {#fetchurl} +:::{.example #ex-fetchers-fetchurl-nixpkgs-version-postfetch} +# Manipulating the content downloaded by `fetchurl` -Two basic fetchers are `fetchurl` and `fetchzip`. Both of these have two required arguments, a URL and a hash. The hash is typically `hash`, although many more hash algorithms are supported. Nixpkgs contributors are currently recommended to use `hash`. This hash will be used by Nix to identify your source. A typical usage of `fetchurl` is provided below. +It might be useful to manipulate the content downloaded by `fetchurl` directly in its derivation. +In this example, we'll adapt [](#ex-fetchers-fetchurl-nixpkgs-version) to append the result of running the `hello` package to the contents we download, purely to illustrate how to manipulate the content. ```nix -{ stdenv, fetchurl }: +{ fetchurl, hello, lib }: +fetchurl { + url = "https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/.version"; -stdenv.mkDerivation { - name = "hello"; - src = fetchurl { - url = "http://www.example.org/hello.tar.gz"; - hash = "sha256-BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB="; - }; + nativeBuildInputs = [ hello ]; + + downloadToTemp = true; + postFetch = '' + ${lib.getExe hello} >> $downloadedFile + mv $downloadedFile $out + ''; + + hash = "sha256-ceooQQYmDx5+0nfg40uU3NNI2yKrixP7HZ/xLZUNv+w="; } ``` -The main difference between `fetchurl` and `fetchzip` is in how they store the contents. `fetchurl` will store the unaltered contents of the URL within the Nix store. `fetchzip` on the other hand, will decompress the archive for you, making files and directories directly accessible in the future. `fetchzip` can only be used with archives. Despite the name, `fetchzip` is not limited to .zip files and can also be used with any tarball. +After building the package, the resulting file will have "Hello, world!" appended to it: + +```shell +$ nix-build +(output removed for clarity) +/nix/store/ifi6pp7q0ag5h7c5v9h1c1c7bhd10c7f-version + +$ cat /nix/store/ifi6pp7q0ag5h7c5v9h1c1c7bhd10c7f-version +23.11 +Hello, world! +``` + +Note that the `hash` specified in the package is different than the hash specified in [](#ex-fetchers-fetchurl-nixpkgs-version), because the contents of the output have changed (even though the actual file that was downloaded is the same). +See [](#chap-pkgs-fetchers-caveats) for more details on how to work with the `hash` attribute when the output changes. +::: + +## `fetchzip` {#sec-pkgs-fetchers-fetchzip} + +Downloads content from a given URL (which is assumed to be an archive), and decompresses the archive for you, making files and directories directly accessible. +`fetchzip` can only be used with archives. +Despite its name, `fetchzip` is not limited to `.zip` files and can also be used with any tarball. -Additional parameters to `fetchurl`: -- `downloadToTemp`: Defaults to `false`. If `true`, saves the source to `$downloadedFile`, to be used in conjunction with `postFetch` -- `postFetch`: Shell code executed after the file has been fetched successfully. Use it for postprocessing, to check or transform the file. +It has two required arguments, a URL and a hash. +The hash is typically `hash`, although many more hash algorithms are supported. +Nixpkgs contributors are currently recommended to use `hash`. +This hash will be used by Nix to identify your source. +A typical usage of `fetchzip` is provided below. + +```nix +{ fetchzip }: +fetchzip { + url = "https://github.com/NixOS/patchelf/releases/download/0.18.0/patchelf-0.18.0.tar.gz"; + hash = "sha256-3ABYlME9R8klcpJ7MQpyFEFwHmxDDEzIYBqu/CpDYmg="; +} +``` ## `fetchpatch` {#fetchpatch} diff --git a/doc/languages-frameworks/dlang.section.md b/doc/languages-frameworks/dlang.section.md new file mode 100644 index 000000000000..6e9edefc5e0f --- /dev/null +++ b/doc/languages-frameworks/dlang.section.md @@ -0,0 +1,69 @@ +# D (Dlang) {#dlang} + +Nixpkgs provides multiple D compilers such as `ldc`, `dmd` and `gdc`. +These can be used like any other package during build time. + +However, Nixpkgs provides a build helper for compiling packages using the `dub` package manager. + +Here's an example: +```nix +{ + lib, + buildDubPackage, + fetchFromGitHub, + ncurses, + zlib, +}: + +buildDubPackage rec { + pname = "btdu"; + version = "0.5.1"; + + src = fetchFromGitHub { + owner = "CyberShadow"; + repo = "btdu"; + rev = "v${version}"; + hash = "sha256-3sSZq+5UJH02IO0Y1yL3BLHDb4lk8k6awb5ZysBQciE="; + }; + + # generated by dub-to-nix, see below + dubLock = ./dub-lock.json; + + buildInputs = [ + ncurses + zlib + ]; + + installPhase = '' + runHook preInstall + install -Dm755 btdu -t $out/bin + runHook postInstall + ''; +} +``` + +Note that you need to define `installPhase` because `dub` doesn't know where files should go in `$out`. + +Also note that running `dub test` is disabled by default. You can enable it by setting `doCheck = true`. + +## Lockfiles {#dub-lockfiles} +Nixpkgs has its own lockfile format for `dub` dependencies, because `dub`'s official "lockfile" format (`dub.selections.json`) is not hash based. + +A lockfile can be generated using the `dub-to-nix` helper package. +* Firstly, install `dub-to-nix` into your shell session by running `nix-shell -p dub-to-nix` +* Then navigate to the root of the source of the program you want to package +* Finally, run `dub-to-nix` and it will print the lockfile to stdout. You could pipe stdout into a text file or just copy the output manually into a file. + +## `buildDubPackage` parameters {#builddubpackage-parameters} + +The `buildDubPackage` function takes an attrset of parameters that are passed on to `stdenv.mkDerivation`. + +The following parameters are specific to `buildDubPackage`: + +* `dubLock`: A lockfile generated by `dub-to-nix` from the source of the package. Can be either a path to the file, or an attrset already parsed with `lib.importJSON`. + The latter useful if the package uses `dub` dependencies not already in the lockfile. (e.g. if the package calls `dub run some-dub-package` manually) +* `dubBuildType ? "release"`: The build type to pass to `dub build` as a value for the `--build=` flag. +* `dubFlags ? []`: The flags to pass to `dub build` and `dub test`. +* `dubBuildFlags ? []`: The flags to pass to `dub build`. +* `dubTestFlags ? []`: The flags to pass to `dub test`. +* `compiler ? ldc`: The D compiler to be used by `dub`. diff --git a/doc/languages-frameworks/index.md b/doc/languages-frameworks/index.md index 67107fb5b687..920e5e7bd431 100644 --- a/doc/languages-frameworks/index.md +++ b/doc/languages-frameworks/index.md @@ -14,6 +14,7 @@ cuda.section.md cuelang.section.md dart.section.md dhall.section.md +dlang.section.md dotnet.section.md emscripten.section.md gnome.section.md diff --git a/doc/manpage-urls.json b/doc/manpage-urls.json index 2cc03af4360f..e878caf042a4 100644 --- a/doc/manpage-urls.json +++ b/doc/manpage-urls.json @@ -320,5 +320,7 @@ "login.defs(5)": "https://man.archlinux.org/man/login.defs.5", "unshare(1)": "https://man.archlinux.org/man/unshare.1.en", "nix-shell(1)": "https://nixos.org/manual/nix/stable/command-ref/nix-shell.html", - "mksquashfs(1)": "https://man.archlinux.org/man/extra/squashfs-tools/mksquashfs.1.en" + "mksquashfs(1)": "https://man.archlinux.org/man/extra/squashfs-tools/mksquashfs.1.en", + "curl(1)": "https://curl.se/docs/manpage.html", + "netrc(5)": "https://man.cx/netrc" } |