Merge staging-next into master (#44009)

* substitute(): --subst-var was silently coercing to "" if the variable does not exist.

* libffi: simplify using `checkInputs`

* pythonPackges.hypothesis, pythonPackages.pytest: simpify dependency cycle fix

* utillinux: 2.32 -> 2.32.1

https://lkml.org/lkml/2018/7/16/532

* busybox: 1.29.0 -> 1.29.1

* bind: 9.12.1-P2 -> 9.12.2

https://ftp.isc.org/isc/bind9/9.12.2/RELEASE-NOTES-bind-9.12.2.html

* curl: 7.60.0 -> 7.61.0

* gvfs: make tests run, but disable

* ilmbase: disable tests on i686. Spooky!

* mdds: fix tests

* git: disable checks as tests are run in installcheck

* ruby: disable tests

* libcommuni: disable checks as tests are run in installcheck

* librdf: make tests run, but disable

* neon, neon_0_29: make tests run, but disable

* pciutils: 3.6.0 -> 3.6.1

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/pciutils/versions.

* mesa: more include fixes

mostly from void-linux (thanks!)

* npth: 1.5 -> 1.6

minor bump

* boost167: Add lockfree next_prior patch

* stdenv: cleanup darwin bootstrapping

Also gets rid of the full python and some of it's dependencies in the
stdenv build closure.

* Revert "pciutils: use standardized equivalent for canonicalize_file_name"

This reverts commit f8db20fb3ae382eba1ba2b160fe24739f43c0bd7.
Patching should no longer be needed with 3.6.1.

* binutils-wrapper: Try to avoid adding unnecessary -L flags

(cherry picked from commit f3758258b8895508475caf83e92bfb236a27ceb9)
Signed-off-by: Domen Kožar <domen@dev.si>

* libffi: don't check on darwin

libffi usages in stdenv broken darwin. We need to disable doCheck for that case.

* "rm $out/share/icons/hicolor/icon-theme.cache" -> hicolor-icon-theme setup-hook

* python.pkgs.pytest: setupHook to prevent creation of .pytest-cache folder, fixes #40273

When `py.test` was run with a folder as argument, it would not only
search for tests in that folder, but also create a .pytest-cache folder.
Not only is this state we don't want, but it was also causing
collisions.

* parity-ui: fix after merge

* python.pkgs.pytest-flake8: disable test, fix build

* Revert "meson: 0.46.1 -> 0.47.0"

With meson 0.47.0 (or 0.47.1, or git)
things are very wrong re:rpath handling
resulting in at best missing libs but
even corrupt binaries :(.

When we run patchelf it masks the problem
by removing obviously busted paths.
Which is probably why this wasn't noticed immediately.

Unfortunately the binary already
has a long series of paths scribbled
in a space intended for a much smaller string;
in my testing it was something like
lengths were 67 with 300+ written to it.

I think we've reported the relevant issues upstream,
but unfortunately it appears our patches
are what introduces the overwrite/corruption
(by no longer being correct in what they assume)

This doesn't look so bad to fix but it's
not something I can spend more time on
at the moment.

--

Interestingly the overwritten string data
(because it is scribbled past the bounds)
remains in the binary and is why we're suddenly
seeing unexpected references in various builds
-- notably this is is the reason we're
seeing the "extra-utils" breakage
that entirely crippled NixOS on master
(and probably on staging before?).

Fixes #43650.

This reverts commit 305ac4dade5758c58e8ab1666ad0197fd305828d.

(cherry picked from commit 273d68eff8f7b6cd4ebed3718e5078a0f43cb55d)
Signed-off-by: Domen Kožar <domen@dev.si>

2 files changed, 29 insertions, 11 deletions

diff --git a/pkgs/stdenv/darwin/default.nix b/pkgs/stdenv/darwin/default.nix
index 366be3915802..b7b3640a9535 100644
--- a/pkgs/stdenv/darwin/default.nix
+++ b/pkgs/stdenv/darwin/default.nix
@@ -188,7 +188,18 @@ in rec {
   };
 
   stage1 = prevStage: let
-    persistent = _: super: { python = super.python.override { configd = null; }; };
+    persistent = self: super: with prevStage; {
+      cmake = super.cmake.override {
+        majorVersion = "3.9";  # FIXME: update ApplicationServices patch
+        isBootstrap = true;
+        useSharedLibraries = false;
+      };
+
+      python = super.callPackage ../../development/interpreters/python/cpython/2.7/boot.nix {
+        CF = null;  # use CoreFoundation from bootstrap-tools
+        configd = null;
+      };
+    };
   in with prevStage; stageFun 1 prevStage {
     extraPreHook = "export NIX_CFLAGS_COMPILE+=\" -F${bootstrapTools}/Library/Frameworks\"";
     extraNativeBuildInputs = [];
@@ -239,6 +250,9 @@ in rec {
         openssh sqlite sed serf openldap db cyrus-sasl expat apr-util
         findfreetype libssh curl cmake autoconf automake libtool cpio;
 
+      # Avoid pulling in a full python and it's extra dependencies for the llvm/clang builds.
+      libxml2 = super.libxml2.override { pythonSupport = false; };
+
       llvmPackages_5 = super.llvmPackages_5 // (let
         libraries = super.llvmPackages_5.libraries.extend (_: _: {
           inherit (llvmPackages_5) libcxx libcxxabi;
@@ -281,6 +295,15 @@ in rec {
         ncurses libffi zlib gmp pcre gnugrep
         coreutils findutils diffutils patchutils;
 
+      # Hack to make sure we don't link ncurses in bootstrap tools. The proper
+      # solution is to avoid passing -L/nix-store/...-bootstrap-tools/lib,
+      # quite a sledgehammer just to get the C runtime.
+      gettext = super.gettext.overrideAttrs (drv: {
+        configureFlags = drv.configureFlags ++ [
+          "--disable-curses"
+        ];
+      });
+
       llvmPackages_5 = super.llvmPackages_5 // (let
         tools = super.llvmPackages_5.tools.extend (llvmSelf: _: {
           inherit (llvmPackages_5) llvm clang-unwrapped;
@@ -303,16 +326,7 @@ in rec {
     extraPreHook = ''
       export PATH_LOCALE=${pkgs.darwin.locale}/share/locale
     '';
-    overrides = lib.composeExtensions persistent (self: super: {
-      # Hack to make sure we don't link ncurses in bootstrap tools. The proper
-      # solution is to avoid passing -L/nix-store/...-bootstrap-tools/lib,
-      # quite a sledgehammer just to get the C runtime.
-      gettext = super.gettext.overrideAttrs (old: {
-         configureFlags = old.configureFlags ++ [
-           "--disable-curses"
-         ];
-      });
-    });
+    overrides = persistent;
   };
 
   stdenvDarwin = prevStage: let
diff --git a/pkgs/stdenv/generic/setup.sh b/pkgs/stdenv/generic/setup.sh
index 26d28609d877..c5feffcea99b 100644
--- a/pkgs/stdenv/generic/setup.sh
+++ b/pkgs/stdenv/generic/setup.sh
@@ -660,6 +660,10 @@ substituteStream() {
                     echo "substituteStream(): ERROR: substitution variables must be valid Bash names, \"$varName\" isn't." >&2
                     return 1
                 fi
+                if [ -z ${!varName+x} ]; then
+                    echo "substituteStream(): ERROR: variable \$$varName is unset" >&2
+                    return 1
+                fi
                 pattern="@$varName@"
                 replacement="${!varName}"
                 ;;