From b6c06e216bb3bface40eb8ea6576b25f9b2971dd Mon Sep 17 00:00:00 2001 From: Charles Strahan Date: Sat, 14 Nov 2015 21:17:29 -0500 Subject: ruby: new bundler infrastructure This improves our Bundler integration (i.e. `bundlerEnv`). Before describing the implementation differences, I'd like to point a breaking change: buildRubyGem now expects `gemName` and `version` as arguments, rather than a `name` attribute in the form of "-". Now for the differences in implementation. The previous implementation installed all gems at once in a single derivation. This was made possible by using a set of monkey-patches to prevent Bundler from downloading gems impurely, and to help Bundler find and activate all required gems prior to installation. This had several downsides: * The patches were really hard to understand, and required subtle interaction with the rest of the build environment. * A single install failure would cause the entire derivation to fail. The new implementation takes a different approach: we install gems into separate derivations, and then present Bundler with a symlink forest thereof. This has a couple benefits over the existing approach: * Fewer patches are required, with less interplay with the rest of the build environment. * Changes to one gem no longer cause a rebuild of the entire dependency graph. * Builds take 20% less time (using gitlab as a reference). It's unfortunate that we still have to muck with Bundler's internals, though it's unavoidable with the way that Bundler is currently designed. There are a number improvements that could be made in Bundler that would simplify our packaging story: * Bundler requires all installed gems reside within the same prefix (GEM_HOME), unlike RubyGems which allows for multiple prefixes to be specified through GEM_PATH. It would be ideal if Bundler allowed for packages to be installed and sourced from multiple prefixes. * Bundler installs git sources very differently from how RubyGems installs gem packages, and, unlike RubyGems, it doesn't provide a public interface (CLI or programmatic) to guide the installation of a single gem. We are presented with the options of either reimplementing a considerable portion Bundler, or patch and use parts of its internals; I choose the latter. Ideally, there would be a way to install gems from git sources in a manner similar to how we drive `gem` to install gem packages. * When a bundled program is executed (via `bundle exec` or a binstub that does `require 'bundler/setup'`), the setup process reads the Gemfile.lock, activates the dependencies, re-serializes the lock file it read earlier, and then attempts to overwrite the Gemfile.lock if the contents aren't bit-identical. I think the reasoning is that by merely running an application with a newer version of Bundler, you'll automatically keep the Gemfile.lock up-to-date with any changes in the format. Unfortunately, that doesn't play well with any form of packaging, because bundler will immediately cause the application to abort when it attempts to write to the read-only Gemfile.lock in the store. We work around this by normalizing the Gemfile.lock with the version of Bundler that we'll use at runtime before we copy it into the store. This feels fragile, but it's the best we can do without changes upstream, or resorting to more delicate hacks. With all of the challenges in using Bundler, one might wonder why we can't just cut Bundler out of the picture and use RubyGems. After all, Nix provides most of the isolation that Bundler is used for anyway. The problem, however, is that almost every Rails application calls `Bundler::require` at startup (by way of the default project templates). Because bundler will then, by default, `require` each gem listed in the Gemfile, Rails applications are almost always written such that none of the source files explicitly require their dependencies. That leaves us with two options: support and use Bundler, or maintain massive patches for every Rails application that we package. Closes #8612 --- .../interpreters/ruby/build-ruby-gem/default.nix | 208 ++++++++++++ .../ruby/build-ruby-gem/gem-post-build.rb | 77 +++++ .../ruby/build-ruby-gem/nix-bundle-install.rb | 153 +++++++++ pkgs/development/interpreters/ruby/bundix/Gemfile | 4 - .../interpreters/ruby/bundix/Gemfile.lock | 18 - .../interpreters/ruby/bundix/default.nix | 25 +- .../interpreters/ruby/bundix/gemset.nix | 22 -- .../ruby/bundler-env/default-gem-config.nix | 142 -------- .../interpreters/ruby/bundler-env/default.nix | 361 ++++----------------- .../interpreters/ruby/bundler-env/gen-bin-stubs.rb | 46 +++ .../ruby/bundler-env/mkrf_conf_xapian.rb | 14 - .../ruby/bundler-env/monkey_patches.rb | 246 -------------- .../interpreters/ruby/bundler-env/package-1.8.rb | 29 -- .../interpreters/ruby/bundler-env/xapian-Rakefile | 38 --- pkgs/development/interpreters/ruby/bundler.nix | 17 +- pkgs/development/interpreters/ruby/gem.nix | 136 -------- .../interpreters/ruby/gemconfig/default.nix | 162 +++++++++ .../ruby/gemconfig/mkrf_conf_xapian.rb | 14 + .../interpreters/ruby/gemconfig/xapian-Rakefile | 38 +++ .../interpreters/ruby/load-ruby-env.nix | 69 ---- pkgs/development/interpreters/ruby/rubygems.nix | 34 +- pkgs/development/libraries/gecode/3.nix | 21 ++ pkgs/development/tools/vagrant/default.nix | 3 +- 23 files changed, 822 insertions(+), 1055 deletions(-) create mode 100644 pkgs/development/interpreters/ruby/build-ruby-gem/default.nix create mode 100644 pkgs/development/interpreters/ruby/build-ruby-gem/gem-post-build.rb create mode 100644 pkgs/development/interpreters/ruby/build-ruby-gem/nix-bundle-install.rb delete mode 100644 pkgs/development/interpreters/ruby/bundix/Gemfile delete mode 100644 pkgs/development/interpreters/ruby/bundix/Gemfile.lock delete mode 100644 pkgs/development/interpreters/ruby/bundix/gemset.nix delete mode 100644 pkgs/development/interpreters/ruby/bundler-env/default-gem-config.nix create mode 100644 pkgs/development/interpreters/ruby/bundler-env/gen-bin-stubs.rb delete mode 100644 pkgs/development/interpreters/ruby/bundler-env/mkrf_conf_xapian.rb delete mode 100644 pkgs/development/interpreters/ruby/bundler-env/monkey_patches.rb delete mode 100644 pkgs/development/interpreters/ruby/bundler-env/package-1.8.rb delete mode 100644 pkgs/development/interpreters/ruby/bundler-env/xapian-Rakefile delete mode 100644 pkgs/development/interpreters/ruby/gem.nix create mode 100644 pkgs/development/interpreters/ruby/gemconfig/default.nix create mode 100644 pkgs/development/interpreters/ruby/gemconfig/mkrf_conf_xapian.rb create mode 100644 pkgs/development/interpreters/ruby/gemconfig/xapian-Rakefile delete mode 100644 pkgs/development/interpreters/ruby/load-ruby-env.nix create mode 100644 pkgs/development/libraries/gecode/3.nix (limited to 'pkgs/development') diff --git a/pkgs/development/interpreters/ruby/build-ruby-gem/default.nix b/pkgs/development/interpreters/ruby/build-ruby-gem/default.nix new file mode 100644 index 000000000000..d050faca2454 --- /dev/null +++ b/pkgs/development/interpreters/ruby/build-ruby-gem/default.nix @@ -0,0 +1,208 @@ +# This builds gems in a way that is compatible with bundler. +# +# Bundler installs gems from git sources _very_ differently from how RubyGems +# installes gem packages, though they both install gem packages similarly. +# +# We monkey-patch Bundler to remove any impurities and then drive its internals +# to install git gems. +# +# For the sake of simplicity, gem packages are installed with the standard `gem` +# program. +# +# Note that bundler does not support multiple prefixes; it assumes that all +# gems are installed in a common prefix, and has no support for specifying +# otherwise. Therefore, if you want to be able to use the resulting derivations +# with bundler, you need to create a symlink forrest first, which is what +# `bundlerEnv` does for you. +# +# Normal gem packages can be used outside of bundler; a binstub is created in +# $out/bin. + +{ lib, ruby, rubygems, bundler, fetchurl, fetchgit, makeWrapper, git, buildRubyGem +} @ defs: + +lib.makeOverridable ( + +{ name ? null +, gemName +, version ? null +, type ? "gem" +, document ? [] # e.g. [ "ri" "rdoc" ] +, platform ? "ruby" +, ruby ? defs.ruby +, stdenv ? ruby.stdenv +, namePrefix ? "${ruby.name}" + "-" +, buildInputs ? [] +, doCheck ? false +, meta ? {} +, patches ? [] +, gemPath ? [] +, dontStrip ? true +, remotes ? ["https://rubygems.org"] +# Assume we don't have to build unless strictly necessary (e.g. the source is a +# git checkout). +# If you need to apply patches, make sure to set `dontBuild = false`; +, dontBuild ? true +, propagatedBuildInputs ? [] +, propagatedUserEnvPkgs ? [] +, buildFlags ? null +, passthru ? {} +, ...} @ attrs: + +if ! builtins.elem type [ "git" "gem" ] +then throw "buildRubyGem: don't know how to build a gem of type \"${type}\"" +else + +let + shellEscape = x: "'${lib.replaceChars ["'"] [("'\\'" + "'")] x}'"; + rubygems = (attrs.rubygems or defs.rubygems).override { + inherit ruby; + }; + src = attrs.src or ( + if type == "gem" + then fetchurl { + urls = map (remote: "${remote}/gems/${gemName}-${version}.gem") remotes; + inherit (attrs) sha256; + } else fetchgit { + inherit (attrs) url rev sha256 fetchSubmodules; + leaveDotGit = true; + } + ); + documentFlag = + if document == [] + then "-N" + else "--document ${lib.concatStringsSep "," document}"; + +in + +stdenv.mkDerivation (attrs // { + inherit ruby rubygems; + inherit doCheck; + inherit dontBuild; + inherit dontStrip; + inherit type; + + buildInputs = [ + ruby rubygems makeWrapper + ] ++ lib.optionals (type == "git") [ git bundler ] + ++ buildInputs; + + name = attrs.name or (namePrefix + gemName); + + inherit src; + + phases = attrs.phases or [ "unpackPhase" "patchPhase" "buildPhase" "installPhase" "fixupPhase" ]; + + unpackPhase = attrs.unpackPhase or '' + runHook preUnpack + + if [[ -f $src && $src == *.gem ]]; then + if [[ -z "$dontBuild" ]]; then + # we won't know the name of the directory that RubyGems creates, + # so we'll just use a glob to find it and move it over. + gempkg="$src" + sourceRoot=source + gem unpack $gempkg --target=container + cp -r container/* $sourceRoot + rm -r container + + # copy out the original gemspec, for convenience during patching / + # overrides. + gem specification $gempkg --ruby > original.gemspec + gemspec=$(readlink -f .)/original.gemspec + else + gempkg="$src" + fi + else + # Fall back to the original thing for everything else. + dontBuild="" + preUnpack="" postUnpack="" unpackPhase + fi + + runHook postUnpack + ''; + + buildPhase = attrs.buildPhase or '' + runHook preBuild + + if [[ "$type" == "gem" ]]; then + if [[ -z "$gemspec" ]]; then + gemspec="$(find . -name '*.gemspec')" + echo "found the following gemspecs:" + echo "$gemspec" + gemspec="$(echo "$gemspec" | head -n1)" + fi + + exec 3>&1 + output="$(gem build $gemspec | tee >(cat - >&3))" + exec 3>&- + + gempkg=$(echo "$output" | grep -oP 'File: \K(.*)') + + echo "gem package built: $gempkg" + fi + + runHook postBuild + ''; + + # Note: + # We really do need to keep the $out/${ruby.gemPath}/cache. + # This is very important in order for many parts of RubyGems/Bundler to not blow up. + # See https://github.com/bundler/bundler/issues/3327 + installPhase = attrs.installPhase or '' + runHook preInstall + + export GEM_HOME=$out/${ruby.gemPath} + mkdir -p $GEM_HOME + + echo "buildFlags: $buildFlags" + + ${lib.optionalString (type == "git") '' + ruby ${./nix-bundle-install.rb} \ + ${gemName} \ + ${attrs.url} \ + ${src} \ + ${attrs.rev} \ + ${version} \ + ${shellEscape (toString buildFlags)} + ''} + + ${lib.optionalString (type == "gem") '' + if [[ -z "$gempkg" ]]; then + echo "failure: \$gempkg path unspecified" 1>&2 + exit 1 + elif [[ ! -f "$gempkg" ]]; then + echo "failure: \$gempkg path invalid" 1>&2 + exit 1 + fi + + gem install \ + --local \ + --force \ + --http-proxy 'http://nodtd.invalid' \ + --ignore-dependencies \ + --build-root '/' \ + --backtrace \ + ${documentFlag} \ + $gempkg $gemFlags -- $buildFlags + + # looks like useless files which break build repeatability and consume space + rm -fv $out/${ruby.gemPath}/doc/*/*/created.rid || true + rm -fv $out/${ruby.gemPath}/gems/*/ext/*/mkmf.log || true + + # write out metadata and binstubs + spec=$(echo $out/${ruby.gemPath}/specifications/*.gemspec) + ruby ${./gem-post-build.rb} "$spec" + ''} + + runHook postInstall + ''; + + propagatedBuildInputs = gemPath ++ propagatedBuildInputs; + propagatedUserEnvPkgs = gemPath ++ propagatedUserEnvPkgs; + + passthru = passthru // { isRubyGem = true; }; + inherit meta; +}) + +) diff --git a/pkgs/development/interpreters/ruby/build-ruby-gem/gem-post-build.rb b/pkgs/development/interpreters/ruby/build-ruby-gem/gem-post-build.rb new file mode 100644 index 000000000000..112a9accc335 --- /dev/null +++ b/pkgs/development/interpreters/ruby/build-ruby-gem/gem-post-build.rb @@ -0,0 +1,77 @@ +require 'rbconfig' +require 'rubygems' +require 'rubygems/specification' +require 'fileutils' + +ruby = File.join(ENV["ruby"], "bin", RbConfig::CONFIG['ruby_install_name']) +out = ENV["out"] +bin_path = File.join(ENV["out"], "bin") +gem_home = ENV["GEM_HOME"] +gem_path = ENV["GEM_PATH"].split(":") +install_path = Dir.glob("#{gem_home}/gems/*").first +gemspec_path = ARGV[0] + +if defined?(Encoding.default_internal) + Encoding.default_internal = Encoding::UTF_8 + Encoding.default_external = Encoding::UTF_8 +end + +gemspec_content = File.read(gemspec_path) +spec = nil +if gemspec_content[0..2] == "---" # YAML header + spec = Gem::Specification.from_yaml(gemspec_content) +else + spec = Gem::Specification.load(gemspec_path) +end + +FileUtils.mkdir_p("#{out}/nix-support") + +# write meta-data +meta = "#{out}/nix-support/gem-meta" +FileUtils.mkdir_p(meta) +FileUtils.ln_s(gemspec_path, "#{meta}/spec") +File.open("#{meta}/name", "w") do |f| + f.write(spec.name) +end +File.open("#{meta}/install-path", "w") do |f| + f.write(install_path) +end +File.open("#{meta}/require-paths", "w") do |f| + f.write(spec.require_paths.join(" ")) +end +File.open("#{meta}/executables", "w") do |f| + f.write(spec.executables.join(" ")) +end + +# add this gem to the GEM_PATH for dependencies +File.open("#{out}/nix-support/setup-hook", "a") do |f| + f.puts("addToSearchPath GEM_PATH #{gem_home}") + spec.require_paths.each do |dir| + f.puts("addToSearchPath RUBYLIB #{install_path}/#{dir}") + end +end + +# create regular rubygems binstubs +FileUtils.mkdir_p(bin_path) +spec.executables.each do |exe| + File.open("#{bin_path}/#{exe}", "w") do |f| + f.write(<<-EOF) +#!#{ruby} +# +# This file was generated by Nix. +# +# The application '#{exe}' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +gem_path = ENV["GEM_PATH"] +ENV["GEM_PATH"] = "\#{gem_path}\#{":" unless gem_path.nil? || gem_path.empty?}#{(gem_path+[gem_home]).join(":")}" + +require 'rubygems' + +load Gem.bin_path(#{spec.name.inspect}, #{exe.inspect}) + EOF + end + + FileUtils.chmod("+x", "#{bin_path}/#{exe}") +end diff --git a/pkgs/development/interpreters/ruby/build-ruby-gem/nix-bundle-install.rb b/pkgs/development/interpreters/ruby/build-ruby-gem/nix-bundle-install.rb new file mode 100644 index 000000000000..647b83b52c38 --- /dev/null +++ b/pkgs/development/interpreters/ruby/build-ruby-gem/nix-bundle-install.rb @@ -0,0 +1,153 @@ +require 'rbconfig' +require 'bundler/vendored_thor' +require 'bundler' +require 'rubygems/command' +require 'fileutils' +require 'pathname' +require 'tmpdir' + +# Options: +# +# name - the gem name +# uri - git repo uri +# repo - path to local checkout +# ref - the commit hash +# version - gem version +# build-flags - build arguments + +ruby = File.join(ENV["ruby"], "bin", RbConfig::CONFIG['ruby_install_name']) +out = ENV["out"] +bin_dir = File.join(ENV["out"], "bin") + +name = ARGV[0] +uri = ARGV[1] +REPO = ARGV[2] +ref = ARGV[3] +version = ARGV[4] +build_flags = ARGV[5] + +# options to pass to bundler +options = { + "name" => name, + "uri" => uri, + "ref" => ref, + "version" => version, +} + +# Monkey-patch Bundler to use our local checkout. +# I wish we didn't have to do this, but bundler does not expose an API to do +# these kinds of things. +Bundler.module_eval do + def self.requires_sudo? + false + end + + def self.root + # we don't have a Gemfile, so it doesn't make sense to try to make paths + # relative to the (non existent) parent directory thereof, so we give a + # nonsense path here. + Pathname.new("/no-root-path") + end + + def self.bundle_path + Pathname.new(ENV["GEM_HOME"]) + end + + def self.locked_gems + nil + end +end + +Bundler::Source::Git.class_eval do + def allow_git_ops? + true + end +end + +Bundler::Source::Git::GitProxy.class_eval do + def checkout + unless path.exist? + FileUtils.mkdir_p(path.dirname) + FileUtils.cp_r(File.join(REPO, ".git"), path) + system("chmod -R +w #{path}") + end + end + + def copy_to(destination, submodules=false) + unless File.exist?(destination.join(".git")) + FileUtils.mkdir_p(destination.dirname) + FileUtils.cp_r(REPO, destination) + system("chmod -R +w #{destination}") + end + end +end + +# UI +verbose = false +no_color = false +Bundler.ui = Bundler::UI::Shell.new({"no-color" => no_color}) +Bundler.ui.level = "debug" if verbose + +# Install +source = Bundler::Source::Git.new(options) +spec = source.specs.search_all(name).first +Bundler.rubygems.with_build_args [build_flags] do + source.install(spec) +end + +msg = spec.post_install_message +if msg + Bundler.ui.confirm "Post-install message from #{name}:" + Bundler.ui.info msg +end + +# Write out the binstubs +if spec.executables.any? + FileUtils.mkdir_p(bin_dir) + spec.executables.each do |exe| + wrapper = File.join(bin_dir, exe) + File.open(wrapper, "w") do |f| + stub = generate_stub(spec.name, exe) + f.write(<<-EOF) +#!#{ruby} +# +# This file was generated by Nix. +# +# The application '#{exe}' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require 'rubygems' +require 'bundler/setup' + +load Gem.bin_path(#{spec.name.inspect}, #{exe.inspect}) +EOF + end + + FileUtils.chmod("+x", wrapper) + end +end + +# Write out metadata +meta = "#{out}/nix-support/gem-meta" +FileUtils.mkdir_p(meta) +FileUtils.ln_s(spec.loaded_from.to_s, "#{meta}/spec") +File.open("#{meta}/name", "w") do |f| + f.write spec.name +end +File.open("#{meta}/install-path", "w") do |f| + f.write source.install_path.to_s +end +File.open("#{meta}/require-paths", "w") do |f| + f.write spec.require_paths.join(" ") +end +File.open("#{meta}/executables", "w") do |f| + f.write spec.executables.join(" ") +end + +# make the lib available during bundler/git installs +File.open("#{out}/nix-support/setup-hook", "a") do |f| + spec.require_paths.each do |dir| + f.puts("addToSearchPath RUBYLIB #{source.install_path}/#{dir}") + end +end diff --git a/pkgs/development/interpreters/ruby/bundix/Gemfile b/pkgs/development/interpreters/ruby/bundix/Gemfile deleted file mode 100644 index 4899cafc3738..000000000000 --- a/pkgs/development/interpreters/ruby/bundix/Gemfile +++ /dev/null @@ -1,4 +0,0 @@ -source "http://rubygems.org" -gem "bundix", - :git => "https://github.com/cstrahan/bundix.git", - :ref => "v1.0.3" diff --git a/pkgs/development/interpreters/ruby/bundix/Gemfile.lock b/pkgs/development/interpreters/ruby/bundix/Gemfile.lock deleted file mode 100644 index f241a3bafd4f..000000000000 --- a/pkgs/development/interpreters/ruby/bundix/Gemfile.lock +++ /dev/null @@ -1,18 +0,0 @@ -GIT - remote: https://github.com/cstrahan/bundix.git - revision: c879cf901ff8084b4c97aaacfb5ecbdb0f95cc03 - ref: v1.0.3 - specs: - bundix (1.0.2) - thor (~> 0.19.1) - -GEM - remote: http://rubygems.org/ - specs: - thor (0.19.1) - -PLATFORMS - ruby - -DEPENDENCIES - bundix! diff --git a/pkgs/development/interpreters/ruby/bundix/default.nix b/pkgs/development/interpreters/ruby/bundix/default.nix index 0196adb8f4c8..b5a49043c60b 100644 --- a/pkgs/development/interpreters/ruby/bundix/default.nix +++ b/pkgs/development/interpreters/ruby/bundix/default.nix @@ -1,9 +1,20 @@ -{ ruby, bundlerEnv }: +{ ruby, fetchgit, buildRubyGem, bundler }: -bundlerEnv { - name = "bundix"; - inherit ruby; - gemset = ./gemset.nix; - gemfile = ./Gemfile; - lockfile = ./Gemfile.lock; +let + thor = buildRubyGem { + gemName = "thor"; + version = "0.19.1"; + type = "gem"; + sha256 = "08p5gx18yrbnwc6xc0mxvsfaxzgy2y9i78xq7ds0qmdm67q39y4z"; + }; + +in buildRubyGem { + gemName = "bundix"; + version = "1.0.4"; + gemPath = [ thor bundler ]; + src = fetchgit { + url = "https://github.com/cstrahan/bundix.git"; + rev = "6dcf1f71c61584f5c9b919ee9df7b0c554862076"; + sha256 = "1w17bvc9srcgr4ry81ispcj35g9kxihbyknmqp8rnd4h5090b7b2"; + }; } diff --git a/pkgs/development/interpreters/ruby/bundix/gemset.nix b/pkgs/development/interpreters/ruby/bundix/gemset.nix deleted file mode 100644 index f8f6546671d9..000000000000 --- a/pkgs/development/interpreters/ruby/bundix/gemset.nix +++ /dev/null @@ -1,22 +0,0 @@ -{ - "bundix" = { - version = "1.0.2"; - source = { - type = "git"; - url = "https://github.com/cstrahan/bundix.git"; - rev = "c879cf901ff8084b4c97aaacfb5ecbdb0f95cc03"; - sha256 = "05kmdnq4qa5h8l3asv05cjpnyplnqqx6hrqybj2cjlzmdxnkkgyj"; - fetchSubmodules = false; - }; - dependencies = [ - "thor" - ]; - }; - "thor" = { - version = "0.19.1"; - source = { - type = "gem"; - sha256 = "08p5gx18yrbnwc6xc0mxvsfaxzgy2y9i78xq7ds0qmdm67q39y4z"; - }; - }; -} \ No newline at end of file diff --git a/pkgs/development/interpreters/ruby/bundler-env/default-gem-config.nix b/pkgs/development/interpreters/ruby/bundler-env/default-gem-config.nix deleted file mode 100644 index 36925bcdf558..000000000000 --- a/pkgs/development/interpreters/ruby/bundler-env/default-gem-config.nix +++ /dev/null @@ -1,142 +0,0 @@ -# The standard set of gems in nixpkgs including potential fixes. -# -# The gemset is derived from two points of entry: -# - An attrset describing a gem, including version, source and dependencies -# This is just meta data, most probably automatically generated by a tool -# like Bundix (https://github.com/aflatter/bundix). -# { -# name = "bundler"; -# version = "1.6.5"; -# sha256 = "1s4x0f5by9xs2y24jk6krq5ky7ffkzmxgr4z1nhdykdmpsi2zd0l"; -# dependencies = [ "rake" ]; -# } -# - An optional derivation that may override how the gem is built. For popular -# gems that don't behave correctly, fixes are already provided in the form of -# derivations. -# -# This seperates "what to build" (the exact gem versions) from "how to build" -# (to make gems behave if necessary). - -{ lib, fetchurl, writeScript, ruby, kerberos, libxml2, libxslt, python, stdenv, which -, libiconv, postgresql, v8_3_16_14, clang, sqlite, zlib, imagemagick -, pkgconfig , ncurses, xapian, gpgme, utillinux, fetchpatch, tzdata, icu, libffi -, cmake, libssh2, openssl, mysql, darwin -}: - -let - v8 = v8_3_16_14; -in - -{ - charlock_holmes = attrs: { - buildInputs = [ which icu zlib ]; - }; - - ffi = attrs: { - buildInputs = [ libffi pkgconfig ]; - }; - - gpgme = attrs: { - buildInputs = [ gpgme ]; - }; - - libv8 = attrs: { - buildInputs = [ which v8 python ]; - buildFlags = [ - "--with-system-v8=true" - ]; - }; - - mysql2 = attrs: { - buildInputs = [ mysql.lib zlib openssl ]; - }; - - ncursesw = attrs: { - buildInputs = [ ncurses ]; - buildFlags = [ - "--with-cflags=-I${ncurses}/include" - "--with-ldflags=-L${ncurses}/lib" - ]; - }; - - nokogiri = attrs: { - buildFlags = [ - "--use-system-libraries" - "--with-zlib-dir=${zlib}" - "--with-xml2-lib=${libxml2}/lib" - "--with-xml2-include=${libxml2}/include/libxml2" - "--with-xslt-lib=${libxslt}/lib" - "--with-xslt-include=${libxslt}/include" - "--with-exslt-lib=${libxslt}/lib" - "--with-exslt-include=${libxslt}/include" - ] ++ lib.optional stdenv.isDarwin "--with-iconv-dir=${libiconv}"; - buildInputs = lib.optional stdenv.isDarwin darwin.libobjc; - }; - - pg = attrs: { - buildFlags = [ - "--with-pg-config=${postgresql}/bin/pg_config" - ]; - }; - - rmagick = attrs: { - buildInputs = [ imagemagick pkgconfig ]; - }; - - rugged = attrs: { - buildInputs = [ cmake pkgconfig openssl libssh2 zlib ]; - }; - - sqlite3 = attrs: { - buildFlags = [ - "--with-sqlite3-include=${sqlite}/include" - "--with-sqlite3-lib=${sqlite}/lib" - ]; - }; - - sup = attrs: { - # prevent sup from trying to dynamically install `xapian-ruby`. - postPatch = '' - cp ${./mkrf_conf_xapian.rb} ext/mkrf_conf_xapian.rb - - substituteInPlace lib/sup/crypto.rb \ - --replace 'which gpg2' \ - '${which}/bin/which gpg2' - ''; - }; - - timfel-krb5-auth = attrs: { - buildInputs = [ kerberos ]; - }; - - therubyracer = attrs: { - buildFlags = [ - "--with-v8-dir=${v8}" - "--with-v8-include=${v8}/include" - "--with-v8-lib=${v8}/lib" - ]; - }; - - tzinfo = attrs: { - postPatch = '' - substituteInPlace lib/tzinfo/zoneinfo_data_source.rb \ - --replace "/usr/share/zoneinfo" "${tzdata}/share/zoneinfo" - ''; - }; - - unf_ext = attrs: { - buildInputs = lib.optional stdenv.isDarwin darwin.libobjc; - }; - - xapian-ruby = attrs: { - # use the system xapian - buildInputs = [ xapian pkgconfig zlib ]; - postPatch = '' - cp ${./xapian-Rakefile} Rakefile - ''; - preInstall = '' - export XAPIAN_CONFIG=${xapian}/bin/xapian-config - ''; - }; -} - diff --git a/pkgs/development/interpreters/ruby/bundler-env/default.nix b/pkgs/development/interpreters/ruby/bundler-env/default.nix index 9fa6e52c4557..c7570d815e3b 100644 --- a/pkgs/development/interpreters/ruby/bundler-env/default.nix +++ b/pkgs/development/interpreters/ruby/bundler-env/default.nix @@ -1,312 +1,75 @@ { stdenv, runCommand, writeText, writeScript, writeScriptBin, ruby, lib -, callPackage, defaultGemConfig, fetchurl, fetchgit, buildRubyGem , bundler_HEAD +, callPackage, defaultGemConfig, fetchurl, fetchgit, buildRubyGem, buildEnv +, rubygems , git +, makeWrapper +, bundler +, tree }@defs: -# This is a work-in-progress. -# The idea is that his will replace load-ruby-env.nix. - { name, gemset, gemfile, lockfile, ruby ? defs.ruby, gemConfig ? defaultGemConfig -, enableParallelBuilding ? false # TODO: this might not work, given the env-var shinanigans. -, postInstall ? null -, documentation ? false +, postBuild ? null +, document ? [] , meta ? {} +, ignoreCollisions ? false , ... }@args: let shellEscape = x: "'${lib.replaceChars ["'"] [("'\\'" + "'")] x}'"; - const = x: y: x; - bundler = bundler_HEAD.override { inherit ruby; }; - inherit (builtins) attrValues; - - gemName = attrs: "${attrs.name}-${attrs.version}.gem"; - - fetchers.path = attrs: attrs.source.path; - fetchers.gem = attrs: fetchurl { - url = "${attrs.source.source or "https://rubygems.org"}/downloads/${gemName attrs}"; - inherit (attrs.source) sha256; - }; - fetchers.git = attrs: fetchgit { - inherit (attrs.source) url rev sha256 fetchSubmodules; - leaveDotGit = true; - }; - - applySrc = attrs: - attrs // { - src = (fetchers."${attrs.source.type}" attrs); - }; - + importedGemset = import gemset; applyGemConfigs = attrs: - if gemConfig ? "${attrs.name}" - then attrs // gemConfig."${attrs.name}" attrs - else attrs; - - needsPatch = attrs: - (attrs ? patches) || (attrs ? prePatch) || (attrs ? postPatch); - - # patch a gem or source tree. - # for gems, the gem is unpacked, patched, and then repacked. - # see: https://github.com/fedora-ruby/gem-patch/blob/master/lib/rubygems/patcher.rb - applyPatches = attrs: - if !needsPatch attrs - then attrs - else attrs // { src = - stdenv.mkDerivation { - name = gemName attrs; - phases = [ "unpackPhase" "patchPhase" "installPhase" ]; - buildInputs = [ ruby ] ++ attrs.buildInputs or []; - patches = attrs.patches or [ ]; - prePatch = attrs.prePatch or "true"; - postPatch = attrs.postPatch or "true"; - unpackPhase = '' - runHook preUnpack - - if [[ -f ${attrs.src} ]]; then - isGem=1 - # we won't know the name of the directory that RubyGems creates, - # so we'll just use a glob to find it and move it over. - gem unpack ${attrs.src} --target=container - cp -r container/* contents - rm -r container - else - cp -r ${attrs.src} contents - chmod -R +w contents - fi - - cd contents - runHook postUnpack - ''; - installPhase = '' - runHook preInstall - - if [[ -n "$isGem" ]]; then - ${writeScript "repack.rb" '' - #!${ruby}/bin/ruby - require 'rubygems' - require 'rubygems/package' - require 'fileutils' - - Encoding.default_internal = Encoding::UTF_8 - Encoding.default_external = Encoding::UTF_8 - - if Gem::VERSION < '2.0' - load "${./package-1.8.rb}" - end - - out = ENV['out'] - files = Dir['**/{.[^\.]*,*}'] - - package = Gem::Package.new("${attrs.src}") - patched_package = Gem::Package.new(package.spec.file_name) - patched_package.spec = package.spec.clone - patched_package.spec.files = files - - patched_package.build(false) - - FileUtils.cp(patched_package.spec.file_name, out) - ''} - else - cp -r . $out - fi - - runHook postInstall - ''; - }; - }; - - instantiate = (attrs: - applyPatches (applyGemConfigs (applySrc attrs)) - ); - - instantiated = lib.flip lib.mapAttrs (import gemset) (name: attrs: - instantiate (attrs // { inherit name; }) + (if gemConfig ? "${attrs.gemName}" + then attrs // gemConfig."${attrs.gemName}" attrs + else attrs); + configuredGemset = lib.flip lib.mapAttrs importedGemset (name: attrs: + applyGemConfigs (attrs // { gemName = name; }) ); - - needsPreInstall = attrs: - (attrs ? preInstall) || (attrs ? buildInputs) || (attrs ? nativeBuildInputs); - - # TODO: support cross compilation? look at stdenv/generic/default.nix. - runPreInstallers = lib.fold (next: acc: - if !needsPreInstall next - then acc - else acc + '' - ${writeScript "${next.name}-pre-install" '' - #!${stdenv.shell} - - export nativeBuildInputs="${toString ((next.nativeBuildInputs or []) ++ (next.buildInputs or []))}" - - source ${stdenv}/setup - - header "running pre-install script for ${next.name}" - - ${next.preInstall or ""} - - ${ruby}/bin/ruby -e 'print ENV.inspect' > env/${next.name} - - stopNest - ''} - '' - ) "" (attrValues instantiated); - - # copy *.gem to ./gems - copyGems = lib.fold (next: acc: - if next.source.type == "gem" - then acc + "cp ${next.src} gems/${gemName next}\n" - else acc - ) "" (attrValues instantiated); - - runRuby = name: env: command: - runCommand name env '' - ${ruby}/bin/ruby ${writeText name command} - ''; - - # TODO: include json_pure, so the version of ruby doesn't matter. - # not all rubies have support for JSON built-in, - # so we'll convert JSON to ruby expressions. - json2rb = writeScript "json2rb" '' - #!${ruby}/bin/ruby - begin - require 'json' - rescue LoadError => ex - require 'json_pure' - end - - puts JSON.parse(STDIN.read).inspect - ''; - - # dump the instantiated gemset as a ruby expression. - serializedGemset = runCommand "gemset.rb" { json = builtins.toJSON instantiated; } '' - printf '%s' "$json" | ${json2rb} > $out - ''; - - # this is a mapping from a source type and identifier (uri/path/etc) - # to the pure store path. - # we'll use this from the patched bundler to make fetching sources pure. - sources = runRuby "sources.rb" { gemset = serializedGemset; } '' - out = ENV['out'] - gemset = eval(File.read(ENV['gemset'])) - - sources = { - "git" => { }, - "path" => { }, - "gem" => { }, - "svn" => { } - } - - gemset.each_value do |spec| - type = spec["source"]["type"] - val = spec["src"] - key = - case type - when "gem" - spec["name"] - when "git" - spec["source"]["url"] - when "path" - spec["source"]["originalPath"] - when "svn" - nil # TODO - end - - sources[type][key] = val if key - end - - File.open(out, "wb") do |f| - f.print sources.inspect - end + hasBundler = builtins.hasAttr "bundler" importedGemset; + bundler = if hasBundler then gems.bundler else defs.bundler.override (attrs: { inherit ruby; }); + rubygems = defs.rubygems.override (attrs: { inherit ruby; }); + gems = lib.flip lib.mapAttrs configuredGemset (name: attrs: + buildRubyGem ((removeAttrs attrs ["source"]) // attrs.source // { + inherit ruby rubygems; + gemName = name; + gemPath = map (gemName: gems."${gemName}") (attrs.dependencies or []); + })); + # We have to normalize the Gemfile.lock, otherwise bundler tries to be + # helpful by doing so at run time, causing executables to immediately bail + # out. Yes, I'm serious. + confFiles = runCommand "gemfile-and-lockfile" {} '' + mkdir -p $out + cp ${gemfile} $out/Gemfile + cp ${lockfile} $out/Gemfile.lock + + cd $out + chmod +w Gemfile.lock + source ${rubygems}/nix-support/setup-hook + export GEM_PATH=${bundler}/${ruby.gemPath} + ${ruby}/bin/ruby -rubygems -e \ + "require 'bundler'; Bundler.definition.lock('Gemfile.lock')" ''; - - # rewrite PATH sources to point into the nix store. - purifiedLockfile = runRuby "purifiedLockfile" {} '' - out = ENV['out'] - sources = eval(File.read("${sources}")) - paths = sources["path"] - - lockfile = File.read("${lockfile}") - - paths.each_pair do |impure, pure| - lockfile.gsub!(/^ remote: #{Regexp.escape(impure)}/, " remote: #{pure}") - end - - File.open(out, "wb") do |f| - f.print lockfile - end - ''; - - needsBuildFlags = attrs: attrs ? buildFlags; - - mkBuildFlags = spec: - "export BUNDLE_BUILD__${lib.toUpper spec.name}='${lib.concatStringsSep " " (map shellEscape spec.buildFlags)}'"; - - allBuildFlags = - lib.concatStringsSep "\n" - (map mkBuildFlags - (lib.filter needsBuildFlags (attrValues instantiated))); - - derivation = stdenv.mkDerivation { - inherit name; - - buildInputs = [ - ruby - bundler - git - ] ++ args.buildInputs or []; - - phases = [ "installPhase" "fixupPhase" ]; - - outputs = [ - "out" # the installed libs/bins - "bundle" # supporting files for bundler - ]; - - installPhase = '' - mkdir -p $bundle - export BUNDLE_GEMFILE=$bundle/Gemfile - cp ${gemfile} $BUNDLE_GEMFILE - cp ${purifiedLockfile} $BUNDLE_GEMFILE.lock - - export NIX_GEM_SOURCES=${sources} - export NIX_BUNDLER_GEMPATH=${bundler}/${ruby.gemPath} - - export GEM_HOME=$out/${ruby.gemPath} - export GEM_PATH=$NIX_BUNDLER_GEMPATH:$GEM_HOME - mkdir -p $GEM_HOME - - ${allBuildFlags} - - mkdir gems - cp ${bundler}/${bundler.ruby.gemPath}/cache/bundler-*.gem gems - ${copyGems} - - ${lib.optionalString (!documentation) '' - mkdir home - HOME="$(pwd -P)/home" - echo "gem: --no-rdoc --no-ri" > $HOME/.gemrc - ''} - - mkdir env - ${runPreInstallers} - - mkdir $out/bin - cp ${./monkey_patches.rb} monkey_patches.rb - export RUBYOPT="-rmonkey_patches.rb -I $(pwd -P)" - bundler install --frozen --binstubs ${lib.optionalString enableParallelBuilding "--jobs $NIX_BUILD_CORES"} - RUBYOPT="" - - runHook postInstall - ''; - - inherit postInstall; - + envPaths = lib.attrValues gems ++ lib.optional (!hasBundler) bundler; + bundlerEnv = buildEnv { + inherit name ignoreCollisions; + paths = envPaths; + pathsToLink = [ "/lib" ]; + postBuild = '' + source ${rubygems}/nix-support/setup-hook + + ${ruby}/bin/ruby ${./gen-bin-stubs.rb} \ + "${ruby}/bin/ruby" \ + "${confFiles}/Gemfile" \ + "$out/${ruby.gemPath}" \ + "${bundler}/${ruby.gemPath}" \ + ${shellEscape (toString envPaths)} + '' + lib.optionalString (postBuild != null) postBuild; passthru = { - inherit ruby; - inherit bundler; - + inherit ruby bundler meta gems; env = let irbrc = builtins.toFile "irbrc" '' - if not ENV["OLD_IRBRC"].empty? + if !(ENV["OLD_IRBRC"].nil? || ENV["OLD_IRBRC"].empty?) require ENV["OLD_IRBRC"] end require 'rubygems' @@ -314,12 +77,12 @@ let ''; in stdenv.mkDerivation { name = "interactive-${name}-environment"; - nativeBuildInputs = [ ruby derivation ]; + nativeBuildInputs = [ ruby bundlerEnv ]; shellHook = '' - export BUNDLE_GEMFILE=${derivation.bundle}/Gemfile - export GEM_HOME=${derivation}/${ruby.gemPath} - export NIX_BUNDLER_GEMPATH=${bundler}/${ruby.gemPath} - export GEM_PATH=$NIX_BUNDLER_GEMPATH:$GEM_HOME + export BUNDLE_GEMFILE=${confFiles}/Gemfile + export BUNDLE_PATH=${bundlerEnv}/${ruby.gemPath} + export GEM_HOME=${bundlerEnv}/${ruby.gemPath} + export GEM_PATH=${bundlerEnv}/${ruby.gemPath} export OLD_IRBRC="$IRBRC" export IRBRC=${irbrc} ''; @@ -331,8 +94,8 @@ let ''; }; }; - - inherit meta; }; -in derivation +in + +bundlerEnv diff --git a/pkgs/development/interpreters/ruby/bundler-env/gen-bin-stubs.rb b/pkgs/development/interpreters/ruby/bundler-env/gen-bin-stubs.rb new file mode 100644 index 000000000000..fac9c9ad9446 --- /dev/null +++ b/pkgs/development/interpreters/ruby/bundler-env/gen-bin-stubs.rb @@ -0,0 +1,46 @@ +require 'rbconfig' +require 'rubygems' +require 'rubygems/specification' +require 'fileutils' + +# args/settings +out = ENV["out"] +ruby = ARGV[0] +gemfile = ARGV[1] +bundle_path = ARGV[2] +bundler_gem_path = ARGV[3] +paths = ARGV[4].split + +# generate binstubs +FileUtils.mkdir_p("#{out}/bin") +paths.each do |path| + next unless File.directory?("#{path}/nix-support/gem-meta") + + name = File.read("#{path}/nix-support/gem-meta/name") + executables = File.read("#{path}/nix-support/gem-meta/executables").split + executables.each do |exe| + File.open("#{out}/bin/#{exe}", "w") do |f| + f.write(<<-EOF) +#!#{ruby} +# +# This file was generated by Nix. +# +# The application '#{exe}' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +ENV["BUNDLE_GEMFILE"] = "#{gemfile}" +ENV["BUNDLE_PATH"] = "#{bundle_path}" + +gem_path = ENV["GEM_PATH"] +ENV["GEM_PATH"] = "\#{gem_path}\#{":" unless gem_path.nil? || gem_path.empty?}#{bundler_gem_path}" + +require 'rubygems' +require 'bundler/setup' + +load Gem.bin_path(#{name.inspect}, #{exe.inspect}) +EOF + FileUtils.chmod("+x", "#{out}/bin/#{exe}") + end + end +end diff --git a/pkgs/development/interpreters/ruby/bundler-env/mkrf_conf_xapian.rb b/pkgs/development/interpreters/ruby/bundler-env/mkrf_conf_xapian.rb deleted file mode 100644 index e19f06e23ac2..000000000000 --- a/pkgs/development/interpreters/ruby/bundler-env/mkrf_conf_xapian.rb +++ /dev/null @@ -1,14 +0,0 @@ -require 'rubygems' -require 'rubygems/command.rb' -require 'rubygems/dependency_installer.rb' -require 'rbconfig' - -begin - Gem::Command.build_args = ARGV -rescue NoMethodError -end - -# create dummy rakefile to indicate success -f = File.open(File.join(File.dirname(__FILE__), "Rakefile"), "w") -f.write("task :default\n") -f.close diff --git a/pkgs/development/interpreters/ruby/bundler-env/monkey_patches.rb b/pkgs/development/interpreters/ruby/bundler-env/monkey_patches.rb deleted file mode 100644 index f68a20212cee..000000000000 --- a/pkgs/development/interpreters/ruby/bundler-env/monkey_patches.rb +++ /dev/null @@ -1,246 +0,0 @@ -require 'bundler' - -# Undo the RUBYOPT trickery. -opt = ENV['RUBYOPT'].dup -opt.gsub!(/-rmonkey_patches.rb -I [^ ]*/, '') -ENV['RUBYOPT'] = opt - -Bundler.module_eval do - class << self - # mappings from original uris to store paths. - def nix_gem_sources - @nix_gem_sources ||= - begin - src = ENV['NIX_GEM_SOURCES'] - eval(Bundler.read_file(src)) - end - end - - # extract the gemspecs from the gems pulled from Rubygems. - def nix_gemspecs - @nix_gemspecs ||= Dir.glob("gems/*.gem").map do |path| - Bundler.rubygems.spec_from_gem(path) - end - end - - # swap out ENV - def nix_with_env(env, &block) - if env - old_env = ENV.to_hash - begin - ENV.replace(env) - block.call - ensure - ENV.replace(old_env) - end - else - block.call - end - end - - # map a git uri to a fetchgit store path. - def nix_git(uri) - Pathname.new(nix_gem_sources["git"][uri]) - end - end -end - -Bundler::Source::Git::GitProxy.class_eval do - def checkout - unless path.exist? - FileUtils.mkdir_p(path.dirname) - FileUtils.cp_r(Bundler.nix_git(@uri).join(".git"), path) - system("chmod -R +w #{path}") - end - end - - def copy_to(destination, submodules=false) - unless File.exist?(destination.join(".git")) - FileUtils.mkdir_p(destination.dirname) - FileUtils.cp_r(Bundler.nix_git(@uri), destination) - system("chmod -R +w #{destination}") - end - end -end - -Bundler::Fetcher.class_eval do - def use_api - true - end - - def fetch_dependency_remote_specs(gem_names) - Bundler.ui.debug "Query Gemcutter Dependency Endpoint API: #{gem_names.join(',')}" - deps_list = [] - - spec_list = gem_names.map do |name| - spec = Bundler.nix_gemspecs.detect {|spec| spec.name == name } - if spec.nil? - msg = "WARNING: Could not find gemspec for '#{name}'" - Bundler.ui.warn msg - nil - else - dependencies = spec.dependencies. - select {|dep| dep.type != :development}. - map do |dep| - deps_list << dep.name - dep - end - - [spec.name, spec.version, spec.platform, dependencies] - end - end - - spec_list.compact! - - [spec_list, deps_list.uniq] - end -end - -Bundler::Source::Rubygems.class_eval do - # We copy all gems into $PWD/gems, and this allows RubyGems to find those - # gems during installation. - def fetchers - @fetchers ||= [ - Bundler::Fetcher.new(URI.parse("file://#{File.expand_path(Dir.pwd)}")) - ] - end - - # Look-up gems that were originally from RubyGems. - def remote_specs - @remote_specs ||= - begin - lockfile = Bundler::LockfileParser.new(Bundler.read_file(Bundler.default_lockfile)) - gem_names = lockfile.specs. - select {|spec| spec.source.is_a?(Bundler::Source::Rubygems)}. - map {|spec| spec.name} - idx = Bundler::Index.new - api_fetchers.each do |f| - Bundler.ui.info "Fetching source index from #{f.uri}" - idx.use f.specs(gem_names, self) - end - idx - end - end -end - -Bundler::Installer.class_eval do - - # WHY: - # This allows us to provide a typical Nix experience, where - # `buildInputs` and/or `preInstall` may set up the $PATH and other env-vars - # as needed. By swapping out the environment per install, we can have finer - # grained control than we would have otherwise. - # - # HOW: - # This is a wrapper around the original `install_gem_from_spec`. - # We expect that a "pre-installer" might exist at `pre-installers/`, - # and if it does, we execute it. - # The pre-installer is expected to dump its environment variables as a Ruby - # hash to `env/`. - # We then swap out the environment for the duration of the install, - # and then set it back to what it was originally. - alias original_install_gem_from_spec install_gem_from_spec - def install_gem_from_spec(spec, standalone = false, worker = 0) - env_dump = "env/#{spec.name}" - if File.exist?(env_dump) - env = eval(Bundler.read_file(env_dump)) - unless env - Bundler.ui.error "The environment variables for #{spec.name} could not be loaded!" - exit 1 - end - Bundler.nix_with_env(env) do - original_install_gem_from_spec(spec, standalone, worker) - end - else - original_install_gem_from_spec(spec, standalone, worker) - end - end - - def generate_bundler_executable_stubs(spec, options = {}) - return if spec.executables.empty? - - out = ENV['out'] - - spec.executables.each do |executable| - next if executable == "bundle" || executable == "bundler" - - binstub_path = "#{out}/bin/#{executable}" - - File.open(binstub_path, 'w', 0777 & ~File.umask) do |f| - f.print <<-TEXT -#!#{RbConfig.ruby} - -old_gemfile = ENV["BUNDLE_GEMFILE"] -old_gem_home = ENV["GEM_HOME"] -old_gem_path = ENV["GEM_PATH"] - -ENV["BUNDLE_GEMFILE"] = - "#{ENV["BUNDLE_GEMFILE"]}" -ENV["GEM_HOME"] = - "#{ENV["GEM_HOME"]}" -ENV["GEM_PATH"] = - "#{ENV["NIX_BUNDLER_GEMPATH"]}:\#{ENV["GEM_HOME"]}\#{old_gem_path ? ":\#{old_gem_path}" : ""}}" - -require 'rubygems' -require 'bundler/setup' - -ENV["BUNDLE_GEMFILE"] = old_gemfile -ENV["GEM_HOME"] = old_gem_home -ENV["GEM_PATH"] = old_gem_path - -load Gem.bin_path('#{spec.name}', '#{executable}') -TEXT - end - end - end -end - -Gem::Installer.class_eval do - # Make the wrappers automagically use bundler. - # - # Stage 1. - # Set $BUNDLE_GEMFILE so bundler knows what gems to load. - # Set $GEM_HOME to the installed gems, because bundler looks there for - # non-Rubygems installed gems (e.g. git/svn/path sources). - # Set $GEM_PATH to include both bundler and installed gems. - # - # Stage 2. - # Setup bundler, locking down the gem versions. - # - # Stage 3. - # Reset $BUNDLE_GEMFILE, $GEM_HOME, $GEM_PATH. - # - # Stage 4. - # Run the actual executable. - def app_script_text(bin_file_name) - return <<-TEXT -#!#{RbConfig.ruby} -# -# This file was generated by Nix's RubyGems. -# -# The application '#{spec.name}' is installed as part of a gem, and -# this file is here to facilitate running it. -# - -old_gemfile = ENV["BUNDLE_GEMFILE"] -old_gem_home = ENV["GEM_HOME"] -old_gem_path = ENV["GEM_PATH"] - -ENV["BUNDLE_GEMFILE"] = - "#{ENV["BUNDLE_GEMFILE"]}" -ENV["GEM_HOME"] = - "#{ENV["GEM_HOME"]}" -ENV["GEM_PATH"] = - "#{ENV["NIX_BUNDLER_GEMPATH"]}:\#{ENV["GEM_HOME"]}\#{old_gem_path ? ":\#{old_gem_path}" : ""}}" - -require 'rubygems' -require 'bundler/setup' - -ENV["BUNDLE_GEMFILE"] = old_gemfile -ENV["GEM_HOME"] = old_gem_home -ENV["GEM_PATH"] = old_gem_path - -load Gem.bin_path('#{spec.name}', '#{bin_file_name}') -TEXT - end -end diff --git a/pkgs/development/interpreters/ruby/bundler-env/package-1.8.rb b/pkgs/development/interpreters/ruby/bundler-env/package-1.8.rb deleted file mode 100644 index 079b65f97ece..000000000000 --- a/pkgs/development/interpreters/ruby/bundler-env/package-1.8.rb +++ /dev/null @@ -1,29 +0,0 @@ -require 'rubygems/installer' -require 'rubygems/builder' - -# Simulate RubyGems 2.0 behavior. - -module Gem::Package - def self.new(gem) - @gem = gem - self - end - - def self.extract_files(dir) - installer = Gem::Installer.new @gem - installer.unpack(dir) - end - - def self.build(skip_validation=false) - builder = Gem::Builder.new(spec) - builder.build - end - - def self.spec=(spec) - @spec = spec - end - - def self.spec - @spec ||= Gem::Installer.new(@gem).spec - end -end diff --git a/pkgs/development/interpreters/ruby/bundler-env/xapian-Rakefile b/pkgs/development/interpreters/ruby/bundler-env/xapian-Rakefile deleted file mode 100644 index 9f0b8e72f08c..000000000000 --- a/pkgs/development/interpreters/ruby/bundler-env/xapian-Rakefile +++ /dev/null @@ -1,38 +0,0 @@ -# encoding: utf-8 -# Install the xapian binaries into the lib folder of the gem -require 'rbconfig' - -c = RbConfig::CONFIG - -def system!(cmd) - puts cmd - system(cmd) or raise -end - -source_dir = 'xapian_source' -bindings = Dir["#{source_dir}/xapian-bindings-*"].first -bindings = File.basename(bindings, ".tar.xz") - -task :default do - system! "tar -xJf #{source_dir}/#{bindings}.tar.xz" - - prefix = Dir.pwd - ENV['LDFLAGS'] = "-L#{prefix}/lib" - - system! "mkdir -p lib" - - Dir.chdir bindings do - ENV['RUBY'] ||= "#{c['bindir']}/#{c['RUBY_INSTALL_NAME']}" - system! "./configure --prefix=#{prefix} --exec-prefix=#{prefix} --with-ruby" - system! "make clean all" - end - - system! "cp -r #{bindings}/ruby/.libs/_xapian.* lib" - system! "cp #{bindings}/ruby/xapian.rb lib" - - system! "rm lib/*.la" - system! "rm lib/*.lai" - - system! "rm -R #{bindings}" - system! "rm -R #{source_dir}" -end diff --git a/pkgs/development/interpreters/ruby/bundler.nix b/pkgs/development/interpreters/ruby/bundler.nix index 3789170f57fe..cdcd12990e27 100644 --- a/pkgs/development/interpreters/ruby/bundler.nix +++ b/pkgs/development/interpreters/ruby/bundler.nix @@ -1,17 +1,10 @@ { buildRubyGem, makeWrapper, ruby, coreutils }: -buildRubyGem { - name = "bundler-1.10.6"; - namePrefix = ""; +buildRubyGem rec { + inherit ruby; + name = "${gemName}-${version}"; + gemName = "bundler"; + version = "1.10.6"; sha256 = "1vlzfq0bkkj4jyq6av0y55mh5nj5n0f3mfbmmifwgkh44g8k6agv"; dontPatchShebangs = true; - postInstall = '' - find $out -type f -perm -0100 | while read f; do - substituteInPlace $f \ - --replace "/usr/bin/env" "${coreutils}/bin/env" - done - - wrapProgram $out/bin/bundler \ - --prefix PATH ":" ${ruby}/bin - ''; } diff --git a/pkgs/development/interpreters/ruby/gem.nix b/pkgs/development/interpreters/ruby/gem.nix deleted file mode 100644 index bbc38226266b..000000000000 --- a/pkgs/development/interpreters/ruby/gem.nix +++ /dev/null @@ -1,136 +0,0 @@ -{ lib, ruby, rubygemsFun, fetchurl, makeWrapper, git } @ defs: - -lib.makeOverridable ( - -{ name -, ruby ? defs.ruby -, rubygems ? (rubygemsFun ruby) -, stdenv ? ruby.stdenv -, namePrefix ? "${lib.replaceStrings ["-"] ["_"] ruby.name}" + "-" -, buildInputs ? [] -, doCheck ? false -, dontBuild ? true -, meta ? {} -, gemPath ? [] -, ...} @ attrs: - -stdenv.mkDerivation (attrs // { - inherit ruby rubygems; - inherit doCheck; - - buildInputs = [ ruby rubygems makeWrapper git ] ++ buildInputs; - - name = namePrefix + name; - - src = if attrs ? src - then attrs.src - else fetchurl { - url = "http://rubygems.org/downloads/${attrs.name}.gem"; - inherit (attrs) sha256; - }; - - phases = [ "unpackPhase" "patchPhase" "buildPhase" "checkPhase" "installPhase" "fixupPhase" ]; - - # The source is expected to either be a gem package or a directory. - # - # - Gem packages are already built, so they don't even need to be unpacked. - # They will skip the buildPhase. - # - A directory containing the sources will need to go through all of the - # usual phases. - unpackPhase= '' - gemRegex="\.gem" - if [[ $src =~ $gemRegex ]] - then - runHook preUnpack - echo "source is a gem package, won't unpack" - gempkg=$src - dontBuild=1 - runHook postUnpack - else - # Fall back to the original thing for everything else. - unpackPhase - fi - ''; - - checkPhase = "true"; - - buildPhase = '' - runHook preBuild - - # TODO: Investigate. The complete working tree is touched by fetchgit. - if [ -d .git ]; then - git reset - fi - - gemspec=$(find . -name '*.gemspec') - echo "found the following gemspecs:" - echo "$gemspec" - - gemspec=$(echo "$gemspec" | head -n1) - echo "building $gemspec" - - exec 3>&1 - output=$(gem build $gemspec | tee >(cat - >&3)) - exec 3>&- - - gempkg=$(echo "$output" | grep -oP 'File: \K(.*)') - - echo "gem package built: $gempkg" - - runHook postBuild - ''; - - installPhase = '' - runHook preInstall - - # NOTE: This does NOT build the unpacked gem, but installs $src directly. - # Gems that have not been downloaded from rubygems.org may need a - # separate buildPhase. - # --ignore-dependencies is necessary as rubygems otherwise always - # connects to the repository, thus breaking pure builds. - GEM_HOME=$out/${ruby.gemPath} \ - gem install \ - --local \ - --force \ - --http-proxy "http://nodtd.invalid" \ - --ignore-dependencies \ - --build-root "/" \ - --backtrace \ - $gempkg $gemFlags -- $buildFlags - - # Yes, we really do need the $out/${ruby.gemPath}/cache. - # This is very important in order for many parts of RubyGems/Bundler to not blow up. - # See https://github.com/bundler/bundler/issues/3327 - - mkdir -p $out/bin - for prog in $out/${ruby.gemPath}/gems/*/bin/*; do - makeWrapper $prog $out/bin/$(basename $prog) \ - --prefix GEM_PATH : "$out/${ruby.gemPath}:$GEM_PATH" \ - --prefix RUBYLIB : "${rubygems}/lib" \ - $extraWrapperFlags ''${extraWrapperFlagsArray[@]} - done - #--prefix RUBYOPT rubygems \ - - # looks like useless files which break build repeatability and consume space - rm -fv $out/${ruby.gemPath}/doc/*/*/created.rid || true - rm -fv $out/${ruby.gemPath}/gems/*/ext/*/mkmf.log || true - - mkdir -p $out/nix-support - - cat > $out/nix-support/setup-hook <= v3.16.14.8, + # otherwise the gem will fail to link to the libv8 binary. + # see: https://github.com/cowboyd/libv8/pull/161 + libv8 = attrs: { + buildInputs = [ which v8 python ]; + buildFlags = [ "--with-system-v8=true" ]; + }; + + mysql2 = attrs: { + buildInputs = [ mysql.lib zlib openssl ]; + }; + + ncursesw = attrs: { + buildInputs = [ ncurses ]; + buildFlags = [ + "--with-cflags=-I${ncurses}/include" + "--with-ldflags=-L${ncurses}/lib" + ]; + }; + + nokogiri = attrs: { + buildFlags = [ + "--use-system-libraries" + "--with-zlib-dir=${zlib}" + "--with-xml2-lib=${libxml2}/lib" + "--with-xml2-include=${libxml2}/include/libxml2" + "--with-xslt-lib=${libxslt}/lib" + "--with-xslt-include=${libxslt}/include" + "--with-exslt-lib=${libxslt}/lib" + "--with-exslt-include=${libxslt}/include" + ] ++ lib.optional stdenv.isDarwin "--with-iconv-dir=${libiconv}"; + buildInputs = lib.optional stdenv.isDarwin darwin.libobjc; + }; + + patron = attrs: { + buildInputs = [ curl ]; + }; + + pg = attrs: { + buildFlags = [ + "--with-pg-config=${postgresql}/bin/pg_config" + ]; + }; + + puma = attrs: { + buildInputs = [ openssl ]; + }; + + rmagick = attrs: { + buildInputs = [ imagemagick pkgconfig ]; + }; + + rugged = attrs: { + buildInputs = [ cmake pkgconfig openssl libssh2 zlib ]; + }; + + sqlite3 = attrs: { + buildFlags = [ + "--with-sqlite3-include=${sqlite}/include" + "--with-sqlite3-lib=${sqlite}/lib" + ]; + }; + + sup = attrs: { + dontBuild = false; + # prevent sup from trying to dynamically install `xapian-ruby`. + postPatch = '' + cp ${./mkrf_conf_xapian.rb} ext/mkrf_conf_xapian.rb + + substituteInPlace lib/sup/crypto.rb \ + --replace 'which gpg2' \ + '${which}/bin/which gpg2' + ''; + }; + + timfel-krb5-auth = attrs: { + buildInputs = [ kerberos ]; + }; + + therubyracer = attrs: { + buildFlags = [ + "--with-v8-dir=${v8}" + "--with-v8-include=${v8}/include" + "--with-v8-lib=${v8}/lib" + ]; + }; + + tzinfo = attrs: { + dontBuild = false; + postPatch = '' + substituteInPlace lib/tzinfo/zoneinfo_data_source.rb \ + --replace "/usr/share/zoneinfo" "${tzdata}/share/zoneinfo" + ''; + }; + + unf_ext = attrs: { + buildInputs = lib.optional stdenv.isDarwin darwin.libobjc; + }; + + xapian-ruby = attrs: { + # use the system xapian + dontBuild = false; + buildInputs = [ xapian pkgconfig zlib ]; + postPatch = '' + cp ${./xapian-Rakefile} Rakefile + ''; + preInstall = '' + export XAPIAN_CONFIG=${xapian}/bin/xapian-config + ''; + }; +} + diff --git a/pkgs/development/interpreters/ruby/gemconfig/mkrf_conf_xapian.rb b/pkgs/development/interpreters/ruby/gemconfig/mkrf_conf_xapian.rb new file mode 100644 index 000000000000..e19f06e23ac2 --- /dev/null +++ b/pkgs/development/interpreters/ruby/gemconfig/mkrf_conf_xapian.rb @@ -0,0 +1,14 @@ +require 'rubygems' +require 'rubygems/command.rb' +require 'rubygems/dependency_installer.rb' +require 'rbconfig' + +begin + Gem::Command.build_args = ARGV +rescue NoMethodError +end + +# create dummy rakefile to indicate success +f = File.open(File.join(File.dirname(__FILE__), "Rakefile"), "w") +f.write("task :default\n") +f.close diff --git a/pkgs/development/interpreters/ruby/gemconfig/xapian-Rakefile b/pkgs/development/interpreters/ruby/gemconfig/xapian-Rakefile new file mode 100644 index 000000000000..9f0b8e72f08c --- /dev/null +++ b/pkgs/development/interpreters/ruby/gemconfig/xapian-Rakefile @@ -0,0 +1,38 @@ +# encoding: utf-8 +# Install the xapian binaries into the lib folder of the gem +require 'rbconfig' + +c = RbConfig::CONFIG + +def system!(cmd) + puts cmd + system(cmd) or raise +end + +source_dir = 'xapian_source' +bindings = Dir["#{source_dir}/xapian-bindings-*"].first +bindings = File.basename(bindings, ".tar.xz") + +task :default do + system! "tar -xJf #{source_dir}/#{bindings}.tar.xz" + + prefix = Dir.pwd + ENV['LDFLAGS'] = "-L#{prefix}/lib" + + system! "mkdir -p lib" + + Dir.chdir bindings do + ENV['RUBY'] ||= "#{c['bindir']}/#{c['RUBY_INSTALL_NAME']}" + system! "./configure --prefix=#{prefix} --exec-prefix=#{prefix} --with-ruby" + system! "make clean all" + end + + system! "cp -r #{bindings}/ruby/.libs/_xapian.* lib" + system! "cp #{bindings}/ruby/xapian.rb lib" + + system! "rm lib/*.la" + system! "rm lib/*.lai" + + system! "rm -R #{bindings}" + system! "rm -R #{source_dir}" +end diff --git a/pkgs/development/interpreters/ruby/load-ruby-env.nix b/pkgs/development/interpreters/ruby/load-ruby-env.nix deleted file mode 100644 index c4356ed5f50f..000000000000 --- a/pkgs/development/interpreters/ruby/load-ruby-env.nix +++ /dev/null @@ -1,69 +0,0 @@ -{ ruby, lib, callPackage, gemFixes, fetchurl, fetchgit, buildRubyGem }@defs: - -# This function builds a set of gems. You first convert your Gemfile to an attrset -# called a "gemset", and then use this function to build the gemset. -# -# A gemset looks like the following: -# -# { -# libv8 = { -# version = "3.16.14.7"; -# src = { -# type = "gem"; -# sha256 = "..."; -# }; -# }; -# therubyracer = { -# version = "0.12.1"; -# dependencies = [ "libv8" ]; -# src = { -# type = "gem"; -# sha256 = "..."; -# }; -# }; -# } -# -# If you use these gems as build inputs, the GEM_PATH will be updated -# appropriately, and command like `bundle exec` should work out of the box. - -{ gemset, ruby ? defs.ruby, fixes ? gemFixes }@args: - -let - const = x: y: x; - - fetchers.path = attrs: attrs.src.path; - fetchers.gem = attrs: fetchurl { - url = "${attrs.src.source or "https://rubygems.org"}/downloads/${attrs.name}-${attrs.version}.gem"; - inherit (attrs.src) sha256; - }; - fetchers.git = attrs: fetchgit { - inherit (attrs.src) url rev sha256 fetchSubmodules; - leaveDotGit = true; - }; - - instantiate = (attrs: - let - defaultAttrs = { - name = "${attrs.name}-${attrs.version}"; - inherit ruby gemPath; - }; - gemPath = map (name: gemset''."${name}") (attrs.dependencies or []); - fixedAttrs = attrs // (fixes."${attrs.name}" or (const {})) attrs; - withSource = fixedAttrs // - (if (lib.isDerivation fixedAttrs.src || builtins.isString fixedAttrs.src) - then {} - else { src = (fetchers."${fixedAttrs.src.type}" fixedAttrs); }); - - in - buildRubyGem (withSource // defaultAttrs) - ); - - gemset' = if builtins.isAttrs gemset then gemset else import gemset; - - gemset'' = lib.flip lib.mapAttrs gemset' (name: attrs: - if (lib.isDerivation attrs) - then attrs - else instantiate (attrs // { inherit name; }) - ); - -in gemset'' diff --git a/pkgs/development/interpreters/ruby/rubygems.nix b/pkgs/development/interpreters/ruby/rubygems.nix index f4942b840919..b6ac04808979 100644 --- a/pkgs/development/interpreters/ruby/rubygems.nix +++ b/pkgs/development/interpreters/ruby/rubygems.nix @@ -1,37 +1,35 @@ -args @ { makeWrapper, ruby, ... }: with args; +{ stdenv, lib, fetchurl, makeWrapper, ruby }: -rec { - name = "rubygems-" + version; +stdenv.mkDerivation rec { + name = "rubygems-${version}"; version = "2.4.1"; src = fetchurl { url = "http://production.cf.rubygems.org/rubygems/${name}.tgz"; sha256 = "0cpr6cx3h74ykpb0cp4p4xg7a8j0bhz3sk271jq69l4mm4zy4h4f"; }; + patches = [ ./gem_hook.patch ]; + buildInputs = [ruby makeWrapper]; - configureFlags = []; - doInstall = fullDepEntry ('' + buildPhase = ":"; + + installPhase = '' ruby setup.rb --prefix=$out/ + wrapProgram $out/bin/gem --prefix RUBYLIB : $out/lib - find $out -type f -name "*.rb" | xargs sed -i "s@/usr/bin/env@$(type -p env)@g" + + find $out -type f -name "*.rb" | + xargs sed -i "s@/usr/bin/env@$(type -p env)@g" + mkdir -pv $out/nix-support cat > $out/nix-support/setup-hook <