path: root/doc/languages-frameworks/python.xml

<section xmlns="http://docbook.org/ns/docbook"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         xml:id="sec-python">

<title>Python</title>

<para>
  Currently supported interpreters are <varname>python26</varname>, <varname>python27</varname>,
  <varname>python33</varname>, <varname>python34</varname>, <varname>python35</varname>
  and <varname>pypy</varname>.
</para>

<para>
  <varname>python</varname> is an alias to <varname>python27</varname> and <varname>python3</varname> is an alias to <varname>python34</varname>.
</para>

<para>
  <varname>python26</varname> and <varname>python27</varname> do not include modules that require
  external dependencies (to reduce dependency bloat). Following modules need to be added as
  <varname>buildInput</varname> explicitly:
</para>

<itemizedlist>
  <listitem><para><varname>python.modules.bsddb</varname></para></listitem>
  <listitem><para><varname>python.modules.curses</varname></para></listitem>
  <listitem><para><varname>python.modules.curses_panel</varname></para></listitem>
  <listitem><para><varname>python.modules.crypt</varname></para></listitem>
  <listitem><para><varname>python.modules.gdbm</varname></para></listitem>
  <listitem><para><varname>python.modules.sqlite3</varname></para></listitem>
  <listitem><para><varname>python.modules.tkinter</varname></para></listitem>
  <listitem><para><varname>python.modules.readline</varname></para></listitem>
</itemizedlist>

<para>For convenience <varname>python27Full</varname> and <varname>python26Full</varname>
are provided with all modules included.</para>

<para>
  Python packages that
  use <link xlink:href="http://pypi.python.org/pypi/setuptools/"><literal>setuptools</literal></link> or <literal>distutils</literal>,
  can be built using the <varname>buildPythonPackage</varname> function as documented below.
</para>

<para>
 All packages depending on any Python interpreter get appended <varname>$out/${python.sitePackages}</varname>
 to <literal>$PYTHONPATH</literal> if such directory exists.
</para>

<variablelist>
  <title>
     Useful attributes on interpreters packages:
  </title>

  <varlistentry>
    <term><varname>libPrefix</varname></term>
    <listitem><para>
        Name of the folder in <literal>${python}/lib/</literal> for corresponding interpreter.
    </para></listitem>
  </varlistentry>

  <varlistentry>
    <term><varname>interpreter</varname></term>
    <listitem><para>
        Alias for <literal>${python}/bin/${executable}.</literal>
    </para></listitem>
  </varlistentry>

  <varlistentry>
    <term><varname>buildEnv</varname></term>
    <listitem><para>
        Function to build python interpreter environments with extra packages bundled together.
        See <xref linkend="ssec-python-build-env" /> for usage and documentation.
    </para></listitem>
  </varlistentry>

  <varlistentry>
    <term><varname>sitePackages</varname></term>
    <listitem><para>
      Alias for <literal>lib/${libPrefix}/site-packages</literal>.
    </para></listitem>
  </varlistentry>

  <varlistentry>
    <term><varname>executable</varname></term>
    <listitem><para>
      Name of the interpreter executable, ie <literal>python3.4</literal>.
    </para></listitem>
  </varlistentry>

</variablelist>
<section xml:id="ssec-build-python-package"><title><varname>buildPythonPackage</varname> function</title>

  <para>
  The function is implemented in <link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/generic/default.nix">
  <filename>pkgs/development/python-modules/generic/default.nix</filename></link>.
  Example usage:

<programlisting language="nix">
twisted = buildPythonPackage {
  name = "twisted-8.1.0";

  src = pkgs.fetchurl {
    url = http://tmrc.mit.edu/mirror/twisted/Twisted/8.1/Twisted-8.1.0.tar.bz2;
    sha256 = "0q25zbr4xzknaghha72mq57kh53qw1bf8csgp63pm9sfi72qhirl";
  };

  propagatedBuildInputs = [ self.ZopeInterface ];

  meta = {
    homepage = http://twistedmatrix.com/;
    description = "Twisted, an event-driven networking engine written in Python";
    license = stdenv.lib.licenses.mit;
  };
};
</programlisting>

  Most of Python packages that use <varname>buildPythonPackage</varname> are defined
  in <link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/python-packages.nix"><filename>pkgs/top-level/python-packages.nix</filename></link>
  and generated for each python interpreter separately into attribute sets <varname>python26Packages</varname>,
  <varname>python27Packages</varname>, <varname>python35Packages</varname>, <varname>python33Packages</varname>,
  <varname>python34Packages</varname> and <varname>pypyPackages</varname>.
  </para>

  <para>
    <function>buildPythonPackage</function> mainly does four things:

    <orderedlist>
      <listitem><para>
        In the <varname>buildPhase</varname>, it calls
        <literal>${python.interpreter} setup.py bdist_wheel</literal> to build a wheel binary zipfile.
      </para></listitem>

      <listitem><para>
        In the <varname>installPhase</varname>, it installs the wheel file using
        <literal>pip install *.whl</literal>.
      </para></listitem>

      <listitem><para>
        In the <varname>postFixup</varname> phase, <literal>wrapPythonPrograms</literal>
        bash function is called to wrap all programs in <filename>$out/bin/*</filename>
        directory to include <literal>$PYTHONPATH</literal> and <literal>$PATH</literal>
        environment variables.
      </para></listitem>

      <listitem><para>
        In the <varname>installCheck</varname> phase, <literal>${python.interpreter} setup.py test</literal>
        is ran.
      </para></listitem>
    </orderedlist>
  </para>

  <para>By default <varname>doCheck = true</varname> is set</para>

  <para>
    As in Perl, dependencies on other Python packages can be specified in the
    <varname>buildInputs</varname> and
    <varname>propagatedBuildInputs</varname> attributes.  If something is
    exclusively a build-time dependency, use
    <varname>buildInputs</varname>; if it’s (also) a runtime dependency,
    use <varname>propagatedBuildInputs</varname>.
  </para>

  <para>
    By default <varname>meta.platforms</varname> is set to the same value
    as the interpreter unless overriden otherwise.
  </para>

  <variablelist>
    <title>
      <varname>buildPythonPackage</varname> parameters
      (all parameters from <varname>mkDerivation</varname> function are still supported)
    </title>

    <varlistentry>
      <term><varname>namePrefix</varname></term>
      <listitem><para>
        Prepended text to <varname>${name}</varname> parameter.
        Defaults to <literal>"python3.3-"</literal> for Python 3.3, etc. Set it to
        <literal>""</literal>
        if you're packaging an application or a command line tool.
      </para></listitem>
    </varlistentry>

    <varlistentry>
      <term><varname>disabled</varname></term>
      <listitem><para>
        If <varname>true</varname>, package is not build for
        particular python interpreter version. Grep around
        <filename>pkgs/top-level/python-packages.nix</filename>
        for examples.
      </para></listitem>
    </varlistentry>

    <varlistentry>
      <term><varname>setupPyBuildFlags</varname></term>
      <listitem><para>
        List of flags passed to <command>setup.py build_ext</command> command.
      </para></listitem>
    </varlistentry>

    <varlistentry>
      <term><varname>pythonPath</varname></term>
      <listitem><para>
        List of packages to be added into <literal>$PYTHONPATH</literal>.
        Packages in <varname>pythonPath</varname> are not propagated
        (contrary to <varname>propagatedBuildInputs</varname>).
      </para></listitem>
    </varlistentry>

    <varlistentry>
      <term><varname>preShellHook</varname></term>
      <listitem><para>
        Hook to execute commands before <varname>shellHook</varname>.
      </para></listitem>
    </varlistentry>

    <varlistentry>
      <term><varname>postShellHook</varname></term>
      <listitem><para>
        Hook to execute commands after <varname>shellHook</varname>.
      </para></listitem>
    </varlistentry>

    <varlistentry>
      <term><varname>makeWrapperArgs</varname></term>
      <listitem><para>
        A list of strings. Arguments to be passed to
        <varname>makeWrapper</varname>, which wraps generated binaries. By
        default, the arguments to <varname>makeWrapper</varname> set
        <varname>PATH</varname> and <varname>PYTHONPATH</varname> environment
        variables before calling the binary. Additional arguments here can
        allow a developer to set environment variables which will be
        available when the binary is run. For example,
        <varname>makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]</varname>.
      </para></listitem>
    </varlistentry>

  </variablelist>

</section>

<section xml:id="ssec-python-build-env"><title><function>python.buildEnv</function> function</title>
  <para>
    Create Python environments using low-level <function>pkgs.buildEnv</function> function. Example <filename>default.nix</filename>:

<programlisting language="nix">
<![CDATA[with import <nixpkgs> {};

python.buildEnv.override {
  extraLibs = [ pkgs.pythonPackages.pyramid ];
  ignoreCollisions = true;
}]]>
</programlisting>

    Running <command>nix-build</command> will create
    <filename>/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env</filename>
    with wrapped binaries in <filename>bin/</filename>.
  </para>

  <para>
  You can also use <varname>env</varname> attribute to create local
  environments with needed packages installed (somewhat comparable to
  <literal>virtualenv</literal>). For example, with the following
  <filename>shell.nix</filename>:

<programlisting language="nix">
<![CDATA[with import <nixpkgs> {};

(python3.buildEnv.override {
  extraLibs = with python3Packages;
    [ numpy
      requests
    ];
}).env]]>
</programlisting>

  Running <command>nix-shell</command> will drop you into a shell where
  <command>python</command> will have specified packages in its path.
  </para>

  <variablelist>
    <title>
      <function>python.buildEnv</function> arguments
    </title>

    <varlistentry>
      <term><varname>extraLibs</varname></term>
      <listitem><para>
        List of packages installed inside the environment.
      </para></listitem>
    </varlistentry>

    <varlistentry>
      <term><varname>postBuild</varname></term>
      <listitem><para>
        Shell command executed after the build of environment.
      </para></listitem>
    </varlistentry>

    <varlistentry>
      <term><varname>ignoreCollisions</varname></term>
      <listitem><para>
         Ignore file collisions inside the environment (default is <varname>false</varname>).
      </para></listitem>
    </varlistentry>
  </variablelist>
</section>

<section xml:id="ssec-python-tools"><title>Tools</title>

<para>Packages inside nixpkgs are written by hand. However many tools
exist in community to help save time. No tool is preferred at the moment.
</para>

<itemizedlist>

  <listitem><para>
    <link xlink:href="https://github.com/proger/python2nix">python2nix</link>
    by Vladimir Kirillov
  </para></listitem>

  <listitem><para>
    <link xlink:href="https://github.com/garbas/pypi2nix">pypi2nix</link>
    by Rok Garbas
  </para></listitem>

  <listitem><para>
    <link xlink:href="https://github.com/offlinehacker/pypi2nix">pypi2nix</link>
    by Jaka Hudoklin
  </para></listitem>

</itemizedlist>

</section>

<section xml:id="ssec-python-development"><title>Development</title>

  <para>
    To develop Python packages <function>buildPythonPackage</function> has
    additional logic inside <varname>shellPhase</varname> to run
    <command>pip install -e . --prefix $TMPDIR/</command> for the package.
  </para>

  <warning><para><varname>shellPhase</varname> is executed only if <filename>setup.py</filename>
  exists.</para></warning>

  <para>
    Given a <filename>default.nix</filename>:

<programlisting language="nix">
<![CDATA[with import <nixpkgs> {};

buildPythonPackage {
  name = "myproject";

  buildInputs = with pkgs.pythonPackages; [ pyramid ];

  src = ./.;
}]]>
</programlisting>

    Running <command>nix-shell</command> with no arguments should give you
    the environment in which the package would be build with
    <command>nix-build</command>.
  </para>

  <para>
    Shortcut to setup environments with C headers/libraries and python packages:

    <programlisting language="bash">$ nix-shell -p pythonPackages.pyramid zlib libjpeg git</programlisting>
  </para>

  <note><para>
    There is a boolean value <varname>lib.inNixShell</varname> set to
    <varname>true</varname> if nix-shell is invoked.
  </para></note>

</section>

<section xml:id="ssec-python-faq"><title>FAQ</title>

<variablelist>

  <varlistentry>
    <term>How to solve circular dependencies?</term>
    <listitem><para>
      If you have packages <varname>A</varname> and <varname>B</varname> that
      depend on each other, when packaging <varname>B</varname> override package
      <varname>A</varname> not to depend on <varname>B</varname> as input
      (and also the other way around).
    </para></listitem>
  </varlistentry>

  <varlistentry>
    <term><varname>install_data / data_files</varname> problems resulting into <literal>error: could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc': Permission denied</literal></term>
    <listitem><para>
      <link xlink:href="https://bitbucket.org/pypa/setuptools/issue/130/install_data-doesnt-respect-prefix">
      Known bug in setuptools <varname>install_data</varname> does not respect --prefix</link>. Example of
      such package using the feature is <filename>pkgs/tools/X11/xpra/default.nix</filename>. As workaround
      install it as an extra <varname>preInstall</varname> step:

      <programlisting>${python.interpreter} setup.py install_data --install-dir=$out --root=$out
sed -i '/ = data_files/d' setup.py</programlisting>
    </para></listitem>
  </varlistentry>

  <varlistentry>
    <term>Rationale of non-existent global site-packages</term>
    <listitem><para>
      There is no need to have global site-packages in Nix. Each package has isolated
      dependency tree and installing any python package will only populate <varname>$PATH</varname>
      inside user environment. See <xref linkend="ssec-python-build-env" /> to create self-contained
      interpreter with a set of packages.
    </para></listitem>
  </varlistentry>

</variablelist>

</section>


<section xml:id="ssec-python-contrib"><title>Contributing guidelines</title>
<para>
  Following rules are desired to be respected:
</para>

<itemizedlist>

  <listitem><para>
    Make sure package builds for all python interpreters. Use <varname>disabled</varname> argument to
    <function>buildPythonPackage</function> to set unsupported interpreters.
  </para></listitem>

  <listitem><para>
    If tests need to be disabled for a package, make sure you leave a comment about reasoning.
  </para></listitem>

  <listitem><para>
    Packages in <link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/python-packages.nix"><filename>pkgs/top-level/python-packages.nix</filename></link>
    are sorted quasi-alphabetically to avoid merge conflicts.
  </para></listitem>

</itemizedlist>

</section>

</section>