cpython: Use --enable-optimizations, for a 16% speedup.

Without this flag, the configure script prints a warning at the end,
like this (reformatted):

  If you want a release build with all stable optimizations active
  (PGO, etc), please run ./configure --enable-optimizations

We're doing a build to distribute to people for day-to-day use,
doing things other than developing the Python interpreter.  So
that's certainly a release build -- we're the target audience for
this recommendation.

---

And, trying it out, upstream isn't kidding!  I ran the standard
benchmark suite that the CPython developers use for performance
work, "pyperformance".  Following its usage instructions:
  https://pyperformance.readthedocs.io/usage.html
I ran the whole suite, like so:

  $ nix-shell -p ./result."$variant" --run '
      cd $(mktemp -d); python -m venv venv; . venv/bin/activate
      pip install pyperformance
      pyperformance run -o ~/tmp/result.'"$variant"'.json
    '

and then examined the results with commands like:

  $ python -m pyperf compare_to --table -G \
      ~/tmp/result.{$before,$after}.json

Across all the benchmarks in the suite, the median speedup was 16%.
(Meaning 1.16x faster; 14% less time).

The middle half of them ranged from a 13% to a 22% speedup.

Each of the 60 benchmarks in the suite got faster, by speedups
ranging from 3% to 53%.

---

One reason this isn't just the default to begin with is that, until
recently, it made the build a lot slower.  What it does is turn on
profile-guided optimization, which means first build for profiling,
then run some task to get a profile, then build again using the
profile.  And, short of further customization, the task it would use
would be nearly the full test suite, which includes a lot of
expensive and slow tests, and can easily take half an hour to run.

Happily, in 2019 an upstream developer did the work to carefully
select a more appropriate set of tests to use for the profile:
  https://github.com/python/cpython/commit/4e16a4a31
  https://bugs.python.org/issue36044
This suite takes just 2 minutes to run.  And the resulting final
build is actually slightly faster than with the much longer suite,
at least as measured by those standard "pyperformance" benchmarks.
That work went into the 3.8 release, but the same list works great
if used on older releases too.

So, start passing that --enable-optimizations flag; and backport
that good-for-PGO set of tests, so that we use it on all releases.

5 files changed, 76 insertions, 0 deletions

diff --git a/pkgs/development/interpreters/python/cpython/2.7/default.nix b/pkgs/development/interpreters/python/cpython/2.7/default.nix
index 77b37c5f5c39..0df49213f5a3 100644
--- a/pkgs/development/interpreters/python/cpython/2.7/default.nix
+++ b/pkgs/development/interpreters/python/cpython/2.7/default.nix
@@ -85,6 +85,9 @@ let
       # backported in debian since 2013.
       # https://bugs.python.org/issue13146
       ./atomic_pyc.patch
+
+      # Backport from CPython 3.8 of a good list of tests to run for PGO.
+      ./profile-task.patch
     ] ++ optionals (x11Support && stdenv.isDarwin) [
       ./use-correct-tcl-tk-on-darwin.patch
     ] ++ optionals stdenv.isLinux [
@@ -135,6 +138,7 @@ let
     '';
 
   configureFlags = [
+    "--enable-optimizations"
     "--enable-shared"
     "--with-threads"
     "--enable-unicode=ucs${toString ucsEncoding}"
diff --git a/pkgs/development/interpreters/python/cpython/2.7/profile-task.patch b/pkgs/development/interpreters/python/cpython/2.7/profile-task.patch
new file mode 100644
index 000000000000..9c085657ac9d
--- /dev/null
+++ b/pkgs/development/interpreters/python/cpython/2.7/profile-task.patch
@@ -0,0 +1,21 @@
+Backport from CPython 3.8 of a good list of tests to run for PGO.
+
+Upstream commit:
+  https://github.com/python/cpython/commit/4e16a4a31
+
+Upstream discussion:
+  https://bugs.python.org/issue36044
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index 00fdd21ce..713dc1e53 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -259,7 +259,7 @@ TCLTK_LIBS=
+ # The task to run while instrumented when building the profile-opt target.
+ # We exclude unittests with -x that take a rediculious amount of time to
+ # run in the instrumented training build or do not provide much value.
+-PROFILE_TASK=-m test.regrtest --pgo -x test_asyncore test_gdb test_multiprocessing test_subprocess
++PROFILE_TASK=-m test.regrtest --pgo test_array test_base64 test_binascii test_binop test_bisect test_bytes test_bz2 test_cmath test_codecs test_collections test_complex test_dataclasses test_datetime test_decimal test_difflib test_embed test_float test_fstring test_functools test_generators test_hashlib test_heapq test_int test_itertools test_json test_long test_lzma test_math test_memoryview test_operator test_ordered_dict test_pickle test_pprint test_re test_set test_sqlite test_statistics test_struct test_tabnanny test_time test_unicode test_xml_etree test_xml_etree_c
+ 
+ # report files for gcov / lcov coverage report
+ COVERAGE_INFO= $(abs_builddir)/coverage.info
diff --git a/pkgs/development/interpreters/python/cpython/3.5/profile-task.patch b/pkgs/development/interpreters/python/cpython/3.5/profile-task.patch
new file mode 100644
index 000000000000..39d5587379ca
--- /dev/null
+++ b/pkgs/development/interpreters/python/cpython/3.5/profile-task.patch
@@ -0,0 +1,21 @@
+Backport from CPython 3.8 of a good list of tests to run for PGO.
+
+Upstream commit:
+  https://github.com/python/cpython/commit/4e16a4a31
+
+Upstream discussion:
+  https://bugs.python.org/issue36044
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index 00fdd21ce..713dc1e53 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -259,7 +259,7 @@ TCLTK_LIBS=
+ # The task to run while instrumented when building the profile-opt target.
+ # We exclude unittests with -x that take a rediculious amount of time to
+ # run in the instrumented training build or do not provide much value.
+-PROFILE_TASK=-m test.regrtest --pgo -x test_asyncore test_gdb test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_main_handling test_multiprocessing_spawn test_subprocess
++PROFILE_TASK=-m test.regrtest --pgo test_array test_base64 test_binascii test_binop test_bisect test_bytes test_bz2 test_cmath test_codecs test_collections test_complex test_dataclasses test_datetime test_decimal test_difflib test_embed test_float test_fstring test_functools test_generators test_hashlib test_heapq test_int test_itertools test_json test_long test_lzma test_math test_memoryview test_operator test_ordered_dict test_pickle test_pprint test_re test_set test_sqlite test_statistics test_struct test_tabnanny test_time test_unicode test_xml_etree test_xml_etree_c
+ 
+ # report files for gcov / lcov coverage report
+ COVERAGE_INFO= $(abs_builddir)/coverage.info
diff --git a/pkgs/development/interpreters/python/cpython/3.6/profile-task.patch b/pkgs/development/interpreters/python/cpython/3.6/profile-task.patch
new file mode 100644
index 000000000000..df55da3a4132
--- /dev/null
+++ b/pkgs/development/interpreters/python/cpython/3.6/profile-task.patch
@@ -0,0 +1,21 @@
+Backport from CPython 3.8 of a good list of tests to run for PGO.
+
+Upstream commit:
+  https://github.com/python/cpython/commit/4e16a4a31
+
+Upstream discussion:
+  https://bugs.python.org/issue36044
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index 00fdd21ce..713dc1e53 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -259,7 +259,7 @@ TCLTK_LIBS=
+ # The task to run while instrumented when building the profile-opt target.
+ # We exclude unittests with -x that take a rediculious amount of time to
+ # run in the instrumented training build or do not provide much value.
+-PROFILE_TASK=-m test.regrtest --pgo
++PROFILE_TASK=-m test.regrtest --pgo test_array test_base64 test_binascii test_binop test_bisect test_bytes test_bz2 test_cmath test_codecs test_collections test_complex test_dataclasses test_datetime test_decimal test_difflib test_embed test_float test_fstring test_functools test_generators test_hashlib test_heapq test_int test_itertools test_json test_long test_lzma test_math test_memoryview test_operator test_ordered_dict test_pickle test_pprint test_re test_set test_sqlite test_statistics test_struct test_tabnanny test_time test_unicode test_xml_etree test_xml_etree_c
+ 
+ # report files for gcov / lcov coverage report
+ COVERAGE_INFO= $(abs_builddir)/coverage.info
diff --git a/pkgs/development/interpreters/python/cpython/default.nix b/pkgs/development/interpreters/python/cpython/default.nix
index 1df70fb99ccb..7dbe6216e728 100644
--- a/pkgs/development/interpreters/python/cpython/default.nix
+++ b/pkgs/development/interpreters/python/cpython/default.nix
@@ -104,6 +104,14 @@ in with passthru; stdenv.mkDerivation {
   ] ++ optionals (isPy37 || isPy38) [
     # Fix darwin build https://bugs.python.org/issue34027
     ./3.7/darwin-libutil.patch
+  ] ++ optionals (pythonOlder "3.8") [
+    # Backport from CPython 3.8 of a good list of tests to run for PGO.
+    (
+      if isPy36 || isPy37 then
+        ./3.6/profile-task.patch
+      else
+        ./3.5/profile-task.patch
+    )
   ] ++ optionals (isPy3k && hasDistutilsCxxPatch) [
     # Fix for http://bugs.python.org/issue1222585
     # Upstream distutils is calling C compiler to compile C++ code, which
@@ -135,6 +143,7 @@ in with passthru; stdenv.mkDerivation {
   PYTHONHASHSEED=0;
 
   configureFlags = [
+    "--enable-optimizations"
     "--enable-shared"
     "--without-ensurepip"
     "--with-system-expat"