core-updates: python-minimal@2.7.13 fails due to missing zlib

Done

Details

3 participants

Danny Milosavljevic
Leo Famulari
Ludovic Courtès

Owner: unassigned

Submitted by: Leo Famulari

Severity: normal

Leo Famulari wrote on 17 Mar 2017 02:46

Recipients:(address . bug-guix@gnu.org)

Message-ID:20170317014622.GA27842@jasmine

python-minimal@2.7.13 is failing to build on core-updates:

https://hydra.gnu.org/build/1912761

I saw this near the beginning of the install phase:

------
configure: error: cannot run /bin/sh
/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Modules/_ctypes/libffi/config.sub
Failed to configure _ctypes module

Python build finished, but the necessary bits to build these modules
were not found:
_bsddb             _curses            _curses_panel   
_sqlite3           _ssl               _tkinter        
bsddb185           dbm                dl              
gdbm               imageop            readline        
sunaudiodev        zlib                               
To find the necessary bits, look in setup.py in detect_modules() for the
module's name.
------

And at the end, it fails:

------
if test "xinstall" != "xno"  ; then \
	case install in \
		upgrade) ensurepip="--upgrade" ;; \
		install|*) ensurepip="" ;; \
	esac; \
	LD_LIBRARY_PATH=/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13 ./python -E -m ensurepip \
		$ensurepip --root=/ ; \
fi
Traceback (most recent call last):
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__main__.py", line 4, in <module>
    ensurepip._main()
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__init__.py", line 205, in _main
    default_pip=args.default_pip,
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__init__.py", line 108, in bootstrap
    _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__init__.py", line 31, in _run_pip
    import pip
zipimport.ZipImportError: can't decompress data; zlib not available
make: *** [Makefile:927: install] Error 1
phase `install' failed after 29.1 seconds
------

Danny Milosavljevic wrote on 17 Mar 2017 15:08

Recipients:(name . Leo Famulari)(address . leo@famulari.name)(address . 26136@debbugs.gnu.org)

Message-ID:20170317150851.56e225f0@scratchpost.org

Hmmm... Python 2.7.13 seems to bundle libffi-3.1. Do we want that?

Also, I checked libffi/configure: If CONFIG_SHELL was set, it would have picked it up.

Also, why does Python 2.7.13 try to configure things in the *install* phase ?

Leo Famulari wrote on 18 Mar 2017 18:26

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 26136@debbugs.gnu.org)

Message-ID:20170318172601.GA2721@jasmine

On Fri, Mar 17, 2017 at 03:08:51PM +0100, Danny Milosavljevic wrote:

Toggle quote (2 lines)

> Hmmm... Python 2.7.13 seems to bundle libffi-3.1. Do we want that?

The python-minimal package *should* use that libffi. I'm not sure why

it's failing now.

Toggle quote (2 lines)

> Also, I checked libffi/configure: If CONFIG_SHELL was set, it would have picked it up.

Do you mean if CONFIG_SHELL is not set, so the build process can't find

libffi?

There is also the question about zlib and the bundled pip: why is it

failing now?

I haven't yet found the changes between 2.7.12 and 2.7.13 that are

causing these issues.

Danny Milosavljevic wrote on 18 Mar 2017 21:19

Recipients:(name . Leo Famulari)(address . leo@famulari.name)(address . 26136@debbugs.gnu.org)

Message-ID:20170318211901.35ed1744@scratchpost.org

Hi Leo,

Toggle quote (2 lines)

>The python-minimal package *should* use that libffi. I'm not sure why it's failing now.

Yeah, sorry. I checked the package definition now and it explicitly disables the system libffi.

Toggle quote (2 lines)

>Do you mean if CONFIG_SHELL is not set, so the build process can't find libffi?

Yes, a workaround for the first problem is:

Toggle diff (38 lines)diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index be452f062..978aeca47 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -192,7 +192,8 @@
                                   '("Lib/subprocess.py"
                                     "Lib/popen2.py"
                                     "Lib/distutils/tests/test_spawn.py"
-                                    "Lib/test/test_subprocess.py"))
+                                    "Lib/test/test_subprocess.py"
+                                    "Modules/_ctypes/libffi/configure"))
                (("/bin/sh") (which "sh")))
 
              ;; Use zero as the timestamp in .pyc files so that builds are

The reason why that helps is because the configure script contains embedded HERE scripts with their own shebangs which patch-shebang doesn't patch.

An alternative workaround is to use this instead:

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index be452f062..04c944d84 100644
--- a/gnu/packages/python.scm12/bin/shore-updates/guix$ /gnu/store/k7029k5va68lka
+++ b/gnu/packages/python.scm
@@ -229,6 +229,8 @@
            ;; before 1980".  Work around this by setting the file times in the
            ;; source tree to sometime in early 1980.
            (lambda _
+             (setenv "CONFIG_SHELL" (which "bash"))
+
              (let ((circa-1980 (* 10 366 24 60 60)))
                (ftw "." (lambda (file stat flag)
                           (utime file circa-1980 circa-1980)

>There is also the question about zlib and the bundled pip: why is it failing now?

The above doesn't fix the pip zlib problem.  It's very strange: The value of the environment variable C_INCLUDE_PATH includes bzip2 but not zlib.  Might make python unable to find zlib.

setup.py does some strange stuff with reading the zlib.h header file manually - maybe that broke.  Then it tries to find the library file for libz.  The value of the environment variable LIBRARY_PATH doesn't contain zlib either.

Danny Milosavljevic wrote on 18 Mar 2017 21:50

Recipients:(name . Leo Famulari)(address . leo@famulari.name)(address . 26136@debbugs.gnu.org)

Message-ID:20170318215057.344e288c@scratchpost.org

Setting the environment variables

CPPFLAGS="-I/gnu/store/y8c62fyb2x4g4qnl54pmlmid1fn8k3mx-zlib-1.2.8/include"

LDFLAGS="-L/gnu/store/y8c62fyb2x4g4qnl54pmlmid1fn8k3mx-zlib-1.2.8/lib"

before running configure makes it work.

(I've also printed inc_dirs in setup.py without that - and zlib didn't appear)

So I guess the general question is how does Python set up inc_dirs ?

And is it normal that C_INCLUDE_PATH and LIBRARY_PATH doesn't contain zlib ? Apparently, since guix-master python2-minimal 2.7.12 doesn't contain it either - and that one works...

So where would it get the other entries?

Leo Famulari wrote on 19 Mar 2017 20:19

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 26136@debbugs.gnu.org)

Message-ID:20170319191950.GA14148@jasmine

On Sat, Mar 18, 2017 at 09:19:01PM +0100, Danny Milosavljevic wrote:

Toggle quote (13 lines)> An alternative workaround is to use this instead:
> 
> diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
> index be452f062..04c944d84 100644
> --- a/gnu/packages/python.scm12/bin/shore-updates/guix$ /gnu/store/k7029k5va68lka
> +++ b/gnu/packages/python.scm
> @@ -229,6 +229,8 @@
>             ;; before 1980".  Work around this by setting the file times in the
>             ;; source tree to sometime in early 1980.
>             (lambda _
> +             (setenv "CONFIG_SHELL" (which "bash"))
> +

I think this is the better solution to the libffi / ctypes issue. It

also fixes python-minimal@3 to use the bundled libffi. We added libffi

to python-minimal@3's inputs in

d0b73960db051baa468fc1a0fbab21fa9f89f885 to work around a similar issue.

In Python 3.7, the bundled libffi will be removed, and we will have to

build our minimal Python-3 variant with an external libffi package:

https://bugs.python.org/issue27976

Here's a patch that fixes this problem for python-minimal@2. I don't

want to change python-minimal@3 this late in the core-updates cycle; we

can do it in a python-updates branch after core-updates.

From 9d7eef243fca8e53fb89bb17a999f0678d191caf Mon Sep 17 00:00:00 2001

libffi.

* gnu/packages/python.scm (python-2)[arguments]: Add 'setenv' phase.

(python2-minimal)[inputs]: Add zlib.

Co-authored-by: Danny Milosavljevic <dannym@scratchpost.org>

---

gnu/packages/python.scm | 17 +++++++++++++++--

1 file changed, 15 insertions(+), 2 deletions(-)

Toggle diff (44 lines)diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index be452f062..95bf53c8f 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -17,7 +17,7 @@
 ;;; Copyright © 2015, 2016 Efraim Flashner <efraim@flashner.co.il>
 ;;; Copyright © 2015 Kyle Meyer <kyle@kyleam.com>
 ;;; Copyright © 2015, 2016 Chris Marusich <cmmarusich@gmail.com>
-;;; Copyright © 2016 Danny Milosavljevic <dannym+a@scratchpost.org>
+;;; Copyright © 2016, 2017 Danny Milosavljevic <dannym+a@scratchpost.org>
 ;;; Copyright © 2016 Lukas Gradl <lgradl@openmailbox.org>
 ;;; Copyright © 2016 Hartmut Goebel <h.goebel@crazy-compilers.com>
 ;;; Copyright © 2016 Daniel Pimentel <d4n1@d4n1.org>
@@ -223,6 +223,19 @@
              (setenv "HOME" (getcwd))
              #t))
           (add-after
+           'unpack 'setenv
+           (lambda _
+             ;; 'Modules/_ctypes/libffi/configure' contains scripts embedded in
+             ;; HEREDOCs whose shebangs aren't patched by the patch-shebang
+             ;; phase.  Setting this variables works around that shortcoming and
+             ;; allows us to keep using the bundled libffi in python-minimal and
+             ;; python2-minimal.
+             ;; XXX The bundled libffi will be removed in Python 3.7:
+             ;; https://bugs.python.org/issue27976
+             ;; https://github.com/python/cpython/commit/935043d1ac19d07ac48233c6175250e66a7bbebd
+             ;; https://github.com/python/cpython/commit/f40d4ddff3c800b3c956a5e8820aabe3aa87cddd
+             (setenv "CONFIG_SHELL" (which "sh"))))
+          (add-after
            'unpack 'set-source-file-times-to-1980
            ;; XXX One of the tests uses a ZIP library to pack up some of the
            ;; source tree, and fails with "ZIP does not support timestamps
@@ -363,7 +376,7 @@ data types.")
      (substitute-keyword-arguments (package-arguments python-2)
        ((#:configure-flags cf)
         `(append ,cf '("--without-system-ffi")))))
-    (inputs '())))                          ;none of the optional dependencies
+    (inputs `(("zlib" ,zlib)))))
 
 (define-public python-minimal
   (package (inherit python)
-- 
2.12.0

Leo Famulari wrote on 19 Mar 2017 20:26

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 26136@debbugs.gnu.org)

Message-ID:20170319192636.GA16604@jasmine

On Sat, Mar 18, 2017 at 09:19:01PM +0100, Danny Milosavljevic wrote:

Toggle quote (12 lines)> >There is also the question about zlib and the bundled pip: why is it
> >failing now?
> 
> The above doesn't fix the pip zlib problem.  It's very strange: The
> value of the environment variable C_INCLUDE_PATH includes bzip2 but
> not zlib.  Might make python unable to find zlib.
> 
> setup.py does some strange stuff with reading the zlib.h header file
> manually - maybe that broke.  Then it tries to find the library file
> for libz.  The value of the environment variable LIBRARY_PATH doesn't
> contain zlib either.

Ideally, the minimal Python variants could use the bundled zlib found at
'Modules/zlib'. We sort of gave up on this with python-minimal@3 the
last time we had this problem, but maybe we can solve it now :)

Leo Famulari wrote on 19 Mar 2017 21:15

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 26136@debbugs.gnu.org)

Message-ID:20170319201545.GA22896@jasmine

On Sun, Mar 19, 2017 at 03:26:36PM -0400, Leo Famulari wrote:

Toggle quote (4 lines)

> On Sat, Mar 18, 2017 at 09:19:01PM +0100, Danny Milosavljevic wrote:

> > >There is also the question about zlib and the bundled pip: why is it

> > >failing now?

Here are some build logs showing differences between master and
core-updates.

For 2.7.12, on the master branch, there is this:

------
if test "xinstall" != "xno"  ; then \
        case install in \
                upgrade) ensurepip="--upgrade" ;; \
                install|*) ensurepip="" ;; \
        esac; \
        LD_LIBRARY_PATH=/tmp/guix-build-python-minimal-2.7.12.drv-0/Python-2.7.12 ./python -E -m ensurepip \
                $ensurepip --root=/ ; \
fi
Ignoring ensurepip failure: pip 8.1.1 requires SSL/TLS
phase `install' succeeded after 31.5 seconds
------

I guess we didn't notice the lack of TLS support because the bundled pip
is never able to reach the internet when used to build packages.

For 2.7.13 without zlib, on core-updates, it does this:

------
if test "xinstall" != "xno"  ; then \
        case install in \
                upgrade) ensurepip="--upgrade" ;; \
                install|*) ensurepip="" ;; \
        esac; \
        LD_LIBRARY_PATH=/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13 ./python -E -m ensurepip \
                $ensurepip --root=/ ; \
fi
Traceback (most recent call last):
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__main__.py", line 4, in <module>
    ensurepip._main()
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__init__.py", line 205, in _main 
    default_pip=args.default_pip,
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__init__.py", line 108, in bootstrap
    _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__init__.py", line 31, in _run_pip
    import pip
zipimport.ZipImportError: can't decompress data; zlib not available
make: *** [Makefile:927: install] Error 1
phase `install' failed after 10.8 seconds
------

If you build 2.7.13 with zlib on core-updates (log not *not* attached):

------
if test "xinstall" != "xno"  ; then \                                                                                                                                                                   case install in \
                upgrade) ensurepip="--upgrade" ;; \
                install|*) ensurepip="" ;; \
        esac; \                                                                                                                                                                                         LD_LIBRARY_PATH=/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13 ./python -E -m ensurepip \                                                                                                    $ensurepip --root=/ ; \                                                                                                                                                         fi
Collecting setuptools                                                                                                                                                                           Collecting pip                                                                                                                                                                                  Installing collected packages: setuptools, pip                                                                                                                                                  Successfully installed pip-9.0.1 setuptools-28.8.0
phase `install' succeeded after 21.6 seconds                  
------

Attachment: python-minimal-2.7.12.log.bz2

Attachment: python-minimal-2.7.13-no-zlib.log.gz

Ludovic Courtès wrote on 19 Mar 2017 22:40

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:87r31t0wtl.fsf@gnu.org

Hi!

Adding libffi and zlib solves the problem, similar to python-minimal@3.

Go for it? :-)

Ludo’.

From 1cb73a54813136632553f03063abdd5cd105a760 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Sun, 19 Mar 2017 22:10:12 +0100
Subject: [PATCH] gnu: python-minimal@2: Add dependencies on libffi and zlib.

* gnu/packages/python.scm (python2-minimal)[inputs]: Add LIBFFI and ZLIB.
[arguments]: Remove.
---
 gnu/packages/python.scm | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

Toggle diff (24 lines)diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index be452f062..00598776a 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -359,11 +359,12 @@ data types.")
   (package (inherit python-2)
     (name "python-minimal")
     (outputs '("out"))
-    (arguments
-     (substitute-keyword-arguments (package-arguments python-2)
-       ((#:configure-flags cf)
-        `(append ,cf '("--without-system-ffi")))))
-    (inputs '())))                          ;none of the optional dependencies
+
+    ;; Keep zlib, which is used by 'pip' (via the 'zipimport' module), which
+    ;; is invoked upon 'make install'.  'pip' also expects 'ctypes' and thus
+    ;; libffi.
+    (inputs `(("libffi" ,libffi)
+              ("zlib" ,zlib)))))
 
 (define-public python-minimal
   (package (inherit python)
-- 
2.12.0

Ludovic Courtès wrote on 20 Mar 2017 00:02

Recipients:(name . Leo Famulari)(address . leo@famulari.name)(address . 26136@debbugs.gnu.org)

Message-ID:87var4zx7n.fsf@gnu.org

ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (8 lines)

> From 1cb73a54813136632553f03063abdd5cd105a760 Mon Sep 17 00:00:00 2001

> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>

> Date: Sun, 19 Mar 2017 22:10:12 +0100

> Subject: [PATCH] gnu: python-minimal@2: Add dependencies on libffi and zlib.

> * gnu/packages/python.scm (python2-minimal)[inputs]: Add LIBFFI and ZLIB.

> [arguments]: Remove.

Pushed as 5927a843b4a91416b5be9e65c69616447f82e162 in core-updates,

following our discussion on IRC.

Thanks,

Ludo’.

Ludovic Courtès wrote on 26 Mar 2017 15:41

control message for bug #26136

Recipients:(address . control@debbugs.gnu.org)

Message-ID:87h92gqhnd.fsf@gnu.org

tags 26136 fixed

close 26136

Your comment

This issue is archived.

To comment on this conversation send an email to 26136@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 26136

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date