Python dlopen()s musl libc

Open

Details

4 participants

Josselin Poiret
Lars-Dominik Braun
Ludovic Courtès
Athena Martin

Owner: unassigned

Submitted by: Athena Martin

Severity: normal

Athena Martin wrote on 27 Jun 2023 02:42

Recipients:(address . bug-guix@gnu.org)

Message-ID:20230626204230.7b3b773a@spock.hosts.alm.website

I've had experiences now with multiple Guix packages, including gajim

(bug 60235) and now python-neovim-remote, which have an issue where

Python tries to dlopen() libc, but finds the system libc instead of

Guix's, resulting on Alpine Linux hosts in a crash with this message:

ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

There are a variety of tracebacks that lead up to this, depending on

the package in question.

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEELAGoSzc6nHIdbhP+S8DcQuG8sXMFAmSaMHYACgkQS8DcQuG8

sXNzwwf/QSs5nNKbTPCDx+ucfoiR1bYTS40WHqBfPvccKoXJ01sGgh332OpRlyMb

ggXdFOjtcTrZqnTNzYOpU5JvE6zWLWNj2Za5Uz4SXWWTX/0E+t93jyQdrDqF0NNr

MheYJ+cSHO5SclaezT60lom3FS9KttQHDaS6IUfDFnMdCtNpnUvFE+hUH55XJmSc

My/kFy1Rv8OgeB5jPj+Kf6I1Fn2n0m68dWenzyjn0GMJl4Wu8qvUlvTz1erDEARE

vKAZdFXmkUKaZEGYL9gC9uf3k1Vf2gUJu0nG/hNhfbb9c6CQbGPiMJNk8xy78DwO

TlDn8/GS/yHryjyeHrS7DbtYPQWz+Q==

=Bjwl

-----END PGP SIGNATURE-----

Ludovic Courtès wrote on 7 Jul 2023 16:00

Recipients:(name . Athena Martin)(address . secure@alm.website)(address . 64309@debbugs.gnu.org)

Message-ID:87mt07kej7.fsf@gnu.org

Hi,

Athena Martin <secure@alm.website> skribis:

Toggle quote (10 lines)> I've had experiences now with multiple Guix packages, including gajim
> (bug 60235) and now python-neovim-remote, which have an issue where
> Python tries to dlopen() libc, but finds the system libc instead of
> Guix's, resulting on Alpine Linux hosts in a crash with this message:
>
> ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory
>
> There are a variety of tracebacks that lead up to this, depending on
> the package in question.

Could you provide a command to reproduce this?

Thanks in advance,

Ludo’.

Ludovic Courtès wrote on 7 Jul 2023 16:00

control message for bug #64309

Recipients:(address . control@debbugs.gnu.org)

Message-ID:87lefrkeiv.fsf@gnu.org

tags 64309 + moreinfo

quit

Athena Martin wrote on 8 Jul 2023 22:16

Re: bug#64309: Python dlopen()s musl libc

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 64309@debbugs.gnu.org)

Message-ID:20230708161658.25be509c@spock.hosts.alm.website

Toggle quote (5 lines)

> > ImportError: libc.musl-x86_64.so.1: cannot open shared object file:

> > No such file or directory

> Could you provide a command to reproduce this?

On an Alpine Linux Edge host, I reproduce with:

$ guix install python-neovim-remote

$ nvr

It's possible that there's some quirk of my specific environment.

(I'm currently in the process of switching to Guix System so I may not

be able to reproduce this forever.)

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEELAGoSzc6nHIdbhP+S8DcQuG8sXMFAmSpxDoACgkQS8DcQuG8

sXN83wf/V7A5dV7nXevxTWjWXnMU5SvzC0zdjhXExa1rVn9xMKKbai8+SSAHzAkF

EGFvq6TUc3oTNG7ErrPpsV/AyIVli1tCcLJhreBt1M75ZHi1vX0ccRveJjrk5vk3

eHhYY903rnFfY9kihCMQqMnBnNynhovr3MJc2gdM3khQgFy8VwruCf23zPQahrdT

C6GEfO76w7dQ10SorJuh1U4gsslxfOZe1bLCprZaOgqaHHO+RAxdKVEgr7o4FVMb

SjSPjWvSwuBocRE8VzkZsRdZpZZTHB0m9OuPPNufIO+joYyhu6EECG1gwxajKNix

yr4oXdPH3vVwV/vcO6tTrV2ByKEjrg==

=/syo

-----END PGP SIGNATURE-----

Josselin Poiret wrote on 9 Jul 2023 10:35

Recipients:(address . 64309@debbugs.gnu.org)

Message-ID:87v8et8oug.fsf@jpoiret.xyz

Hi Athena,

Athena Martin via Bug reports for GNU Guix <bug-guix@gnu.org> writes:

Toggle quote (10 lines)> On an Alpine Linux Edge host, I reproduce with:
>
> $ guix install python-neovim-remote
> $ nvr
>
> It's possible that there's some quirk of my specific environment.
>
> (I'm currently in the process of switching to Guix System so I may not
> be able to reproduce this forever.)

Can you do `LD_DEBUG=libs nvr` so that we get a log of what ld's trying
to load?

Best,
-- 
Josselin Poiret

Athena Martin wrote on 9 Jul 2023 22:22

Recipients:(name . Josselin Poiret)(address . dev@jpoiret.xyz)

Message-ID:20230709162221.39fb0e66@spock.hosts.alm.website

Toggle quote (3 lines)

> Can you do `LD_DEBUG=libs nvr` so that we get a log of what ld's

> trying to load?

I've attached the full log. The first mention of musl is on line 200 and
everything seems to happen pretty fast, here's the most relevant
portion:

     16405:	find library=libc.musl-x86_64.so.1 [0]; searching
     16405:	 search cache=/gnu/store/kj6wzba6p192baizq99b489rs8bynpn7-python-3.10.7/etc/ld.so.cache
     16405:	 search path=/gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib		(system search path)
     16405:	  trying file=/gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libc.musl-x86_64.so.1
     16405:	
Traceback (most recent call last):
  File "/gnu/store/dsgxdqs620pp284bfm1drbsjqpb36i4n-python-neovim-remote-2.5.1/bin/.nvr-real", line 4, in <module>
    import nvr.nvr as mod
  File "/home/alm/.local/lib/python3.10/site-packages/nvr/__init__.py", line 1, in <module>
    from .nvr import main
  File "/home/alm/.local/lib/python3.10/site-packages/nvr/nvr.py", line 34, in <module>
    import psutil
  File "/home/alm/.local/lib/python3.10/site-packages/psutil/__init__.py", line 102, in <module>
    from . import _pslinux as _psplatform
  File "/home/alm/.local/lib/python3.10/site-packages/psutil/_pslinux.py", line 26, in <module>
    from . import _psutil_linux as cext
ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

After that it starts calling fini()s and terminates.

Attachment: stderr.log

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEELAGoSzc6nHIdbhP+S8DcQuG8sXMFAmSrFv0ACgkQS8DcQuG8

sXOxAAgAjot+pPtMO79F5afeokuVMiERpmPlw0uDSLsQ/+ivt2WaupOAm7Ag9sdg

MgeecEF1qWRVlVzLa5kxp9t+gb+AfWt4Ykh+eguTQSGnpecwVChuyW6BezkW1/iP

F5udkxgS6gaPk0DOoCsxiapSpwUKUW4DuUdf7GAXYRCJZCOZ1PVxdjMEjukEWE4v

bSF07LbF5Gy/5/8/rygEoxEgj/pVM5ARwPWHaPjUztLXxBnML22UHbN+kudZqLUH

MuDw1HxEJOE0SHLOWa5jbwAiO912tIaRP4CiVTyt0fsWekRhXbFUH2VbiLEeb77v

YI2DjpSG/NP/xJLz/gjADA8NAUHyEA==

=BNXr

-----END PGP SIGNATURE-----

Josselin Poiret wrote on 10 Jul 2023 09:13

Recipients:(name . Athena Martin)(address . secure@alm.website)

Message-ID:87pm508cir.fsf@jpoiret.xyz

Hi Athena,

So it's not the LD_DEBUG output that hold a clue, but rather the Python

traceback.

Athena Martin <secure@alm.website> writes:

Toggle quote (12 lines)>   File "/gnu/store/dsgxdqs620pp284bfm1drbsjqpb36i4n-python-neovim-remote-2.5.1/bin/.nvr-real", line 4, in <module>
>     import nvr.nvr as mod
>   File "/home/alm/.local/lib/python3.10/site-packages/nvr/__init__.py", line 1, in <module>
>     from .nvr import main
>   File "/home/alm/.local/lib/python3.10/site-packages/nvr/nvr.py", line 34, in <module>
>     import psutil
>   File "/home/alm/.local/lib/python3.10/site-packages/psutil/__init__.py", line 102, in <module>
>     from . import _pslinux as _psplatform
>   File "/home/alm/.local/lib/python3.10/site-packages/psutil/_pslinux.py", line 26, in <module>
>     from . import _psutil_linux as cext
> ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

The nvr package in ~/.local seems to be used instead of a Guix package.
That locally installed nvr package expects to use the host's libc, but
since the python interpreter being used has a fixed RPATH and system
search path it won't find it.

.nvr-real should definitely be using the Python code inside the store, I
wonder why that isn't being done.  Maybe our sitecustomize.py is
misbehaving?  Can you do `guix shell python-neovim-remote python --
python3` then type `import sys.path; sys.path`?

Best,
-- 
Josselin Poiret

Athena Martin wrote on 10 Jul 2023 21:47

Recipients:(name . Josselin Poiret)(address . dev@jpoiret.xyz)

Message-ID:20230710154734.2b71b56b@spock.hosts.alm.website

Toggle quote (8 lines)

> So it's not the LD_DEBUG output that hold a clue, but rather the

> Python traceback.

> The nvr package in ~/.local seems to be used instead of a Guix

> package. That locally installed nvr package expects to use the host's

> libc, but since the python interpreter being used has a fixed RPATH

> and system search path it won't find it.

Ah, I see.

Toggle quote (6 lines)

> .nvr-real should definitely be using the Python code inside the

> store, I wonder why that isn't being done. Maybe our

> sitecustomize.py is misbehaving? Can you do `guix shell

> python-neovim-remote python -- python3` then type `import sys.path;

> sys.path`?

$ guix shell python-neovim-remote python -- python3

Toggle quote (1 lines)

>>> sys.path

['', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python310.zip', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/lib-dynload', '/home/alm/.local/lib/python3.10/site-packages', '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/site-packages']

(It's not from the environment:)

$ guix shell python-neovim-remote python --pure -- python3

Toggle quote (1 lines)

>>> sys.path

Toggle quote (1 lines)

>>> os.environ

environ({'HOME': '/home/alm', 'LOGNAME': 'alm', 'PAGER': 'less', 'DISPLAY': ':0', 'USER': 'alm', 'TERM': 'xterm-256color', 'PATH': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/bin', 'GUIX_PYTHONPATH': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages', 'GUIX_ENVIRONMENT': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile'})

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEELAGoSzc6nHIdbhP+S8DcQuG8sXMFAmSsYFYACgkQS8DcQuG8

sXPssAf9EpHPxtQE5Z6rrhegm0NzQhRde1l10Ygdl6S1fEwv7nfi5sMc5HoZqmFJ

0bsPdV7zH28ja3HdTzwVGim4ZlwGQoU6JDzpJRzfb9mT6CYIwo/2/qHExWl233Vk

9AqPg+4Z08x2oYRDJ1l2+pPCJXs5Es58M/2cpj29Y7MIxgkUgPJGsKuwhev2Akd5

XqY4URLFWRUswFws4xFDwyr7I0wy00uNbWxZJVQU38F7rP3n4zT/+fuRNalDk2/A

53SU8E8vegFlY2zWbTxcOAhgitRvMtmFz1DEK1TqIBQIc+0s5xpktnhXN0p4uJbK

+itMeRwHilBS+GbJqXFL05z5HylJ6Q==

=kHyC

-----END PGP SIGNATURE-----

Athena Martin wrote on 10 Jul 2023 22:21

Recipients:(name . Josselin Poiret)(address . dev@jpoiret.xyz)

Message-ID:20230710162122.190051b8@spock.hosts.alm.website

Toggle quote (5 lines)

> The nvr package in ~/.local seems to be used instead of a Guix

> package. That locally installed nvr package expects to use the

> host's libc, but since the python interpreter being used has a

> fixed RPATH and system search path it won't find it.

I've just checked and temporarily removing .local/lib/python3.10 makes

Guix's nvr work, and takes the .local site-packages off sys.path.

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEELAGoSzc6nHIdbhP+S8DcQuG8sXMFAmSsaEIACgkQS8DcQuG8

sXP/uwgAvwU4yWcpnff4eX3I5rhCXgDEoOly3UMZpovPF26dqW9RpFQxgrJF5Aar

6+vGASglCZpSSvwbsDcUhNaL0oeWvXUvmiYrc70mmkR2j0TRLdI8PDD5q7tWr0CZ

rfiOL/i4GS0wso+qD0L4mEWPdDNe8Vx2ob2DDS3JS7VLKXAYo8wu/m5Bu9mVjf6n

SQ+/SVxAl9MDtFM2YzlAJnEMhjB7AfMtRFdrKxEtwURyjSZfAWm3QZqiuSUgbndP

c8CtP+YhD0gRMfRTfdQB07TyCF66Kl2zkairo0XkOh36s4DwlIZ1gN77GcedDY+I

35IOv6PxY0bqQ4IILukHTdpPAxkJMg==

=vq0j

-----END PGP SIGNATURE-----

Josselin Poiret wrote on 11 Jul 2023 10:34

Recipients:(name . Athena Martin)(address . secure@alm.website)

Message-ID:87jzv6979d.fsf@jpoiret.xyz

Hi Athena,

Athena Martin <secure@alm.website> writes:

Toggle quote (12 lines)> $ guix shell python-neovim-remote python -- python3
>>>> sys.path
> ['', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python310.zip', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/lib-dynload', '/home/alm/.local/lib/python3.10/site-packages', '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/site-packages']
>
> (It's not from the environment:)
>
> $ guix shell python-neovim-remote python --pure -- python3
>>>> sys.path
> ['', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python310.zip', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/lib-dynload', '/home/alm/.local/lib/python3.10/site-packages', '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/site-packages']
>>>> os.environ
> environ({'HOME': '/home/alm', 'LOGNAME': 'alm', 'PAGER': 'less', 'DISPLAY': ':0', 'USER': 'alm', 'TERM': 'xterm-256color', 'PATH': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/bin', 'GUIX_PYTHONPATH': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages', 'GUIX_ENVIRONMENT': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile'})

So, looks like the default Python behavior is to load the usercustomize
after the sitecustomize [1], which leads to exactly the behavior you're
experiencing.  I don't know what we should do here, maybe pass `-s` to
the shebang line of Python to disable loading the usercustomize?  That
would probably be a world-rebuild though.  CC'ing the Python team to see
what they think.

[1] https://docs.python.org/3/library/site.html

Best,
-- 
Josselin Poiret

Lars-Dominik Braun wrote on 11 Jul 2023 14:43

Recipients:(name . Josselin Poiret)(address . dev@jpoiret.xyz)

Message-ID:ZK1OeNPPIGHDZGpw@noor.fritz.box

Hi,

there’s a similar issue #63912 and a thread on the guix-devel

mailinglist at

https://lists.gnu.org/archive/html/guix-devel/2023-06/msg00066.html

Toggle quote (7 lines)

> So, looks like the default Python behavior is to load the usercustomize

> after the sitecustomize [1], which leads to exactly the behavior you're

> experiencing. I don't know what we should do here, maybe pass `-s` to

> the shebang line of Python to disable loading the usercustomize? That

> would probably be a world-rebuild though. CC'ing the Python team to see

> what they think.

I think the problem is bigger than usercustomize. Any custom PYTHONPATH
also slips through and causes this issue, as well as any custom
GUIX_PYTHONPATH, because the executable wrapper appends it (think nested
`guix shell` invokations with different versions of a library for
an example where this could go wrong).

Guix-managed Python packages (libraries nor applications) should
generally not pick up dependencies from random paths – only those
from their package description, so we can keep Guix’ promise of being
self-contained.

I have experimented with customizing Python’s importing mechanism
through a custom MetaPathFinder. It works by adding a __guix_pythonpath__
variable to every Python package’s __init__.py file and modifying the
module loader’s search path accordingly if such a variable exists. It
would provide exactly that guarantee, but it’s just a PoC at this
point – see attached file.

Apart from that I don’t see a good short-term solution right now. It’s
just how Python works.

Cheers,
Lars

# Credits to
# https://tenthousandmeters.com/blog/python-behind-the-scenes-11-how-the-python-import-system-works/
# for the very good explanation of how Python’s import statement works.

import sys, os
from importlib.abc import MetaPathFinder
from importlib.machinery import SourceFileLoader, ModuleSpec, PathFinder

class GuixPythonFinder (MetaPathFinder):
    def find_spec (self, fullname, path, target=None):
        # Short-circuit for non-top-level imports, which already have a path.
        if path:
            return None

        attrname = '__guix_pythonpath__'
        searchPath = None
        # Search for our caller.
        frame = sys._getframe ()
        while frame:
            # If he has a search path, use it. This is mainly for executable
            # scripts with `__name__ == '__main__'`.
            searchPath = frame.f_globals.get (attrname, None)
            if not searchPath:
                # Otherwise check the top-level package for search paths
                # declared in __init__.py
                package = frame.f_globals.get ('__package__')
                if package:
                    module = sys.modules.get (package)
                    if module:
                        searchPath = getattr (module, attrname, None)
            if searchPath is not None:
                break
            frame = frame.f_back

        # If we have a caller…
        if searchPath is not None:
            return PathFinder.find_spec (fullname, searchPath, target=target)
        else:
            # Otherwise we’re not responsible for this module.
            return None

sys.meta_path.insert (0, GuixPythonFinder ())

Josselin Poiret wrote on 12 Jul 2023 10:50

Recipients:(name . Lars-Dominik Braun)(address . lars@6xq.net)

Message-ID:87h6q98qe7.fsf@jpoiret.xyz

Hi Lars,

Lars-Dominik Braun <lars@6xq.net> writes:

Toggle quote (18 lines)> I think the problem is bigger than usercustomize. Any custom PYTHONPATH
> also slips through and causes this issue, as well as any custom
> GUIX_PYTHONPATH, because the executable wrapper appends it (think nested
> `guix shell` invokations with different versions of a library for
> an example where this could go wrong).
>
> Guix-managed Python packages (libraries nor applications) should
> generally not pick up dependencies from random paths – only those
> from their package description, so we can keep Guix’ promise of being
> self-contained.
>
> I have experimented with customizing Python’s importing mechanism
> through a custom MetaPathFinder. It works by adding a __guix_pythonpath__
> variable to every Python package’s __init__.py file and modifying the
> module loader’s search path accordingly if such a variable exists. It
> would provide exactly that guarantee, but it’s just a PoC at this
> point – see attached file.

Woah, looks like a neat solution. Do you think it would scale for all

our Python packages without manual intervention? If so, this would

definitely be the way forward.

Toggle quote (3 lines)

> Apart from that I don’t see a good short-term solution right now. It’s

> just how Python works.

I mostly agree with you, but for this rather common case of having also
a usercustomize it would be nice to circumvent it.  In general, I don't
think we ever want a Guix-produced Python script to load the
usercustomize, hence my suggestion.  The other case of PYTHONPATH is
also annoying but can be tamed by modifying the env variable
temporarily.

Best,
-- 
Josselin Poiret

Lars-Dominik Braun wrote on 16 Jul 2023 10:35

Recipients:(name . Josselin Poiret)(address . dev@jpoiret.xyz)

Message-ID:ZLOr5K4bTkZceqyS@noor.fritz.box

Hi Josselin,

Toggle quote (4 lines)

> Woah, looks like a neat solution. Do you think it would scale for all

> our Python packages without manual intervention? If so, this would

> definitely be the way forward.

I hope so, but haven’t tried it yet.

Toggle quote (7 lines)

> I mostly agree with you, but for this rather common case of having also

> a usercustomize it would be nice to circumvent it. In general, I don't

> think we ever want a Guix-produced Python script to load the

> usercustomize, hence my suggestion. The other case of PYTHONPATH is

> also annoying but can be tamed by modifying the env variable

> temporarily.

True, PYTHONPATH can be unset more easily than moving the user site

dir. I’ll have a look at #64573, which should work around this issue.

Cheers,

Lars

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 64309@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 64309

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date