Wrapped python programs get native-inputs in PYTHONPATH

The 'wrap' phase of 'python-build-system' adds native-inputs to

PYTHONPATH of the wrapped programs. This causes unnecessary runtime

dependencies and may create problems when cross-compiling.

Given that all python dependencies are now supposed to be propagated,

perhaps we can simply remove the wrap phase? It only touches PYTHONPATH.

Marius Bakke <mbakke@fastmail.com> skribis:

We cannot simply remove it: it is meant for packages that provide

standalone programs, such as ‘certbot’. For these, it automatically

wraps bin/* and we don’t even have to have propagated inputs, which is

pretty cool.

However, you are right that native inputs shouldn’t be there. Probably

we need to add some filtering in the ‘wrap’ phase?

Ludo’.

On Tue, Dec 20, 2016 at 03:00:50PM +0100, Ludovic Courtès wrote:

While looking at the beets package definition with fresh eyes, I realize

that it will surely break once this change is made. So it will make a

good test case for this improvement :)

I'm working on fixing this bug. I have modified

guix/build/python-build-system.scm for the same. In particular, I have

added #:use-module (guix packages) because I need the functions

`package-name' and `package-transitive-target-inputs'. But, when I try

building any python-build-system package with something like

"./pre-inst-env guix build scons", I get a ("no code for module" (guix

packages)) error. What am I missing?

Full backtrace follows:

Backtrace:

In ice-9/eval.scm:

432: 19 [eval # ()]

In system/base/compile.scm:

152: 18 [compile-file "/gnu/store/38rxls2y8v8zjva7xq7zv99rv6c81bq3-module-import/guix/build/python-build-system.scm" ...]

43: 17 [call-once #<procedure ae84c0 at system/base/compile.scm:56:5 ()>]

In ice-9/boot-9.scm:

174: 16 [with-throw-handler #t ...]

In system/base/compile.scm:

59: 15 [#<procedure ae8480 at system/base/compile.scm:58:9 ()>]

155: 14 [#<procedure ae8500 at system/base/compile.scm:153:8 (port)> #<closed: file 0>]

218: 13 [read-and-compile #<input: /gnu/store/38rxls2y8v8zjva7xq7zv99rv6c81bq3-module-import/guix/build/python-build-system.scm 6> ...]

234: 12 [lp () #f #<module (#{ g152}#) ae7a20>]

182: 11 [lp (#<procedure compile-tree-il (x e opts)>) (define-module # # ...) ...]

In ice-9/boot-9.scm:

2404: 10 [save-module-excursion #<procedure c967e0 at language/scheme/compile-tree-il.scm:29:3 ()>]

In language/scheme/compile-tree-il.scm:

31: 9 [#<procedure c967e0 at language/scheme/compile-tree-il.scm:29:3 ()>]

In ice-9/psyntax.scm:

1106: 8 [expand-top-sequence ((define-module # # # ...)) () ((top)) ...]

989: 7 [scan ((define-module # # # ...)) () ((top)) ...]

279: 6 [scan ((#(syntax-object let # ...) (#) (# #) ...)) () ...]

In ice-9/eval.scm:

411: 5 [eval # ()]

In ice-9/boot-9.scm:

2954: 4 [define-module* (guix build python-build-system) #:filename ...]

2929: 3 [resolve-imports ((# # gnu:) (#) (#) (#) ...)]

2870: 2 [resolve-interface (guix packages) #:select ...]

In unknown file:

?: 1 [scm-error misc-error #f "~A ~S" ("no code for module" (guix packages)) #f]

In ice-9/boot-9.scm:

109: 0 [#<procedure ae8440 at ice-9/boot-9.scm:100:6 (thrown-k . args)> misc-error ...]

ice-9/boot-9.scm:109:20: In procedure #<procedure ae8440 at ice-9/boot-9.scm:100:6 (thrown-k . args)>:

ice-9/boot-9.scm:109:20: no code for module (guix packages)

builder for `/gnu/store/lsrxzd1hdsgpb9zhrdv02dsxihpmxmww-module-import-compiled.drv' failed with exit code 1

@ build-failed /gnu/store/lsrxzd1hdsgpb9zhrdv02dsxihpmxmww-module-import-compiled.drv - 1 builder for `/gnu/store/lsrxzd1hdsgpb9zhrdv02dsxihpmxmww-module-import-compiled.drv' failed with exit code 1

cannot build derivation `/gnu/store/pm2r6cbcfij8d7nzdxkzqx1dq880mhg8-scons-2.3.4.drv': 1 dependencies couldn't be built

guix build: error: build failed: build of `/gnu/store/pm2r6cbcfij8d7nzdxkzqx1dq880mhg8-scons-2.3.4.drv' failed

Hello,

Arun Isaac <arunisaac@systemreboot.net> skribis:

[...]

“Build-side” modules, which typically live in (guix build …), should not

depend on “host-side” modules such as (guix packages). That’s because

if we did that, we’d effectively end up importing all of Guix on the

build side, but then we’d also have to serialize data structures such as

packages to pass them from one side to the other. (I hope this makes

sense to you, but if it doesn’t maybe the intro of

https://gnu.org/s/guix/manual/html_node/G_002dExpressions.html can

shed some light.)

So in short, we cannot use ‘package-name’ and

‘package-transitive-target-inputs’ in this module.

(Time passes…)

I wasn’t sure how to fix this bug myself so I gave it a try and ended up

with the patch below, but I haven’t tested in detail. (You’ll notice

(guix build-system python) is hard to work with because it doesn’t use

gexps yet.)

How does it look?

Thanks,

Ludo’.

It makes some sense, but I'll read up more.

Ok.

I'm travelling now. I'll get back on Monday and study this patch.

I tried your patch. But native-inputs are still there in the wrapped

PYTHONPATH. I printed out the results from various steps from the wrap

phase. They are as follows:

`build-time-inputs' is

("/gnu/store/5wbi6vsi12bmpzygwhys3c0cc6sjdw40-python-beautifulsoup4-4.5.3"

"/gnu/store/kc5gl1j85y9ispjsgbg5f5c6jdy1k32s-python-flask-0.11.1"

"/gnu/store/3ib43rpca6qzwfcav8s9ivm3dikpxq87-python-mock-1.0.1"

"/gnu/store/6q4hszis7z6x44ig2zlwjwlcjy3vvfmp-python-mpd2-0.5.5"

"/gnu/store/5wx6yffx6dppd2dr91qfi1jlpqlp9f5m-python-nose-1.3.7"

"/gnu/store/kqjfgfwzm7sq43fnng1w23mjyr2xrd75-python-pathlib-1.0.1"

"/gnu/store/b9yn8bfd43pvrp2wyinf6rf2qw113r5p-python-pyxdg-0.25"

"/gnu/store/g4vfl4wy81y14fjhlpmgqh0i01cg37mb-python-pylast-1.6.0"

"/gnu/store/nim5r75b1kzawln3a1bgag2f12vdmb8x-python-rarfile-2.8"

"/gnu/store/wbaaacm1m7bzbhi3z3scwbz2p4pf2ybg-python-responses-0.5.1")

`path' is

("/gnu/store/5wbi6vsi12bmpzygwhys3c0cc6sjdw40-python-beautifulsoup4-4.5.3/lib/python3.5/site-packages"

"/gnu/store/r3f04sda4gh5vbsa7zb5x0ygf326cp75-python-flask-0.11.1/lib/python3.5/site-packages"

"/gnu/store/3ib43rpca6qzwfcav8s9ivm3dikpxq87-python-mock-1.0.1/lib/python3.5/site-packages"

"/gnu/store/5mkb6im3dijkw4f5146mjz4c49qgcklq-python-mpd2-0.5.5/lib/python3.5/site-packages"

"/gnu/store/9z450rcsza3x448ky83fkd5636ck9aqd-python-nose-1.3.7/lib/python3.5/site-packages"

"/gnu/store/kqjfgfwzm7sq43fnng1w23mjyr2xrd75-python-pathlib-1.0.1/lib/python3.5/site-packages"

"/gnu/store/b9yn8bfd43pvrp2wyinf6rf2qw113r5p-python-pyxdg-0.25/lib/python3.5/site-packages"

"/gnu/store/g4vfl4wy81y14fjhlpmgqh0i01cg37mb-python-pylast-1.6.0/lib/python3.5/site-packages"

"/gnu/store/nim5r75b1kzawln3a1bgag2f12vdmb8x-python-rarfile-2.8/lib/python3.5/site-packages"

"/gnu/store/wbaaacm1m7bzbhi3z3scwbz2p4pf2ybg-python-responses-0.5.1/lib/python3.5/site-packages"

"/gnu/store/m6ir6mj7laxcwsjrz0lvxm7q04w0anhp-python-discogs-client-2.2.1/lib/python3.5/site-packages"

"/gnu/store/v3xiyw0cj08qmpn08jaiznd8ak3nvdga-python-jellyfish-0.5.6/lib/python3.5/site-packages"

"/gnu/store/pwxvvi24950x6rgvl4f46z89sbpa2b7a-python-munkres-1.0.8/lib/python3.5/site-packages"

"/gnu/store/nws3867b5q78ri461wr3rdwgqzasvv2a-python-musicbrainzngs-0.6/lib/python3.5/site-packages"

"/gnu/store/wsw2nr30nhz29dww1hfjb6lgqn2ckrgy-python-mutagen-1.36/lib/python3.5/site-packages"

"/gnu/store/n7fh268b1g1g87s4djvm672p2xmwkd5r-python-pyyaml-3.12/lib/python3.5/site-packages"

"/gnu/store/y3ncl6ia04vqdkhnhgzb3iabcsczj0zb-python-unidecode-0.04.20/lib/python3.5/site-packages"

"/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5/site-packages"

"/gnu/store/z2mmzglkykws7hc9k2sr40fidiv62cpa-python-werkzeug-0.11.15/lib/python3.5/site-packages"

"/gnu/store/5w60y9zyjskv179sf4pmq2kzpff887dv-python-click-6.7/lib/python3.5/site-packages"

"/gnu/store/f4qqgaql0f95alxrz9r52n3m5ahjrc1s-python-jinja2-2.8/lib/python3.5/site-packages"

"/gnu/store/n0l3szawn661n2rjk6nnbajzzc02pmfj-python-itsdangerous-0.24/lib/python3.5/site-packages"

"/gnu/store/rpnfm1v7vnwxl3wpbadz2bz25s2x1bbg-python-six-1.10.0/lib/python3.5/site-packages"

"/gnu/store/64ahrdmjmmm8id5327i5kqjj8ppvi2kl-python-cookies-2.2.1/lib/python3.5/site-packages"

"/gnu/store/2iyrxbr6k1qdcrhfpv3cz1nh1znwnga5-python-requests-2.13.0/lib/python3.5/site-packages"

"/gnu/store/0c28plwcf33lnibf27h8ijig467pphiw-python-oauthlib-1.0.3/lib/python3.5/site-packages"

"/gnu/store/m48bcrmrzrcb79p6dhaq0diwr3aacfq3-python-markupsafe-0.23/lib/python3.5/site-packages")

(remove build-time-dependency? path) is

("/gnu/store/r3f04sda4gh5vbsa7zb5x0ygf326cp75-python-flask-0.11.1/lib/python3.5/site-packages"

"/gnu/store/5mkb6im3dijkw4f5146mjz4c49qgcklq-python-mpd2-0.5.5/lib/python3.5/site-packages"

"/gnu/store/9z450rcsza3x448ky83fkd5636ck9aqd-python-nose-1.3.7/lib/python3.5/site-packages"

"/gnu/store/m6ir6mj7laxcwsjrz0lvxm7q04w0anhp-python-discogs-client-2.2.1/lib/python3.5/site-packages"

"/gnu/store/v3xiyw0cj08qmpn08jaiznd8ak3nvdga-python-jellyfish-0.5.6/lib/python3.5/site-packages"

"/gnu/store/pwxvvi24950x6rgvl4f46z89sbpa2b7a-python-munkres-1.0.8/lib/python3.5/site-packages"

"/gnu/store/nws3867b5q78ri461wr3rdwgqzasvv2a-python-musicbrainzngs-0.6/lib/python3.5/site-packages"

"/gnu/store/wsw2nr30nhz29dww1hfjb6lgqn2ckrgy-python-mutagen-1.36/lib/python3.5/site-packages"

"/gnu/store/n7fh268b1g1g87s4djvm672p2xmwkd5r-python-pyyaml-3.12/lib/python3.5/site-packages"

"/gnu/store/y3ncl6ia04vqdkhnhgzb3iabcsczj0zb-python-unidecode-0.04.20/lib/python3.5/site-packages"

"/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5/site-packages"

"/gnu/store/z2mmzglkykws7hc9k2sr40fidiv62cpa-python-werkzeug-0.11.15/lib/python3.5/site-packages"

"/gnu/store/5w60y9zyjskv179sf4pmq2kzpff887dv-python-click-6.7/lib/python3.5/site-packages"

"/gnu/store/f4qqgaql0f95alxrz9r52n3m5ahjrc1s-python-jinja2-2.8/lib/python3.5/site-packages"

"/gnu/store/n0l3szawn661n2rjk6nnbajzzc02pmfj-python-itsdangerous-0.24/lib/python3.5/site-packages"

"/gnu/store/rpnfm1v7vnwxl3wpbadz2bz25s2x1bbg-python-six-1.10.0/lib/python3.5/site-packages"

"/gnu/store/64ahrdmjmmm8id5327i5kqjj8ppvi2kl-python-cookies-2.2.1/lib/python3.5/site-packages"

"/gnu/store/2iyrxbr6k1qdcrhfpv3cz1nh1znwnga5-python-requests-2.13.0/lib/python3.5/site-packages"

"/gnu/store/0c28plwcf33lnibf27h8ijig467pphiw-python-oauthlib-1.0.3/lib/python3.5/site-packages"

"/gnu/store/m48bcrmrzrcb79p6dhaq0diwr3aacfq3-python-markupsafe-0.23/lib/python3.5/site-packages")

Somehow, the paths in `build-time-inputs' are not the same as those in

`path'. Notice the different hashes. Hence `remove' does not succeed in

filtering them out.

What is this part for?

Arun Isaac <arunisaac@systemreboot.net> writes:

This is to ensure that no references to any native input are retained in

the output. (Think of it as an assertion that is checked at the end of

the build.)

~~ Ricardo

Have you been able to figure out why some of the items differ in the two lists?

Hi! Are you asking me? I haven't worked on this in a very long

time. But I'll give it another shot now and get back to you.

Hi,

The two lists are the same if `guix build' is invoked with the

`--no-grafts' flag. I don't understand grafts very well and am not sure

how that can be fixed. Any ideas?

See different output below with and without the `--no-grafts' flag.

$ guix build python-mock

$ guix build --no-grafts python-mock

Regards,

Arun

This fixes a longstanding issue. The fix is made trivial by the changes made

in #60847 adding cross-compilation support for the pyproject build system.

Additionally, it's based on #60868, which touches the wrap phase to use

wrap-script instead of wrap-program.

Maxim Cournoyer (1):

build-system/pyproject: Do not wrap native-inputs.

guix/build/pyproject-build-system.scm | 16 ++++++++++++----

1 file changed, 12 insertions(+), 4 deletions(-)

base-commit: 9da36aa1e207ff8a8fb2af8bc9442f9c935dc8db

--

2.38.1

Fixes https://issues.guix.gnu.org/25235.

Filter out native inputs from the values in GUIX_PYTHONPATH.

---

guix/build/pyproject-build-system.scm | 16 ++++++++++++----

1 file changed, 12 insertions(+), 4 deletions(-)

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Yay, great to see this fixed!

I think there’s no need to delay+thunk things since we’re using them

right away. How about:

(define native-input-directories

(match native-inputs

…))

(let ((var `("GUIX_PYTHONPATH" prefix

,(remove (lambda (x) … native-input-directories …) …))))

…)

?

(Same for ‘%guile’ above, but I don’t see it in ‘master’; maybe that’s

from another branch?)

Thanks,

Ludo’.

Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

It's a (small) optimization, that was intended to delay things until

they are really needed that was already used for %bash (now %guile) in

the wrap phase of (guix build python-build-system).

The idea is to delay the evaluation of native-input-directories or the

lookup for guile in the inputs until it's really necessary. For

example, the computations are avoided when there are no "bin" or "sbin"

sub-directories in the outputs.

Yes, the change build on both #60847 (Enable cross-compilation for the

pyproject-build-system) and #60868 (Use wrap-script instead of

wrap-program), which should be applied first. %guile appears in #60868,

and changes that make the above easy (native-inputs always appearing as

native-inputs, even for native builds) are from #60847.

--

Thanks,

Maxim

block 25235 by 60847

quit

I'm still on my first cup of coffee...

If a python input is in inputs and in native-inputs would it be included

in the wrapper or not? At first glance I'd say no. Does the search path

do deduplication? If it doesn't then it'd work to only remove the first

instance matching from native-input-dirs.

On Mon, Jan 16, 2023 at 04:29:19PM -0500, Maxim Cournoyer wrote:

--

Efraim Flashner <efraim@flashner.co.il> ????? ?????

GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351

Confidentiality cannot be guaranteed on emails sent or received unencrypted

Hi Efraim,

Efraim Flashner <efraim@flashner.co.il> writes:

It wouldn't be included with the current simple logic, which simply

remove any native input from the items of GUIX_PYTHONPATH. Good catch!

I'll send a v2 with the refined check.

--

Thanks,

Maxim

This fixes a longstanding issue. The fix is made trivial by the changes made

in #60847 adding cross-compilation support for the pyproject build system.

Additionally, it's based on #60868, which touches the wrap phase to use

wrap-script instead of wrap-program.

Changes in v2:

- Add missing copyright line

- Rework wrap phase to avoid removing inputs found in both native-inputs and

inputs

- Enclose wrap computations in an 'unless' form and streamline

Maxim Cournoyer (1):

build-system/pyproject: Do not wrap native-inputs.

guix/build/pyproject-build-system.scm | 39 ++++++++++++++++-----------

1 file changed, 24 insertions(+), 15 deletions(-)

base-commit: 9da36aa1e207ff8a8fb2af8bc9442f9c935dc8db

--

2.39.1

Fixes https://issues.guix.gnu.org/25235.

Filter out native inputs from the values in GUIX_PYTHONPATH.

---

Changes in v2:

- Add missing copyright line

- Rework wrap phase to avoid removing inputs found in both native-inputs and inputs

- Enclose wrap computations in an 'unless' form and streamline

guix/build/pyproject-build-system.scm | 39 ++++++++++++++++-----------

1 file changed, 24 insertions(+), 15 deletions(-)

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

LGTM!

I still don’t see what branch this snippet is on, what am I missing?

:-)

Thanks,

Ludo’.

This fixes a longstanding issue. The fix is made trivial by the changes made

in #60847 adding cross-compilation support for the pyproject build system.

Additionally, it's based on #60868, which touches the wrap phase to use

wrap-script instead of wrap-program.

Changes in v3:

- Rebase

Changes in v2:

- Add missing copyright line

- Rework wrap phase to avoid removing inputs found in both native-inputs and inputs

- Enclose wrap computations in an 'unless' form and streamline

Maxim Cournoyer (1):

build-system/pyproject: Do not wrap native-inputs.

guix/build/pyproject-build-system.scm | 39 ++++++++++++++++-----------

1 file changed, 24 insertions(+), 15 deletions(-)

base-commit: 490e386e10ca2b7496c3da7ed85299ed3e9c2404

--

2.39.1

Fixes https://issues.guix.gnu.org/25235.

Filter out native inputs from the values in GUIX_PYTHONPATH.

---

Changes in v3:

- Rebase

Changes in v2:

- Add missing copyright line

- Rework wrap phase to avoid removing inputs found in both native-inputs and inputs

- Enclose wrap computations in an 'unless' form and streamline

guix/build/pyproject-build-system.scm | 39 ++++++++++++++++-----------

1 file changed, 24 insertions(+), 15 deletions(-)

Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

It's based on https://issues.guix.gnu.org/60868,itself based on

https://issues.guix.gnu.org/60847. I've tried to express this

relationship by marking the dependent changes blocked by the submissions

they are based on.

I've just rebased the 3 submissions and added you in CC; hopefully that

helps!

--

Thanks,

Maxim

Hi Lars, Ricardo,

I'm advancing with my patch series, which I can submit soon. I was

curious about why this 25235 patch isn't in python-team branch yet since

it's also a very welcome change to the pyproject-build-system.

Cheers,

Nicolas

-------------------- Start of forwarded message --------------------

native-inputs.

To: 25235@debbugs.gnu.org

Cc: arunisaac@systemreboot.net, Maxim Cournoyer <maxim.cournoyer@gmail.com>,

ludo@gnu.org, Lars-Dominik Braun <lars@6xq.net>, efraim@flashner.co.il,

rekado@elephly.net, jgart <jgart@dismail.de>, marius@gnu.org

Fixes https://issues.guix.gnu.org/25235.

Filter out native inputs from the values in GUIX_PYTHONPATH.

---

Changes in v3:

- Rebase

Changes in v2:

- Add missing copyright line

- Rework wrap phase to avoid removing inputs found in both native-inputs and inputs

- Enclose wrap computations in an 'unless' form and streamline

guix/build/pyproject-build-system.scm | 39 ++++++++++++++++-----------

1 file changed, 24 insertions(+), 15 deletions(-)

Hi,

I believe this change is based on #60847, which Ludo objected to

(https://issues.guix.gnu.org/60847#19-lineno32).Thus it cannot be merged.

Lars