‘binutils-mesboot0’ includes non-zero timestamps in ar archives

  • Open
  • quality assurance status badge
Details
3 participants
  • Jan Nieuwenhuizen
  • Ludovic Courtès
  • Maxim Cournoyer
Owner
unassigned
Submitted by
Ludovic Courtès
Severity
important
Merged with
L
L
Ludovic Courtès wrote on 18 Jan 2021 18:29
(address . bug-guix@gnu.org)
87im7ujgqz.fsf@inria.fr
Hi!

On #bootstrappable, mid-kid reported that ‘binutils-mesboot0’ in
commencement.scm lacks ‘--enable-deterministic-archives’. So I checked
if this had an effect by running:

guix build -e '(@@ (gnu packages commencement) gcc-core-mesboot0)' \
--check -K

and yes, it does:

Toggle snippet (7 lines)
$ diff -ru /gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3{,-check}
Binary files /gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3/lib/gcc-lib/i686-unknown-linux-gnu/2.95.3/libc.a and /gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3-check/lib/gcc-lib/i686-unknown-linux-gnu/2.95.3/libc.a differ
Binary files /gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3/lib/gcc-lib/i686-unknown-linux-gnu/2.95.3/libgcc.a and /gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3-check/lib/gcc-lib/i686-unknown-linux-gnu/2.95.3/libgcc.a differ
Binary files /gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3/lib/libgcc2.a and /gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3-check/lib/libgcc2.a differ
Binary files /gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3/lib/libiberty.a and /gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3-check/lib/libiberty.a differ

Apparently Binutils 2.14 didn’t have ‘--enable-deterministic-archives’
so we’ll have to patch it.

There are a few other Binutils variants in commencement.scm that we’ll
have to check.

Ludo’.
L
L
Ludovic Courtès wrote on 20 Jan 2021 09:45
control message for bug #45962
(address . control@debbugs.gnu.org)
87pn20f13k.fsf@gnu.org
severity 45962 important
quit
L
L
Ludovic Courtès wrote on 3 Mar 2021 14:48
Re: bug#45962: ‘binutils-mesboot0’ includes non-zero timestamps in ar archives
(address . 45962@debbugs.gnu.org)
87lfb4s5z7.fsf@gnu.org
Hi,

Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (4 lines)
> On #bootstrappable, mid-kid reported that ‘binutils-mesboot0’ in
> commencement.scm lacks ‘--enable-deterministic-archives’. So I checked
> if this had an effect by running:

[...]

Toggle quote (3 lines)
> Apparently Binutils 2.14 didn’t have ‘--enable-deterministic-archives’
> so we’ll have to patch it.

Sikonas on #bootstrappable provided a patch for that (thanks!) so I went
ahead and gave it a try on ‘core-updates’ (Guix patch attached).

The binutils source gets patched and repacked, but then decompressing it
fails:

Toggle snippet (37 lines)
building /gnu/store/fwz150xjaqbh8n02z6gsmpm9w8lxckak-binutils-mesboot0-2.14.drv...
starting phase `set-SOURCE-DATE-EPOCH'
phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds
starting phase `set-paths'
environment variable `PATH' set to `/gnu/store/70w6gnl3vzv2rsjhyjg0wsxl3pbnpjb5-bash-mesboot0-2.05b/bin:/gnu/store/2wkjilvkdyb2wrb20ba82dhdpnx2732n-bzip2-mesboot-1.0.8/bin:/gnu/store/4nxicfzj8a390f4scxcgglhdqgbyhlkw-gzip-mesboot-1.2.4/bin:/gnu/store/6fgjdcl2qrrs7fvdznkx1gcbw6wjfrn0-patch-mesboot-2.5.9/bin:/gnu/store/4lpagdav2gm022v1fgzcs8323xrggz4b-sed-mesboot0-1.18/bin:/gnu/store/y70rava2vqf1nilr8smg54qqxdclvrz6-gash-utils-boot-0.1.0/bin:/gnu/store/y9fy4r99q6pqisrw81fn92gk8mzbyzn1-tcc-boot-0.9.27/bin:/gnu/store/krnwfrki71dj6zicl2qwv6bdyqvsxgcg-make-mesboot0-3.80/bin:/gnu/store/xxnxdnjdlav4s8v3lfhg7x7amrqcrv57-gash-boot-0.2.0/bin:/gnu/store/aglbpf6bihv35pwpvdy6chxv318hsafq-bootar-1a/bin:/gnu/store/lgi9x15a0w35mcpd7g1kb9274r6wy4pv-guile-bootstrap-2.0/bin'
environment variable `BASH_LOADABLES_PATH' unset
environment variable `C_INCLUDE_PATH' set to `/gnu/store/2wkjilvkdyb2wrb20ba82dhdpnx2732n-bzip2-mesboot-1.0.8/include:/gnu/store/y9fy4r99q6pqisrw81fn92gk8mzbyzn1-tcc-boot-0.9.27/include'
environment variable `LIBRARY_PATH' set to `/gnu/store/2wkjilvkdyb2wrb20ba82dhdpnx2732n-bzip2-mesboot-1.0.8/lib:/gnu/store/y70rava2vqf1nilr8smg54qqxdclvrz6-gash-utils-boot-0.1.0/lib:/gnu/store/y9fy4r99q6pqisrw81fn92gk8mzbyzn1-tcc-boot-0.9.27/lib:/gnu/store/xxnxdnjdlav4s8v3lfhg7x7amrqcrv57-gash-boot-0.2.0/lib:/gnu/store/aglbpf6bihv35pwpvdy6chxv318hsafq-bootar-1a/lib:/gnu/store/lgi9x15a0w35mcpd7g1kb9274r6wy4pv-guile-bootstrap-2.0/lib'
phase `set-paths' succeeded after 0.0 seconds
starting phase `install-locale'
warning: failed to install 'en_US.utf8' locale: Invalid argument
phase `install-locale' succeeded after 0.0 seconds
starting phase `unpack'
xz: cannot decompress from stdin
Backtrace:
In ice-9/boot-9.scm:
157: 5 [catch #t #<catch-closure c928e0> ...]
In unknown file:
?: 4 [apply-smob/1 #<catch-closure c928e0>]
In ice-9/boot-9.scm:
63: 3 [call-with-prompt prompt0 ...]
In ice-9/eval.scm:
432: 2 [eval # #]
In gash/guix-utils.scm:
138: 1 [call-with-decompressed-port xz ...]
In unknown file:
?: 0 [scm-error misc-error #f "~A ~S" ("decompressed-port failure" (7)) #f]

ERROR: In procedure scm-error:
ERROR: decompressed-port failure (7)
error: in phase 'unpack': uncaught exception:
srfi-34 #<condition &invoke-error [program: "tar" arguments: ("xvf" "/gnu/store/ywf5j03423yiawix3z21xa14hyyvd83z-binutils-2.14.tar.xz") exit-status: 1 term-signal: #f stop-signal: #f] 1215400>
phase `unpack' failed after 0.3 seconds
command "tar" "xvf" "/gnu/store/ywf5j03423yiawix3z21xa14hyyvd83z-binutils-2.14.tar.xz" failed with status 1
builder for `/gnu/store/fwz150xjaqbh8n02z6gsmpm9w8lxckak-binutils-mesboot0-2.14.drv' failed with exit code 1

Maxime, does that ring a bell? Could it be that this bootstrap ‘xz’ is
less capable, or could it be a Gash-Utils bug?

Thanks,
Ludo’.
Toggle diff (99 lines)
diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index 890d57941f..5c523ae7ad 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -997,13 +997,15 @@ $MES -e '(mescc)' module/mescc.scm -- \"$@\"
(inherit binutils)
(name "binutils-mesboot0")
(version "2.14")
- (source (origin
- (method url-fetch)
- (uri (string-append "mirror://gnu/binutils/binutils-"
- version ".tar.gz"))
- (sha256
- (base32
- "1w8xp7k44bkijr974x9918i4p1sw4g2fcd5mxvspkjpg38m214ds"))))
+ (source (bootstrap-origin
+ (origin
+ (method url-fetch)
+ (uri (string-append "mirror://gnu/binutils/binutils-"
+ version ".tar.gz"))
+ (sha256
+ (base32
+ "1w8xp7k44bkijr974x9918i4p1sw4g2fcd5mxvspkjpg38m214ds"))
+ (patches (search-patches "binutils-2.14-deterministic-ar.patch")))))
(inputs '())
(propagated-inputs '())
(native-inputs (%boot-tcc-inputs))
diff --git a/gnu/packages/patches/binutils-2.14-deterministic-ar.patch b/gnu/packages/patches/binutils-2.14-deterministic-ar.patch
new file mode 100644
index 0000000000..7f9310e994
--- /dev/null
+++ b/gnu/packages/patches/binutils-2.14-deterministic-ar.patch
@@ -0,0 +1,66 @@
+Old binutils do not have support for creating deterministic archives.
+Backported from upstream commit 36e4dce69dd23bea9ea2258dea35f034b6d6351c
+
+Patch by Chris Demetriou <cgd@google.com> (2009), adapted by
+Andrius Štikonas <andrius@stikonas.eu> (2021).
+
+--- a/bfd/archive.c 2021-03-01 00:05:54.888301655 +0000
++++ b/bfd/archive.c 2021-03-02 21:53:51.001617689 +0000
+@@ -1396,10 +1396,6 @@
+ {
+ /* Assume we just "made" the member, and fake it. */
+ struct bfd_in_memory *bim = (struct bfd_in_memory *) member->iostream;
+- time (&status.st_mtime);
+- status.st_uid = getuid ();
+- status.st_gid = getgid ();
+- status.st_mode = 0644;
+ status.st_size = bim->size;
+ }
+ else if (stat (filename, &status) != 0)
+@@ -1408,6 +1404,11 @@
+ return NULL;
+ }
+
++ status.st_mtime = 0;
++ status.st_uid = 0;
++ status.st_gid = 0;
++ status.st_mode = 0644;
++
+ amt = sizeof (struct ar_hdr) + sizeof (struct areltdata);
+ ared = (struct areltdata *) bfd_zalloc (abfd, amt);
+ if (ared == NULL)
+@@ -2003,13 +2004,11 @@
+ stat (arch->filename, &statbuf);
+ memset ((char *) (&hdr), 0, sizeof (struct ar_hdr));
+ sprintf (hdr.ar_name, RANLIBMAG);
+- /* Remember the timestamp, to keep it holy. But fudge it a little. */
+- bfd_ardata (arch)->armap_timestamp = statbuf.st_mtime + ARMAP_TIME_OFFSET;
+ bfd_ardata (arch)->armap_datepos = (SARMAG
+ + offsetof (struct ar_hdr, ar_date[0]));
+- sprintf (hdr.ar_date, "%ld", bfd_ardata (arch)->armap_timestamp);
+- sprintf (hdr.ar_uid, "%ld", (long) getuid ());
+- sprintf (hdr.ar_gid, "%ld", (long) getgid ());
++ sprintf (hdr.ar_date, "%ld", 0);
++ sprintf (hdr.ar_uid, "%ld", 0);
++ sprintf (hdr.ar_gid, "%ld", 0);
+ sprintf (hdr.ar_size, "%-10d", (int) mapsize);
+ strncpy (hdr.ar_fmag, ARFMAG, 2);
+ for (i = 0; i < sizeof (struct ar_hdr); i++)
+@@ -2082,6 +2081,8 @@
+ struct ar_hdr hdr;
+ unsigned int i;
+
++ return TRUE;
++
+ /* Flush writes, get last-write timestamp from file, and compare it
+ to the timestamp IN the file. */
+ bfd_flush (arch);
+@@ -2169,7 +2170,7 @@
+ memset ((char *) (&hdr), 0, sizeof (struct ar_hdr));
+ hdr.ar_name[0] = '/';
+ sprintf (hdr.ar_size, "%-10d", (int) mapsize);
+- sprintf (hdr.ar_date, "%ld", (long) time (NULL));
++ sprintf (hdr.ar_date, "%ld", 0);
+ /* This, at least, is what Intel coff sets the values to. */
+ sprintf ((hdr.ar_uid), "%d", 0);
+ sprintf ((hdr.ar_gid), "%d", 0);
J
J
Jan Nieuwenhuizen wrote on 3 Mar 2021 18:54
(name . Ludovic Courtès)(address . ludo@gnu.org)
8735xc86ne.fsf@gnu.org
Ludovic Courtès writes:

Hello!

Toggle quote (14 lines)
> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> On #bootstrappable, mid-kid reported that ‘binutils-mesboot0’ in
>> commencement.scm lacks ‘--enable-deterministic-archives’. So I checked
>> if this had an effect by running:

> [...]
>
>> Apparently Binutils 2.14 didn’t have ‘--enable-deterministic-archives’
>> so we’ll have to patch it.
>
> Sikonas on #bootstrappable provided a patch for that (thanks!) so I went
> ahead and gave it a try on ‘core-updates’ (Guix patch attached).

Great!

Toggle quote (3 lines)
> The binutils source gets patched and repacked, but then decompressing it
> fails:

[..]

Toggle quote (3 lines)
> Maxime, does that ring a bell? Could it be that this bootstrap ‘xz’ is
> less capable, or could it be a Gash-Utils bug?

Currently, we avoid using non-bootstrapped binaries in the bootstrap,
that includes 'xz'; earlier in the bootstrap that includes also 'patch'.

See also gcc-core-mesboot0: it applies the patch in a manual phase. So
I'm not sure if we want to start depending on 'xz' an this stage?

Greetings,
Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
M
M
Maxim Cournoyer wrote on 4 Mar 2021 17:14
(name . Ludovic Courtès)(address . ludo@gnu.org)
87k0qmq4ka.fsf@gmail.com
Hi,

Ludovic Courtès <ludo@gnu.org> writes:

[...]

Toggle quote (11 lines)
> ERROR: In procedure scm-error:
> ERROR: decompressed-port failure (7)
> error: in phase 'unpack': uncaught exception:
> srfi-34 #<condition &invoke-error [program: "tar" arguments: ("xvf" "/gnu/store/ywf5j03423yiawix3z21xa14hyyvd83z-binutils-2.14.tar.xz") exit-status: 1 term-signal: #f stop-signal: #f] 1215400>
> phase `unpack' failed after 0.3 seconds
> command "tar" "xvf" "/gnu/store/ywf5j03423yiawix3z21xa14hyyvd83z-binutils-2.14.tar.xz" failed with status 1
> builder for `/gnu/store/fwz150xjaqbh8n02z6gsmpm9w8lxckak-binutils-mesboot0-2.14.drv' failed with exit code 1
>
> Maxime, does that ring a bell? Could it be that this bootstrap ‘xz’ is
> less capable, or could it be a Gash-Utils bug?

From my IRC logs:

2020-09-20 22:32:01 apteryx janneke: haha! the xz-bootstrap supports
--memlimit with % after all, my mistake was really silly... forgetting a
space between the args passed as XZ_DEFAULTS

I recall a similar error I had hit when working on adding multi-core
compression support to xz, but it ended up being just a mistake on my
part; the xz-bootstrap supported the required arguments just fine after
all.

HTH,

Maxim
M
M
Maxim Cournoyer wrote on 4 Mar 2021 17:39
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)
87ft1aq3fb.fsf@gmail.com
Jan Nieuwenhuizen <janneke@gnu.org> writes:

Toggle quote (34 lines)
> Ludovic Courtès writes:
>
> Hello!
>
>> Ludovic Courtès <ludo@gnu.org> skribis:
>>
>>> On #bootstrappable, mid-kid reported that ‘binutils-mesboot0’ in
>>> commencement.scm lacks ‘--enable-deterministic-archives’. So I checked
>>> if this had an effect by running:
>
>> [...]
>>
>>> Apparently Binutils 2.14 didn’t have ‘--enable-deterministic-archives’
>>> so we’ll have to patch it.
>>
>> Sikonas on #bootstrappable provided a patch for that (thanks!) so I went
>> ahead and gave it a try on ‘core-updates’ (Guix patch attached).
>
> Great!
>
>> The binutils source gets patched and repacked, but then decompressing it
>> fails:
>
> [..]
>
>> Maxime, does that ring a bell? Could it be that this bootstrap ‘xz’ is
>> less capable, or could it be a Gash-Utils bug?
>
> Currently, we avoid using non-bootstrapped binaries in the bootstrap,
> that includes 'xz'; earlier in the bootstrap that includes also 'patch'.
>
> See also gcc-core-mesboot0: it applies the patch in a manual phase. So
> I'm not sure if we want to start depending on 'xz' an this stage?

I see; so what Ludovic got surprised by is the fact that when adding
patches or a snippet to an origin it gets repacked as an xz tarball.
That's nothing new (it's how it is on the master branch too); but it can
indeed be surprising.

Maxim
L
L
Ludovic Courtès wrote on 8 Mar 2021 15:13
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)
87ft15d98v.fsf@gnu.org
Howdy Janneke!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

Toggle quote (6 lines)
> Currently, we avoid using non-bootstrapped binaries in the bootstrap,
> that includes 'xz'; earlier in the bootstrap that includes also 'patch'.
>
> See also gcc-core-mesboot0: it applies the patch in a manual phase. So
> I'm not sure if we want to start depending on 'xz' an this stage?

Probably not, good point!

I can think of two possibilities, then: (1) apply the patch in a phase
rather than via the ‘patches’ field, and (2) arrange so that
‘patch-and-repack’ does not compress the patched code or compresses it
with the bootstrap gzip.

My understanding is that #2 may be doable now thanks to Maxim’s recent
changes. I’ll take a look!

Thanks,
Ludo’.
J
J
Jan Nieuwenhuizen wrote on 10 Mar 2021 06:52
(name . Ludovic Courtès)(address . ludo@gnu.org)
87blbrsggs.fsf@gnu.org
Ludovic Courtès writes:

Hey Ludo!

Toggle quote (7 lines)
> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>
> I can think of two possibilities, then: (1) apply the patch in a phase
> rather than via the ‘patches’ field, and (2) arrange so that
> ‘patch-and-repack’ does not compress the patched code or compresses it
> with the bootstrap gzip.

Oh, that would be nice: we could remove all these clumsy manual patching
stages. Also, we may be able to remove XZ from the bootstrap chain, if
no XZ-repackaging happens and only build a final version.

Toggle quote (3 lines)
> My understanding is that #2 may be doable now thanks to Maxim’s recent
> changes. I’ll take a look!

Great!
Greetings,
Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
L
L
Ludovic Courtès wrote on 20 Sep 2021 09:08
control message for bug #45962
(address . control@debbugs.gnu.org)
87o88nrafn.fsf@gnu.org
merge 45962 50031
quit
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 45962@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 45962
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch