[PATCH] guix: Compress and decompress xz archives in parallel.

  • Done
  • quality assurance status badge
Details
6 participants
  • Efraim Flashner
  • Kei Kebreau
  • Leo Famulari
  • Ludovic Courtès
  • Marius Bakke
  • Glenn Morris
Owner
unassigned
Submitted by
Efraim Flashner
Severity
normal
E
E
Efraim Flashner wrote on 23 Mar 2017 10:47
E1cr0d6-0004an-8g@eggs.gnu.org
* guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
* guix/utils.scm (decompressed-port, compressed-port,
compressed-output-port): Same.
---
guix/scripts/pack.scm | 3 ++-
guix/utils.scm | 7 ++++---
2 files changed, 6 insertions(+), 4 deletions(-)

Toggle diff (61 lines)
diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
index 626c592e1..9e91bc22a 100644
--- a/guix/scripts/pack.scm
+++ b/guix/scripts/pack.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2015, 2017 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2017 Efraim Flashner <efraim@flashner.co.il>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -59,7 +60,7 @@
(compressor "lzip" "lz"
#~(#+(file-append lzip "/bin/lzip") "-9"))
(compressor "xz" "xz"
- #~(#+(file-append xz "/bin/xz") "-e"))
+ #~(#+(file-append xz "/bin/xz") "-e -T0"))
(compressor "bzip2" "bz2"
#~(#+(file-append bzip2 "/bin/bzip2") "-9"))))
diff --git a/guix/utils.scm b/guix/utils.scm
index bc90686de..fb962df8b 100644
--- a/guix/utils.scm
+++ b/guix/utils.scm
@@ -5,6 +5,7 @@
;;; Copyright © 2014 Ian Denhardt <ian@zenhack.net>
;;; Copyright © 2016 Mathieu Lirzin <mthl@gnu.org>
;;; Copyright © 2015 David Thompson <davet@gnu.org>
+;;; Copyright © 2017 Efraim Flashner <efraim@flashner.co.il>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -154,7 +155,7 @@ a symbol such as 'xz."
(match compression
((or #f 'none) (values input '()))
('bzip2 (filtered-port `(,%bzip2 "-dc") input))
- ('xz (filtered-port `(,%xz "-dc") input))
+ ('xz (filtered-port `(,%xz "-dc -T0") input))
('gzip (filtered-port `(,%gzip "-dc") input))
(else (error "unsupported compression scheme" compression))))
@@ -164,7 +165,7 @@ a symbol such as 'xz."
(match compression
((or #f 'none) (values input '()))
('bzip2 (filtered-port `(,%bzip2 "-c") input))
- ('xz (filtered-port `(,%xz "-c") input))
+ ('xz (filtered-port `(,%xz "-c -T0") input))
('gzip (filtered-port `(,%gzip "-c") input))
(else (error "unsupported compression scheme" compression))))
@@ -221,7 +222,7 @@ program--e.g., '(\"--fast\")."
(match compression
((or #f 'none) (values output '()))
('bzip2 (filtered-output-port `(,%bzip2 "-c" ,@options) output))
- ('xz (filtered-output-port `(,%xz "-c" ,@options) output))
+ ('xz (filtered-output-port `(,%xz "-c -T0" ,@options) output))
('gzip (filtered-output-port `(,%gzip "-c" ,@options) output))
(else (error "unsupported compression scheme" compression))))
--
2.12.1
G
G
Glenn Morris wrote on 24 Mar 2017 20:33
control message for bug 26225
(address . control@debbugs.gnu.org)
E1crUxR-00085h-5y@fencepost.gnu.org
reassign 26225 guix-patches
G
G
Glenn Morris wrote on 24 Mar 2017 20:36
Re: bug#26225: [PATCH] guix: Compress and decompress xz archives in parallel.
(address . 26225@debbugs.gnu.org)(name . Efraim Flashner)(address . efraim@flashner.co.il)
5vvaqyii1c.fsf@fencepost.gnu.org
Reassigned to guix-patches.
(You must include a Package: line if reporting by bcc(?) ).

Efraim Flashner wrote:

Toggle quote (66 lines)
> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
> * guix/utils.scm (decompressed-port, compressed-port,
> compressed-output-port): Same.
> ---
> guix/scripts/pack.scm | 3 ++-
> guix/utils.scm | 7 ++++---
> 2 files changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
> index 626c592e1..9e91bc22a 100644
> --- a/guix/scripts/pack.scm
> +++ b/guix/scripts/pack.scm
> @@ -1,5 +1,6 @@
> ;;; GNU Guix --- Functional package management for GNU
> ;;; Copyright © 2015, 2017 Ludovic Courtès <ludo@gnu.org>
> +;;; Copyright © 2017 Efraim Flashner <efraim@flashner.co.il>
> ;;;
> ;;; This file is part of GNU Guix.
> ;;;
> @@ -59,7 +60,7 @@
> (compressor "lzip" "lz"
> #~(#+(file-append lzip "/bin/lzip") "-9"))
> (compressor "xz" "xz"
> - #~(#+(file-append xz "/bin/xz") "-e"))
> + #~(#+(file-append xz "/bin/xz") "-e -T0"))
> (compressor "bzip2" "bz2"
> #~(#+(file-append bzip2 "/bin/bzip2") "-9"))))
>
> diff --git a/guix/utils.scm b/guix/utils.scm
> index bc90686de..fb962df8b 100644
> --- a/guix/utils.scm
> +++ b/guix/utils.scm
> @@ -5,6 +5,7 @@
> ;;; Copyright © 2014 Ian Denhardt <ian@zenhack.net>
> ;;; Copyright © 2016 Mathieu Lirzin <mthl@gnu.org>
> ;;; Copyright © 2015 David Thompson <davet@gnu.org>
> +;;; Copyright © 2017 Efraim Flashner <efraim@flashner.co.il>
> ;;;
> ;;; This file is part of GNU Guix.
> ;;;
> @@ -154,7 +155,7 @@ a symbol such as 'xz."
> (match compression
> ((or #f 'none) (values input '()))
> ('bzip2 (filtered-port `(,%bzip2 "-dc") input))
> - ('xz (filtered-port `(,%xz "-dc") input))
> + ('xz (filtered-port `(,%xz "-dc -T0") input))
> ('gzip (filtered-port `(,%gzip "-dc") input))
> (else (error "unsupported compression scheme" compression))))
>
> @@ -164,7 +165,7 @@ a symbol such as 'xz."
> (match compression
> ((or #f 'none) (values input '()))
> ('bzip2 (filtered-port `(,%bzip2 "-c") input))
> - ('xz (filtered-port `(,%xz "-c") input))
> + ('xz (filtered-port `(,%xz "-c -T0") input))
> ('gzip (filtered-port `(,%gzip "-c") input))
> (else (error "unsupported compression scheme" compression))))
>
> @@ -221,7 +222,7 @@ program--e.g., '(\"--fast\")."
> (match compression
> ((or #f 'none) (values output '()))
> ('bzip2 (filtered-output-port `(,%bzip2 "-c" ,@options) output))
> - ('xz (filtered-output-port `(,%xz "-c" ,@options) output))
> + ('xz (filtered-output-port `(,%xz "-c -T0" ,@options) output))
> ('gzip (filtered-output-port `(,%gzip "-c" ,@options) output))
> (else (error "unsupported compression scheme" compression))))
L
L
Leo Famulari wrote on 31 Mar 2017 00:22
20170330222235.GB10357@jasmine
Efraim Flashner wrote:
Toggle quote (15 lines)
> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
> * guix/utils.scm (decompressed-port, compressed-port,
> compressed-output-port): Same.

> --- a/guix/utils.scm
> +++ b/guix/utils.scm
> @@ -154,7 +155,7 @@ a symbol such as 'xz."
> (match compression
> ((or #f 'none) (values input '()))
> ('bzip2 (filtered-port `(,%bzip2 "-dc") input))
> - ('xz (filtered-port `(,%xz "-dc") input))
> + ('xz (filtered-port `(,%xz "-dc -T0") input))
> ('gzip (filtered-port `(,%gzip "-dc") input))
> (else (error "unsupported compression scheme" compression))))

Parallel decompression isn't implemented yet, but the extra '-T0' option
doesn't seem to break anything.

The changes LGTM.

I sent a companion patch to https://bugs.gnu.org/26316 that enables
threaded compression when building source tarballs.
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAljdhSsACgkQJkb6MLrK
fwhkRBAAz6Rd76JFEETSgiWnlatIAoOraOBSk8jXNNgMixc4jCtYaQEIS3M7cOA/
QRz/VTlDZgONfhSxEPOp0vHAMrxhwdGSMIW9vqstb+mnAxzS4zIHW/nxrUQ5mkLi
IgzxQqQcT3+9tPxjqcEYC74qRo2rHFGbY3KM9Ksb+G5C7LsbC2wGAG+pik3a7OtH
mvOa2GVAE09vNldeW3P8spt595oDqye+Er3jjY0BZEHkpxK6ZG1qVHL0SP9QTkeT
pkEFIJSjX9ofeLt1atrd+b+oQ5/37K7+WxYzyJEskHLjl5ZziknyQk8K9nwzDz+A
bWWn1qW6BhddnXMY/7eRFACNJCU99w6ng03K6YlLumZxizpkLN+ISGy/7rVjcWC+
Yr9lUguLRoO0yXo0SXNBjFvedxHUKMarqcbwKLMCf9jvHOwio5E/jQifvLJvJXei
bHyhmd3bkJm7hY0YWYfOGkLh/VjbDmfHShzLaDcb6dIsTsaFTfpTa/ll+5rqREpX
Bpkhdc+LXz/J6cUAU1+Ry+hR7Lu2XlnSKcYWnjPIagyttg1BqxHPXyGji7Z3Fmrn
pwp2wxSagRdVGOOCbA0SxdPZ/TbG92yAMjhU8y1TyFzPBgMJQqUrnvOynpgZlHFp
820uI1rWMFcqoZrkvPCEyWPdX4S+YGDslqv61Xwdw45DTf7ihGQ=
=ViJO
-----END PGP SIGNATURE-----


K
K
Kei Kebreau wrote on 31 Mar 2017 19:56
(name . Leo Famulari)(address . leo@famulari.name)
87h9295nze.fsf@openmailbox.org
Leo Famulari <leo@famulari.name> writes:

Toggle quote (24 lines)
> Efraim Flashner wrote:
>> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
>> * guix/utils.scm (decompressed-port, compressed-port,
>> compressed-output-port): Same.
>
>> --- a/guix/utils.scm
>> +++ b/guix/utils.scm
>> @@ -154,7 +155,7 @@ a symbol such as 'xz."
>> (match compression
>> ((or #f 'none) (values input '()))
>> ('bzip2 (filtered-port `(,%bzip2 "-dc") input))
>> - ('xz (filtered-port `(,%xz "-dc") input))
>> + ('xz (filtered-port `(,%xz "-dc -T0") input))
>> ('gzip (filtered-port `(,%gzip "-dc") input))
>> (else (error "unsupported compression scheme" compression))))
>
> Parallel decompression isn't implemented yet, but the extra '-T0' option
> doesn't seem to break anything.
>
> The changes LGTM.
>
> I sent a companion patch to <https://bugs.gnu.org/26316> that enables
> threaded compression when building source tarballs.

LGTM as well.
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEg7ZwOtzKO2lLzi2m5qXuPBlGeg0FAljemFUACgkQ5qXuPBlG
eg15DQ//WCd4qGdVJRgi+mo8Z2Oq9XF1xR7P4ud4J5kRGFqZbWzOysFldBC6a4AN
I9f2uLX0lh1+uj+WMUSyZxHO4pwiB1PC2K9EK4Clx5WwZLC9gqxueWhIQ41XI6Rq
qgHHlNxfZhaTeI6qglpLtxdZUZqF1RDrjvBZXSVd4oYty/EhRvaJbKYhkUN+CaeV
PVwcg2Wh5K7SeYC1IuMBTp6fppue/QmfLUhgr9F6IzYxXN9UoSImxXz1whQlj1f4
/dV4TzbGGPIbczWpVuTE1jxa20yFoOm3YnkkAU9F8111fC3G8UPXjWWT9mQPYf2/
Xrup/5dyX5FtORU1492Y0imZWTgvAoIoTDkjpe68r62ztzuuR+pv6pZwKdnXZ7Do
jHow9nF6bINquK7x/FYcRO2buIAQrj18KgohrsMa4/WbCKNqr7oUz8MoLwpFOH/S
SVTfDZlP52ay8koSnhe2e+TXH6UyIkJ9MPejkPiBxS1s4yP90BnF9w9eexXLYDty
eimR9uq/E+e11My58eFCT8mIj+hNs4r6DQEwpBGXSPtyBSWwFSuCXdxIBY7OxYJ8
iQHcALnelvdCv99Hk+k0mEVCMyT4ChHrWtkajIZpsUxkEUOPApUSzdEp7jmAU/BY
OqPOtehOZsR2tVY629VNsBFWD5YCedzh7CbEfx4B0uSg2dyrimQ=
=iwIp
-----END PGP SIGNATURE-----

L
L
Leo Famulari wrote on 1 Apr 2017 01:01
(name . Kei Kebreau)(address . kei@openmailbox.org)
20170331230158.GA9642@jasmine
On Fri, Mar 31, 2017 at 01:56:37PM -0400, Kei Kebreau wrote:
Toggle quote (28 lines)
> Leo Famulari <leo@famulari.name> writes:
>
> > Efraim Flashner wrote:
> >> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
> >> * guix/utils.scm (decompressed-port, compressed-port,
> >> compressed-output-port): Same.
> >
> >> --- a/guix/utils.scm
> >> +++ b/guix/utils.scm
> >> @@ -154,7 +155,7 @@ a symbol such as 'xz."
> >> (match compression
> >> ((or #f 'none) (values input '()))
> >> ('bzip2 (filtered-port `(,%bzip2 "-dc") input))
> >> - ('xz (filtered-port `(,%xz "-dc") input))
> >> + ('xz (filtered-port `(,%xz "-dc -T0") input))
> >> ('gzip (filtered-port `(,%gzip "-dc") input))
> >> (else (error "unsupported compression scheme" compression))))
> >
> > Parallel decompression isn't implemented yet, but the extra '-T0' option
> > doesn't seem to break anything.
> >
> > The changes LGTM.
> >
> > I sent a companion patch to <https://bugs.gnu.org/26316> that enables
> > threaded compression when building source tarballs.
>
> LGTM as well.

AFAICT, this doesn't cause mass rebuilds.
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAlje3+AACgkQJkb6MLrK
fwif8g/+Nxyq4Jp56kYniociZcomjXKfPz2tCvd06pNtQObyuZlIQqNCU66bQIoD
A09A3aQPbnc88eTqv3bE1a6KJH7h9suPzvls2/j5ddah5MHDZABh+YT3yCNQW7no
xseINtlkbYxGXo5aCcJ6cEjLmzF60FmgN93/wpNJOXXMXAtyF4IWccgB/lovRQNe
Zvm6RLxY95Xfj5kiw0SOsNTRnxOtwY2oRvNrcKYKSeaBoaSPOaD8RQNS0Y1tBH8E
dSvb52NenO8LqqQScY3QDvj2LpoLs6KsGUE7YKGJNQj4lU/HAJWT9yxyCjwHO0kr
ZEAmYfX7CeO4y54lo9h3uuqArJIhq0/glotqxqikWmWKpRvK9clpthPRD07bWz5u
0cNmbJw7+a/ykqDo6PmPtWItN+wn0DvFufIUBJCJhd0jSat9VfUVeJoTCqfP1par
5k5s4P46BFZS9npPDQPBcVqVaEI8HX+VNAGpfYByE9Q4/mieKGVFFMsgD9E7EiMT
ojtjYPcSBnQgogY6qjZKSQhBj4Q3GdQVUUKjPFTZhyncS/mDaPbTnoXfUTx3KzmA
+kSEjKEXZkTwr7pvTkBWydvSToVOq36ovX997rlbFL6lDh2ONBY7BLrqtNicQ0Jc
2qTXtftNLOlUqwCmbiFyFXiIZu4zUSFn71dp0EQbi3/anyDSruU=
=oLYK
-----END PGP SIGNATURE-----


M
M
Marius Bakke wrote on 1 Apr 2017 08:52
Re: bug#26225: [PATCH] guix: Compress and decompress xz archives in parallel.
878tnkfwlv.fsf@kirby.i-did-not-set--mail-host-address--so-tickle-me
Leo Famulari <leo@famulari.name> writes:

Toggle quote (24 lines)
> Efraim Flashner wrote:
>> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
>> * guix/utils.scm (decompressed-port, compressed-port,
>> compressed-output-port): Same.
>
>> --- a/guix/utils.scm
>> +++ b/guix/utils.scm
>> @@ -154,7 +155,7 @@ a symbol such as 'xz."
>> (match compression
>> ((or #f 'none) (values input '()))
>> ('bzip2 (filtered-port `(,%bzip2 "-dc") input))
>> - ('xz (filtered-port `(,%xz "-dc") input))
>> + ('xz (filtered-port `(,%xz "-dc -T0") input))
>> ('gzip (filtered-port `(,%gzip "-dc") input))
>> (else (error "unsupported compression scheme" compression))))
>
> Parallel decompression isn't implemented yet, but the extra '-T0' option
> doesn't seem to break anything.
>
> The changes LGTM.
>
> I sent a companion patch to <https://bugs.gnu.org/26316> that enables
> threaded compression when building source tarballs.

Is (parallel-job-count) accessible here? It would be nice to respect it.
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAljfTiwACgkQoqBt8qM6
VPrBWQgApc4bdfvSCVtABSMaB+kRw9WSK5S/jcuYEIocUNxnqJJeQo+MF8nYNBPi
i0g1NlWrXr8nonRO0iuMIZLbi74iy4sv4ghGjRXIO4SZfWzsc8ESh+rZH9j2Wnq0
AEDn7oo5klBgWlHnHzsV5DabwQm/6fYFC0ozoBuZw6Xxhnxdg+FMUT8+7DhwexZG
tzfjOD6eoWGVA11N8693EW8KWEO+ViDZNteN5dmCtb7TN7h/1hw4jY3vOn78VLVn
p6ZuH5qxJYDc0e/jcR+PACdX4DZE3ucmt1YS6A0oj38vHPrmjbuRRY4g3+FVmVRk
wtgfbuGFw4Gv4kS/lmG3Uj3HJBnFeg==
=nXr1
-----END PGP SIGNATURE-----

L
L
Leo Famulari wrote on 2 Apr 2017 23:21
Re: bug#26225: [PATCH] guix: Compress and decompress xz archives in parallel.
(name . Marius Bakke)(address . mbakke@fastmail.com)
20170402212118.GA28696@jasmine
On Sat, Apr 01, 2017 at 08:52:28AM +0200, Marius Bakke wrote:
Toggle quote (18 lines)
> > Efraim Flashner wrote:
> >> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
> >> * guix/utils.scm (decompressed-port, compressed-port,
> >> compressed-output-port): Same.
> >
> >> --- a/guix/utils.scm
> >> +++ b/guix/utils.scm
> >> @@ -154,7 +155,7 @@ a symbol such as 'xz."
> >> (match compression
> >> ((or #f 'none) (values input '()))
> >> ('bzip2 (filtered-port `(,%bzip2 "-dc") input))
> >> - ('xz (filtered-port `(,%xz "-dc") input))
> >> + ('xz (filtered-port `(,%xz "-dc -T0") input))
> >> ('gzip (filtered-port `(,%gzip "-dc") input))
> >> (else (error "unsupported compression scheme" compression))))
>
> Is (parallel-job-count) accessible here? It would be nice to respect it.

I agree, it would be good if we respected it here, if possible.
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAljha04ACgkQJkb6MLrK
fwjmyhAA4Kh8awEp7TBs8a83qSg1wzjOhMtmpZ3ocYpi70KWtQ9ZiyOTrWZMiy36
EvfbJweY+wUUqcSxbytP3nGRjRyxXSFyMw7KmTMvmGTlp2vLrw8Jn8s1mrrDvUoU
cWZC5Y8lT5LUTo/vYikk0qjppBjiLtwIe+voHGqKD+2rs9pySP5iHiT5rduRpWXO
R0UkLfsQkoDjkmzRbxNbwIi1TtNcUxjwhRcUEdnX4xQE1G9M4zdFIWarfos6nffD
fTfOf/7ClPmGB9eZi2vfPZgOJv8GM5t/u4yJVY/goKSqim6hnzmRpMzBKpqHbYu1
YVacG2ulnCpVbDTlN+yMeasVmBGW2c2Pzpz+Ecr1GKjGWq9D8SIigBsnK6GI9N9O
+f2DOmA3aWoKGnSui2InYVVAcGPlK435+Caycn4OSYZMkcOzn7E0rTowCBoYsNy7
n+8piwJ1UWb1qhImuxKNnJqFYTmef8DgMBbhS4PJXw/ZKYRm4qmk+u6uq9UPYOBg
FkoSuISww/9qDBQ8Y/Gqz1SHu6CEIPYJzA8pDJyPWrbtPqv46fKoIS4rlnZREGFb
YGJ2XSiirr5tMi7ukQCoJogL8jG2mpbGrxzw148QtSVAleJPte4BDb2f50XVja25
1pjvLXSt7nuHJ9yJRnwwREIBssUXu3/9ZWvb9/YG5KTzo7tbEB0=
=JfVK
-----END PGP SIGNATURE-----


L
L
Ludovic Courtès wrote on 7 Apr 2017 22:59
control message for bug #26225
(address . control@debbugs.gnu.org)
87o9w8szlf.fsf@gnu.org
tags 26225 fixed
close 26225
?
Your comment

This issue is archived.

To comment on this conversation send an email to 26225@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 26225
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch