Recurring timeout error while using offload

  • Open
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Maxim Cournoyer
Owner
unassigned
Submitted by
Maxim Cournoyer
Severity
important
M
M
Maxim Cournoyer wrote on 1 Sep 2021 07:35
Recurring timeout eror while using offload
(name . bug-guix)(address . bug-guix@gnu.org)
87k0k0eu40.fsf@gmail.com
Hello Guix,

I'm encountered timeout error that seem to be caused by a particular
payload rather than networking issues. With a specific payload (a
bunch of rust package to build on the core-updates-frozen branch), it
seems to be triggered every time.

Here's the derivations to be built:

Toggle snippet (22 lines)
$ ./pre-inst-env guix build rust -n
The following derivations would be built:
/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv
/gnu/store/bmc82kpr11gpk1xy7d3l6hy37cs58i7d-rustc-1.54.0-src.tar.xz.drv
/gnu/store/x1v25nclqps677j1g5dvyck2njaxy0ya-rust-1.53.0.drv
/gnu/store/9v5yl0fg4dj1n82pkznksasmdjgi541w-rust-1.52.1.drv
/gnu/store/72572pjayh2j5frr187i7axjzfd01kp6-rust-1.51.0.drv
/gnu/store/0z13dkldzsxf87f4kgi3b2545bm39nnh-rustc-1.51.0-src.tar.xz.drv
/gnu/store/q9mbicnmrpd1c98vmi5y0gyjnpr3gqbn-rust-1.50.0.drv
/gnu/store/h5959ysczr862ldwh6f85h10895c4csp-rust-1.49.0.drv
/gnu/store/izgl0l21r5lwd3d3glxqwmigl0315b26-rustc-1.49.0-src.tar.xz.drv
/gnu/store/n0dpka76b9q0w4r2k912qq5p2a57bz6c-rust-1.48.0.drv
/gnu/store/a9qwa9cl90xrm2dpa3hnd3ak54nqss8n-rust-1.47.0.drv
/gnu/store/7ib14fx50651av5bq5g195cqf2azsvhc-rust-1.46.0.drv
/gnu/store/h3mxv9zlx8q3x0m0s5gvin2n1ndliivw-rust-1.45.2.drv
/gnu/store/4i5xm0wcasilh55f9gfcrzp2sjdkd507-rust-1.44.1.drv
/gnu/store/m7z7jdd2my4p9p5mjwa0cna4fzhd1v8j-rust-1.43.0.drv
/gnu/store/pw4ynvgbnk1j6fxlh37a9wd8fyi7bjnc-rustc-1.50.0-src.tar.xz.drv
/gnu/store/g9izbhyhc6jcalxgbxc6aas3aa66h2aa-rustc-1.52.1-src.tar.xz.drv
/gnu/store/jbr3f7360x3ww5rn6nnb803yxq7ppck3-rustc-1.53.0-src.tar.xz.drv

And here is what the error looks like:

Toggle snippet (29 lines)
rustc-1.52.1-src/src/llvm-project/clang/test/Misc/message-length.c
rustc-1.52.1-src/src/llvm-project/clang/test/Driver/fmessage-length.c
rustc-1.52.1-src/src/llvm-project/compiler-rt/test/hwasan/TestCases/stack-history-length.c
rustc-1.52.1-src/src/llvm-project/clang/test/Sema/vector_swizzle_length.c
rustc-1.52.1-src/src/llvm-project/clang/test/Sema/array-bounds-ptr-arith.c
rustc-1.52.1-src/src/llvm-project/clang/test/Analysis/ptr-arith.c
rustc-1.52.1-src/vendor/openssl-src/openssl/crypto/engine/tb_asnmth.c
rustc-1.52.1-src/src/llvm-project/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_cnth.c
rustc-1.52.1-src/vendor/openssl-src/openssl/crypto/pem/pem_oth.c
rustc-1.52.1-src/vendor/libgit2-sys/libgit2/src/transports/auth.c
guix offload: error: failed to connect to '127.0.0.1': Timeout connecting to 127.0.0.1
cannot build derivation `/gnu/store/4i5xm0wcasilh55f9gfcrzp2sjdkd507-rust-1.44.1.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/h3mxv9zlx8q3x0m0s5gvin2n1ndliivw-rust-1.45.2.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/7ib14fx50651av5bq5g195cqf2azsvhc-rust-1.46.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/a9qwa9cl90xrm2dpa3hnd3ak54nqss8n-rust-1.47.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/n0dpka76b9q0w4r2k912qq5p2a57bz6c-rust-1.48.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/h5959ysczr862ldwh6f85h10895c4csp-rust-1.49.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/q9mbicnmrpd1c98vmi5y0gyjnpr3gqbn-rust-1.50.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/72572pjayh2j5frr187i7axjzfd01kp6-rust-1.51.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/9v5yl0fg4dj1n82pkznksasmdjgi541w-rust-1.52.1.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/x1v25nclqps677j1g5dvyck2njaxy0ya-rust-1.53.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv' failed

real 2m50.854s
user 0m11.302s
sys 0m0.661s

It occurs shortly after heavy scrolling starts.

I've verified that it's not my network link at least on my local
network; any ideas what I should look into for debugging? Perhaps the
logs/strace of the remote SSH server?

Thanks,

Maxim
M
M
Maxim Cournoyer wrote on 1 Sep 2021 18:57
(address . 50312@debbugs.gnu.org)
87pmtscjy5.fsf@gmail.com
Hello,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

[...]

Toggle quote (40 lines)
> And here is what the error looks like:
>
> rustc-1.52.1-src/src/llvm-project/clang/test/Misc/message-length.c
> rustc-1.52.1-src/src/llvm-project/clang/test/Driver/fmessage-length.c
> rustc-1.52.1-src/src/llvm-project/compiler-rt/test/hwasan/TestCases/stack-history-length.c
> rustc-1.52.1-src/src/llvm-project/clang/test/Sema/vector_swizzle_length.c
> rustc-1.52.1-src/src/llvm-project/clang/test/Sema/array-bounds-ptr-arith.c
> rustc-1.52.1-src/src/llvm-project/clang/test/Analysis/ptr-arith.c
> rustc-1.52.1-src/vendor/openssl-src/openssl/crypto/engine/tb_asnmth.c
> rustc-1.52.1-src/src/llvm-project/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_cnth.c
> rustc-1.52.1-src/vendor/openssl-src/openssl/crypto/pem/pem_oth.c
> rustc-1.52.1-src/vendor/libgit2-sys/libgit2/src/transports/auth.c
> guix offload: error: failed to connect to '127.0.0.1': Timeout connecting to 127.0.0.1
> cannot build derivation `/gnu/store/4i5xm0wcasilh55f9gfcrzp2sjdkd507-rust-1.44.1.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/h3mxv9zlx8q3x0m0s5gvin2n1ndliivw-rust-1.45.2.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/7ib14fx50651av5bq5g195cqf2azsvhc-rust-1.46.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/a9qwa9cl90xrm2dpa3hnd3ak54nqss8n-rust-1.47.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/n0dpka76b9q0w4r2k912qq5p2a57bz6c-rust-1.48.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/h5959ysczr862ldwh6f85h10895c4csp-rust-1.49.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/q9mbicnmrpd1c98vmi5y0gyjnpr3gqbn-rust-1.50.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/72572pjayh2j5frr187i7axjzfd01kp6-rust-1.51.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/9v5yl0fg4dj1n82pkznksasmdjgi541w-rust-1.52.1.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/x1v25nclqps677j1g5dvyck2njaxy0ya-rust-1.53.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv': 1 dependencies couldn't be built
> guix build: error: build of `/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv' failed
>
> real 2m50.854s
> user 0m11.302s
> sys 0m0.661s
>
> It occurs shortly after heavy scrolling starts.
>
> I've verified that it's not my network link at least on my local
> network; any ideas what I should look into for debugging? Perhaps the
> logs/strace of the remote SSH server?
>
> Thanks,
>
> Maxim

Now that I've manually built them by small batches of 4 (that would not
fail), I reached the last item rust-1.54, which failed like so:

Toggle snippet (35 lines)
rustc-1.54.0-src/vendor/lzma-sys/xz-5.2/tests/files/good-0cat-empty.xz
source is at 'rustc-1.54.0-src'
Backtrace:
12 (primitive-load "/gnu/store/j2hl7fmsbh7jmixpy8dy07finqp…")
In ice-9/eval.scm:
619:8 11 (_ #(#(#<directory (guile-user) 7ffff5fdbc80> "rus…") #))
In ice-9/boot-9.scm:
142:2 10 (dynamic-wind _ _ #<procedure 7ffff5fc8e00 at ice-9/eva…>)
In system/base/compile.scm:
352:28 9 (compile _ #:from _ #:to _ #:env _ #:optimization-level …)
265:44 8 (_ _ _)
265:44 7 (_ _ _)
265:44 6 (_ _ _)
261:27 5 (_ _ _)
In ice-9/boot-9.scm:
2835:4 4 (save-module-excursion _)
In language/bytecode/spec.scm:
35:19 3 (_)
In ice-9/boot-9.scm:
260:13 2 (for-each #<procedure delete-file-recursively (dir #:k…> …)
In guix/build/utils.scm:
446:23 1 (delete-file-recursively "vendor/jemalloc-sys/jemalloc" …)
In unknown file:
0 (lstat "vendor/jemalloc-sys/jemalloc")

ERROR: In procedure lstat:
In procedure lstat: No such file or directory: "vendor/jemalloc-sys/jemalloc"
note: keeping build directory `/tmp/guix-build-rustc-1.54.0-src.tar.xz.drv-0'
builder for `/gnu/store/bmc82kpr11gpk1xy7d3l6hy37cs58i7d-rustc-1.54.0-src.tar.xz.drv' failed with exit code 1
build of /gnu/store/bmc82kpr11gpk1xy7d3l6hy37cs58i7d-rustc-1.54.0-src.tar.xz.drv failed
View build log at '/var/log/guix/drvs/bm/c82kpr11gpk1xy7d3l6hy37cs58i7d-rustc-1.54.0-src.tar.xz.drv'.
cannot build derivation `/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv' failed

Perhaps this failure was badly handled in the parallel build an reported
as a timeout? I'm just guessing, but reporting it as it may be at
cause.

Thanks,

Maxim
L
L
Ludovic Courtès wrote on 13 Sep 2021 14:05
control message for bug #50312
(address . control@debbugs.gnu.org)
87y2808yvf.fsf@gnu.org
severity 50312 important
quit
M
M
Maxim Cournoyer wrote on 14 Apr 2023 15:17
(address . control@debbugs.gnu.org)
87bkjqk3gm.fsf@gmail.com
retitle 50312 Recurring timeout error while using offload
quit
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 50312@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 50312
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch