Daemon vulnerability allowing takeover of build users

  • Done
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Reepca Russelstein
Owner
unassigned
Submitted by
Ludovic Courtès
Severity
important
L
L
Ludovic Courtès wrote on 21 Oct 00:02 +0200
(address . guix-patches@gnu.org)
87y12ih1q9.fsf@gnu.org
(Due to email issues, I’m sending the message below on behalf of
Reepca Russelstein <reepca@russelstein.xyz>.)

For a very long time, guix-daemon has helpfully made the outputs of
failed derivation builds available at the same location they were at in
the build container (see
This has proven quite useful for debugging of various packages, but
unfortunately it is implemented by a plain "rename" of the top-level
store items from the chroot's store to the real store. This does not
change the permissions or ownership of these files, which allows a
setuid / setgid binary created by a malicious build to become exposed to
the rest of the users, who can then use it to gain control over that
build user. They can exploit this control to overwrite the output of
any builds run by that user using /proc/PID/fd and SIGSTOP.

Also, there is a window of time for /successful/ build outputs between
when they are moved from the chroot and when their permissions are
canonicalized, which likewise allows for setuid / setgid binaries to be
exposed to other users (see

The first patch fixes the former, the second patch fixes the latter. We
then need to update the guix package to use these new commits, which I
leave to whoever applies this to do since my local repository is in a
rather unclean state and a fresh work tree may take some time to be
ready to run 'make update-guix-package'.
From e936861263d9bafdfbe395c12526f2dc48ac17d7 Mon Sep 17 00:00:00 2001
Message-ID: <e936861263d9bafdfbe395c12526f2dc48ac17d7.1729457080.git.reepca@russelstein.xyz>
From: Reepca Russelstein <reepca@russelstein.xyz>
Date: Sun, 20 Oct 2024 15:36:06 -0500
Subject: [PATCH 1/2] nix: build: sanitize failed build outputs prior to
exposing them.

The only thing keeping a rogue builder and a local user from collaborating to
usurp control over the builder's user during the build is the fact that
whatever files the builder may produce are not accessible to any other users
yet. If we're going to make them accessible, we should probably do some
sanity checking to ensure that sort of collaborating can't happen.

Currently this isn't happening when failed build outputs are moved from the
chroot as an aid to debugging.

* nix/libstore/build.cc (secureFilePerms): new function.
(DerivationGoal::buildDone): use it.

Change-Id: I9dce1e3d8813b31cabd87a0e3219bf9830d8be96
---
nix/libstore/build.cc | 36 +++++++++++++++++++++++++++++++++++-
1 file changed, 35 insertions(+), 1 deletion(-)

Toggle diff (58 lines)
diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc
index d23c0944a4..67ebfe2f14 100644
--- a/nix/libstore/build.cc
+++ b/nix/libstore/build.cc
@@ -1301,6 +1301,34 @@ void replaceValidPath(const Path & storePath, const Path tmpPath)
MakeError(NotDeterministic, BuildError)
+/* Recursively make the file permissions of a path safe for exposure to
+ arbitrary users, but without canonicalising its permissions, timestamp, and
+ user. Throw an exception if a file type that isn't explicitly known to be
+ safe is found. */
+static void secureFilePerms(Path path)
+{
+ struct stat st;
+ if (lstat(path.c_str(), &st)) return;
+
+ switch(st.st_mode & S_IFMT) {
+ case S_IFLNK:
+ return;
+
+ case S_IFDIR:
+ for (auto & i : readDirectory(path)) {
+ secureFilePerms(path + "/" + i.name);
+ }
+ /* FALLTHROUGH */
+
+ case S_IFREG:
+ chmod(path.c_str(), (st.st_mode & ~S_IFMT) & ~(S_ISUID | S_ISGID | S_IWOTH));
+ break;
+
+ default:
+ throw Error(format("file `%1%' has an unsupported type") % path);
+ }
+}
+
void DerivationGoal::buildDone()
{
trace("build done");
@@ -1372,9 +1400,15 @@ void DerivationGoal::buildDone()
build failures. */
if (useChroot && buildMode == bmNormal)
foreach (PathSet::iterator, i, missingPaths)
- if (pathExists(chrootRootDir + *i))
+ if (pathExists(chrootRootDir + *i)) {
+ try {
+ secureFilePerms(chrootRootDir + *i);
rename((chrootRootDir + *i).c_str(), i->c_str());
+ } catch(Error & e) {
+ printMsg(lvlError, e.msg());
+ }
+ }
if (diskFull)
printMsg(lvlError, "note: build failure may have been caused by lack of free disk space");

--
2.45.2
From d096d653cc69118e05f49247ab312d0096b16656 Mon Sep 17 00:00:00 2001
Message-ID: <d096d653cc69118e05f49247ab312d0096b16656.1729457080.git.reepca@russelstein.xyz>
In-Reply-To: <e936861263d9bafdfbe395c12526f2dc48ac17d7.1729457080.git.reepca@russelstein.xyz>
References: <e936861263d9bafdfbe395c12526f2dc48ac17d7.1729457080.git.reepca@russelstein.xyz>
From: Reepca Russelstein <reepca@russelstein.xyz>
Date: Sun, 20 Oct 2024 15:39:02 -0500
Subject: [PATCH 2/2] nix: build: sanitize successful build outputs prior to
exposing them.

There is currently a window of time between when the build outputs are exposed
and when their metadata is canonicalized.

* nix/libstore/build.cc (DerivationGoal::registerOutputs): wait until after
metadata canonicalization to move successful build outputs to the store.

Change-Id: Ia995136f3f965eaf7b0e1d92af964b816f3fb276
---
nix/libstore/build.cc | 23 ++++++++++++++---------
1 file changed, 14 insertions(+), 9 deletions(-)

Toggle diff (43 lines)
diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc
index 67ebfe2f14..43a8a37184 100644
--- a/nix/libstore/build.cc
+++ b/nix/libstore/build.cc
@@ -2369,15 +2369,6 @@ void DerivationGoal::registerOutputs()
Path actualPath = path;
if (useChroot) {
actualPath = chrootRootDir + path;
- if (pathExists(actualPath)) {
- /* Move output paths from the chroot to the store. */
- if (buildMode == bmRepair)
- replaceValidPath(path, actualPath);
- else
- if (buildMode != bmCheck && rename(actualPath.c_str(), path.c_str()) == -1)
- throw SysError(format("moving build output `%1%' from the chroot to the store") % path);
- }
- if (buildMode != bmCheck) actualPath = path;
} else {
Path redirected = redirectedOutputs[path];
if (buildMode == bmRepair
@@ -2463,6 +2454,20 @@ void DerivationGoal::registerOutputs()
canonicalisePathMetaData(actualPath,
buildUser.enabled() && !rewritten ? buildUser.getUID() : -1, inodesSeen);
+ if (useChroot) {
+ if (pathExists(actualPath)) {
+ /* Now that output paths have been canonicalized (in particular
+ there are no setuid files left), move them outside of the
+ chroot and to the store. */
+ if (buildMode == bmRepair)
+ replaceValidPath(path, actualPath);
+ else
+ if (buildMode != bmCheck && rename(actualPath.c_str(), path.c_str()) == -1)
+ throw SysError(format("moving build output `%1%' from the chroot to the store") % path);
+ }
+ if (buildMode != bmCheck) actualPath = path;
+ }
+
/* For this output path, find the references to other paths
contained in it. Compute the SHA-256 NAR hash at the same
time. The hash is stored in the database so that we can
--
2.45.2
L
L
Ludovic Courtès wrote on 21 Oct 00:28 +0200
control message for bug #73919
(address . control@debbugs.gnu.org)
87jze21k9j.fsf@gnu.org
tags 73919 + security
quit
L
L
Ludovic Courtès wrote on 21 Oct 00:29 +0200
(address . control@debbugs.gnu.org)
87iktm1k9f.fsf@gnu.org
severity 73919 important
quit
L
L
Ludovic Courtès wrote on 21 Oct 01:28 +0200
Re: bug#73919: Daemon vulnerability allowing takeover of build users
(address . 73919@debbugs.gnu.org)(name . Reepca Russelstein)(address . reepca@russelstein.xyz)
87cyju1hio.fsf@gnu.org
The two patches by Reepca have been pushed, followed by an update of the
‘guix’ package, followed by a news entry:

c9e51ab38d * news: Add news entry for build user takeover vulnerability fix.
5966e0fdc7 * gnu: guix: Update to 5ab3c4c.
5ab3c4c1e4 * daemon: Sanitize successful build outputs prior to exposing them.
558224140d * daemon: Sanitize failed build outputs prior to exposing them.

Now’s the time to upgrade your system!

Ludo’.
R
R
Reepca Russelstein wrote on 21 Oct 00:35 +0200
News entry
(address . 73919@debbugs.gnu.org)(address . ludo@gnu.org)
87ldyijtcl.fsf@russelstein.xyz
Here's a news entry describing the vulnerability. The "TBD" commit
should be replaced with the one that updated the guix package.
From 532996c5908fb14cc8d102865280fb203c075c9c Mon Sep 17 00:00:00 2001
From: Reepca Russelstein <reepca@russelstein.xyz>
Date: Sun, 20 Oct 2024 17:32:23 -0500
Subject: [PATCH] etc: news: add news entry for build user takeover
vulnerability fix.

* etc/news.scm: add entry about build user takeover vulnerability.
---
etc/news.scm | 32 ++++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)

Toggle diff (45 lines)
diff --git a/etc/news.scm b/etc/news.scm
index a90f92a..20cc3d7 100644
--- a/etc/news.scm
+++ b/etc/news.scm
@@ -33,6 +33,38 @@
(channel-news
(version 0)
+ (entry (commit "TBD")
+ (title
+ (en "Daemon vulnerability allowing takeover of build users fixed"))
+ (body
+ (en "A vulnerability allowing a local user to execute arbitrary code
+as any of the build users has been identified and fixed. Most notably, this
+allows any local user to alter the result of any local build, even if it
+happens inside a container. The only requirements to exploit this
+vulnerability are the ability to start a derivation build and the ability to
+run arbitrary code with access to the store in the root PID namespace on the
+machine that build occurs on. This largely limits the vulnerability to
+multi-user systems.
+
+This vulnerability is caused by the fact that @command{guix-daemon} does not
+change ownership and permissions on the outputs of failed builds when it moves
+them to the store, and is also caused by there being a window of time between
+when it moves outputs of successful builds to the store and when it changes
+their ownership and permissions. Because of this, a build can create a binary
+with both setuid and setgid bits set and have it become visible to the outside
+world once the build ends. At that point any process that can access the
+store can execute it and gain the build user's privileges. From there any
+process owned by that build user can be manipulated via procfs and signals at
+will, allowing the attacker to control the output of its builds.
+
+You are advised to upgrade @command{guix-daemon}. Run @command{info \"(guix)
+Upgrading Guix\"}, for info on how to do that. Additionally, if there is any
+risk that a builder may have already created these setuid binaries (for
+example on accident), run @command{guix gc} to remove all failed build
+outputs.
+
+See @uref{https://issues.guix.gnu.org/73919} for more information on this
+vulnerability.")))
(entry (commit "2fae63df2138b74d30e120364f0f272871595862")
(title
(en "Core packages updated")
--
2.45.2
-----BEGIN PGP SIGNATURE-----

iQFLBAEBCAA1FiEEdNapMPRLm4SepVYGwWaqSV9/GJwFAmcVhaoXHHJlZXBjYUBy
dXNzZWxzdGVpbi54eXoACgkQwWaqSV9/GJwCxAf9GFxSOrjV4np2fuUO0eTtT3SU
FHBuQe/da2OHSD6xaB2kBpaKXIylNOI9+pseOIa3tve0YatvR1BV0Vn6oUmfY2nb
d61JVCrkUj1raiqpwyOvO+Cd0IOot20m4vv+OMA8lTk51l7cnPymfBeJPnhx3Wa0
oUBnxarxMbrtwkwXrC2cdFs03viXDOuKzzGKBP9ixDePcIxMffi8wOcXPr2QHWHH
GN5+DYmrimoQQyXnq1EKpo9/qh+V7PI7G2/CHnFkeGUc7+hNgvNGkV7NC6twgKz/
YezpgyDTgMnlTIHVAN3qooMtUY8kowHw/0fI5p3ETbJ5P3BwVQcNcoJO9pb6XA==
=gZP/
-----END PGP SIGNATURE-----

R
R
Reepca Russelstein wrote on 21 Oct 07:10 +0200
[PATCH guix-artwork] add security advisory post about recent vulnerability
(address . 73919@debbugs.gnu.org)
87a5eyhwgz.fsf@russelstein.xyz
Patch attached, let me know if anything needs revision (or make the
change yourself, I'm going to have to sleep soon).

- reepca
-----BEGIN PGP SIGNATURE-----

iQFLBAEBCAA1FiEEdNapMPRLm4SepVYGwWaqSV9/GJwFAmcV4lwXHHJlZXBjYUBy
dXNzZWxzdGVpbi54eXoACgkQwWaqSV9/GJxaYAf8C16kJ1BlZ3BMdnNrnekCYafZ
wYnCMBUGUZD/z+Gz2n+vFta2PsVy6fDx7YXDC6wvVX1xJHXQkS96Rsfr5QQQ+Whs
ZC5MA3DYKxdid8qGAUGUlSEFt8iTMZS1bxI6oGURkixA5SQ9yddBvC5aEJgaD/4w
KMvy9E21VLqfdszZRcLlvZXfksvJtowvrQSYXN3sIIBJpunI/7WFAkq9pgL+rQ/x
0e7JGdLeKz4NS6qd4/F6gU/m7p/kvbLSWh/8D6oK4SHxlJoBdpfScgbgGfrk/PfP
1bRDkZcimX7hrli7D84givo8Y+cnxtPNB7r+d5TExhBdd0I09LfflonbVAouFw==
=hr6Z
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 21 Oct 17:36 +0200
Re: bug#73919: Daemon vulnerability allowing takeover of build users
(name . Reepca Russelstein)(address . reepca@russelstein.xyz)(address . 73919@debbugs.gnu.org)
87cyjtead9.fsf_-_@gnu.org
Hello,

Reepca Russelstein <reepca@russelstein.xyz> skribis:

Toggle quote (7 lines)
> From 22adb50845bf4af7b1d6fd78e2515c3387356ce1 Mon Sep 17 00:00:00 2001
> From: Reepca Russelstein <reepca@russelstein.xyz>
> Date: Mon, 21 Oct 2024 00:04:32 -0500
> Subject: [PATCH] website: Add 2024-10-21 security advisory post
>
> * website/posts/2024-10-21-security-advisory.md: new file.

I pushed this patch a few hours ago and followed up with minor edits:


Ludo’.
L
L
Ludovic Courtès wrote on 21 Oct 22:51 +0200
control message for bug #73919
(address . control@debbugs.gnu.org)
87y12hch7l.fsf@gnu.org
close 73919
quit
R
R
Reepca Russelstein wrote on 23 Oct 20:20 +0200
Re: bug#73919: Daemon vulnerability allowing takeover of build users
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 73919@debbugs.gnu.org)
875xpiisua.fsf@russelstein.xyz
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (12 lines)
>> From 22adb50845bf4af7b1d6fd78e2515c3387356ce1 Mon Sep 17 00:00:00 2001
>> From: Reepca Russelstein <reepca@russelstein.xyz>
>> Date: Mon, 21 Oct 2024 00:04:32 -0500
>> Subject: [PATCH] website: Add 2024-10-21 security advisory post
>>
>> * website/posts/2024-10-21-security-advisory.md: new file.
>
> I pushed this patch a few hours ago and followed up with minor edits:
>
> https://guix.gnu.org/blog/2024/build-user-takeover-vulnerability/
>

Thanks, the changes look good, except for one part:

Toggle quote (2 lines)
> Your are strongly advised

- reepca
-----BEGIN PGP SIGNATURE-----

iQFLBAEBCAA1FiEEdNapMPRLm4SepVYGwWaqSV9/GJwFAmcZPn0XHHJlZXBjYUBy
dXNzZWxzdGVpbi54eXoACgkQwWaqSV9/GJzodQf/QAfIvBqPrZ9evGq+qYxO4U/r
/XfomFRFdHlcDSkcztxdnRvtOtKXG8x83ScK7r348jIv1T4CeI2DKkk2ktXjuIjS
4OXs21gEgTuEUvx8rEptk58zxShryyryV+tdzI0RjopTzt+0631sc4g0POZngv8j
7LtPjwy1f4DIuN5NnPy+p4HCkUP5fJAOKxBLCYKYL//fj3JZDiA66Pla3CAqY1VW
vi8FolQyXIdD7LDWoULYWvSJQQ6gr0VtWCRWgVCdXI7LiJq5Q2qUiis+NCEA8mbU
S8AQuBd/kA3+CxGjkQpeErnOrRpO0COprbKoDKzFE55a8cAA6zXP0RDJiAy9EA==
=HoCa
-----END PGP SIGNATURE-----

?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 73919@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 73919
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch