(address . guix-patches@gnu.org)
Here is a rough draft of my toolchain post.
I hope it can make an interesting contribution to the Guix literature.
- Mitchell
From 4f6c43091ffd67cdbc5f041e496f61bc8a06070e Mon Sep 17 00:00:00 2001
From: Mitchell Schmeisser <mitchellschmeisser@librem.one>
Date: Fri, 24 Feb 2023 13:02:05 -0500
Subject: [PATCH] website: Add custom toolchain blog post
* website/posts/custom-toolchains-with-guix.md: New file.
---
website/posts/custom-toolchains-with-guix.md | 557 +++++++++++++++++++
1 file changed, 557 insertions(+)
create mode 100644 website/posts/custom-toolchains-with-guix.md
Toggle diff (540 lines)
diff --git a/website/posts/custom-toolchains-with-guix.md b/website/posts/custom-toolchains-with-guix.md
new file mode 100644
index 0000000..f73f1ab
--- /dev/null
+++ b/website/posts/custom-toolchains-with-guix.md
@@ -0,0 +1,557 @@
+# Table of Contents
+
+1. [Overview](#org2633a51)
+2. [Anatomy of a toolchain](#orgc440e9e)
+3. [Bootstrapping a Toolchain](#orgd42b6c3)
+4. [Defining the Packages](#org55042c5)
+ 1. [Binutils](#org67da1ec)
+ 2. [GCC sans libc](#org82d6f83)
+ 3. [Newlib(-nano)](#orgf6bafbc)
+ 4. [Complete toolchain](#org052f2a2)
+5. [Integrating with Zephyr Build System](#orgc3f87f4)
+ 1. [Testing](#org9f3c314)
+
+All code is available at [guix-zephyr](https://github.com/paperclip4465/guix-zephyr) channel.
+
+
+<a id="org2633a51"></a>
+
+# Overview
+
+In order to deploy embedded software using Guix we first need to teach Guix
+how to build it. Since Guix bootstraps everything this means we must teach Guix
+how to build our toolchain.
+
+The [Zephyr Project](https://zephyrproject.org) uses its own fork of GCC with custom configs for
+the architectures supported by the project.
+
+
+<a id="orgc440e9e"></a>
+
+# Anatomy of a toolchain
+
+Toolchains are responsible for taking high level descriptions of programs
+and lowering them down to a series of equivalent machine instructions.
+This process involves more than just a compiler. The compiler uses the `binutils`
+to manipulate it's internal representation down to a given architecture.
+It also needs the use of the C standard library as well as a few other libraries
+needed for some compiler optimizations.
+
+The C library provides the interface to the underlying kernel. System calls like `write`
+and `read` are provided by `Glibc` on most Linux distributions.
+
+In embedded systems smaller implementations like `newlib` and `newlib-nano` are used.
+
+
+<a id="orgd42b6c3"></a>
+
+# Bootstrapping a Toolchain
+
+In order to compile GCC we need a C library that's been compiled for
+our target architecture. How can we cross compile our C library if we
+need our C library to build a cross compiler? The solution is to build
+a simpler compiler that doesn't require the C library to function.
+It will not be capable of as many optimizations and it will be very slow,
+however it will be able to build the C libraries as well as the complete version
+of GCC.
+
+In order to build the simpler compiler we need to compile the `binutils` to
+work with our target architecture.
+The `binutils` can be bootstrapped with our host GCC and have no target dependencies.
+
+[For more information read this.](https://crosstool-ng.github.io/docs/toolchain-construction/)
+
+Doesn't sound so bad right? It isn't… in theory.
+However internet forums since time immemorial have been
+littered with the laments of those who came before.
+From incorrect versions of `ISL` to the wrong C library being linked
+or the host linker being used, etc.
+The one commonality between all of these issues is the environment.
+Building a cross toolchain is difficult because isolating build
+environments is difficult.
+
+In fact as of `v0.14.2` the zephyr SDK repository took down the build
+instructions and posted a sign that read
+"Building this is too complicated, don't worry about it."<sup><a id="fnr.1" class="footref" href="#fn.1" role="doc-backlink">1</a></sup>
+
+We will neatly side step all of these problems and not
+risk destroying or polluting our host system with garbage
+by using Guix to manage our environments for us.
+
+Our toolchain only requires the first pass compiler because
+newlib(-nano) is statically linked and introduced to the toolchain
+by normal package composition.
+
+
+<a id="org55042c5"></a>
+
+# Defining the Packages
+
+All of the base packages are defined in `zephyr/packages/zephyr.scm`.
+Zephyr modules are defined in `zephyr/packages/zephyr-xyz.scm`, following
+the pattern of other module systems implemented by Guix.
+
+
+<a id="org67da1ec"></a>
+
+## Binutils
+
+First thing we need to build is the `arm-zephyr-eabi` binutils.
+This is very easy in Guix.
+
+ (define-module (zephyr packages zephyr)
+ #:use-module (guix packages))
+
+ (define-public arm-zephyr-eabi-binutils
+ (let ((xbinutils (cross-binutils "arm-zephyr-eabi")))
+ (package
+ (inherit xbinutils)
+ (name "arm-zephyr-eabi-binutils")
+ (version "2.38")
+ (source
+ (origin (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/zephyrproject-rtos/binutils-gdb")
+ (commit "6a1be1a6a571957fea8b130e4ca2dcc65e753469")))
+ (file-name (git-file-name name version))
+ (sha256 (base32 "0ylnl48jj5jk3jrmvfx5zf8byvwg7g7my7jwwyqw3a95qcyh0isr"))))
+ (arguments
+ `(#:tests? #f
+ ,@(substitute-keyword-arguments (package-arguments xbinutils)
+ ((#:configure-flags flags)
+ `(cons "--program-prefix=arm-zephyr-eabi-" ,flags)))))
+ (native-inputs
+ (append
+ (list texinfo
+ bison
+ flex
+ gmp
+ dejagnu)
+ (package-native-inputs xbinutils)))
+ (home-page "https://zephyrproject.org")
+ (synopsis "binutils for zephyr RTOS"))))
+
+The function `cross-binutils` returns a package which has been
+configured for the given gnu triplet. We simply inherit that package
+and replace the source.
+The zephyr build system expects the binutils to be prefixed with
+`arm-zephyr-eabi-` which is accomplished by adding another flag to the
+`#:configure-flags` argument.
+
+We can test our package definition using the `-L` flag with `guix build`
+to add our packages.
+
+ guix build -L guix-zephyr zephyr-binutils
+
+ /gnu/store/...-zephyr-binutils-2.38
+
+This directory contains the results of `make install`.
+
+
+<a id="org82d6f83"></a>
+
+## GCC sans libc
+
+This one is a bit more involved. Don't be afraid!
+This version of GCC wants ISL version 0.15. It's easy enough
+to make that happen. Inherit the current version of ISL and swap
+out the source and update the version. For most packages the build process doesn't
+change that much between versions.
+
+ (define-public isl-0.15
+ (package
+ (inherit isl)
+ (version "0.15")
+ (source (origin
+ (method url-fetch)
+ (uri (list (string-append "mirror://sourceforge/libisl/isl-"
+ version ".tar.gz")))
+ (sha256
+ (base32
+ "11vrpznpdh7w8jp4wm4i8zqhzq2h7nix71xfdddp8xnzhz26gyq2"))))))
+
+Like the binutils, there is a function for creating cross-gcc packages.
+This one accepts keywords specifying which binutils and libc to use.
+If libc isn't given (like here), gcc is configured with many options disabled
+to facilitate being built without libc. Therefore we need to add the extra options
+we want <sup><a id="fnr.2" class="footref" href="#fn.2" role="doc-backlink">2</a></sup>
+
+ (define-public gcc-arm-zephyr-eabi-12
+ (let ((xgcc (cross-gcc "arm-zephyr-eabi"
+ #:xbinutils zephyr-binutils)))
+ (package
+ (inherit xgcc)
+ (version "12.1.0")
+ (source (origin (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/zephyrproject-rtos/gcc")
+ (commit "0218469df050c33479a1d5be3e5239ac0eb351bf")))
+ (file-name (git-file-name (package-name xgcc) version))
+ (sha256
+ (base32 "1s409qmidlvzaw1ns6jaanigh3azcxisjplzwn7j2n3s33b76zjk"))
+ (patches
+ (search-patches "gcc-12-cross-environment-variables.patch"
+ "gcc-cross-gxx-include-dir.patch"))))
+ (native-inputs
+ (modify-inputs (package-native-inputs xgcc)
+ ;; Get rid of stock ISL
+ (delete "isl")
+ ;; Add additional dependencies that xgcc doesn't have
+ ;; including our special ISL
+ (prepend flex
+ perl
+ python-3
+ gmp
+ isl-0.15
+ texinfo
+ python
+ mpc
+ mpfr
+ zlib)))
+ (arguments
+ (substitute-keyword-arguments (package-arguments xgcc)
+ ((#:phases phases)
+ `(modify-phases ,phases
+ (add-after 'unpack 'fix-genmultilib
+ (lambda _
+ (substitute* "gcc/genmultilib"
+ (("#!/bin/sh") (string-append "#!" (which "sh"))))
+ #t))
+
+ (add-after 'set-paths 'augment-CPLUS_INCLUDE_PATH
+ (lambda* (#:key inputs #:allow-other-keys)
+ (let ((gcc (assoc-ref inputs "gcc")))
+ ;; Remove the default compiler from CPLUS_INCLUDE_PATH to
+ ;; prevent header conflict with the GCC from native-inputs.
+ (setenv "CPLUS_INCLUDE_PATH"
+ (string-join
+ (delete (string-append gcc "/include/c++")
+ (string-split (getenv "CPLUS_INCLUDE_PATH")
+ #\:))
+ ":"))
+ (format #t
+ "environment variable `CPLUS_INCLUDE_PATH' changed to ~a~%"
+ (getenv "CPLUS_INCLUDE_PATH"))
+ #t)))))
+
+ ((#:configure-flags flags)
+ ;; The configure flags are largely identical to the flags used by the
+ ;; "GCC ARM embedded" project.
+ `(append (list "--enable-multilib"
+ "--with-newlib"
+ "--with-multilib-list=rmprofile"
+ "--with-host-libstdcxx=-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm"
+ "--enable-plugins"
+ "--disable-decimal-float"
+ "--disable-libffi"
+ "--disable-libgomp"
+ "--disable-libmudflap"
+ "--disable-libquadmath"
+ "--disable-libssp"
+ "--disable-libstdcxx-pch"
+ "--disable-nls"
+ "--disable-shared"
+ "--disable-threads"
+ "--disable-tls"
+ "--with-gnu-ld"
+ "--with-gnu-as"
+ "--enable-initfini-array")
+ (delete "--disable-multilib" ,flags)))))
+ (native-search-paths
+ (list (search-path-specification
+ (variable "CROSS_C_INCLUDE_PATH")
+ (files '("arm-zephyr-eabi/include")))
+ (search-path-specification
+ (variable "CROSS_CPLUS_INCLUDE_PATH")
+ (files '("arm-zephyr-eabi/include"
+ "arm-zephyr-eabi/c++"
+ "arm-zephyr-eabi/c++/arm-zephyr-eabi")))
+ (search-path-specification
+ (variable "CROSS_LIBRARY_PATH")
+ (files '("arm-zephyr-eabi/lib")))))
+ (home-page "https://zephyrproject.org")
+ (synopsis "GCC for zephyr RTOS"))))
+
+This GCC can be built like so.
+
+ guix build -L guix-zephyr gcc-cross-sans-libc-arm-zephyr-eabi
+
+ /gnu/store/...-gcc-cross-sans-libc-arm-zephyr-eabi-12.1.0-lib
+ /gnu/store/...-gcc-cross-sans-libc-arm-zephyr-eabi-12.1.0
+
+Great! We now have our stage-1 compiler.
+
+
+<a id="orgf6bafbc"></a>
+
+## Newlib(-nano)
+
+The newlib package package is quite straight forward (relatively).
+It is mostly adding in the relevent configuration flags and patching
+the files the `patch-shebangs` phase missed.
+
+ (define-public zephyr-newlib
+ (package
+ (name "zephyr-newlib")
+ (version "3.3")
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/zephyrproject-rtos/newlib-cygwin")
+ (commit "4e150303bcc1e44f4d90f3489a4417433980d5ff")))
+ (sha256
+ (base32 "08qwjpj5jhpc3p7a5mbl7n6z7rav5yqlydqanm6nny42qpa8kxij"))))
+ (build-system gnu-build-system)
+ (arguments
+ `(#:out-of-source? #t
+ #:configure-flags '("--target=arm-zephyr-eabi"
+ "--enable-newlib-io-long-long"
+ "--enable-newlib-io-float"
+ "--enable-newlib-io-c99-formats"
+ "--enable-newlib-retargetable-locking"
+ "--enable-newlib-lite-exit"
+ "--enable-newlib-multithread"
+ "--enable-newlib-register-fini"
+ "--enable-newlib-extra-sections"
+ "--disable-newlib-wide-orient"
+ "--disable-newlib-fseek-optimization"
+ "--disable-newlib-supplied-syscalls"
+ "--disable-newlib-target-optspace"
+ "--disable-nls")
+ #:phases
+ (modify-phases %standard-phases
+ (add-after 'unpack 'fix-references-to-/bin/sh
+ (lambda _
+ (substitute* '("libgloss/arm/cpu-init/Makefile.in"
+ "libgloss/arm/Makefile.in"
+ "libgloss/libnosys/Makefile.in"
+ "libgloss/Makefile.in")
+ (("/bin/sh") (which "sh")))
+ #t)))))
+ (native-inputs
+ `(("xbinutils" ,zephyr-binutils)
+ ("xgcc" ,gcc-arm-zephyr-eabi-12)
+ ("texinfo" ,texinfo)))
+ (home-page "https://www.sourceware.org/newlib/")
+ (synopsis "C library for use on embedded systems")
+ (description "Newlib is a C library intended for use on embedded
+ systems. It is a conglomeration of several library parts that are easily
+ usable on embedded products.")
+ (license (license:non-copyleft
+ "https://www.sourceware.org/newlib/COPYING.NEWLIB"))))
+
+And the build.
+
+ guix build -L guix-zephyr zephyr-newlib
+
+ /gnu/store/...-zephyr-newlib-3.3
+
+
+<a id="org052f2a2"></a>
+
+## Complete toolchain<sup><a id="fnr.3" class="footref" href="#fn.3" role="doc-backlink">3</a></sup>
+
+Now that we've got the individual tools it's time to create our complete toolchain.
+For this we need to do some package transformations.
+Because these transformations are going to have to be done for every combination of
+gcc/newlib it is best to create a function which we can reuse for every version
+of the SDK.
+
+ (define (arm-zephyr-eabi-toolchain xgcc newlib version)
+ "Produce a cross-compiler zephyr toolchain package with the compiler XGCC and the C
+ library variant NEWLIB."
+ (let ((newlib-with-xgcc (package (inherit newlib)
+ (native-inputs
+ (alist-replace "xgcc" (list xgcc)
+ (package-native-inputs newlib))))))
+ (package
+ (name (string-append "arm-zephyr-eabi"
+ (if (string=? (package-name newlib-with-xgcc)
+ "newlib-nano")
+ "-nano" "")
+ "-toolchain"))
+ (version version)
+ (source #f)
+ (build-system trivial-build-system)
+ (arguments
+ '(#:modules ((guix build union)
+ (guix build utils))
+ #:builder
+ (begin
+ (use-modules (ice-9 match)
+ (guix build union)
+ (guix build utils))
+ (let ((out (assoc-ref %outputs "out")))
+ (mkdir-p out)
+ (match %build-inputs
+ (((names . directories) ...)
+ (union-build (string-append out "/arm-zephyr-eabi")
+ directories)
+ #t))))))
+ (inputs
+ `(("binutils" ,zephyr-binutils)
+ ("gcc" ,xgcc)
+ ("newlib" ,newlib-with-xgcc)))
+ (synopsis "Complete GCC tool chain for ARM zephyrRTOS development")
+ (description "This package provides a complete GCC tool chain for ARM
+ bare metal development with zephyr rtos. This includes the GCC arm-zephyr-eabi cross compiler
+ and newlib (or newlib-nano) as the C library. The supported programming
+ language is C.")
+ (home-page (package-home-page xgcc))
+ (license (package-license xgcc)))))
+
+This function creates a special package which consists of the toolchain in a special directory hierarchy, i.e `arm-zephyr-eabi/`.
+Our complete toolchain definition looks like this.
+
+ (define-public arm-zephyr-eabi-toolchain-0.15.0
+ (arm-zephyr-eabi-toolchain
+ gcc-arm-zephyr-eabi-12
+ zephyr-newlib
+ "0.15.0"))
+
+To build:
+
+ guix build -L guix-zephyr arm-zephyr-eabi-toolchain
+
+ /gnu/store/...-arm-zephyr-eabi-toolchain-0.15.0
+
+
+<a id="orgc3f87f4"></a>
+
+# Integrating with Zephyr Build System
+
+Zephyr uses CMake as it's build system. It contains numerous CMake
+files in both the so-called `ZEPHYR_BASE`, the zephyr source code
+repository, as well as a handful in the SDK
+which help select the correct toolchain for a given board.
+
+There are standard locations the build system will look for the
+SDK. We are not using any of them. Our SDK lives in the store,
+immutable forever. According to [this](https://docs.zephyrproject.org/latest/develop/west/without-west.html) the variable
+`ZEPHYR_SDK_INSTALL_DIR` needs to point to our custom spot.
+
+We also need to grab the cmake files from the [repository](https://github.com/zephyrproject-rtos/sdk-ng) and create a
+file `sdk_version` which contains the version string `ZEPHYR_BASE`
+uses to find a compatible SDK.
+
+Along with the SDK proper we need to include a number of python
+packages required by the build system<sup><a id="fnr.4" class="footref" href="#fn.4" role="doc-backlink">4</a></sup>.
+
+ (define-public zephyr-sdk
+ (package
+ (name "zephyr-sdk")
+ (version "0.15.0")
+ (home-page "https://zephyrproject.org")
+ (source (origin (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/zephyrproject-rtos/sdk-ng")
+ (commit "v0.15.0")))
+ (file-name (git-file-name name version))
+ (sha256 (base32 "04gsvh20y820dkv5lrwppbj7w3wdqvd8hcanm8hl4wi907lwlmwi"))))
+ (build-system trivial-build-system)
+ (arguments
+ `(#:modules ((guix build union)
+ (guix build utils))
+ #:builder
+ (begin
+ (use-modules (guix build union)
+ (ice-9 match)
+ (guix build utils))
+ (let* ((out (assoc-ref %outputs "out"))
+ (cmake-scripts (string-append (assoc-ref %build-inputs "source")
+ "/cmake"))
+ (sdk-out (string-append out "/zephyr-sdk-0.15.0")))
+ (mkdir-p out)
+
+ (match (assoc-remove! %build-inputs "source")
+ (((names . directories) ...)
+ (union-build sdk-out directories)))
+
+ (copy-recursively cmake-scripts
+ (string-append sdk-out "/cmake"))
+
+ (with-directory-excursion sdk-out
+ (call-with-output-file "sdk_version"
+ (lambda (p)
+ (format p "0.15.0")))
+ #t)))))
+ (propagated-inputs
+ (list
+ arm-zephyr-eabi-toolchain-0.15.0
+ zephyr-binutils
+ python-3
+ python-pyelftools
+ python-pykwalify
+ python-pyyaml
+ python-packaging
+ dtc))
+ (native-search-paths
+ (list (search-path-specification
+ (variable "ZEPHYR_SDK_INSTALL_DIR")
+ (files '("")))))
+ (synopsis "SDK for zephyrRTOS")
+ (description "zephyr-sdk contains bundles a complete gcc toolchain as well
+ as host tools like dtc, openocd, qemu, and required python packages.")
+ (license license:apsl2)))
+
+
+<a id="org9f3c314"></a>
+
+## Testing
+
+In order to test we will need an environment with the SDK installed.
+We can take advantage of `guix shell` to avoid installing test
+packages into our home environment. This way if it causes problems we
+can just exit the shell and try again.
+
+ guix shell -L guix-zephyr zephyr-sdk cmake ninja git
+
+`ZEPHYR_BASE` can be cloned into a temporary workspace to test our
+toolchain functionality.<sup><a id="fnr.5" class="footref" href="#fn.5" role="doc-backlink">5</a></sup>
+
+ mkdir /tmp/zephyr-project
+ cd /tmp/zephyr-project
+ git clone https://github.com/zephyrproject-rtos/zephyr
+ export ZEPHYR_BASE=/tmp/zephyr-project/zephyr
+
+In order to build for the test board (k64f in this case) we need to
+get a hold of the vendor Hardware Abstraction Layers and CMSIS.<sup><a id="fnr.6" class="footref" href="#fn.6" role="doc-backlink">6</a></sup>
+
+ git clone https://github.com/zephyrproject-rtos/hal_nxp &&
+ git clone https://github.com/zephyrproject-rtos/cmsis
+
+To inform the build system about this module we pass it in with
+`-DZEPHYR_MODULES=` which is a semicolon separated list of paths
+containing a module.yml file.
+
+To build the hello world sample we use the following incantation.
+
+ cmake -Bbuild $ZEPHYR_BASE/samples/hello_world \
+ -GNinja \
+ -DBOARD=frdm_k64f \
+ -DBUILD_VERSION=3.1.0 \
+ -DZEPHYR_MODULES="/tmp/zephyr-project/hal_nxp;/tmp/zephyr-project/cmsis" \
+ && ninja -Cb
This message was truncated. Download the full message here.