Reconfiguring the build process

2021-02-01

Up until now HitchHiker has used a build process that closely followed that used by the Linux From Scratch book. There was a temporary toolchain, hacked to use a different library directory and prefix

so as not to conflict with the host system or with the libraries and tools being installed as the finished system. This temporary toolchain was in itself a complete development environment that included everything required to bootstrap a full system. To build to final system, is was necessary to chroot into a new directory containing the temporary tools and root filesystem.

This method had some advantages, but also involved compiling most packages at least twice. It also required quite a lot of extra work to do any kind of cross compiling. In fact, I had set up an RPI4 specifically as a compilation machine for Armv7l, as it was basically impractical to use my x86_64 laptop to develop for Arm.

In an effort to increase the flexibility of the HitchHiker build tree, support more architectures, and to streamline the build I am experimenting with using a sysroot compiler for the majority of the build. In testing, with minimal effort I was able to set up cross toolchains for armv7l, aarch64, riscv64, and i486 as well as a native x86_64 toolchain using the same build tree and only adjusting the ${arch} variable in config.mk. In testing I was able to compile almost the complete userland for x86_64, aarch64 and riscv64 without resorting to a chroot, and all on my main machine. My current plan is to compile most of the base system using a sysroot toolchain, either native or cross, including the build of the native toolchain, and then to chroot into the near complete root filesystem to finish up any packages that cannot be easily cross compiled. At this juncture I believe the only package in the base system that will be truly problematic for cross-compilation is Perl, which uses an ill conceived homegrown configure script instead of autotools. However this is not quite verified for all packages just yet.

In the case of a package refusing to cross-compile, I can then use the method outlined in the previous post and compile with the native toolchain using qemu user-mode emulation.

In addition to the vast reduction of packages being compiled twice or more, this also has the advantage of running on a fast machine. Additionally, with most of the work not requiring the chroot trick, there is no emulation layer to slow the process down. A future goal will be to fix the issues with Perl and push patches upstream if possible, as Perl is a hard build dependency of the kernel and therefore must be considered an essential part of the base system.

To give some idea of the resource savings this brings about, here's a quick table outlining the number of passes required for certain packages under the old layout vs the new. Note that it is still necessary to make multiple passes at Binutils and Gcc, as we have to build our sysroot/cross toolchain, but there is a rather significant reduction in the amount of other software being built twice.

Tags for this post: