Using QEMU to produce Debian filesystems for multiple architectures

By Jay Rodgers

This article discusses producing Debian filesystems for multiple architectures, if you just want to download the actual tarballs, they can be found on the Github releases page. Please remember to star the repository on Github if you find it helpful.

I’ve previously written at some length about the importance of supporting systems based on non-traditional architectures with our code.

For some time I’ve been supporting open-source builds of Visual Studio Code for ARM processors, with scripts to install VS Code to devices like Chromebooks and the Raspberry Pi, primarily for use in education (and, surprisingly to me, I’ve heard from some folks using this in the datacenter too!).

Certainly this helps people get this specific software on their device, but it’s not enough. It doesn’t do much to push the status quo at large or to move the needle in support of these devices.

I’m currently working on a project that I hope will (at least for some codebases) significantly lower the bar to supporting non-traditional architectures and low-end devices. I’d like to discuss the first part of that project today, which is now up on Github.

Specifically, we’re going to be building Debian root filesystems for each of the architectures supported by Stretch, automatically, in the cloud. The rationale for using Debian rather than Ubuntu is that Debian officially supports a much wider set of architectures. Unfortunately I was unable to find root filesystem packages for Debian available to download anywhere, so we’re going to have to generate them ourselves!

What is a root filesystem, and what would I use it for?
In Linux parlance, a root filesystem is simply the collection of files and directories that make up the very root directory of a bootable Linux operating system as compiled for a particular architecture (i.e. exactly what the name implies).
This is useful for a range of scenarios where you need to build or test code for more than one architecture, but without access to a physical device for compiling or testing on.
The excellent QEMU emulator will allow for entering these directories as if they were a seperate, already booted operating system (known as a “change of root”, using the chroot command), while emulating the architecture — in effect allowing code intended for a different architecture to be run on the host.

One of several problems with packaging a root filesystem for reuse is that it rather quickly becomes out-of-date. When the filesystem is built, the most up-to-date packages are downloaded and installed, but over time (as updates are released for the packages in the filesystem) it takes progressively longer to bring the filesystem up-to-date after downloading it — to the point were it eventually becomes faster just to build a new one from scratch.

This simply won’t do if our goal is to provide some support to encourage treating these alternate architectures as first-class targets. We need to be consistently up-to-date. We need to automate this, which will be the topic of my next article in this series.

For now though, on to building these filesystems.

Debian provides a couple of tools to create new root filesystems. The one we’ll be using here is debootstrap.

The first step is to ensure that our environment has the tools we need, specifically debootstrap and qemu-user-static (more on that later!).

apt-get install -y debootstrap qemu-user-static

This will install both packages without further input from the user, essential if we want the build server to run these commands in an automated fashion.

debootstrap --foreign --verbose --arch=armhf --variant=minbase stretch rootfs

Now that debootstrap is installed, we can call it to create a root filesystem for the given architecture.

The foreign argument tells debootstrap that we need to produce a filesystem for a different architecture than the one we’re running on. This is important as debootstrap will postpone installing packages as we’re not in a position to run armhf code (yet!).

The verbose argument simply tells debootstrap not to be conservative with logging information to the console — this is how we’ll follow along with the execution of the commands on the build server to ensure everything is behaving as we expected.

The arch argument tells debootstrap we want to prepare a filesystem for armhf, but we’ll use variables to switch this out for the other architectures we need later.

The variant argument in effect tells debootstrap how expansive of an operating system we want to produce in the new filesystem. The minbase option installs the least amount of packages necessary to produce a widely useful operating system — which is what we want in our case, as if this is for use in arbitrary builds we won’t know the dependencies ahead of time.

Finally we indicate which Debian release we want to build a filesystem for. Stretch is (at time of writing) the current stable release, so we’ll go with that, and we specify which subdirectory the new filesystem should be created in (rootfs).

We’ve now hit the first speedbump. We need to be able to emulate our new filesystem, but the execution of the packages inside the filesystem depend on virtual devices that the filesystem won’t have access to. We need to take the virtual devices from our host and bind them to the new filesystem so it can execute in the context of an operating system.

cd rootfs
mount --bind /dev dev/
mount --bind /sys sys/
mount --bind /proc proc/
mount --bind /dev/pts dev/pts/
cd ..

We enter the filesystem, and mount the virtual devices in our host (/dev, /sys, /proc and /dev/pts) into the same locations within the filesystem — making them available for later.

cp /usr/bin/qemu-arm-static rootfs/usr/bin/
chmod +x rootfs/usr/bin/qemu-arm-static

In order for us to execute the compiled armhf code in our file system later in the process, we need to use the QEMU emulator we installed into /usr/bin. Unfortunately, once we change our root directory into the new filesystem, we’ll be pointing at the /usr/bin of the new filesystem, which does not contain the emulator. So, we need to copy it into place first — and use chmod +x to make it executable.

chroot rootfs /debootstrap/debootstrap --second-stage --verbose

It’s at this point we’ll enter the filesystem to complete the debootstrap process. The chroot command will change our root directory to the rootfs sub-directory we created with debootstrap earlier.

The verbose argument is given again to ensure we get as much information as possible from our logs.

The second-stage argument tells debootstrap that we’re now running on the target architecture. We’re not, of course, but thanks to QEMU debootstrap will happily execute as if we were to complete the setup of the filesystem.

rm rootfs/usr/bin/qemu-arm-static

Just for housekeeping sake, we’ll remove QEMU from the root filesystem when we’re done, so that it’s effectively in the same state it would be if we were working on physical armhf hardware. This is quite important from an emulation standpoint, as leaving the emulator in place could confuse matters later with regards to testing for this architecture.

cd rootfs;
tar -zcvf $OUTPUT_DIR/prebootstrap_stretch_minbase_armhf_rootfs.tar.gz *;

Now that the root filesystem is built, all we need to do is compress it into a tarball and send it to our output directory, as denoted by the $OUTPUT_DIR environment variable.

In the next article in this series I’ll discuss taking these scripts and setting up continuous delivery in the cloud, so that the filesystems stay up-to-date automatically.

Thanks for reading, and please let me know if you have any feedback on debootstrap or Debian for cross-compilation, and by all means yell at me if you see anything in here that’s not quite right and I’ll correct it. You can view my other stories here, and contact me on twitter (@headmelted).

Also, please pin down that applause button and (Follow) below if you’ve found the article helpful. It’s really encouraging and it helps me understand which topics people find interesting!