Monday, 1 May 2017

Building a minimal (64bit) Linux system with a vanilla kernel on the Raspberry Pi 3

One of the first things I wanted to do with the new Raspberry Pi (3) was create my own (simple) Linux distribution.

The Pi 3 is based on a BCM2837 SoC - which has a 64-bit ARMv8 CPU (opposed to ARMv7 in the Pi 2) - however the upstream kernels provided from the Pi Foundation are unfortunately all 32-bit - so for this tutorial I will concentrate on providing a 64-bit kernel so we can fully utilise its power.

We will firstly need to understand how the boot process works with Raspberry Pies - as unlike a normal desktop computer which use a BIOS to initiate a bootloader such as Grub - instead the Rasperry Pi has a closed source firmware in the SoC (System on a Chip).

This firmware is read-only / can't be modified in any way - this enables the second-stage bootloader to be read from a FAT32 formatted partition on the SD-Card.

The second-stage bootloader (bootcode.bin) is used to retrieve and program the GPU firmware (start.elf) from the SD-Card, as well as starting up the CPU. There is also a additional file called fixup.dat that configures the SDRAM between the GOU and CPU.

A kernel is then loaded - by default (on the Rapsberry Pi 3) this is named either kernel7.img (32 bit) or kernel8.img (64 bit) and is a Linux kernel - however of course this doesn't necessarily have to be.

The three files above (bootcode.bin, fixup.dat and kernelX.img) are required as a minimum in order to get the Pi up and running.

For a more detailed overview of how the boot process works please see this article.

The Pi Foundation maintains its own kernel tree for the Pi - which as of right now is 4.9 - however the mainline kernel version also works pretty well too!

To start with lets firstly obtain the latest vanilla / mainline kernel - which at this moment is 4.11 - we can download this from here:

and then cross-compile it - I am going to be using Fedora for this - however a lot of people also do this on Debian / Ubuntu:

mkdir /home/user/workspace
tar zxvf linux-4.11-rc8.tar.xz

now let's also ensure that we are going to have the relevant utilities to compile the kernel:

yum groupinstall "Development Tools" "Development Libraries" aarch64-linux-gnu-gcc

Ensure that the kernel .config file is clean / in it's default state with:

make mrproper

The kernel config files (defconfig) are located within:


Within the Pi Foundation upstream kernel tree you can get hold of bcmrpi_defconfig - which as it stands seems to be the most stable configuration - however as i'm trying to make this generic as possible I am going to use the default defconfig for ARM64.

make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig

We should also backup the .config file we have generated so we don't lose it next time we cleanup the configuration:

cp .config backup-conf.txt

and finally compile the kernel:

make -j2 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-

(where '-j' defines how many cores you wish to utilise during the compilation.)

We should then find the kernel in arch/arm64/boot/Image.gz

Building the root filesystem

For the rootfs I will be using busybox (so I don't over complicate things) - the latest version is currently 1.26.2:

cd /home/limited/workspace

and again we will cross-compile busybox:

cd busybox*
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig

or for the GUI config:

make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig

and then compile it with:

mkdir /home/limited/workspace/rootfs
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- install CONFIG_PREFIX=/home/limited/workspace/rootfs

* Specifying the 'CONFIG_PREFIX' allows us to specify where the structure / root for the compiled files will end up. *

This is where everything failed for me - the compiler started complaining about missing glibc headers - however it turns out that Fedora does not provide these as the cross platform toolchain only works for compiling kernels - not userspace programs!

So I ended up download Debian Stretch (currently testing) to compile busybox instead.

The Debian package is called: gcc-aarch64-linux-gnu

sudo apt-get install gcc-aarch64-linux-gnu

and attempt to compile as above.

We also need to ensure that we have the appropriate shared libraries for busybox - usually i'd just use ldd on the executable - however I would need to run an arm version on ldd to get this working and because i'm feeling lazy i'm going to cheat a little and install the glibc library:

cd /home/limited/workspace
tar xvf glibc*
mkdir buildc && cd buildc
../glibc-2.25/configure aarch64-linux-gnu- --target=aarch64-linux-gnu- --build=i686-pc-linux-gnu --prefix= --enable-add-ons
make install install_root=/home/limited/workspace/rootfs

We also need to create the directory structure for the rootfs:

mkdir proc sys dev etc/init.d usr/lib

We also need to ensure that the /proc and /sys filesystems mount on boot and that the dev nodes are populated:

vi etc/init.d/rcS

and add the following:

mount -t proc none /proc
mount -t sysfs none /sys
echo /sbin/mdev > /proc/sys/kernel/hotplug
/sbin/mdev -s

ensuring it's also executable:

chmod +x etc/init.d/rcS

TODO: Add user / SSH support.

Testing with QEMU

We should have a pretty bare bones filesystem - although we'll spin it up with QEMU firstly to ensure that everything comes up ok:

qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 512 -kernel Image --append "console=ttyAMA0" -initrd rootfs.img -append "root=/dev/ram rdinit=/sbin/init"

* Note: The last bit (append) is very important - as it instructs the kernel to use the the inird system as the root and ensures that the first program to run is /sbin/init. *

Testing on the Raspberry Pi

We'll now move the filesystem over to a new disk, along with the kernel and grub.

Our disk will have a 1GB boot partition formatted with FAT32 and a root partition of 15GB (we will skip swap etc. for this tutorial.)

Install GRUB to the new disk:

sudo grub-install --target=arm64-efi /dev/sdb

Build busybox for ARM:
Raspberry Pi Foundation:


Post a Comment