One of the first things I wanted to do with the new Raspberry Pi (3) was create my own (simple) Linux distribution.
The Pi 3 is based on a BCM2837 SoC - which has a 64-bit ARMv8 CPU (opposed to ARMv7 in the Pi 2) - however the upstream kernels provided from the Pi Foundation are unfortunately all 32-bit - so for this tutorial I will concentrate on providing a 64-bit kernel so we can fully utilise its power.
We will firstly need to understand how the boot process works with Raspberry Pies - as unlike a normal desktop computer which use a BIOS to initiate a bootloader such as Grub - instead the Rasperry Pi has a closed source firmware in the SoC (System on a Chip).
This firmware is read-only / can't be modified in any way - this enables the second-stage bootloader to be read from a FAT32 formatted partition on the SD-Card.
The second-stage bootloader (bootcode.bin) is used to retrieve and program the GPU firmware (start.elf) from the SD-Card, as well as starting up the CPU. There is also a additional file called fixup.dat that configures the SDRAM between the GOU and CPU.
A kernel is then loaded - by default (on the Rapsberry Pi 3) this is named either kernel7.img (32 bit) or kernel8.img (64 bit) and is a Linux kernel - however of course this doesn't necessarily have to be.
The three files above (bootcode.bin, fixup.dat and kernelX.img) are required as a minimum in order to get the Pi up and running.
For a more detailed overview of how the boot process works please see this article.
The Pi Foundation maintains its own kernel tree for the Pi - which as of right now is 4.9 - however the mainline kernel version also works pretty well too!
To start with lets firstly obtain the latest vanilla / mainline kernel - which at this moment is 4.11 - we can download this from here:
https://cdn.kernel.org/pub/linux/kernel/v4.x/testing/linux-4.11-rc8.tar.xz
and then cross-compile it - I am going to be using Fedora for this - however a lot of people also do this on Debian / Ubuntu:
mkdir /home/user/workspace
wget https://cdn.kernel.org/pub/linux/kernel/v4.x/testing/linux-4.11-rc8.tar.xz
tar zxvf linux-4.11-rc8.tar.xz
now let's also ensure that we are going to have the relevant utilities to compile the kernel:
yum groupinstall "Development Tools" "Development Libraries" aarch64-linux-gnu-gcc
Ensure that the kernel .config file is clean / in it's default state with:
make mrproper
The kernel config files (defconfig) are located within:
arch/arm64/configs
Within the Pi Foundation upstream kernel tree you can get hold of bcmrpi_defconfig - which as it stands seems to be the most stable configuration - however as i'm trying to make this generic as possible I am going to use the default defconfig for ARM64.
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
We should also backup the .config file we have generated so we don't lose it next time we cleanup the configuration:
cp .config backup-conf.txt
and finally compile the kernel:
make -j2 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
(where '-j' defines how many cores you wish to utilise during the compilation.)
We should then find the kernel in arch/arm64/boot/Image.gz
Building the root filesystem
For the rootfs I will be using busybox (so I don't over complicate things) - the latest version is currently 1.26.2:
cd /home/limited/workspace
wget https://www.busybox.net/downloads/busybox-1.26.2.tar.bz2
and again we will cross-compile busybox:
cd busybox*
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
or for the GUI config:
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig
and then compile it with:
mkdir /home/limited/workspace/rootfs
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- install CONFIG_PREFIX=/home/limited/workspace/rootfs
* Specifying the 'CONFIG_PREFIX' allows us to specify where the structure / root for the compiled files will end up. *
This is where everything failed for me - the compiler started complaining about missing glibc headers - however it turns out that Fedora does not provide these as the cross platform toolchain only works for compiling kernels - not userspace programs!
So I ended up download Debian Stretch (currently testing) to compile busybox instead.
The Debian package is called: gcc-aarch64-linux-gnu
sudo apt-get install gcc-aarch64-linux-gnu
and attempt to compile as above.
We also need to ensure that we have the appropriate shared libraries for busybox - usually i'd just use ldd on the executable - however I would need to run an arm version on ldd to get this working and because i'm feeling lazy i'm going to cheat a little and install the glibc library:
cd /home/limited/workspace
wget http://ftp.gnu.org/gnu/libc/glibc-2.25.tar.bz2
tar xvf glibc*
mkdir buildc && cd buildc
../glibc-2.25/configure aarch64-linux-gnu- --target=aarch64-linux-gnu- --build=i686-pc-linux-gnu --prefix= --enable-add-ons
make
make install install_root=/home/limited/workspace/rootfs
We also need to create the directory structure for the rootfs:
mkdir proc sys dev etc/init.d usr/lib
We also need to ensure that the /proc and /sys filesystems mount on boot and that the dev nodes are populated:
vi etc/init.d/rcS
and add the following:
#!bin/sh
mount -t proc none /proc
mount -t sysfs none /sys
echo /sbin/mdev > /proc/sys/kernel/hotplug
/sbin/mdev -s
ensuring it's also executable:
chmod +x etc/init.d/rcS
TODO: Add user / SSH support.
Testing with QEMU
We should have a pretty bare bones filesystem - although we'll spin it up with QEMU firstly to ensure that everything comes up ok:
qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 512 -kernel Image --append "console=ttyAMA0" -initrd rootfs.img -append "root=/dev/ram rdinit=/sbin/init"
* Note: The last bit (append) is very important - as it instructs the kernel to use the the inird system as the root and ensures that the first program to run is /sbin/init. *
Testing on the Raspberry Pi
We'll now move the filesystem over to a new disk, along with the kernel and grub.
Our disk will have a 1GB boot partition formatted with FAT32 and a root partition of 15GB (we will skip swap etc. for this tutorial.)
Install GRUB to the new disk:
sudo grub-install --target=arm64-efi /dev/sdb
Sources:
Build busybox for ARM: http://wiki.beyondlogic.org/index.php?title=Cross_Compiling_BusyBox_for_ARM
Raspberry Pi Foundation: https://www.raspberrypi.org/documentation/linux/kernel/building.md
0 comments:
Post a Comment