# Running syzkaller on `arm64` Linux targets *Reply-to:* - Will Deacon `` *Revision history:* - 03/12/19 - Initial draft *** THIS IS A WORK-IN-PROGRESS AND IS CURRENTLY INCOMPLETE! *** *** ## Disclaimer This guide assumes that you have some familiarity with Linux systems, an `arm64` development board and a desktop PC running a Debian derivative on the same local network. ## Introduction Effectively testing a large, complex, collaborative project such as the Linux kernel is an exercise fraught with difficulties: Which tree should be the testing focus? How can code coverage be guaranteed across the configuration space? Is there a reliance upon implicit behaviours between subsystems? Does the code run as expected across multiple machines or architectures? Is there the potential for undefined behaviour? Consequently, ensuring that even a seemingly trivial piece of kernel code works as intended relies heavily on bug reports from users and results from targeted tests. This approach clearly has its limits and inevitably tends to focus on maintaining stability of "common-case" functionality so that routine operations on mainstream architectures rarely suffer from visible regressions between major releases of the kernel. Outside of this scope, however, regressions and bugs are just as significant when it comes to establishing security and portability guarantees of the kernel. [Fuzzing] [1] is a largely automated technique that can be used to explore these unusual corners of the kernel methodically and has been adopted by the hugely successful [syzkaller] [2] project to find [hundreds of bugs] [3] in the mainline kernel, many of which turned out not to be so esoteric after all. This document is a work-in-progress guide for setting up a syzkaller system targetting the `arm64` architecture (`AArch64` if you're fancy) based on my own experience trying to do this, and finding it a bit more difficult than I would have liked. ## Syzkaller design Although it's probably possible to run syzkaller entirely on the machine being tested, it also wouldn't be much use because if when the kernel crashes, syzkaller itself will die and you'll be stuck without much in the way of debugging information required to identify the cause of the problem. A much better option is to run the syzkaller tests inside a virtual machine (VM) so that the damage is *hopefully* contained to that VM instance. This also means you can run the sucker as root without it accidentally trashing your data or going crazy and uploading your SSH keys to the web. To accomplish this, syzkaller provides a tool called the `syz-manager`, which is responsible for interacting with target VMs by sending them binaries to execute and collecting their output. The `syz-manager` process also reports the progress of each target via a local webserver and, on detecting a crash, works to generate a reproducer binary for analysis. This is a fairly intensive task, so I would recommend running the `syz-manager` on your desktop machine, communicating with the targets over the local network. To summarise, the components we're going to configure are as follows: - The *manager*: an `x86` machine for the heavy lifting. If you're fortunate enough to have access to one of those fabled 'ARM servers', you could probably use that, but I'll assume `x86` because it means we're going to do a spot of [cross compilation] [4]. Hostname: `manager`. - The *target (host)*: an `arm64` board running Linux and KVM. Hostname: `target-host`. - The *target (guest)*: a small VM running on the target (host). This is the kernel being tested. Hostname: `target-guest`. ## Configuring the manager ### golang Syzkaller is written in [`Go`] [5] and requires a toolchain version of 1.11+. The build packaged by recent distributions (e.g. Debian [buster] [6]) should be sufficient to install using: manager:~# apt-get install golang-go Failing that, you can try an [official prebuilt binary] [7]. Once it's installed, you should be able to run: manager:~$ go version go version go1.13.4 linux/amd64 ### Cross-compiler Before we can build syzkaller, we need to get ourselves an `arm64` cross-compiler. Thankfully, we can grab pre-built binaries from [arm.com] [8], although I have little faith in the link remaining stable. Make sure you grab the one named '*x86_64 Linux hosted cross compiler for the AArch64 GNU/Linux target (aarch64-linux-gnu)*': manager:~$ wget "https://developer.arm.com/-/media/Files/downloads/gnu-a/8.3-2019.03/binrel/gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz" manager:~$ tar -xJf gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz You will then need to ensure that the contents of `gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu/bin/` are visible on your `$PATH`. Alternatively, you might be able to use the host `clang`, but I haven't tried that myself. Once successful, you should be able to run something like: manager:~$ aarch64-linux-gnu-gcc --version aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0 Copyright © 2018 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ### syzkaller Then it's time to pull down the syzkaller sources and build 'em. As it turns out, `go` already knows how to do this for us: manager:~$ go get -u -d github.com/google/syzkaller/... manager:~$ cd ~/go/src/github.com/google/syzkaller/ manager:~/go/src/github.com/google/syzkaller$ make TARGETARCH=arm64 Once that's done, you should see a mixture of `x86` and `arm64` binaries for the manager and the target respectively: manager:~/go/src/github.com/google/syzkaller$ file bin/syz-manager bin/syz-manager: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped manager:~/go/src/github.com/google/syzkaller$ file bin/linux_arm64/syz-executor bin/linux_arm64/syz-executor: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, for GNU/Linux 3.7.0, BuildID[sha1]=0740fa499330983c0fddcbfb477ca6f90e601510, with debug_info, not stripped Eventually, the static binaries will be sent over to the target and executed there. Cool, huh? ## Configuring the target (host) It's unfortunately difficult to write a generic guide for configuring an arbitrary `arm64` board to act as a host machine because so many important details vary wildly between different embedded systems. Even silly things like the serial port are designed like it's the wild west, with incompatible modifications all over the place and vendors competing for the highest baud rate as though it's some sort of [pissing contest] [9]. Instead, I'll list some of the requirements you need to satisfy and then I'll explain how I configured the vastly underpowered [*La Frite*] [10] board which I received as a freebie from the thoroughly excellent [Kernel Recipes] [11] conference. If you're ever given the opportunity to attend, then you definitely should. The target (host) system needs to satisfy at least all of the following: - An `arm64` CPU booting in 64-bit mode. - 1GB of memory. - 8GB of persistent storage. This can be a simple USB stick or SD card and you might get away with something smaller if you're frugal with the filesystem. - A functional ethernet interface. - A hardware random number generator or some other reliable source of entropy. - Preferably running some form of Linux distribution since you need to get `sshd` up and running. - **Important:** A bootloader that enters the kernel at `EL2`. Look for a line in the kernel `dmesg` that reads '` CPU: All CPU(s) started at EL2`'. If you can't find it, hurl the thing in the bin and organise a protest. From this point on, I'm going to assume the target (host) is running Debian and that you have access to a root shell. If you're already in that state, then skip ahead to configuring KVMtool. *** ### La Frite 'La Frite' is a low-powered, low-cost development board based on the [`S805x`] [12] SoC from Amlogic. It just about ticks the boxes above but, more importantly, I had a couple kicking around to sacrifice to the syzkaller gods. If you feel to the need to purchase one, they [appear to be on sale] [13]. #### Assembly Nope, not assembly *language* but actual assembly of the board and its cheap aluminium chassis instead. If you have trouble then your best bet is probably to look at the images I've linked to in this section. The process requires mounting the eMMC on the PCB before applying the [thermal heatpad](img/board-side.jpg) to the SoC itself and clamping shut with the two outer plates. The final touches are some [delightful rubber feet](img/board-bottom.jpg) that don't seem to serve any real purpose. [Looking down at the completed enclosure](img/board-top.jpg), the 40-pin GPIO header should be visible with the 'u-boot button' accessible in the far corner near the USB ports. Make sure you have your questionably-standards-compliant USB A-to-A cable handy, since you'll need it in the next step. **Word of warning!** Rumour has it that the eMMC spacers are *directional* and, if mounted incorrectly, could lead to intermittent eMMC disconnects and/or probe failures during boot. [See what you think](img/emmc-spacers.jpg) but I couldn't spot the directionality although my eyesight is, frankly, appalling. However, I did experience some issues with eMMC probing and so ended up wrapping it in tape (why not?) which seems to have helped a bit, but hasn't completely solved the problem. Hmm. Did I mention the board was free? #### Cables On the cables front, you're going to need: - The dodgy USB A-to-A cable I mentioned earlier. - A standard 3.3V serial adapter so you can connect TX/RX/GND to the GPIO header. The FTDI chip has always worked best for me. - A micro-USB power supply (5V, 2.5A). - An ethernet cable. Link the serial adapter to the UART pins exposed on the GPIO header as follows: - TX to pin 3 (this is RX on the SoC) - RX to pin 5 (this is TX on the SoC) - GND to pin 6 The silkscreen has a little arrow to identify pin 1, with pins 39 and 40 being numbered at the other end so you can figure out how the sequence works. The baud rate is an impressively modest 115200. Don't hook up the other cables just yet. #### Flashing firmware Although the board comes with some pre-installed firmware, there were some eMMC issues when running with earlier versions so it's best just to nuke it with the latest and greatest before going any further. Before we can do that, we need to grab a copy of the bespoke flashing tool: manager:~$ git clone https://github.com/libre-computer-project/pyamlboot.git manager:~$ cd ~/pyamlboot manager:~/pyamlboot$ git checkout gxl You'll also need the Python USB libraries installed: manager:~/pyamlboot# apt-get install python3-usb **Yet another warning!** If you haven't guessed already, we're about to run a random python script from the internet as `root`. I encourage you to read the thing before doing so. Now, connect one end of the dodgy USB A-to-A cable to your computer, and *with the u-boot button held down* connect the other end to the USB port closest to the 40-pin GPIO header on the board. I found that you didn't need extra power: it should light up like a Christmas tree without the micro-USB connected. If you have the serial console up, you might see some junk: GXL:BL1:9ac50e:bb16dc;FEAT:ADFC318C:0;POC:0;RCY:0;USB:0; If it stops here, then that's Fritese for "Waiting for firmware" and we can finally run that script that you've audited: manager:~/pyamlboot# ./flash-firmware.sh aml-s805x-ac I like the part where it downloads the random temporary file best. Anywho, the serial console should be littered with messages, hopefully finishing with something to indicate that the flashing was either successful or not needed. If it failed, maybe you can try again. #### Installing a root filesystem Disconnect and immediately reconnect the dodgy USB cable: you'll see the firmware booting on the serial console. Hammer `Escape` until it drops you into a menu entitled `*** U-Boot Boot Menu ***`. So good they named it twice. From this menu, select the `eMMC USB Drive Mode` option. You should then see the eMMC show up as a USB mass storage device on your desktop machine: usb 2-2: Manufacturer: Libre Computer usb-storage 2-2:1.0: USB Mass Storage device detected scsi host4: usb-storage 2-2:1.0 scsi 4:0:0:0: Direct-Access Linux UMS disk 0 ffff PQ: 0 ANSI: 2 sd 4:0:0:0: Attached scsi generic sg1 type 0 sd 4:0:0:0: [sdb] 15269888 512-byte logical blocks: (7.82 GB/7.28 GiB) sd 4:0:0:0: [sdb] Attached SCSI removable disk In my case, it's `sdb`, so when running the next few commands take care to substitute that with whatever you ended up with. Grab a pre-baked Debian image for flashing: manager:~$ wget http://share.loverpi.com/board/libre-computer-project/libre-computer-board/image/debian/libre-computer-aml-s805x-ac-debian-buster-headless-4.19.64%2B-2019-08-05.zip manager:~$ unzip libre-computer-aml-s805x-ac-debian-buster-headless-4.19.64+-2019-08-05.zip manager:~# dd if=libre-computer-aml-s805x-ac-debian-buster-headless-4.19.64+-2019-08-05.img of=/dev/sdb bs=4M This can take a little while, so put the kettle on and come back later (it takes about 2 mins). When it's completed, disconnect the USB cable and discard it. #### Initial configuration Connect the ethernet and mini-USB power supply. After a few seconds, you should see the Linux kernel booting at last. You'll get dropped to a login after `systemd` has done its thing: the credentials are `libre:computer` . Although this works as a basic setup, I tweaked it slightly to make it a bit more friendly. I recommend you do the same: #### Change the default user and group names #### libre-computer:~$ sudo su libre-computer:~# echo ttyAML0 >> /etc/securetty libre-computer:~# passwd # libre-computer:~# exit libre-computer:~$ exit # # You should customise this unless you're also called Will libre-computer:~# usermod -l will -d /home/will -c "" -m libre libre-computer:~# groupmod -n will libre libre-computer:~# exit # libre-computer:~$ passwd # #### Update the system #### libre-computer:~# apt-get update libre-computer:~# apt-get dist-upgrade # #### Change the hostname #### libre-computer:~# echo "target-host" > /etc/hostname libre-computer:~# sed -i 's/libre-computer/target-host/' /etc/hosts #### Avoid silly timeout when mounting swap #### libre-computer:~# vim /etc/fstab # #### Avoid reboot taking ages thanks to the NIC not shutting down properly #### libre-computer:~# vim /etc/systemd/system.conf # # Note that this won't take effect until after a reboot #### Enable the GRUB menu during boot #### libre-computer:~# vim /etc/default/grub # libre-computer:~# update-grub #### Reboot the system #### libre-computer:~# reboot #### Mainline kernel (optional) This isn't required by syzkaller, but I thought I'd include it here because it's fairly straightforward once you've got this far and it might save you from being stuck forever on the 4.19 kernel shipped with the Debian filesystem. Take a copy of the kernel sources on your desktop: manager:~$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git manager:~$ cd linux I've had success with v5.4, but later releases should work too: manager:~/linux$ git reset --hard v5.4 Then simply build a `defconfig` Debian kernel package: manager:~/linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig manager:~/linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j $(nproc) bindeb-pkg For some reason, this puts all of the output in the parent directory, so it will make a mess there. You might decide to clean it up later on. I remember getting annoyed in the early days of ACPI on `arm64` when the enterprise folks would poke fun at the community for continuously breaking the devicetree bindings used by the kernel. Well, it turns out they were right, so we need to update the kernel and the devicetree blob at the same time. We'll `scp` them over to the target from the build machine: manager:~/linux$ scp arch/arm64/boot/dts/amlogic/meson-gxl-s805x-libretech-ac.dtb ../linux-image-5.4.0_5.4.0-1_arm64.deb 10.0.0.251:~/ Then on the target: target-host:~# cp meson-gxl-s805x-libretech-ac.dtb /boot/efi/dtb/libre-computer/aml-s805x-ac/platform.dtb target-host:~# dpkg -i linux-image-5.4.0_5.4.0-1_arm64.deb Say a short prayer, then reboot. If all goes well, you'll boot back into the mainline kernel and everything should work. There's a big `[FAILED]` entry in the `systemd` log for an LED Trigger, but I haven't bothered to figure out what's going on there because I don't care and it still glows too much for my liking even without whatever is broken. #### Support I am by no means an expert on this board, I just wasted a weekend beating it into submission. The real experts are reachable via IRC at `#librecomputer` and `#linux-amlogic` on [freenode] [14]. I'm grateful for the help they gave me when I was about to reach for the hammer. There is also a *tonne* of information in the dedicated forums over at [loverpi] [15]. *** ### KVMtool You could use [QEMU] [16] here instead if you prefer, but I found with the limited eMMC space on my target, it was just a little large when installing the version packaged with Debian. Before we go any further, we need to grab a bunch of essential utilities and dependencies: target-host:~$ apt-get install gcc git libfdt-dev make Then grab the sources: target-host:~$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git target-host:~$ cd kvmtool target-host:~/kvmtool$ make -j4 Before we give our new binary a go, we'll ensure that we're in the right group: target-host:~$ sudo usermod -a -G kvm,netdev $(whoami) then log out and back in again. If you built your own mainline kernel as described earlier, then you can give it a spin along the lines of: target-host:~/kvmtool$ ./lkvm run -p "earlycon" <(zcat /boot/vmlinuz-5.4.0) Which should boot to a basic guest shell. ### Network bridge The manager is going to want to `ssh` into both the guest (for running tests) and the host (for observing a crashed guest). This means we need to set up a network bridge on the host so that the two are independently addressable. We can do this by bodging a new network interface using `macvtap`: target-host:~# cat << EOF > /etc/network/interfaces.d/kvmtap0 auto kvmtap0 iface kvmtap0 inet manual pre-up ip link add link eth0 name kvmtap0 type macvtap mode bridge post-down ip link del kvmtap0 EOF We also need to persuade udev to make the `/dev/tap*` nodes accessible to the `netdev` group: target-host:~# cat << EOF > /etc/udev/rules.d/99-tap.rules KERNEL=="tap[0-9]*",GROUP="netdev",MODE="0660" EOF The interface should appear magically following a reboot, but we can also just raise it now: target-host:~# service udev restart target-host:~# ifup kvmtap0 **Note:** Although this will allow the manager machine to communicate with both the target host and the target guest on the local network, due to the way that `macvtap` works, the guest will *not* be visible to the host. ## Configuring the target (guest) The guest doesn't need a lot, other than a specially-configured kernel and a filesystem with an SSH daemon. ### Kernel configuration If you already have a kernel source tree after building a mainline kernel for 'La Frite' earlier on, then we can reuse that. Otherwise, you'll want to pull them down onto your desktop box: manager:~$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git manager:~$ cd linux This is the kernel that we'll be fuzzing, so now is the time to apply whatever untested rubbish you have on top. We'll use `defconfig` as a base: manager:~/linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig Syzkaller requires some [additional kernel configuration options] [17] to operate usefully: # Enable KCOV manager:~/linux$ ./scripts/config -e KCOV -e KCOV_INSTRUMENT_ALL -e KCOV_ENABLE_COMPARISONS # Enable KASAN manager:~/linux$ ./scripts/config -e KASAN -e KASAN_INLINE # Enable fault injection manager:~/linux$ ./scripts/config -e FAULT_INJECTION -e FAULT_INJECTION_DEBUG_FS -e FAILSLAB -e FAIL_PAGE_ALLOC -e FAIL_MAKE_REQUEST -e FAIL_IO_TIMEOUT -e FAIL_FUTEX # Enable kernel debugging options manager:~/linux$ ./scripts/config -e LOCKDEP -e PROVE_LOCKING -e DEBUG_ATOMIC_SLEEP -e PROVE_RCU -e DEBUG_VM -e FORTIFY_SOURCE -e HARDENED_USERCOPY -e LOCKUP_DETECTOR -e SOFTLOCKUP_DETECTOR -e HARDLOCKUP_DETECTOR -e BOOTPARAM_HARDLOCKUP_PANIC -e DETECT_HUNG_TASK -e WQ_WATCHDOG # Enable virtio-rng manager:~/linux$ ./scripts/config -e HW_RANDOM -e HW_RANDOM_VIRTIO At which point we can build a kernel fit for fuzzing: manager:~/linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- olddefconfig manager:~/linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j $(nproc) Image Then transfer the `Image` to the `arm64` host: manager:~/linux$ scp arch/arm64/boot/Image target-host:~/ ### Minimal root filesystem Although you could probably build something manually with `busybox`, I find it easiest just to use Debian once again. You can build the image directly on the target, but first we need an empty image with a filesystem: target-host:~$ truncate -s 4G debian-buster-arm64.img target-host:~$ /sbin/mkfs.ext4 debian-buster-arm64.img Building the image this way also means we don't waste disk space, since `truncate` creates a sparse file. We can mount the image as follows: target-host:~# losetup /dev/loop0 debian-buster-arm64.img target-host:~# mount /dev/loop0 /mnt At which point we're ready to create our new filesystem: target-host:~# apt-get install debootstrap target-host:~# debootstrap buster /mnt Once that's done (you might want to grab another cup of tea), we just need to change the root password and hostname so that we can log in: target-host:~# chroot /mnt target-host:/# passwd # target-host:/# echo "target-guest" > /etc/hostname target-host:/# sed -i 's/localhost/localhost target-guest/' /etc/hosts target-host:/# exit Finally, we can clean up: target-host:~# umount /mnt target-host:~# losetup -d /dev/loop0 ### Networking and SSH At this point, we should be able to boot our shiny new guest environment: target-host:~# ./kvmtool/lkvm run -n mode=tap,tapif=/dev/tap$(cat /sys/class/net/kvmtap0/ifindex),guest_mac=$(cat /sys/class/net/kvmtap0/address) --rng -d debian-buster-arm64.img -k Image You can tweak the amount of guest memory and number of virtual CPUs that it has by passing `-m` and `-c` respectively. Once the guest has booted, you can log in on the serial console as root, using the password you chose when creating the filesystem. From there, let's get the network up and running: target-guest:~# echo << EOF > /etc/network/interfaces.d/eth0 auto eth0 iface eth0 inet dhcp EOF target-guest:~# ifup eth0 Unless you're very good at typing passwords, having syzkaller use key-based authentication for its SSH sessions is essential. We'll get `sshd` going with public key authentication for the `root` user, which is how `syz-manager` will connect to the guest later on. From inside the guest: target-guest:~# apt-get install openssh-server target-guest:~# echo "PermitRootLogin yes" >> /etc/ssh/sshd_config target-guest:~# service ssh restart Then back on the `x86` manager machine: manager:~$ ssh-keygen -q -N "" -f syzkaller This will generate a public/private key pair that can be used for authentication. We just need to install the public key on the target (host and guest), which we can do from the comfort of the manager! manager:~$ ssh-copy-id -i syzkaller target-host manager:~$ ssh-copy-id -i syzkaller root@target-guest For each invocation, it should report something along the lines of: Number of key(s) added: 1 Now you can connect to the guest and remove the line we added to `sshd_config` earlier on: manager:~$ ssh -i syzkaller root@target-guest "sed -i '\$d' /etc/ssh/sshd_config" ## Fuzzing like it's 1999 > TODO: Teach syzkaller how to deal with this setup! > We're in a position where the manager can ssh to the target using key > authentication, spawn a guest and then ssh into the guest. ## Known issues / wishlist - Fuzzing the compat layer for an `arm64` kernel - Shell script interface to target - Stack unwinding issues? [1]: https://en.wikipedia.org/wiki/Fuzzing [2]: https://github.com/google/syzkaller [3]: http://events19.linuxfoundation.org/wp-content/uploads/2017/11/Syzbot-and-the-Tale-of-Thousand-Kernel-Bugs-Dmitry-Vyukov-Google.pdf [4]: https://en.wikipedia.org/wiki/Cross_compiler#GCC_and_cross_compilation [5]: https://golang.org [6]: https://packages.debian.org/buster/golang-go [7]: https://golang.org/dl/ [8]: https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads [9]: https://en.wikipedia.org/wiki/Pissing_contest [10]: https://libre.computer/products/boards/aml-s805x-ac/ [11]: https://kernel-recipes.org/ [12]: http://wiki.loverpi.com/soc:amlogic-s805x [13]: https://www.loverpi.com/collections/la-frite [14]: https://freenode.net/ [15]: https://forum.loverpi.com/categories/libre-computer-aml-s805x-ac [16]: https://www.qemu.org/ [17]: https://github.com/google/syzkaller/blob/master/docs/linux/kernel_configs.md