Linux Kernel Toolchain Setup

In the last blog, we looked at how to compile a vulnerable Linux kernel version when you intend to do N-day research for kernel vulnerabilities. But with only the kernel boot image, you cannot run your exploits. You need to simulate a Linux distribution environment. But we would like it to be limited to only the things required for research. In this blog, we will outline how to setup an environment, using various tools, tailored to the needs of the CVE to be exploited. The environment will include a filesystem, networking capabilities and system libraries like libc.

We assume that you are running Ubuntu 24.04 on an x86-64 machine and you have a built Linux kernel image.

Breakdown

To break it down into the things we need, we get a total of 3 things:

1. A machine emulator
2. Filesystem image for the files required
3. Debugging setup

Filesystem

We will start with creating a filesystem image with all the files required for the debugging setup and running Linux with a shell and everything. There are two ways you can go about it:

1. initramfs
2. buildroot

initramfs is a filesystem loaded by the kernel into ram at boot. It is generally used to house critical files for the boot process e.g., LVM disk encryption. It is very easy to hack together to quick start with a vulnerability setup, you don't always have to rebuild it and it generally very small in size.
But it is limited in its flexibility since it needs to be decompressed and loaded in ram on each run. You need to decompress it, add files to it, compress it and run the kernel again. You don't want it to get too big in size because that would make the boot quite slow.

On the other hand, buildroot is a project to create filesystem images with all the functionality you need and in the format you like. It can create ext2 , ext3 and qcow2 images that can be loaded into emulators like a normal hard drive and don't need any compression or decompression for modification. You mount it like a normal image and make the modifications you want. Since it won't load in the main memory, it can be of any size you like. You can also add any libraries and utilities to the image you like to use it as a normal Linux system. All of the cumbersome work of downloading and building the utilities and libraries is handled by buildroot. It's specially useful when you need custom libraries for testing of vulnerabilities in external devices or drivers.

But it can be bit daunting at first compared to initramfs.

I personally prefer buildroot because it is very easy to reproduce and is extremely granular and flexible.

I will outline both methods so you can decide what suits you case better.

git clone https://github.com/mirror/busybox
cd busybox
git checkout $(git tag --sort=committerdate | tail -1)

mkdir build
make O=build menuconfig

cd build
make -j$(nproc) && make install

mkdir initramfs
cd initramfs
mkdir -pv
{bin,dev,sbin,home/user,etc,proc,sys/kernel/debug,usr/{bin,sbin},lib,lib64
,mnt/root,root}
cp -av /path/to/busybox/build/_install/* .

root:x:0:0:root:/root:/bin/sh
user:x:1000:1000:User:/home/user:/bin/sh

root:x:0:
user:x:1000:

#!/bin/sh
mount -t proc none /proc
mount -t sysfs none /sys
mount -t devtmpfs none /dev
echo -e "\nBoot took $(cut -d' ' -f1 /proc/uptime) seconds\n"

setsid /bin/cttyhack setuidgid 1000 /bin/sh

umount /proc
umount /sys
umount /dev
poweroff -d 0 -f

chmod +x init
find . -print0 | cpio -0 -R 0:0 -o -H newc | gzip > ../initramfs.cpio

mkdir fs
cd fs
gzip -d -c -k ../initramfs.cpio.gz | cpio -i

Buildroot

As you just saw, initramfs involves a lot of manual effort to hack together and it is quite raw in nature. This inherently is not maintainable and can be quite cumbersome if we need to do it multiple times.

Buildroot is an alternative for it and is quite flexible. We will create an environment similar to initramfs.

We will start by shallow cloning the buildroot repository.

git clone --depth 1 https://gitlab.com/buildroot.org/buildroot

Now you need to do a few modifications to the config. Run the following command to bring the menu:

make menuconfig

Now change the target architecture to your target's in Target options -> Target architecture -> x86_64 (say) and change the toolchain to external in Toolchain -> Toolchain type -> External toolchain . This makes buildroot use pre-built toolchain rather than building one; building one takes a lot of time.

Now change the root password in System -> Root password and to add an unprivileged user to the image, add system/users.txt path in System -> Path to the users tables .

To create an ext2 image, choose ext2 in Filesystem images -> ext2/3/4 images and change the Filesystem images -> ext2/3/4 -> exact size to whatever you like. Some thing like 200M , 500M or 1G should work fine depending upon the files you want to include in the image. Also disable the tar image, we don't need that.

You can explore other options and tailor them to your liking. You can add custom utilities, shells, libraries and debugging binaries in Target packages . For example, if you want to communicate with netfilter subsystem via netlink protocol, you will need libnftnl and libmnl which can be enabled in Target packages -> Libraries -> Networking . Also standard libraries are included by default.
Some people like to have a ssh server in the image, for that enable Target packages ->
Networking applications -> openssh -> server and Target packages ->
Networking applications -> dhcpd .
Note that in order for this to work, the kernel should have been built with
CONFIG_VIRTIO_PCI=y , CONFIG_PCI=y , CONFIG_VIRTIO_NET=y and CONFIG_VIRTIO=y .
‍
To add the unprivileged user details, create file in system/users.txt with the following contents:

user 1000 user 1000 - /home/user /bin/sh - Unprivileged user

This makes a user named user with group user , UID and GID 1000 , /home/user home directory and no password or additional groups. You can view the syntax here

Now run make -j$(nproc) to build the image, it can take 20 to 60 mins depending on your machine and network. After the build is complete, you will find the build in output/images folder named rootfs.ext2 (by default).

Note: If you run into issue of automake segfaulting, run the following command to updatethe perl interpreter. You will need to rebuild automake as well.

sudo apt install perlbrew
perlbrew init
perlbrew install -n -j $(nproc) perl-5.40.3
perlbrew switch perl-5.40.3
make host-automake-dirclean
make -j$(nproc)

Compared to initramfs, buildroot is quite simple, easy to setup and flexible. This also takes care of doing all the initialization of drivers and filesystem for us. Now to modify the contents,
you can mount the image using the following command:

sudo mount rootfs.ext2 /mnt/foo

Make the modifications in /mnt/foo folder and unmount using the following command:

sudo umount /mnt/foo

You can learn more about buildroot here

Qemu

Now that we have the filesystem image, we need a program to simulate a machine to which we will give the filsystem image to boot from. Qemu is one such tool. Qemu is an incredibly powerful tool that supports emulation and virtualization. Emulation is when your host machine and the simulated machine have different architectures and Virtualization is when they are of the same architectures. Covering everything about Qemu is out of the scope of this post since it supports everything from booting, hard-drives, network cards, wifi modems and peripherals.
We will use Qemu to define features of our simulated machine e.g., cpu capabilities, cores, threads and attach network card and filesystem image.
To install qemu, run the following command in a terminal:

sudo apt-get install -y qemu-system

This will install qemu-system-<arch> binaries for each architecture e.g., qemu-system-x86_64 for x86-64 and qemu-system-arm64 for ARM64.

We will use Qemu to define the following properties of our emulated/virtualized environment:

1. CPU
2. CPU capabilities
3. Memory size
4. Kernel
5. Kernel command line
6. initramfs image
7. Hard drive image
8. Serial interfacing
9. GDB debugging
10. Network card

CPU

The CPU architecture is selected by running the appropriate qemu-system binary e.g., qemu-system-x86_64 for x86-64. To configure the cores and threads of this CPU, you will need -smp switch. You can specify the cores and threads in the format -smp cores= <cores>,threads=<threads> .

CPU Capabilities

For CPU capabilities like SMEP and SMAP, we will use -cpu switch with qemu64 CPU. This will work for x86-64 CPUs, you can also use host for Host CPU passthrough if you are virtualizing. The full list of CPUs can be found here. A CPU with SMEP and SMAP enabled will look like -cpu qemu64,+smep,+smap . Removing ,+smep,+smap will disable these features.

Memory Size

For memory size, you will need to specify -m switch with the amount of memory you want the VM to have. For example, if you want to allocate 1 Gigabyte of memory, you will specify -m 1G .

Kernel

The kernel boot image is given to qemu by using the -kernel switch. Remember, you need to specify path of bzImage / vmlinuz not vmLinux (the ELF file).

Kernel Command Line

Kernel command line means the boot options we give to the kernel to orchestrate its runtime behavior. This includes (but not limited to) specifying the root device, console, logging level, KASLR, KPTI, OOPS, panic etc. This is done by giving the options as a string to the - append qemu switch. For the complete list of options, check out kernel docs. An example command line will look like root=/dev/sda loglevel=3 quiet . This specifies /dev/sda as the root device, log level to be critical and not print boot messages.

Interesting options

The following options are the ones you will need to vulnerability research and exploit development.

1. console
   console is used to specify the console where the interactions will take place. In our setup, we need it to be the serial interface exposed by qemu. console=ttyS0
2. init
   init tells the kernel where the init script is placed. By default, it executes /sbin/init . You can make it a symlink to the init script or, alternatively, specify the init script path using init boot option. In the above description of creation of images, you don't need to specify it. init=/init
3. root
   root tells the kernel what filesystem is used as the root filesystem. You don't need to specify it for initramfs but for buildroot you will need it. root=/dev/sda
4. loglevel
   loglevel tells the kernel what verbose logging level to use. It can be helpful in cases where you want to look at the logs or troubleshoot. Its range is 0-7 , 7 being the most verbose. loglevel=3
5. earlyprintk
   earlyprintk is useful when the kernel crashes before the normal console is initialized. In order to enable it, we need to specify the serial console. earlyprintk=serial
6. debug
   debug is an alias for loglevel=7 .
7. quiet
   quiet is an alias for loglevel=4
8. nokaslr
   nokaslr disables the Kernel Address Space Layout Randomization (KASLR). A must have, otherwise your vmlinux symbols will not match
9. oops
   An OOPS is some unexpected behavior in Linux kernel that can potentially make the kernel unstable or dysfunctional. The kernel tries to continue by killing the offending process but during exploit development, you don't want that. You want the crash to halt the kernel. This is called panic in Linux kernel. To alter this, specify oops=panic
10. panic
    panic option tells the kernel the time to wait before rebooting when a panic occurs. panic=1
11. 11. pti / kpti
    pti and kpti deal with Kernel Page Table Isolation (KPTI) that separates the kernel's virtual memory page table from the user-space's virtual memory page table. To turn it on, pti=on OR kpti=1 should be added in the command line. And to turn it off, pti=off OR kpti=0 should be added instead.
12. nosmep & nosmap
    nosmep and nosmap are used to disable Supervisor Mode Execution Protection and Supervisor Mode Access Protection respectively, even if you added the capabilities in the CPU option.

initramfs and hard drive image

If you are using an initramfs image, you will specify it by using -initrd option. If you are using a hard drive image (ext2, qcow2), you can specify the image path using - hda option.

Serial interfacing

By default, qemu start a separate UI for the logs and interfacing. But we don't need this, we want the qemu process to start right in shell. You can do this by specifying -nographic flag. Qemu also provides an interactive interface, called Qemu monitor. This allows you to check various registers and alter the state. If you want to use it, -monitor stdio should be given to qemu. If you want to disable it, set the monitor to /dev/null .
If you want to disable reboots (for example for panics), specify -no-reboot .

GDB

For debugging the kernel, we will use Gnu DeBugger (GDB). It comes with almost all Linux distributions and supports various features that will be helpful. If you don't know about GDB, you can learn about it here. GDB allows connecting to remote servers by specifying the remote server's host and port. GDB server interface is implemented by qemu. In order to enable it, you need to specify -gdb tcp::<port> option with your preferred port. There is - s switch which is a shortcut for -gdb tcp::1234 . After running qemu, you will need to run target remote :1234 in GDB if you are running on the same host.
GDB will attach to the kernel when you give it the target command. But what if you need to debug the early boot process, you can specify -S switch to halt the execution of qemu until GDB is connected.

Network card

We will use the network card to access the ssh server that we added to the buildroot image. For this, we need to forward the guest's ssh port to the host.

-net nic -net user,hostfwd=tcp::<host-port>-:22

To expose the ssh port to host on 2220 port

-net nic -net user,hostfwd=tcp::2220-:22

Example usage with initramfs

#!/bin/sh
ARCH="x86_64"
CPU="qemu64,+smep,+smap"
CORES=2
THREADS=2
MEM="256M"
KERNEL="linux-5.16.10/arch/x86/boot/bzImage"
KERNELCMD="console=ttyS0 quiet oops=panic panic=1"
INITRAMFS="initramfs.cpio.gz"
GDBPORT=1337
qemu-system-$ARCH -nographic -no-reboot \
 -cpu $CPU \
 -smp cores=$CORES,threads=$THREADS \
 -m $MEM \
 -kernel $KERNEL \
 -initrd $INITRAMFS \
 -append "$KERNELCMD" \
 -gdb tcp::$GDBPORT

Example usage with buildroot with ssh

#!/bin/sh
ARCH="x86_64"
CPU="qemu64,+smep,+smap"
CORES=2
THREADS=2
MEM="256M"
KERNEL="linux-5.16.10/arch/x86/boot/bzImage"
KERNELCMD="console=ttyS0 root=/dev/sda quiet oops=panic panic=1"
IMG="rootfs.ext2"
GDBPORT=1337
SSHPORT=2220
qemu-system-$ARCH -nographic -no-reboot \
 -cpu $CPU \
 -smp cores=$CORES,threads=$THREADS \
 -m $MEM \
 -kernel $KERNEL \
 -hda $IMG \
 -append "$KERNELCMD" \
 -net nic -net user,hostfwd=tcp::$SSHPORT-:22 \
 -gdb tcp::$GDBPORT

Debugging Setup

Now that you have kernel with a filesystem running attached with gdb. But vanilla gdb doesn't provide good enough feature out of the box to help us with kernel debugging. For example, if you are debugging a kernel module, you will have to manually find all sections' address using the /sys/module/<module>/sections/<section> extension and then manually plug those into gdb.

Kernel's GDB scripts

Fortunately, you don't have to do this manually. Linux kernel provides a few scripts that helpyou with usual debugging tasks in gdb. You will find this script in the source root namedvmlinux-gdb.py . You can run this by running source vmlinux-gdb.py in gdb.
This file is not generated by default. In order to generate it, you need to compile the kernel with CONFIG_DEBUG_INFO=y and CONFIG_GDB_SCRIPTS=y . You can generate the scripts later with make scripts_gdb

This script adds the following non-exhaustive list of useful commands/functions:

1. lx-symbols
   (Re-)loads symbols of kernel and loaded modules. Saves a lot of manual work when trying to find a module's symbols especially
2. lx-lsmod
   Lists loaded modules in the kernel with their base address and other modules that reference one
3. lx-dmesg
   Prints the kernel's dmesg buffer's contents
4. lx-version
   Prints the kernel version header. Can also be obtained using file bzImage
5. lx-ps
   Prints running processes with their PIDs and task address
6. lx-mounts
   Prints VFS mounts with their mount, uper block address, devname, path and options
7. lx-cpus
   Prints available CPUs info (online, active, present)
8. lx-cmdline
   Prints the kernel command line (supplied using -append in qemu)
9. $lx_current()
   This function returns the current process' task address
10. $lx_module()
    This returns module's base address when given its name. $lx_module("hax")
11. $lx_per_cpu()
    Returns the supplied variable name from a CPU's (default current) per-cpu variable list.
    $lx_per_cpu([, cpu number])
12. $lx_task_by_pid()
    Returns the task struct of the PID supplied
13. $lx_thread_info()/$lx_thread_info_by_pid()
    Returns the address of thread_info of a given PID. This includes the thread's flags, syscalls done, status and CPU its running on.

Third-party GDB scripts

In addition to the kernel's GDB scripts, other GDB scripts also provide helpful kernel debugging and exploit development functionalities.

1. bata24-gef
   This script has extensive features for kernel debugging, kmalloc heap analysis, module search, physical pages and many more. This uses heuristics to determine stuff so it can be used when you don't have a kernel build without debugging symbols. But it tends to be unstable, so you may have to edit it based on the commands you want to use.
2. Pwndbg
   Pwndbg has a kernel specific section. It provides commands for Android's binderfs, Buddy allocator, netfilter tables, MSRs, basic slab info and some other useful features.
3. GEF
   Although the core gef doesn't have any kernel specific commands, gef-extras has commands for ftrace and for finding kernel symbols

Helpful tip: Struct's members

When developing exploits, more often than not, we need some struct's members offsets. You can find the offsets of a particular struct using either pahole utility or GDB.
For pahole running the following command to get members and their offsets and sizes for a particular struct

pahole -C <struct name> vmlinux

For gdb, first launch gdb using gdb vmlinux then run the following command:

(gdb) ptype/o struct <struct name>

Kernel's configs reference

# initramfs
CONFIG_BLK_DEV_INITRD=y
# Buildroot with ssh
CONFIG_PCI=y
CONFIG_VIRTIO=y
CONFIG_VIRTIO_NET=y
CONFIG_VIRTIO_PCI=y
# Debugging scripts
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
CONFIG_GDB_SCRIPTS=y

Summary

In this blog, we looked at how to setup a filesystem for kernel and how to run the kernel with it, optionally with ssh capability.

Then we looked at how you can use different tools to simplify the debugging process using the kernel's GDB scripts and third-party GDB scripts with a note on how to determine offsets of struct members.\

In the next blog, we will do a Root Cause Analysis (RCA) of a kernel vulnerability walking you through the process using the same steps we outlined in the previous blog.

Stay tuned!