Debugging the Linux kernel is the bedrock of any activity on the kernel, from learning newbies to expert contributors activities. There are many ways to debug a kernel, such as printing messages on the console or on log files, using pseudo-breakpoints where printing is not feasible, using a virtual machine and so on.

Debugging the kernel of a running operating system may be tricky, but now the Qemu emulator supports cross-platform kernel and module debugging at, avoiding to mess up with the running kernel on your computer and do that in a controlled environment.

Target Environment

Architecture: x86_64
Kernel version: mainline
Debugger: GDB
OS: Ubuntu

Install QEMU and GDB

Luckily, there are packages available for multiple distros of Qemu

# For Debian/Ubuntu distributions
$ sudo apt install qemu-system

# For Arch distributions
$ sudo pacman -S qemu

and GDB

# For Debian/Ubuntu distributions
$ sudo apt install gdb

# For Arch distributions
$ pacman -S gdb

Syzkaller and Syzbot

Syzkaller is an unsupervised coverage-guided kernel fuzzer, which test the kernel functionalities with different configurations and inputs in order to find bugs through dynamic analysis. Initialy, developed for the Linux kernel, nowadays, it supports different kernels (Android, FreeBSD, etc.)

Syzbot is higher-level automation on top of Syzkaller that:

continously build the kernel
run multiple Syzkaller instances to test all the different kernels
reports found bugs and track them on the web ui

For the purpose of this article, we will setup the debugging, generaly, for the mainline Kernel version, without taking any bug reported on Syzbot website. Btw a very similar setup is needed in case of walking through a real bug from Syzbot: you only need to edit the vmlinux and kernel image related to the bug report page accordingly.

Setup Syzkaller and Kernel Image

Let’s start by cloning and building syzkaller:

$ cd $HOME
$ git clone https://github.com/google/syzkaller
$ cd syzkaller
$ make

Create the kernel image:

# inside the syzkaller folder
cd tools
chmod +x create-image.sh
./create-image.sh
# Add --distribution buster to specify the distribution: ./create-image.sh --distribution buster

and you should locate a new image called bullseye.img within the folder.

Build the Kernel

Clone the latest mainline Linux release from the The Linux Kernel Archives web page

$ cd $HOME
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux_mainline
$ cd linux_mainline

Edit kernel configuration file .conf in the root of the tree, adding:

# Coverage collection.
CONFIG_KCOV=y

# Debug enable
CONFIG_DEBUG_INFO=y

# Debug info for symbolization.
CONFIG_DEBUG_INFO_DWARF4=y

# Memory bug detector
CONFIG_KASAN=y
CONFIG_KASAN_INLINE=y

# Required for Debian Stretch and later
CONFIG_CONFIGFS_FS=y
CONFIG_SECURITYFS=y

Generate the kernel configuration file based to the current configuration and build the kernel:

# Update configuration
$ make oldconfig

# Build the kernel
$ make -j`nproc` # Use nproc just to optimize the build according to your processing unit.

Spawn the Qemu VM

Open a terminal and start the Qemu GDB server:

$ export KERNEL=linux_mainline
$ export IMAGE=syzkaller/linux_image

$ cd $HOME
$ qemu-system-x86_64 \
        -m 2G \ # assign 2GB memory
        -cpu host\ 
        -smp 2 \ # assign 2 cpu to the vm
        -kernel $KERNEL/arch/x86/boot/bzImage \ # kernel image
        -append "console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0 nokaslr" \ # kernel boot options
        -drive file=$IMAGE/bullseye.img,format=raw \ # drive image
        -net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10021-:22 \ # network interface
        -net nic,model=e1000 \
        -enable-kvm \ 
        -nographic \
        -s \ # open gdbserver on TCP port 1234
        -S \ # wait for client at startup
        -pidfile vm.pid \
        2>&1 | tee vm.log

Start the GDB client in another terminal:

$ cd linux_mainline
$ gdb -ex ‘target remote :1234’ ./vmlinux

# GDB starts...

gdb -ex 'target remote :1234' ./vmlinux GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./vmlinux...
warning: File "/home/alessandro/linux_work/linux_mainline/scripts/gdb/vmlinux-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
 add-auto-load-safe-path /home/alessandro/linux_work/linux_mainline/scripts/gdb/vmlinux-gdb.py
line to your configuration file "/home/alessandro/.config/gdb/gdbinit".
To completely disable this security protection add
 set auto-load safe-path /
line to your configuration file "/home/alessandro/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
 info "(gdb)Auto-loading safe path"
Remote debugging using :1234
0x000000000000fff0 in gdt_page ()
(gdb) c
Continuing.
(gdb) ^C
Thread 2 received signal SIGINT, Interrupt.
[Switching to Thread 1.2]
default_idle () at arch/x86/kernel/process.c:743
743  raw_local_irq_disable();

PS: press c (continue) after GDB has started.

On the first terminal with Qemu you should see something like:

...
[  OK  ] Finished Permit User Sessions.
[  OK  ] Started Getty on tty1.
[  OK  ] Started Getty on tty2.
[  OK  ] Started Getty on tty3.
[  OK  ] Started Getty on tty4.
[  OK  ] Started Getty on tty5.
[  OK  ] Started Getty on tty6.
[  OK  ] Started Serial Getty on ttyS0.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started OpenBSD Secure Shell server.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Finished Update UTMP about System Runlevel Changes.
[    8.203958] audit: type=1400 audit(1726757653.078:6): avc:  denied  { checkpoint_restore }1

Debian GNU/Linux 11 syzkaller ttyS0

syzkaller login: root
Linux syzkaller 6.11.0-04557-g2f27fce67173 #6 SMP PREEMPT_DYNAMIC Wed Sep 18 15:28:32 CEST 2024 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
A valid context for root could not be obtained.
Last login: Wed Sep 18 23:17:45 UTC 2024 on ttyS0
root@syzkaller:~#

Your debugging environment is ready! Now, you can use GDB to debug your kernel.

Troubleshooting

GDB not showing method names

GDB doesn’t show method name and line information

Thread 2 received signal SIGINT, Interrupt.
[Switching to Thread 1.2]
0xffffffffa3ddc63f in ?? ()
(gdb)

-> disable KASLR by adding nokaslr as boot parameter

Qemu not responding

Qemu not responding -> kill the process and retry

$ ps -aux | grep qemu

alessan+   48324  6.5  3.6 3761520 892584 pts/2  Sl+  16:53   0:11 qemu-system-x86_64 ...

$ kill -9 48423

Other resources

While this article targets the x86_64 kernel, here a list of resources to setup Syzkaller on different architectures:

Check the QEMU documentation and GDB documentation for more details.