Setups For Debugging QEMU with GDB and DDD

 

QEMU Build Considerations For Debugging

Default (distro) installations for QEMU usually include stripped binaries with no debugging info. For instance:

$ file /usr/bin/qemu-system-x86_64 
/usr/bin/qemu-system-x86_64: ELF 64-bit LSB shared object [...] stripped

and an attempt to run the stripped binary with gdb(1) will result in:

$ gdb -q --args qemu-system-x86_64 [...]
Reading symbols from /usr/bin/qemu-system-x86_64...(no debugging symbols found)...done.

This scenario necessitates a build from source for QEMU binaries with debugging info. A few GNU/Linux toolchain related issues for general binary build with debugging info will be noted before discussing pertinent QEMU build considerations.

GNU/Linux Toolchain and Debug Info

GCC Debugging Options

gcc(1) exports several options that control the level of debugging information to be included in the final binary. Generic options can be specified via -gLEVEL where LEVEL is 0, 1, 2, or 3. Level 0 produces no debug information at all; level 3 produces the most. The default, i.e. -g, is level 2. Consult the gcc(1) manpage for more details.

Debugging Info and GCC Optimized Code

From gcc(1):

GCC allows you to use -g with -O.  The shortcuts taken by optimized
code may occasionally produce surprising results: some variables
you declared may not exist at all; flow of control may briefly move
where you did not expect it; some statements may not be executed
because they compute constant results or their values were already
at hand; some statements may execute in different places because
they were moved out of loops.

Position Independent Executables

Currently, a default qemu-system-$ARCH build results in a Position Independent Executable (PIE) shared object. Like ordinary executables, these binaries can be run directly as the main program. But, unlike the former, PIE are loadable at arbitrary locations in the address space of a process, and can even be dynamically loaded and used as a module by another executable program instance. The virtual addresses in a PIE object files are therefore likely to be different from the runtime addresses. On the other hand, with ordinary executables, the virtual addresses in the object file generally correspond to the runtime addresses.

Now, depending on the situation, one might wish to disable PIE generation in order to allow cross-referencing between the addresses of the static symbols/locations in the object file and the load addresses in the address space of a QEMU execution instance.

QEMU Build Options for Debugging

The exact set of debugging options will depend on user requirements. Mileage will vary. A few of the debugging related options are shown below:

$ ./configure --help

Usage: configure [options]
Options: [defaults in brackets after descriptions]
(...)
    --extra-cflags=CFLAGS    append extra C compiler flags QEMU_CFLAGS
    --extra-ldflags=LDFLAGS  append extra linker flags LDFLAGS
(...)
    --enable-debug-tcg       enable TCG debugging
    --disable-debug-tcg      disable TCG debugging (default)
    --enable-debug-info      enable debugging information (default)
    --disable-debug-info     disable debugging information
    --enable-debug           enable common debug build options
(...)
    --disable-strip          disable stripping binaries
    --disable-werror         disable compilation abort on warning
(...)
    --enable-pie             build Position Independent Executables
    --disable-pie            do not build Position Independent Executables
(...)

The --disable-strip option prevents the default make install target from stripping debugging info (and other symbols) from the binaries during the installation process.

Consider the following build instances. The make V=1 command is run here after a successful build in order to view the CFLAGS and LDFLAGS used.

• Building with Default Debug Info

  $ ./configure --target-list=i386-softmmu,x86_64-softmmu,arm-softmmu,arm-linux-user --enable-kvm
  
  $ make -jN
  
  $ make V=1
  [...] CFLAGS="-O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -pthread -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -g -fPIE -DPIE -m64 [...] LDFLAGS="-Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g " [...]
  

  i.e. -O2 enabled along with -g in CFLAGS. PIE generation enabled via -fPIE (compiler) and -pie (static linker) flags.

• Enabling Common Debug Options

  $ ./configure --target-list=i386-softmmu,x86_64-softmmu,arm-softmmu,arm-linux-user --enable-kvm --enable-debug
  
  $ make -jN
  
  $ make V=1
  [...] CFLAGS="-pthread -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -g -fPIE -DPIE -m64 [...] LDFLAGS="-Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g " [...]
  

  Notice that --enable-debug disables gcc(1)'s -O2 optimization level.

• Configuring For Extra Debugging Info

  Along with enabling common debug options, the following build additionally disables both PIE binary generation and default strip(1) action during make install:

  $ ./configure --target-list=i386-softmmu,x86_64-softmmu,arm-softmmu,arm-linux-user --enable-kvm --enable-debug --extra-cflags="-g3" --extra-ldflags="-g3" --disable-strip --disable-pie --prefix=${PWD}/../v2.1.3
  
  $ make -jN
  
  $ make V=1
  [...] CFLAGS="-pthread -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -g -m64 [...] -g3 [...] LDFLAGS="-Wl,--warn-common -m64 -g -g3 " [...]
  
  $ make install
  
  $ file ../v2.1.3/bin/qemu-system-x86_64 
  /tmp/qemu/v2.1.3/bin/qemu-system-x86_64: ELF 64-bit LSB executable [...] not stripped
  

  Note that the -g3 switch overrides the default QEMU -g (i.e. -g2)¹ option. The following simple program can be used to verify this:

  $ cat vipi.c
  #include <stdio.h>
  #define VIPI "vipi"
  int main(void){ printf("%s
  ", VIPI); return 0; }
  
  $ cat gdbc_vipi
  break main 
  run
  macro expand VIPI
  continue
  quit
  
  $ gcc -Wall vipi.c -g 
  $ gdb -q -command=gdbc_vipi a.out | grep expands
  expands to: VIPI
  
  $ gcc -Wall vipi.c -g -g3
  $ gdb -q -command=gdbc_vipi a.out | grep expands
  expands to: "vipi"
  

  i.e. gdb(1) macro expansion was possible after appending -g3 (to override -g) on gcc(1)'s commandline.

Setups for Debugging QEMU with gdb(1)

Simple Setups

Some of the examples presented in this section require Linux guest configuration for serial port system console and login terminal. Refer to QEMU Serial Port System Console for a background coverage - in addition to the basic QEMU commandline for specifying serial port terminal devices and backends.

QEMU Graphical Mode

$ gdb -q --args /tmp/qemu/v2.3.1/bin/qemu-system-x86_64 
    -enable-kvm -smp 2 -m 1G 
    -kernel vmlinuz -initrd initrd.img 
    -drive file=demo.img,if=virtio 
    -append "root=/dev/vda1 rw console=ttyS0"

Reading symbols from /tmp/qemu/v2.1.3/bin/qemu-system-x86_64...done.
(gdb) break do_device_add 
Breakpoint 1 at 0x553305: file qdev-monitor.c, line 662.
(gdb) r

This will result in a graphical boot instance e.g:

with the gdb(1) console on stdio of the QEMU launch term, and the Human Monitor Interface (HMI) and serial port system console on the default QEMU SDL virtual consoles (VC) (accessible via CTRL+N where N is 2, 3, ...)².

gdb(1) Redirection for QEMU Serial Line Terminals

Executing QEMU via gdb(1) results in GDB's console taking over stdio of the launch term. Now, typically with graphical QEMU boots, it lends to convinience working with the HMI from a separate xtem, rather than having to constantly keep toggling between guest VGA VC and the HMI VC. However, since gdb(1) already uses this interface as its controlling terminal, the HMI's console will have to be redirected to another terminal interface on the host.

gdb(1) exports the tty option to redirect the output of the program being debugged to a separate terminal. To redirect the stdin, stdout and stderr of a QEMU serial line terminal to a separate xterm(1) on the VM host:

• Prepare a "target" xterm(1): Since the corresponding pseudoterminal slave (PTS) device, i.e. /dev/pts/N, was already established as the controlling terminal by the xterm(1)'s startup shell for the terminal session, a foreground process that allows redirection of the session's local keyboard stdin need first be executed before using gdb(1) redirection on this PTS e.g:

  $ tty
  /dev/pts/2
  
  $ sleep 10d
  

• Then, to redirect the HMI to this terminal:

  $ gdb -q --args /tmp/qemu/v2.3.1/bin/qemu-system-x86_64 
      -enable-kvm -smp 2 -m 1G 
      -kernel vmlinuz -initrd initrd.img 
      -drive file=demo.img,if=virtio 
      -append "root=/dev/vda1 rw console=ttyS0" 
      -monitor stdio
  
  Reading symbols from /tmp/qemu/v2.1.3/bin/qemu-system-x86_64...done.
  (gdb) tty /dev/pts/2
  (gdb) r
  

  With this setup, guest serial port system console will be redirected to the default (SDL) VC, while the HMI will be accessible from the xterm(1) @ /dev/pts/2.

• Or, to redirect both the HMI and the guest serial port system console to this separate xterm(1):

  $ gdb -q --args /tmp/qemu/v2.3.1/bin/qemu-system-x86_64 
      -enable-kvm -smp 2 -m 1G 
      -kernel vmlinuz -initrd initrd.img 
      -drive file=demo.img,if=virtio 
      -append "root=/dev/vda1 rw console=ttyS0" 
      -serial mon:stdio
  
  Reading symbols from /tmp/qemu/v2.1.3/bin/qemu-system-x86_64...done.
  (gdb) tty /dev/pts/2
  Breakpoint 1 at 0x553305: file qdev-monitor.c, line 662.
  (gdb) r
  

  Here, both HMI and guest serial port system console will multiplexed on the "target" xterm(1).

• NOTE: gdb(1) might emit the following warning message on the "target" xterm(1) display upon connection:

  warning: GDB: Failed to set controlling terminal: Operation not permitted
  

  Basically, this means that a controlling terminal was already established for /dev/pts/N (by the xterm(1)'s startup shell when it opened the terminal device). A terminal session (i.e. the parent process that opened terminal device file and its children) has the controlling terminal if it receives the signal-generating terminal characters, i.e. SIGINT (CTRL+C), SIGQUIT (CTRL+), and SIGTSTP (CTRL+Z). Only one session at a time can have the controlling terminal and the terminal driver delivers the corresponding signal to the members of the foreground process group.

  Unless terminal attributes are modified, then while local keyboard input of the xterm(1) session will now be accessible by a remote program using gdb(1) redirection, the sleep 10d foreground process will exit when the user types any of these signal-generating terminal characters; stdin access will then be lost for the program being debugged and will now get redirected back to the controlling process of this terminal i.e. the startup shell. Fortunately, QEMU here modifies the "target" terminal's attributes via tcsetattr(3) in qemu-char.c:qemu_chr_set_echo_stdio() to acquire the controlling terminal for its serial line³.

QEMU Nographic Mode

In nographic mode, QEMU's HMI and guest serial port system console/terminal are automatically multiplexed and redirected to stdio of the launch terminal. So, for simple redirection, GDB's tty option can be used to redirect these QEMU serial device interfaces to a seperate terminal on the host:

On the "target" xterm(1):

$ tty
/dev/pts/2

$ sleep 10d

Then launch QEMU via gdb(1), e.g:

$ gdb -q --args /tmp/qemu/v2.1.3/bin/qemu-system-x86_64 
    -enable-kvm -smp 2 -m 1G -kernel vmlinuz -initrd initrd.img 
    -drive file=demo.img,if=virtio 
    -append "root=/dev/vda1 rw console=ttyS0" -nographic
Reading symbols from /tmp/qemu/v2.1.3/bin/qemu-system-x86_64...done.
(gdb) tty /dev/pts/2
(gdb) r

This should result in a nographic QEMU boot instance with the HMI and guest serial port system console getting redirected to the xterm(1) @ /dev/pts/2.

QEMU Options for Serial Line Terminal Redirection

QEMU serial line terminals support a number of options that allow for more sophisticated schemes for program output than the simple redirection provided by gdb(1)'s tty option. A few configurations are presented below. See Redirecting QEMU Serial Line Terminals for a background coverage of the QEMU commandline options used.

Net Console

The example below illustrates HMI redirection via TCP Net Console.

• Fire-up a listening nc(1) instance on one xterm(1):

  $ nc -l 4555
  

• Then launch QEMU via gdb(1) on a separate xterm(1):

  $ sudo gdb -q --args /tmp/qemu/v2.3.1/bin/qemu-system-x86_64 
      -enable-kvm -smp 2 -m 1G 
      -kernel vmlinuz -initrd initrd.img 
      -drive file=demo.img,if=virtio 
      -append "root=/dev/vda1 rw console=ttyS0" 
      -net nic 
      -net tap,ifname=tap0,script=qemu_br0_ifup.sh,downscript=qemu_br0_ifdown.sh 
      -monitor tcp:127.0.0.1:4555 [-nographic]
  

  where the qemu_br0_if*.sh scripts were used to setup for a QEMU TAP configuration on the host's Linux bridge interface. See A QEMU TAP Networking Setup for the host network interface configuration and for the definitions the of qemu_br0_if* scripts used here.

At this point the HMI should be accessible via the xterm(1) interface of the listening nc(1) server:

$ nc -l 4555
QEMU 2.1.3 monitor - type 'help' for more information
(qemu)

Using Separate Terminals on the Host

As explained in Redirecting QEMU Serial Line Terminals, while Net Console setups are quite straightforward, these interfaces present certain limitations with respect to operations expected of a terminal. Also included in that entry is a configuration that enables using host terminal devices with services such as SSH, for remote (across a network) terminals that overcome the limitations of Net Consoles.

To obtain the HMI on a separate xterm(1) on the host, first peform the following set of commands on the "target" xterm(1):

$ tty
/dev/pts/2

$ sleep 10d

Then on the QEMU-via-gdb(1) launch xterm(1):

$ gdb -q --args /tmp/qemu/v2.1.3/bin/qemu-system-x86_64 
    -enable-kvm -smp 2 -m 1G -kernel vmlinuz -initrd initrd.img 
    -drive file=demo.img,if=virtio 
    -append "root=/dev/vda1 rw console=ttyS0" -monitor /dev/pts/2
Reading symbols from /tmp/qemu/v2.1.3/bin/qemu-system-x86_64...done.
(gdb) r

This should result in graphical QEMU boot instance with the HMI's stdin, stdout and stderr being redirected to the xterm(1) @ /dev/pts/2. Unlike the interface provided by the Net Console example above, TAB key completion and command history should work here as expected.

To redirect HMI and guest serial port system console to separate host terminal backends, open an additional xterm(1) for the guest serial port system console:

$ tty
/dev/pts/10

$ sleep 10d

and start a gdb(1) debugging session with, say:

$ gdb -q --args /tmp/qemu/v2.1.3/bin/qemu-system-x86_64 
    -enable-kvm -smp 2 -m 1G -kernel vmlinuz -initrd initrd.img 
    -drive file=demo.img,if=virtio 
    -append "root=/dev/vda1 rw console=ttyS0" 
    -monitor /dev/pts/2 
    -chardev tty,id=pts10,path=/dev/pts/10 
    -device isa-serial,chardev=pts10
Reading symbols from /tmp/qemu/v2.1.3/bin/qemu-system-x86_64...done.
(gdb) r

This should result in graphical QEMU boot instances similar to:

then,

A Demo QEMU Debug Session with gdb(1)

This section presents an example of debugging operation of a QEMU virtual device. The ivshmem PCI device is considered. See Writing a Linux PCI Device Driver tutorial for an introduction the ivshmem framework. The programs presented in that entry will be used here.

Among other things, this framework enables inter-VM notification via the eventfd(2) mechanism. The ivshmem device interprets an eventfd(2) notification as an IRQ and interrupts the guest. Now, from eventfd(2):

 eventfd()  creates  an  "eventfd  object"  that can be used as an event
 wait/notify mechanism by userspace applications, and by the  kernel  to
 notify  userspace  applications  of  events.   The  object  contains an
 unsigned 64-bit integer (uint64_t) counter that is  maintained  by  the
 kernel.   This  counter  is initialized with the value specified in the
 argument initval.

Upon QEMU's reception of eventfd(2) notification from ne_ivshmem_send_qeventfd (or another ivshmem enabled VM instance for that matter), the QEMU I/O thread reads out this host kernel maintained eventfd(2) counter⁴. This thread's code execution control path eventually invokes the hw/misc/ivshmem.c:ivshmem_IntrStatus_write() function, passing it the counter value in val:

160 static void ivshmem_IntrStatus_write(IVShmemState *s, uint32_t val)
161 {
162     IVSHMEM_DPRINTF("IntrStatus write(w) val = 0x%04x
", val);
163 
164     s->intrstatus = val;
165 
166     ivshmem_update_irq(s, val);
167 }

As shown, a copy of this value is stored in the device's state object. The s->intrstatus member is the virtual status register of the ivshmem device and its value is what is returned to the device driver in the guest when a read is performed on the status register. Finally, this function invokes hw/misc/ivshmem.c:ivshmem_update_irq() which performs the IRQ notification via pci_set_irq():

/* hw/misc/ivshmem.c */
127 static void ivshmem_update_irq(IVShmemState *s, int val)
128 {
129     PCIDevice *d = PCI_DEVICE(s);
130     int isr;
131     isr = (s->intrstatus & s->intrmask) & 0xffffffff;
132 
133     /* don't print ISR resets */
134     if (isr) {
135         IVSHMEM_DPRINTF("Set IRQ to %d (%04x %04x)
",
136            isr ? 1 : 0, s->intrstatus, s->intrmask);
137     }
138 
139     pci_set_irq(d, (isr != 0));
140 }

A gdb(1) session is presented next to illustrate how the above eventfd(2) related code commentary was verified.

gdb(1) Debugging Session

Notice that the val value passed to hw/misc/ivshmem.c:ivshmem_update_irq() is actually not used in the function (it was already used to initialize the status register in the caller hw/misc/ivshmem.c:ivshmem_IntrStatus_write()). Nevertheless, a breakpoint will be set against the former to allow a one-liner trace of both val value and IRQ notification.

The shared memory server was instantiated with the default settings:

$ ./ivshmem_server 
listening socket: /tmp/ivshmem_socket
shared object: ivshmem
shared object size: 1048576 (bytes)
vm_sockets (0) =

Waiting (maxfd = 4)

Separate terminals for the HMI and guest serial port system console were then prepared, as described in the previous sections, before executing the following QEMU instance via gdb(1):

$ gdb -q --args /tmp/qemu/v2.1.3/bin/qemu-system-x86_64 
    -enable-kvm -smp 2 -m 1G 
    -kernel vmlinuz -initrd initrd.img 
    -drive file=demo.img,if=virtio 
    -append "root=/dev/vda1 rw console=ttyS0" 
    -monitor /dev/pts/2 
    -chardev tty,id=pts10,path=/dev/pts/10 
    -device isa-serial,chardev=pts10 
    -nographic 
    -chardev socket,path=/tmp/ivshmem_socket,id=ivshmemid 
    -device ivshmem,chardev=ivshmemid,size=1,msi=off
Reading symbols from /tmp/qemu/v2.1.3/bin/qemu-system-x86_64...done.
(gdb) break ivshmem_update_irq 
Breakpoint 1 at 0x4951ff: file /tmp/qemu/hw/misc/ivshmem.c, line 128.
(gdb) command 1
Type commands for breakpoint(s) 1, one per line.
End with a line saying just "end".
>silent
>printf "val is %d
", val
>c
>end
(gdb) r

Once QEMU booted, a login via guest serial port terminal and loading of the guest ivshmem driver was performed:

root@vm:~/linux_pci# echo 8 > /proc/sys/kernel/printk
root@vm:~/linux_pci# insmod ne_ivshmem_ldd_basic.ko

Device driver load produced the following output on the gdb(1) console:

[Switching to Thread 0x7fffe7fff700 (LWP 19070)]
val is -1
val is 0
val is 0

An immediate observation was that since no eventfd(2) notification had begun, the code execution control path(s) leading to these invocations of these ivshmem_update_irq() instances must be due to some other internal QEMU task(s) - in this case, the thread identified by (LWP 19070) - rather than the I/O thread. Results of further probing shown here explain the cause of these invocations.

Now, on the host, executing ne_ivshmem_send_qeventfd resulted in periodic (1Hz) IRQs on the QEMU ivshmem device via eventfd(2), with the following output getting generated on the gdb(1) console and guest serial port system console:

As shown in the screenshot above, two consecutive execution instances of ne_ivshmem_send_qeventfd were performed. The 2xx.xxxxxx guest kernel timestamps correspond to the ne_ivshmem_ldd_basic.ko print out for the second run. A brief commentary on the events displayed in the screenshot due to the second round of ne_ivshmem_send_qeventfd execution will now be made.

Upon firing ne_ivshmem_send_qeventfd, the guest kernel ivshmem device driver printed:

[  255.549676] ivshmem_interrupt:71:: interrupt (status = 0x0002)
[  256.552918] ivshmem_interrupt:71:: interrupt (status = 0x0001)
[  257.554352] ivshmem_interrupt:71:: interrupt (status = 0x0001)
[  258.556009] ivshmem_interrupt:71:: interrupt (status = 0x0001)
...

with gdb(1) displaying the following corresponding output:

[Switching to Thread 0x7ffff7fb6940 (LWP 19066)]
val is 2
[Switching to Thread 0x7fffe7fff700 (LWP 19070)]
val is 0
[Switching to Thread 0x7ffff7fb6940 (LWP 19066)]
val is 1
[Switching to Thread 0x7fffe7fff700 (LWP 19070)]
val is 0
val is 0
[Switching to Thread 0x7ffff7fb6940 (LWP 19066)]
val is 1
[Switching to Thread 0x7fffe7fff700 (LWP 19070)]
val is 0
[Switching to Thread 0x7ffff7fb6940 (LWP 19066)]
val is 1
[Switching to Thread 0x7fffe7fff700 (LWP 19070)]
val is 0
val is 0
...

From GDB's output, it can easily be deduced that (LWP 19066) is the QEMU I/O thread whose code execution control path reads out the corresponding eventfd(2) counter from the host kernel before invoking ivshmem_upate_irq(). As explained here, the other thread, i.e. (LWP 19070), executes the QEMU control path due to a guest read of the ivshmem device's status register (or more precisely, when the vCPU accesses the memory mapped status register during the execution of the ne_ivshmem_ldd_basic.c:ivshmem_interrupt() ISR).

Now, the gdb(1) events due to QEMU I/O thread (LWP 19060) trigger full processing of the ne_ivshmem_ldd_basic.c:ivshmem_interrupt() ISR i.e. IRQ_HANDLED unlike other Linux invocations of the ISR. Notice that in the first event, the host kernel eventfd(2) counter value was 2, i.e. "val is 2" (GBD event) and "status = 0x0002" (guest ISR message). This information translates to a case where ne_ivshmem_send_qeventfd performed two periodic cycles, updating the eventfd(2) counter in the host kernel twice, before QEMU had a chance to read out the counter.

When gdb(1) print out eventually reached the bottom of its xterm(1) window, it printed⁵:

---Type <return> to continue, or q <return> to quit---
[Switching to Thread 0x7ffff7fb6940 (LWP 19066)]
val is 1

and froze QEMU execution until the RETURN key was pressed - which made gdb(1) create some room for more print out. Since the ne_ivshmem_send_qeventfd program on the host had merrily kept on sending periodic notifications (@1sec) during the time the QEMU execution via gdb(1) was frozen, the host kernel correspondingly incremented the eventfd(2) counter until QEMU resumed execution. This is the reason why val is 3 in the following snippet:

val is 3
[Switching to Thread 0x7fffe7fff700 (LWP 19070)]
val is 0
val is 0
val is 0
val is 0
[Switching to Thread 0x7ffff7fb6940 (LWP 19066)]
val is 1

The guest's timestamp info corroborates this: the timestamp in the line corresponding to status = 0x0003 below, i.e. 262.940289, is approximately three seconds apart from the previous 258.556009 timestamp:

    [  258.556009] ivshmem_interrupt:71:: interrupt (status = 0x0001)
    [  262.940289] ivshmem_interrupt:71:: interrupt (status = 0x0003)
    [  263.561084] ivshmem_interrupt:71:: interrupt (status = 0x0001)

Notice that with this QEMU configuration, the guest uses the host kernel's Time Stamp Counter (TSC) info for timing.

Naturally, during the time QEMU execution was frozen, no interaction with the guest serial port terminal was possible; whether keyboard input or guest console output.

Attaching gdb(1) to a QEMU instance

gdb(1) includes a facility that enables attaching the debugger to a live program instance, instead of having to instantiate it via gdb(1). Attaching gdb(1) to a QEMU process is a simple matter of:

• Determining the QEMU process' PID, e.g:

  $ ps -a -C qemu | grep qemu
  21761 pts/5    00:00:51 qemu-system-x86
  

• Attaching gdb(1) to this PID -- root priviledges may be required for this operation:

  $ sudo gdb -q
  (gdb) attach 21761
  Attaching to process 21761
  Reading symbols from [...]qemu-system-x86_64...done.
  Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done.
  Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
  [...]
  Reading symbols from /usr/lib/x86_64-linux-gnu/libogg.so.0...(no debugging symbols found)...done.
  Loaded symbols for /usr/lib/x86_64-linux-gnu/libogg.so.0
  0x00007f19c7c84b13 in ppoll () from /lib/x86_64-linux-gnu/libc.so.6
  

  Since the goal is to debug/trace QEMU execution, the (no debugging symbols found) messages by the loaded shared library dependencies are of no consequence here. The important thing is that the binary of this QEMU instance was compiled with the necessary debugging info.

  Upon attaching, gdb(1) will freeze program execution. At this point, setting breakpoints and performing other gdb(1) related tasks may be done before resuming QEMU execution with:

  (gdb) c
  Continuing.
  [Thread 0x7f1966eee700 (LWP 21795) exited]
  [Thread 0x7f19117fa700 (LWP 21820) exited]
  

Setups for Debugging QEMU with ddd(1)

The Data Display Debugger (DDD) presents a rich GUI with menus and interfaces that greatly facilitate the debugging process. At least on Linux, ddd(1) is a frontend to gdb(1) by default and presents a gdb(1) terminal where the user can enter gdb(1) commands. Alternatively, a Command Tool window is available for sending commands to gdb(1). Its layout also includes several other GUI windows for source code browsing (Source Window), viewing "in-step" machine code execution (Machine Code Window), displaying data (Data Window), etc. In addition to easier source code browsing in DDD via the Source Window, its Data Window presents a particularly powerful feature for viewing data structures - notably lists and queues - which make DDD sometimes preferable to use than plain, text-based gdb(1).

On Debian based systems:

$ sudo apt-get install ddd

Rather than dwell on the GUI features of DDD, this section will only present an example of a QEMU launch command line with ddd(1). A recommended guide to learning DDD usage is listed in the Resources section below.

Prepare one "target" xterm(1) for the HMI and another for the guest serial port terminal as illustrated in previous sections - and for the reasons given in A Note on pty vs tty QEMU Options. Then fire-up a QEMU instance via ddd(1) e.g:

$ ddd --args /tmp/qemu/v2.1.3/bin/qemu-system-x86_64 
  -smp 2 -enable-kvm -m 1G 
    -kernel vmlinuz -initrd initrd.img  
    -drive file=demo.img,if=virtio   
    -append "root=/dev/vda1 rw console=ttyS0"  
    -monitor /dev/pts/3  
    -chardev tty,path=/dev/pts/4,id=pts4 
    -device isa-serial,chardev=pts4 [-nographic]

Here, the HMI is redirected to the xterm(1) @ /dev/pts/3 while the guest serial port system console and login terminal will appear on xterm(1) @ /dev/pts/4.

Also See

• QEMU Serial Port System Console for a background on configuring a Linux guest's system console over QEMU serial port.

• Redirecting QEMU Serial Line Terminals for various configurations for redirecting the HMI and guest serial port terminal to different host backends.

• QEMU Human Monitor Interface for tips on using the HMI.

Resources

• qemu(1), gcc(1), and gdb(1) manpages

• Books on Using GDB/DDD:

  • Debugging with GDB - RHEL

  • The Art Of Debugging with GDB, DDD, and Eclipse, Normal Matloff and Peter Jay Salzman, 2008, No Starch Press, Inc.

Footnotes

^{1. ld(1) ignores the -g option and only provides it for compatibility with other tools. [go back]}

^{2. HMI and guest serial port system console on QEMU SDL VC interfaces seemingly deprecated in QEMU 2.x [go back]}

^{3. For an authoritative discussion on controlling terminals, see The Linux Programming Interface: A Linux and UNIX System Programming Handbook, Micheal Kerrisk, No Starch Press, Inc. 2010. The definitive guide to Linux System Programming. Yes. [go back]}

^{4. The host kernel increments the eventfd(2) counter @ write(2) by ne_ivshmem_send_qeventfd, and resets it upon a read(2) by QEMU. [go back]}

^{5. See Screen Size (below) for a discussion on controlling gdb(1)'s screen output. [go back]}

Appendix

Potential Issues

Currently, all the examples included in this entry were performed against qemu-system-x86_64 with KVM enabled. If running with dynamic translation instead (i.e. using Tiny Code Generator (TCG)), and encounter signals e.g:

$ gdb -q --args qemu-system-x86_64 -machine accel=tcg [...] 
Reading symbols from [...]/qemu-system-x86_64...done.
(gdb) r

Program received signal SIGUSR1, User defined signal 1.
[Switching to Thread 0x7fffdd496700 (LWP 19752)]
0x000000000040f33f in int128_make64 (a=4096)
        at [...]/qemu/include/qemu/int128.h:16
16  {

interrupting gdb(1) execution of QEMU, then the handle command may be used to instruct gdb(1) to pass on the signals without stopping, say:

(gdb) handle SIGUSR1 nostop noprint
Signal        Stop   Print  Pass to program  Description
SIGUSR1       No     No     Yes              User defined signal 1

(gdb) c
Continuing.

Help on using the handle command can obtained via:

(gdb) help handle
Specify how to handle a signal.
Args are signals and actions to apply to those signals.
[...]
The special arg "all" is recognized to mean all signals except those
used by the debugger, typically SIGTRAP and SIGINT.
Recognized actions include "stop", "nostop", "print", "noprint",
"pass", "nopass", "ignore", or "noignore".
[...]

while signal state info can be viewed via:

(gdb) info signal
Signal        Stop  Print  Pass to program  Description

SIGHUP        Yes   Yes    Yes              Hangup
SIGINT        Yes   Yes    No               Interrupt
SIGQUIT       Yes   Yes    Yes              Quit
[...]
SIGUSR1       No    No     Yes              User defined signal 1
SIGUSR2       Yes   Yes    Yes              User defined signal 2   
[...]

or,

(gdb) info handle
Signal        Stop  Print  Pass to program  Description

SIGHUP        Yes   Yes    Yes              Hangup
SIGINT        Yes   Yes    No               Interrupt
SIGQUIT       Yes   Yes    Yes              Quit
[...]
SIGUSR1       No    No     Yes              User defined signal 1
SIGUSR2       Yes   Yes    Yes              User defined signal 2   
[...]

Specifying Source Directories

Debugging info in executables will contain the names of the source files, but may not include directory/path info. gdb(1) supports a source path which is a list of directories to search for source files. Each time gdb(1) wants a source file, it tries all directories in the list, and the order that they are listed.

(gdb) show directories 
Source directories searched: $cdir:$cwd

where:

• $cdir is the directory in which the source files were compiled into object code.
• $cwd is the current working directory.

If gdb(1) cannot find a source file in the source path, and if the object program records a directory, gdb(1) tries that directory.

To add other directories to source path, e.g:

(gdb) directory /home/siro/qemu:/usr/local/src/qemu
Source directories searched: /home/siro/qemu:/usr/local/src/qemu:$cdir:$cwd

To reset the source path:

(gdb) directory
Reinitialize source path to empty? (y or n) y
Source directories searched: $cdir:$cwd

(gdb) show directories
Source directories searched: $cdir:$cwd

For help:

(gdb) help directory

Screen Size

A gdb(1) command may result in a large amount of information getting output to the screen. By default, gdb(1) pauses and prompts for user action at the end of each page (or "screenfull") of output e.g:

#7  0x000000000041768d in address_space_read_full (as=0xeff600, 
    addr=4273807364, attrs=..., buf=0x7ffff7fe3028 "", len=4)
    at /usr/local/src/qemu/v2.7.0-rc4/exec.c:2691
---Type <return> to continue, or q <return> to quit---

i.e. type RET to display more output, or q to discard the remaining output. Also notice that the screen width setting controls the line wrapping.

Typically, gdb(1) knows the size (height and width) of the screen from the terminal driver software, e.g:

$ stty size
24 80

will yield:

(gdb) show height
Number of lines gdb thinks are in a page is 24.
(gdb) show width
Number of characters gdb thinks are in a line is 80.

The set height and set width commands can be used to override these settings. A height of zero lines means that gdb(1) will not pause during output regardless of how long the output is:

(gdb) set height 0
(gdb) show height
Number of lines gdb thinks are in a page is unlimited.

This is useful if output is to a file or to an editor buffer. Similarly, a width of zero prevents gdb(1) from wrapping its output:

(gdb) set width 0
(gdb) show width
Number of characters gdb thinks are in a line is unlimited.