什么是SystemTap以及如何使用它？

环境

• Red Hat Enterprise Linux (all versions)

问题

• What is SystemTap and how to use it?

决议

SystemTap is an innovative tool which allows for simplified information gathering on the running Linux kernel. The purpose of using SystemTap is to obtain information on either performance issues or functional problems (bugs). By using SystemTap, developers and system administrators can debug problems and gather profiling and performance data without having to create and install instrumented kernels or other packages.

In essence, SystemTap provides the infrastructure (a command line interface and scripting language) needed to gather information. The actual job that SystemTap performs relies on user-developed scripts tailored to a specific purpose. Currently, there are a number of example SystemTap scripts pre-made for general use.

The operation of SystemTap is quite simple. The stap command takes as an argument the name of a SystemTap file (called a probe). There may be additional command line arguments passed depending on the probe. SystemTap translates the probe into C and compiles the C source as a kernel module. It then inserts the resulting kernel module into the running kernel to perform the probe functions defined in the script. The output is printed to the console, or can be redirected to file.

Requirements

Because SystemTap compiles scripts from C code and launches probes for kernel instrumentation, it requires several packages in order to function. See the Installing SystemTap chapter of the SystemTap Beginners Guide for detailed installation instructions.

The easiest way to satisfy the requirements is to simply subscribe the system to the relevant debuginfo channels in RHN, then run the following commands which should set up the environment for SystemTap:

Raw

yum install systemtap
stap-prep

To set the environment up manually, in addition to the systemtap package, the following packages must also be installed:

• kernel-devel for the running kernel
• kernel-debuginfo for the running kernel
• kernel-debuginfo-common for the running kernel
• gcc
• systemtap

For example, for a 2.6.32-71.18.2.el6.x86_64 kernel you'll need:

Raw

kernel-debuginfo-2.6.32-71.18.2.el6.x86_64
kernel-debuginfo-common-x86_64-2.6.32-71.18.2.el6.x86_64
kernel-devel-2.6.32-71.18.2.el6.x86_64

As well as the gcc and systemtap packages.

The gcc and kernel-devel packages are available on Red Hat Network and can be installed using yum (RHEL 5 onwards).

Legacy

On RHEL 4, use up2date. For Red Hat Enterprise Linux (RHEL) 4, the kernel-debuginfo package is not available via up2date and must be installed from the ISO image available on Red Hat Network or from ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/4AS/en/os/Debuginfo/ (substitute 4AS in the URL with 4WS, 4ES, or 4Desktop, depending on the variant of of RHEL you have installed. Note: Red Hat Enterprise Linux 4 does not have kernel-debuginfo-common package.

For RHEL 5, you can download the kernel-debuginfo and kernel-debuginfo-common packages from ftp://ftp.redhat.com/pub/redhat/linux/enterprise/5Server/en/os/.

Using Systemtap

You can verify the SystemTap environment with the stap-report command.

An example script is shown below:

Raw

#! /usr/bin/env stap

# Using statistics and maps to examine kernel memory allocations

global kmalloc

probe kernel.function("__kmalloc") {
        kmalloc[execname()] <<< $size
}

# Exit after 10 seconds
probe timer.ms(10000) { exit () }

probe end {
        foreach ([name] in kmalloc) {
                printf("Allocations for %s\n", name)
                printf("Count:   %d allocations\n", @count(kmalloc[name]))
                printf("Sum:     %d Kbytes\n", @sum(kmalloc[name])/1000)
                printf("Average: %d bytes\n", @avg(kmalloc[name]))
                printf("Min:     %d bytes\n", @min(kmalloc[name]))
                printf("Max:     %d bytes\n", @max(kmalloc[name]))
                print("\nAllocations by size in bytes\n")
                print(@hist_log(kmalloc[name]))
                printf("-------------------------------------------------------\n\n");
        }
}

This script, drawn from the SystemTap project wiki, can be used to print information kernel memory allocations of the system. The script can be invoked as follows:

Raw

stap kmalloc2.stp

For issues during the compile or loading of the module within the stap command, append the parameter -vv to show more verbose output.

SystemTap will then translate the probe into C, compile the C program, and insert the probe into the running kernel. Truncated output is below:

Raw

-------------------------------------------------------

Allocations for httpd
Count:   10 allocations
Sum:     0 Kbytes
Average: 0 bytes
Min:     0 bytes
Max:     0 bytes

Allocations by size in bytes
value |-------------------------------------------------- count
    0 |@@@@@@@@@@                                         10
    1 |                                                    0
    2 |                                                    0

-------------------------------------------------------

Allocations for sendmail
Count:   2 allocations
Sum:     0 Kbytes
Average: 24 bytes
Min:     24 bytes
Max:     24 bytes

Allocations by size in bytes
value |-------------------------------------------------- count
    4 |                                                   0
    8 |                                                   0
   16 |@@                                                 2
   32 |                                                   0
   64 |                                                   0

-------------------------------------------------------

This is a simple example that just touches the surface of the capabilities offered by SystemTap. System Administrators could use this information to better understand kernel memory allocation on the running system and adjust kernel tuning parameters accordingly. Application developers can use this information as an overview of which applications are receiving more kernel memory allocations, which can be used as a starting point for deeper application profiling.

Additional Information about SystemTap

• Complement to this guide
• SystemTap Beginners Guide
• The SystemTap project Wiki: http://sourceware.org/systemtap/wiki/HomePage
• The SystemTap project Homepage: http://sourceware.org/systemtap/
• Red Hat Magazine overview of SystemTap: http://www.redhat.com/magazine/011sep05/features/systemtap/