Linux Performance
hi-res: observability + static + perf-tools/bcc (svg)
slides: observability
slides: static, benchmarking, tuning
sar, perf-tools, bcc/BPF:
Images license: creative commons Attribution-ShareAlike 4.0.
slides: observability
slides: static, benchmarking, tuning
sar, perf-tools, bcc/BPF:
Images license: creative commons Attribution-ShareAlike 4.0.
This page links to various Linux performance material I've created, including the tools maps on the right. The first is a hi-res version combining observability, static performance tuning, and perf-tools/bcc (see discussion). The remainder were designed for use in slide decks and have larger fonts and arrows, and show: Linux observability tools, Linux benchmarking tools, Linux tuning tools, and Linux sar. For even more diagrams, see my slide decks below.
Tools
- perf_events: perf one-liners, examples, visualizations.
- eBPF tools: eBPF tracing tools and examples with bcc.
- perf-tools: perf analysis tools using ftrace (github).
- bcc: perf analysis tools using eBPF (github).
- ktap: one-liners, examples, and scripts.
- Flame Graphs: using perf, SystemTap, and ktap.
Documentation
- Linux Performance Analysis in 60,000 Milliseconds shows the first ten commands to use in an investigation (video, PDF). Written by myself and the performance engineering team at Netflix (2015).
- My post Performance Tuning Linux Instances on EC2 includes the tunables we're using at Netflix (2015).
- A gdb Debugging Full Example (Tutorial), including the use of some perf/debugging tools (2016).
- Posts about eBPF and bcc (2015-6):Linux eBPF
bcc: Taming Linux 4.3+ Tracing Superpowers
tcpconnect and tcpaccept for Linux (bcc)
Linux eBPF Stack Trace Hack (bcc)
Linux eBPF Off-CPU Flame Graph (bcc)
Linux Wakeup and Off-Wake Profiling (bcc)
Linux chain graph prototype (bcc)
Linux eBPF/bcc uprobes
Linux BPF/bcc Road Ahead
Ubuntu Xenial bcc/BPF
Linux bcc/BPF Tracing Security Capabilities
Linux MySQL Slow Query Tracing with bcc/BPF
Linux bcc/BPF ext4 Latency Tracing
Linux bcc/BPF Run Queue (Scheduler) Latency
Linux bcc/BPF Node.js USDT Tracing
Linux bcc tcptop
Linux 4.9's Efficient BPF-based Profiler
DTrace for Linux 2016
Linux 4.x Tracing Tools: Using BPF Superpowers
Linux bcc/BPF tcplife: TCP Lifespans
Golang bcc/BPF Function Tracing - A post about Linux 4.7 Hist Triggers (2016).
- My lwn.net article Ftrace: The Hidden Light Switch shows a use case for Linux ftrace (Aug, 2014).
- Posts about ftrace-based perf-tools (2014-5):iosnoop for Linux
- ,
- ,
- ,
- ,
Page Cache Hit Ratio- ,
- ,
- ,
- Posts about perf-based perf-tools: perf Hacktogram.
- Posts about perf_events (2014-5):perf CPU Sampling
- ,
- ,
- ,
- ,
- ,
perf off-CPU Time Flame Graphs- ,
- ,
- (
- )
- The blog post strace Wow Much Syscall discusses strace(1) for production use, and compares it to advanced tracing tools (2014).
- USE Method: Linux Performance Checklist; also see the USE Method page for the description of this methodology.
- Systems Performance: Enterprise and the Cloud (Prentice Hall, 2013) uses Linux distributions as the primary example.