网管告警：
告警主机：YiDHLWJKFZ-js-app-16 主机IP：192.168.***.*** 告警项目：system.cpu.util[,iowait] 告警时间：2019.02.25 15:18:48 告警等级：Warning 告警信息：Disk I/O is overloaded on YiDHLWJKFZ-js-app-16 问题详情：CPU iowait time:20.14 % 当前状态：PROBLEM:20.14 % 事件ID：1003377
top查看：（wa值为17.7%）
[root@localhost vmuser]# top top - 15:31:31 up 1:17, 1 user, load average: 3.56, 3.45, 3.40 Tasks: 232 total, 1 running, 231 sleeping, 0 stopped, 0 zombie Cpu(s): 1.0%us, 2.6%sy, 0.0%ni, 78.2%id, 17.7%wa, 0.2%hi, 0.3%si, 0.0%st Mem: 16467984k total, 2823808k used, 13644176k free, 331524k buffers Swap: 16383992k total, 0k used, 16383992k free, 565436k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 389 root 20 0 0 0 0 D 5.3 0.0 2:16.37 jbd2/sda3-8 2483 postfix 20 0 104m 46m 2244 S 3.7 0.3 1:45.87 qmgr 1531 root 20 0 242m 3484 920 S 3.0 0.0 1:17.92 rsyslogd 23237 postfix 20 0 62740 3556 2616 S 1.7 0.0 0:00.05 local 2476 root 20 0 62360 2844 2128 S 1.3 0.0 0:44.87 master 23230 postfix 20 0 62584 2984 2256 S 1.3 0.0 0:00.05 cleanup 23238 root 20 0 62740 3564 2616 D 1.3 0.0 0:00.04 local 2488 postfix 20 0 62540 2980 2144 S 1.0 0.0 0:29.32 trivial-rewrite 23229 postfix 20 0 62584 2984 2256 S 1.0 0.0 0:00.06 cleanup 2482 postfix 20 0 62440 2900 2164 D 0.7 0.0 0:39.73 pickup 23231 postfix 20 0 62584 2988 2256 D 0.7 0.0 0:00.03 cleanup
iostat查看：（iowait值一直过高）
[root@localhost vmuser]# iostat -x 2 5 Linux 2.6.32-71.el6.x86_64 (localhost.localdomain) 02/25/2019 _x86_64_ (8 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 2.96 0.00 2.14 19.78 0.00 75.12 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 12.58 361.59 28.73 1621.40 616.25 15618.31 9.84 3.72 2.25 0.53 86.86 sdb 307.71 0.01 6.02 0.00 1255.11 0.05 208.45 0.14 22.52 14.07 8.47 avg-cpu: %user %nice %system %iowait %steal %idle 0.84 0.00 2.12 21.38 0.00 75.66 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 465.00 36.00 1641.00 288.00 16508.00 10.02 3.27 1.98 0.58 97.30 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 1.44 0.00 4.49 15.09 0.00 78.97 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 1167.00 18.00 4016.50 144.00 40720.00 10.13 3.61 0.88 0.22 87.95 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.44 0.00 1.19 24.88 0.00 73.50 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 351.00 22.50 879.00 180.00 9696.00 10.96 3.25 3.68 1.10 98.90 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.87 0.00 2.31 20.07 0.00 76.75 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 435.50 19.00 2666.50 152.00 24512.00 9.18 3.23 1.21 0.35 93.00 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 [root@localhost vmuser]#
iotop查看：
[root@localhost vmuser]# iotop Total DISK READ: 124.32 K/s | Total DISK WRITE: 6.69 M/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 389 be/3 root 0.00 B/s 236.99 K/s 0.00 % 92.26 % [jbd2/sda3-8] 2482 be/4 postfix 124.32 K/s 0.00 B/s 0.00 % 88.09 % pickup -l -t fifo -u 2483 be/4 postfix 0.00 B/s 0.00 B/s 0.00 % 6.21 % qmgr -l -t fifo -u 23557 be/4 postfix 0.00 B/s 58.28 K/s 0.00 % 6.02 % cleanup -z -t unix -u 1532 be/4 root 0.00 B/s 27.20 K/s 0.00 % 3.82 % rsyslogd -c 4 23564 be/4 postfix 0.00 B/s 158.28 K/s 0.00 % 1.79 % cleanup -z -t unix -u 23545 be/4 postfix 0.00 B/s 260.31 K/s 0.00 % 1.62 % local -t unix unix -u 23570 be/4 postfix 0.00 B/s 19.43 K/s 0.00 % 0.75 % bounce -z -t unix -u 23556 be/4 postfix 0.00 B/s 58.28 K/s 0.00 % 0.75 % cleanup -z -t unix -uth :/usr/local/sa/sa-agent/main/~ocal/sa/sa-agent com.transfar.sa.agent.main.Bootstrap 17 rt/4 root 0.00 B/s 54.39 B/s 0.00 % 0.67 % bounce -z -t unix -u 9 23578 rt/4 postfix 0.00 B/s 38.85 B/s 0.00 % 0.60 % bounce -z -t unix -u 19 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/5] 20 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/5] 21 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/6] 23 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/6] 23 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/6]]
关闭不必要的postfix进程
[root@localhost vmuser]# service postfix stop
Shutting down postfix: [ OK ]
[root@localhost vmuser]#