mysql占用磁盘IO过高的解决办法

一、现象

最近发现Mysql服务器磁盘IO一直很高

[root@push-5-221 ~]# iostat -k -d -x 1 10
Linux 3.10.0-957.el7.x86_64 (push-5-221)        2019年07月05日  _x86_64_        (8 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.08    0.00   14.89     0.09   177.79    23.88     0.34   23.05    5.42   23.05  11.01  16.40
dm-0              0.00     0.00    0.00    0.18     0.09     3.62    39.49     0.07  380.21    5.59  387.46  12.69   0.24
dm-1              0.00     0.00    0.00    0.00     0.00     0.01     8.98     0.00   91.25   12.04   92.94   6.55   0.00
dm-2              0.00     0.00    0.00   14.79     0.00   174.15    23.56     0.33   22.27    8.84   22.27  10.96  16.20

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00   83.00     0.00   650.50    15.67     2.55   30.66    0.00   30.66  12.02  99.80
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00   83.00     0.00   636.50    15.34     2.55   30.66    0.00   30.66  12.02  99.80

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00   88.00     0.00   684.50    15.56     2.60   29.42    0.00   29.42  11.35  99.90
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00   87.00     0.00   677.00    15.56     2.60   29.76    0.00   29.76  11.49 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00   87.00     0.00   687.50    15.80     2.48   28.77    0.00   28.77  11.51 100.10
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00   88.00     0.00   699.00    15.89     2.48   28.48    0.00   28.48  11.36 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00   88.00     0.00   756.50    17.19     2.48   28.22    0.00   28.22  11.35  99.90
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00   86.00     0.00   740.00    17.21     2.48   28.87    0.00   28.87  11.62  99.90

%util已经接近100%了，IO性能严重瓶颈，这台服务器有两个Mysql实例，3306是主库，3307是另一台Mysql的从库，进入从库查看从库状态

mysql> show slave statusG;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 172.28.5.223
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: master-223.000012
          Read_Master_Log_Pos: 511363808
               Relay_Log_File: push-5-221-relay-bin.000034
                Relay_Log_Pos: 382425277
        Relay_Master_Log_File: master-223.000012
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 382425062
              Relay_Log_Space: 511364283
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 2345
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 2233306
                  Master_UUID: 165a203c-8021-11e9-bc40-6c2b5992e592
             Master_Info_File: /home/mysql-5.7.26/data/3307/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Reading event from the relay log
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
1 row in set (0.00 sec)

ERROR: 
No query specified

mysql> 

Seconds_Behind_Master: 2345 发现从库同步延时很高，通过iotop查看

Total DISK READ :       0.00 B/s | Total DISK WRITE :     683.85 K/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     721.19 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                        
 7061 be/4 mysql       0.00 B/s  153.28 K/s  0.00 % 72.27 % mysqld --defaults-file=/etc/mysq~mysql/3307/mysql.sock --port=3307
 1766 be/4 mysql       0.00 B/s   27.51 K/s  0.00 %  8.51 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 2730 be/4 mysql       0.00 B/s   11.79 K/s  0.00 %  7.50 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 2164 be/4 mysql       0.00 B/s   11.79 K/s  0.00 %  7.47 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 8932 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  7.41 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 1840 be/4 mysql       0.00 B/s   35.37 K/s  0.00 %  7.14 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
  936 be/4 mysql       0.00 B/s   27.51 K/s  0.00 %  6.77 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 1296 be/4 mysql       0.00 B/s   15.72 K/s  0.00 %  6.31 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 1201 be/4 mysql       0.00 B/s   15.72 K/s  0.00 %  5.32 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 2728 be/4 mysql       0.00 B/s    7.86 K/s  0.00 %  5.05 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 2539 be/4 mysql       0.00 B/s    7.86 K/s  0.00 %  5.00 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 2531 be/4 mysql       0.00 B/s   47.16 K/s  0.00 %  4.21 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 6895 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  3.63 % mysqld --defaults-file=/etc/mysq~mysql/3307/mysql.sock --port=3307
 2303 be/4 mysql       0.00 B/s    7.86 K/s  0.00 %  3.54 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
  461 be/4 mysql       0.00 B/s   15.72 K/s  0.00 %  3.40 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 2523 be/4 mysql       0.00 B/s    3.93 K/s  0.00 %  3.36 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 2520 be/4 mysql       0.00 B/s    7.86 K/s  0.00 %  3.35 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 2316 be/4 mysql       0.00 B/s    7.86 K/s  0.00 %  2.85 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 2537 be/4 mysql       0.00 B/s   15.72 K/s  0.00 %  2.81 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 2731 be/4 mysql       0.00 B/s    7.86 K/s  0.00 %  2.42 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 8940 be/4 mysql       0.00 B/s    3.93 K/s  0.00 %  2.07 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 2310 be/4 mysql       0.00 B/s   15.72 K/s  0.00 %  1.91 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 8935 be/4 mysql       0.00 B/s  157.21 K/s  0.00 %  1.41 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 2302 be/4 mysql       0.00 B/s    3.93 K/s  0.00 %  1.30 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
 6899 be/4 mysql       0.00 B/s   31.44 K/s  0.00 %  0.93 % mysqld --defaults-file=/etc/mysq~mysql/3307/mysql.sock --port=3307
32593 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.61 % [kworker/3:1]
 7060 be/4 mysql       0.00 B/s   27.51 K/s  0.00 %  0.00 % mysqld --defaults-file=/etc/mysq~mysql/3307/mysql.sock --port=3307
 2535 be/4 mysql       0.00 B/s   15.72 K/s  0.00 %  0.00 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306
13824 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % systemd-udevd 
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % systemd --system --deserialize 17

发现3307和3306实例占用IO最高，进入3307实例，查看sync_binlog变量

mysql> show variables like '%sync_binlog%'
    -> ;
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| sync_binlog   | 1     |
+---------------+-------+
1 row in set (0.01 sec)

mysql> 

发现设置的是1，也就是说每次事务提交都会将binlog的缓存写入磁盘，严重影响磁盘效率，将变量设置为1000,

mysql> set global sync_binlog=1000;
Query OK, 0 rows affected (0.00 sec)

mysql> show variables like '%sync_binlog%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| sync_binlog   | 1000  |
+---------------+-------+
1 row in set (0.00 sec)

再查看innodb_flush_log_at_trx_commit参数

mysql> show variables like '%innodb_flush_log%';
+--------------------------------+-------+
| Variable_name                  | Value |
+--------------------------------+-------+
| innodb_flush_log_at_timeout    | 1     |
| innodb_flush_log_at_trx_commit | 1     |
+--------------------------------+-------+
2 rows in set (0.00 sec)

设置为1，也就是说每次事务提交，都会将innodb日志缓存写入磁盘，对磁盘效率影响很大，将它设置为2，每次事务提交时mysql都会把log buffer的数据写入log file，但是flush(刷到磁盘)操作并不会同时进行。该模式下，MySQL会每秒执行一次 flush(刷到磁盘)操作

mysql> set global innodb_flush_log_at_trx_commit=2;
Query OK, 0 rows affected (0.00 sec)

mysql> show variables like '%innodb_flush_log%';
+--------------------------------+-------+
| Variable_name                  | Value |
+--------------------------------+-------+
| innodb_flush_log_at_timeout    | 1     |
| innodb_flush_log_at_trx_commit | 2     |
+--------------------------------+-------+
2 rows in set (0.00 sec)

此时再查看从库同步状态

mysql> show slave statusG;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 172.28.5.223
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: master-223.000012
          Read_Master_Log_Pos: 539225447
               Relay_Log_File: push-5-221-relay-bin.000034
                Relay_Log_Pos: 490154357
        Relay_Master_Log_File: master-223.000012
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 490154142
              Relay_Log_Space: 539225922
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 1378
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 2233306
                  Master_UUID: 165a203c-8021-11e9-bc40-6c2b5992e592
             Master_Info_File: /home/mysql-5.7.26/data/3307/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: System lock
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
1 row in set (0.00 sec)

ERROR: 
No query specified

发现，同步延迟已经在很快缩小了，很快变为0，完全同步，这时通过iostat 查看io状态

[root@push-5-221 ~]# iostat -k -d -x 1 10/s  0.00 %  0.00 % [kworker/2:0H]
Linux 3.10.0-957.el7.x86_64 (push-5-221)        2019年07月05日  _x86_64_        (8 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.08    0.00   14.91     0.09   178.29    23.92     0.34   23.05    5.42   23.06  11.01  16.43
dm-0              0.00     0.00    0.00    0.18     0.09     3.62    39.48     0.07  380.15    5.59  387.40  12.69   0.24
dm-1              0.00     0.00    0.00    0.00     0.00     0.01     8.97     0.00   91.84   12.04   93.52   6.63   0.00
dm-2              0.00     0.00    0.00   14.81     0.00   174.66    23.59     0.33   22.28    8.84   22.28  10.96  16.23

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00  114.00     0.00  2013.00    35.32     3.14   27.40    0.00   27.40   8.75  99.80
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00  113.00     0.00  2007.50    35.53     3.14   27.65    0.00   27.65   8.83  99.80

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00  108.91     0.00  3026.24    55.57     3.10   27.85    0.00   27.85   9.05  98.51
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00  111.88     0.00  3121.29    55.80     3.10   27.11    0.00   27.11   8.81  98.51

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00   98.00     0.00  2648.00    54.04     2.81   29.63    0.00   29.63   9.71  95.20
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00   95.00     0.00  2551.50    53.72     2.81   30.59    0.00   30.59  10.02  95.20

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00  103.00     0.00  2037.00    39.55     2.97   28.67    0.00   28.67   9.12  93.90
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00  104.00     0.00  2051.00    39.44     2.97   28.39    0.00   28.39   9.03  93.90

发现%util依旧差不多100%，磁盘IO性能还是很低，我们再把3306实例的变量设置改一下

[root@push-5-221 ~]# mysql -uroot -p -S /var/lib/mysql/3306/mysql.sock 
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 1476252
Server version: 5.7.26-log MySQL Community Server (GPL)

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

mysql> show variables like '%sync_binlog%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| sync_binlog   | 1     |
+---------------+-------+
1 row in set (0.01 sec)

mysql> set global sync_binlog=1000;
Query OK, 0 rows affected (0.00 sec)

mysql> show variables like '%innodb_flush_log%';
+--------------------------------+-------+
| Variable_name                  | Value |
+--------------------------------+-------+
| innodb_flush_log_at_timeout    | 1     |
| innodb_flush_log_at_trx_commit | 1     |
+--------------------------------+-------+
2 rows in set (0.00 sec)

mysql> set global innodb_flush_log_at_trx_commit=2;
Query OK, 0 rows affected (0.00 sec)

mysql> show variables like '%innodb_flush_log%';
+--------------------------------+-------+
| Variable_name                  | Value |
+--------------------------------+-------+
| innodb_flush_log_at_timeout    | 1     |
| innodb_flush_log_at_trx_commit | 2     |
+--------------------------------+-------+
2 rows in set (0.00 sec)

mysql> 

此时，再通过iostat查看IO

[root@push-5-221 ~]# iostat -k -d -x 1 10
Linux 3.10.0-957.el7.x86_64 (push-5-221)        2019年07月05日  _x86_64_        (8 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.08    0.00   14.92     0.09   178.36    23.92     0.34   23.06    5.42   23.06  11.01  16.43
dm-0              0.00     0.00    0.00    0.18     0.09     3.62    39.48     0.07  380.15    5.59  387.39  12.69   0.24
dm-1              0.00     0.00    0.00    0.00     0.00     0.01     8.97     0.00   91.84   12.04   93.52   6.63   0.00
dm-2              0.00     0.00    0.00   14.81     0.00   174.72    23.59     0.33   22.28    8.84   22.28  10.96  16.23

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00   21.00     0.00   604.00    57.52     0.38   18.10    0.00   18.10  13.76  28.90
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00   21.00     0.00   604.00    57.52     0.38   18.10    0.00   18.10  13.76  28.90

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00   34.00     0.00  2220.50   130.62     1.29   37.88    0.00   37.88  12.88  43.80
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00   34.00     0.00  2220.50   130.62     1.29   37.88    0.00   37.88  12.88  43.80

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00   42.00     0.00  2056.00    97.90     1.29   30.67    0.00   30.67   9.40  39.50
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00   42.00     0.00  2056.00    97.90     1.29   30.67    0.00   30.67   9.40  39.50

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00   21.00     0.00   550.00    52.38     0.34   16.19    0.00   16.19  13.76  28.90
dm-0              0.00     0.00    0.00    2.00     0.00     1.50     1.50     0.05   24.00    0.00   24.00  24.00   4.80
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00   19.00     0.00   548.50    57.74     0.29   15.37    0.00   15.37  12.74  24.20

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00   18.00     0.00   524.00    58.22     0.34   18.61    0.00   18.61  13.33  24.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00   18.00     0.00   524.00    58.22     0.34   18.61    0.00   18.61  13.33  24.00

%util已经顺利降到40%左右了。