使用cgroups来控制磁盘IO带宽

磨砺技术珠矶，践行数据之道，追求卓越价值

回到上一级页面：PostgreSQL内部结构与源代码研究索引页    回到顶级页面：PostgreSQL索引页

[作者 高健@博客园  luckyjackgao@gmail.com]

可资参考的资料：

http://kaivanov.blogspot.com/2012/07/setting-up-linux-cgroups-control-groups.html

http://www.oracle.com/technetwork/articles/servers-storage-admin/resource-controllers-linux-1506602.html

首先用 hdparm测试磁盘读取的最大带宽:

hdparm --direct -t /dev/sda

测试结果为200MB/s以上。

然后给用户postgres设置参数，保持磁盘读写I/O为10MB/秒以下（此用户名下所有进程总和）。

# Configuration file generated by cgsnapshot
mount {
    cpuset = /cgroup/cpuset;
    cpu = /cgroup/cpu;
    cpuacct = /cgroup/cpuacct;
    memory = /cgroup/memory;
    devices = /cgroup/devices;
    freezer = /cgroup/freezer;
    net_cls = /cgroup/net_cls;
    blkio = /cgroup/blkio;
}

group io-test {
    perm {
          task{
              uid=postgres;
              gid=postgres;
          }
          
          admin{
             uid=root;
             gid=root; 
          }

    } blkio {
        blkio.throttle.write_iops_device="";
        blkio.throttle.read_iops_device="";
        blkio.throttle.write_bps_device="8:0 10485760";
        blkio.throttle.read_bps_device="8:0 10485760";
        blkio.reset_stats="";
        blkio.weight="500";
        blkio.weight_device="";
    }
}

[postgres@cent6 Desktop]$ 

再看

[postgres@cent6 Desktop]$ cat /etc/cgrules.conf
# /etc/cgrules.conf
#
#Each line describes a rule for a user in the forms:
#
#<user>            <controllers>        <destination>
#<user>:<process name>    <controllers>        <destination>
#
#Where:
# <user> can be:
#        - an user name
#        - a group name, with @group syntax
#        - the wildcard *, for any user or group.
#        - The %, which is equivalent to "ditto". This is useful for
#          multiline rules where different cgroups need to be specified
#          for various hierarchies for a single user.
#
# <process name> is optional and it can be:
#     - a process name
#     - a full command path of a process
#
# <controller> can be:
#      - comma separated controller names (no spaces)
#      - * (for all mounted controllers)
#
# <destination> can be:
#      - path with-in the controller hierarchy (ex. pgrp1/gid1/uid1)
#
# Note:
# - It currently has rules based on uids, gids and process name.
#
# - Don't put overlapping rules. First rule which matches the criteria
#   will be executed.
#
# - Multiline rules can be specified for specifying different cgroups
#   for multiple hierarchies. In the example below, user "peter" has
#   specified 2 line rule. First line says put peter's task in test1/
#   dir for "cpu" controller and second line says put peter's tasks in
#   test2/ dir for memory controller. Make a note of "%" sign in second line.
#   This is an indication that it is continuation of previous rule.
#
#
#<user>      <controllers>      <destination>
#
#john          cpu        usergroup/faculty/john/
#john:cp       cpu        usergroup/faculty/john/cp
#@student      cpu,memory    usergroup/student/
#peter           cpu        test1/
#%           memory        test2/
#@root            *        admingroup/
#*        *        default/
postgres       blkio            io-test/
# End of file
#
#
#
[postgres@cent6 Desktop]$ 

先用dd命令实际测试看看：

[postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile2 bs=4K count=1024 oflag=direct
1024+0 records in
1024+0 records out
4194304 bytes (4.2 MB) copied, 0.372953 s, 11.2 MB/s
[postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile2 bs=8K count=1024 oflag=direct
1024+0 records in
1024+0 records out
8388608 bytes (8.4 MB) copied, 0.733823 s, 11.4 MB/s
[postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile2 bs=8K count=1024 oflag=direct
1024+0 records in
1024+0 records out
8388608 bytes (8.4 MB) copied, 0.733256 s, 11.4 MB/s
[postgres@cent6 Desktop]$ 
[postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile2 bs=16K count=1024 oflag=direct
1024+0 records in
1024+0 records out
[postgres@cent6 Desktop]$ 

当然，如果dd执行时，开数据块太多，又是这种完全空的块，误差就会增大：

[postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile1 bs=10240 count=300000
300000+0 records in
300000+0 records out
3072000000 bytes (3.1 GB) copied, 57.6779 s, 53.3 MB/s
[postgres@cent6 Desktop]$

用实际的下载动作进行测试：

[postgres@cent6 Desktop]$ wget http://centos.arcticnetwork.ca/6.4/isos/x86_64/CentOS-6.4-x86_64-LiveCD.iso

再看看IO状态：

[postgres@cent6 Desktop]$ iostat -x 20
Linux 2.6.32-279.el6.x86_64 (cent6.gao)     09/10/2013     _x86_64_    (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.38    0.00    3.55   12.16    0.00   82.91

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               7.50  1747.20    7.77   24.88   535.75 14224.09   452.17    12.54  384.25   4.41  14.40
dm-0              0.00     0.00   13.79 1771.99   524.65 14224.07     8.26  1975.37 1106.16   0.09  16.73
dm-1              0.00     0.00    0.38    0.00     3.03     0.00     8.00     0.00    3.68   2.34   0.09

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.65    0.00    2.14    0.73    0.00   93.48

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     1.82    1.09    1.56    49.64    25.44    28.24     0.03   11.78   3.57   0.95
dm-0              0.00     0.00    1.04    3.18    48.80    25.44    17.58     0.03    7.35   2.15   0.91
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.65    0.00    1.65    0.98    0.00   95.72

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00    60.28    0.00    1.44     0.00   492.12   341.14     0.02   11.79   8.86   1.28
dm-0              0.00     0.00    0.00   61.51     0.00   492.12     8.00     0.59    9.54   0.21   1.28
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

除了一开始的峰值，util长期在2%-3%之间，基本达到效果。

[作者 高健@博客园  luckyjackgao@gmail.com]

磨砺技术珠矶，践行数据之道，追求卓越价值

回到上一级页面：PostgreSQL内部结构与源代码研究索引页    回到顶级页面：PostgreSQL索引页