• 05:Sysbench压测-innodb_deadlock_detect参数对性能的影响


    sysbench压测-innodb_deadlock_detect参数对性能的影响

    一、OLTP测试前准备

    • 基本信息:
      • 主机信息
    CPU 内存 硬盘 系统版本 MySQL版本 sysbench版本
    Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz *2 2G 虚拟机硬盘 CentOS release 6.9 (Final) 5.7.18 1.1.0-76
    • sysbench 目录(/software/sysbench)
    • 脚本(mysql_oltp_sysbench.sh)
    • 创建dbtest库
    # line V1.2
    # mail:			
    # data:		    	2017-10-26
    # file_name: 		mysql_oltp_sysbench.sh
    # 修改符合公司环境     	
    
    #通过sysbench测试mysql相关性能,并将关键数据存储于‘dbtest.sysbenc_test’表中
    
    #----------自定义部分----------
    #定义记录测试结果的mysql连接相关参数,本例我在测试机上记录测试结果
    m_user='root'
    m_passwd='iforgot'
    m_port='3306'
    m_host='localhost'
    #测试结果存储于哪个库
    m_db='dbtest'
    #测试结果存储于哪个表
    m_table='sysbench_test'
    
    #sysbench lua脚本目录
    lua_dir=/software/sysbench/tests/include/oltp_legacy
    
    #sysbench 参数
    SYSBENCH_PARAMETER="--mysql-table-engine=innodb --oltp-table-size=5000000 --oltp-tables-count=16 --oltp-test-mode=complex --rand-type=uniform --rand-init=on --report-interval=10  --max-time=1500 "
    
    #画图维度(关注的三个指标,也就是会画三张图)
    target="request_per_second transactions_per_second 95_pct_time"
    
    #定义错误日志文件
    log=/tmp/mysql_oltp.log
    #定义分析结果文件
    data=/tmp/mysql_oltp.dat
    
    #定义测试线程
    threds_num='8 24 48 64 96 128 160 196 256'
    #每种场景的测试次数,分析时取平均值
    times=3
    #----------自定义部分结束---------- 
    
    # sysbenc cleanup and perpare
    #sysbench $lua_dir/parallel_prepare.lua --mysql-user=$6 --mysql-password=$7 --mysql-port=$5 --mysql-host=$4 $SYSBENCH_PARAMETER  --num-threads=8 cleanup
    
    #sysbench $lua_dir/parallel_prepare.lua --mysql-user=$6 --mysql-password=$7 --mysql-port=$5 --mysql-host=$4 $SYSBENCH_PARAMETER  --num-threads=8 prepare 
    
    #测试函数
    sb_test() {
     
        if [ "$3" == "read-only" ];then read_only='on';else read_only='off';fi    #根据脚本参数确定是否read-only
     
        #创建记录测试信息的表
        echo -e "
    ---------------
    创建测测试结果表$m_db.$m_table
    ---------------"
        return=$(mysql -u$m_user -p$m_passwd -P$m_port -h$m_host <<EOF 2>&1
            CREATE TABLE IF NOT EXISTS $m_db.$m_table (
            scenario varchar(30) NOT NULL DEFAULT '' COMMENT '测试场景',
            server_name varchar(15) NOT NULL COMMENT '被测DB name',
            test_type varchar(15) NOT NULL COMMENT 'read-only,read-write,insert等',
            sb_threads int(11) NOT NULL DEFAULT '0' COMMENT 'sysbench 测试线程',
            server_load decimal(12,2) NOT NULL DEFAULT '0.00' COMMENT '以当前线程测试完后立刻记录一分钟负载值',
            request_read int(11) NOT NULL DEFAULT '0',
            request_write int(11) NOT NULL DEFAULT '0',
            transactions_per_second decimal(12,2) NOT NULL DEFAULT '0.00',
            request_per_second decimal(12,2) NOT NULL DEFAULT '0.00',
            95_pct_time decimal(12,2) NOT NULL DEFAULT '0.00' COMMENT '单位毫秒'
            ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
    EOF
    )
        if [ $? -ne 0 ];then
            echo $return|sed 's/[Ww]arning:.*password on the command.*insecure.//'
            #echo "create table $m_db.$m_table failed"
            exit -1
        fi
    
        #开始测试,每种条件测$times次,分析时取平均值
        echo -e "
    ---------------
    场景:$2 模式:$3
    ---------------"
        for i in `seq $times`;do
            for sb_threds in $threds_num;do    #按照指定的sysbench线程测试
                printf "  %-10s %s
    " $sb_threds线程 第$i次运行...
                 
                #result 作为每次最小测试单元的结果,根据sysbench测试结果各参数的出现顺序
                #以request_read、request_write、request_total、request_per_second、total_time、95_pct_time为顺序插入表中
                #下条命令中,egerp之后的操作是为了对sysbench的输出做筛选和格式化,以便插入数据库
    
     		#sysbench 参数
    		#--oltp_table_count=1:指定测试过程中表的个数,0.5新增,0.4整个测试过程只有一个表。  
    		#--oltp-table-size=:指定表的大小,如果指定1000,那么它会往表里初始化1000条数据  
    		#--rand-init=on:是否随机初始化数据,如果不随机化那么初始好的数据每行内容除了主键不同外其他完全相同。  
    		#--num-threads=:测试过程中并发线程数,看测试要求来定并发压力。  
    		#--otlp-read-only=off:知否只读测试  
    		#--report-interval=10:每隔多久打印一次统计信息,单位秒,0.5新增  
    		#--rand-type=special:数据分布模式,special表示存在热点数据,uniform表示非热点数据模式,还有其他几个选项。  
    		#--rand-spec-pct=5:这个与上面那个选项相关,热点数据的百分比,我们公司的一个应用测试出来是4.9%的热点数据。  
    		#--mysql-table-engine=$type:表的存储引擎类型,innodb/myisam/tokudb/这些都可以。  
    		#--max-time=8000:这个命令跑多长时间,单位秒,与之相反的是指定请求数--max-requests  
    
    
                sysbench $lua_dir/oltp.lua --mysql-user=$6 --mysql-password=$7 --mysql-port=$5 --mysql-host=$4 
                          --num-threads=$sb_threds $SYSBENCH_PARAMETER --oltp-skip-trx=on --oltp-read-only=$read_only  run &> $log
                if [ $? -ne 0 ];then
                    echo -e "
    Sysbench error! For more information see $log"
                    exit -1
                fi
                result=$(cat $log | egrep  "read:|write:|transactions:|queries:|95th percentile:" | sed -r -e "s/[0-9]+ (//g" -e "s/ per sec.)//g" -e "s/m?s$//g"| awk  '{printf("%s ",$NF)}' | sed "s/ /,/g" | sed "s/,$//g"  |  sed "s/(//g")
               
    
                #测试完成后立刻记录系统一分钟负载值,可近似认为测试过程中proxy的负载抽样
                load=`uptime|awk -F: '{print $NF}'| awk -F , '{print $1}'`
                if [ -s $load ];then
                    load=0.00
                fi
                #本次测试结果写入数据库
                return=$(mysql -u$m_user -p$m_passwd -P$m_port -h$m_host  $m_db <<EOF 2>&1
                    INSERT INTO $m_table (scenario,server_name,test_type,sb_threads,server_load,request_read,
                                                request_write,transactions_per_second,request_per_second,95_pct_time) 
                    VALUES ('$2','$4','$3','$sb_threds','$load',$result);
    EOF
    )
    #	   echo "INSERT INTO $m_table (scenario,server_name,test_type,sb_threads,server_load,request_read,request_write,transactions_per_second,request_per_second,95_pct_time) VALUES ('$2','$4','$3','$sb_threds','$load',$result)"     
    # exit 1    
                if [ $? -ne 0 ];then
                    echo -e "
    ----------$sb_threds线程测试,第$i次插入数据库时失败----------"
                    #错误输出中排除mysql安全提示
                    echo $return|sed 's/[Ww]arning:.*password on the command.*insecure.//'
                    #echo "INSERT VALUES ('$2','$4','$3',$sb_threds,$load,$result)"
                    exit -2
                fi
                sleep 60    #让库歇一会,也让一分钟负载能够恢复到测试前的值
            done
     
        done
    }
     
    #结果分析函数
    sb_analyse() {
        #2>&1|grep部分为避免安全提示,使用该技巧就无法获取SQL执行的返回码了
        mysql -u$m_user -p$m_passwd -h$m_host -P$m_port <<EOF 2>&1|grep -v 'password on the command line'
            SELECT
            scenario, 
            server_name,
            test_type,
            sb_threads,
            convert(avg(server_load),decimal(12,2)) as server_load,
            convert(avg(request_read),decimal(12,0)) as request_read,
            convert(avg(request_write),decimal(12,0)) as request_write,
            convert(avg(transactions_per_second),decimal(12,2)) as transactions_per_second,
            convert(avg(request_per_second),decimal(12,2)) as request_per_second,
            convert(avg(95_pct_time),decimal(12,2)) as 95_pct_time
            FROM $m_db.$m_table group by scenario,server_name,test_type,sb_threads
    EOF
    }
     
    #画图函数
    sb_chart() {
        sb_analyse >$data
     
        for chart_type in $target;do
            col_num=0    #该行及下面这个for循环用于取得三个指标在数据中的列号
            for col_name in `cat $data |awk 'NR<2 {print}'`;do
                let col_num++
                if [ $col_name == $chart_type ];then break;fi
            done
             
            if [ $chart_type == "transactions_per_second" ];then    #根据图表特点为不同的chart_type设置gunplot不同的key position
                key_pos="bottom right"
                unit=""
            elif [ $chart_type == "request_per_second" ];then    #根据图表特点为不同的chart_type设置gunplot不同的key position
                key_pos="bottom right"
                unit=""
            elif [ $chart_type == "95_pct_time" ];then
                key_pos="top left"
                unit="(ms)"
            fi
     
            plot_cmd="set term png size 800,600;set output '/tmp/$chart_type.png';set title '$chart_type $unit';set grid;set key $key_pos;plot "
             
            if [ $# -eq 0 ];then
                #对分析结果中所有场景进行画图
                for scenario in `mysql -u$m_user -p$m_passwd -h$m_host -P$m_port -s -e "select distinct(scenario) from $m_db.$m_table" 2>/dev/null`;do
                    sb_analyse | awk -v scenario=$scenario '$1 == scenario {print}' > /tmp/"$scenario.dat"
                    plot_cmd=${plot_cmd}"'/tmp/"$scenario.dat"' using $col_num:xtic(4) title '$scenario' with linespoints lw 2,"
                done
                plot_cmd=$(echo $plot_cmd | sed 's/,$//g')
                echo $plot_cmd | gnuplot
            else
                #只绘制指定的场景
                for scenario in $*;do
                    sb_analyse | awk -v scenario=$scenario '$1 == scenario {print}' > /tmp/"$scenario.dat"
                    plot_cmd=${plot_cmd}"'/tmp/"$scenario.dat"' using $col_num:xtic(4) title '$scenario' with linespoints lw 2,"
                done
                plot_cmd=$(echo $plot_cmd | sed 's/,$//g')
                echo "$plot_cmd" | gnuplot
            fi
        done
    }
     
    #脚本使用说明/参数判断
    if [ $# -eq 1 ] && [ $1 == "-h" -o $1 == "--help" ];then
        echo -e "
    Usage: $0 test (test_scenario) (test_type) (mysql_host) (mysql_port) (mysql_user) (mysql_password)
           $0 analyse
           $0 chart [scenario]...
    "
        echo ----------
        echo -e "测试: 子命令test"
        echo -e "      test_scenario: 自定义的测试场景名"
        echo -e "      test_type: read-only 或 read-write, 表示测试模式"
        echo -e "      其余4参数表示待测试MySQL连接相关信息,密码若包含特殊字符,将其置于单引号内"
        echo -e "----------"
        echo -e "分析: 子命令analyse"
        echo -e "----------"
        echo -e "画图: 子命令chart"
        echo -e "      会在/tmp/下生成request_per_second.png transactions_per_second.png 95_pct_time.png 三张图"        
        echo -e "      chart (对分析结果中的所有测试场景画图)"
        echo -e "      chart scenario ... (对指定的测试场景画图,场景名依据先前自定义的名称)
    "
        exit -1
    elif [ "$1" == "test" -a  $# -eq 7 ];then
        sb_test $1 $2 $3 $4 $5 $6 $7
    elif [ "$1" == "analyse" -a $# -eq 1 ];then
        sb_analyse
    elif [ "$1" == "chart" ];then
        #chart函数可不接参数,也可接任意个'测试场景'作为参数
        arg=($*)
        arg_len=${#arg[@]}
        sb_chart ${arg[@]:1:$arg_len-1}
    else
        echo -e "
    Usage: $0 test (test_scenario) (test_type) (mysql_host) (mysql_port) (mysql_user) (mysql_password)
           $0 analyse
           $0 chart [scenario]...
    "
    fi
    
    

    初始化测试库环境(总共16个测试表,每个表 500000 条记录,填充随机生成的数据):

    # cd /software/sysbench
    
    -- 在脚本里面把 # sysbenc cleanup and perpare 这一行下面两个语句去掉注释,或者自己执行
    # sysbench $lua_dir/parallel_prepare.lua --mysql-user=$6 --mysql-password=$7 --mysql-port=$5 --mysql-host=$4 $SYSBENCH_PARAMETER  --num-threads=8 cleanup
    # sysbench $lua_dir/parallel_prepare.lua --mysql-user=$6 --mysql-password=$7 --mysql-port=$5 --mysql-host=$4 $SYSBENCH_PARAMETER  --num-threads=8 prepare 
    
    -- 自己执行
    
    /usr/local/bin/sysbench /software/sysbench/tests/include/oltp_legacy/oltp.lua  --mysql-user=xxx --mysql-password=xxx --mysql-host=localhost  --mysql-socket=/r2/mysqldata/mysql.sock --mysql-port=3306 --mysql-table-engine=innodb --oltp-table-size=5000000 --oltp-tables-count=16 --oltp-test-mode=complex --rand-type=uniform --rand-init=on --report-interval=10  --max-time=1500  --num-threads=8   cleanup
    
    /usr/local/bin/sysbench /software/sysbench/tests/include/oltp_legacy/oltp.lua  --mysql-user=xxx --mysql-password=xxx --mysql-host=localhost  --mysql-socket=/r2/mysqldata/mysql.sock --mysql-port=3306 --mysql-table-engine=innodb --oltp-table-size=5000000 --oltp-tables-count=16 --oltp-test-mode=complex --rand-type=uniform --rand-init=on --report-interval=10  --max-time=1500  --num-threads=8   prepare 
    
    
    • 关于这几个参数的解释:

      • -- /software/sysbench/tests/include/oltp_legacy/oltp.lua 表示调用 oltp.lua 脚本进行 oltp 模式测试

      • --oltp_tables_count=16 表示会生成 16 个测试表

      • --oltp-table-size=5000000 表示每个测试表填充数据量为 5000000

      • --rand-init=on 表示每个测试表都是用随机数据来填充的

    如果在本机,也可以使用 –mysql-socket 指定 socket 文件来连接。

    真实测试场景中,数据表建议不低于10个,单表数据量不低于500万行,当然了,要视服务器硬件配置而定。如果是配备了SSD或者PCIE SSD这种高IOPS设备的话,则建议单表数据量最少不低于1亿行。

    二、进行OLTP测试

    • 执行脚本测试,使用sh 脚本 test (测试名称) (测试类型) (mysql_host) (mysql_port) (mysql_user) (mysql_password),如下:
    [root@dbproxy01 software]# sh  mysql_oltp_sysbench.sh test deadlock_check_on read-write localhost 3306 root 'iforgot'   
    
    ---------------
    创建测测试结果表dbtest.sysbench_test
    ---------------
    
    ---------------
    场景:deadlock_check_on 模式:read-write
    ---------------
      8线程    第1次运行...
      24线程   第1次运行...
      48线程   第1次运行...
      64线程   第1次运行...
      96线程   第1次运行...
      128线程  第1次运行...
      160线程  第1次运行...
      196线程  第1次运行...
      256线程  第1次运行...
      8线程    第2次运行...
      24线程   第2次运行...
      48线程   第2次运行...
      64线程   第2次运行...
      96线程   第2次运行...
      128线程  第2次运行...
      160线程  第2次运行...
      196线程  第2次运行...
      256线程  第2次运行...
      8线程    第3次运行...
      24线程   第3次运行...
      48线程   第3次运行...
      64线程   第3次运行...
      96线程   第3次运行...
      128线程  第3次运行...
      160线程  第3次运行...
      196线程  第3次运行...
      256线程  第3次运行...
    [root@dbproxy01 software]# sh  mysql_oltp_sysbench.sh test deadlock_check_off read-write localhost 3306 root 'iforgot'     
    
    ---------------
    创建测测试结果表dbtest.sysbench_test
    ---------------
    
    ---------------
    场景:deadlock_check_off 模式:read-write
    ---------------
      8线程    第1次运行...
      24线程   第1次运行...
      48线程   第1次运行...
      64线程   第1次运行...
      96线程   第1次运行...
      128线程  第1次运行...
      160线程  第1次运行...
      196线程  第1次运行...
      256线程  第1次运行...
      8线程    第2次运行...
      24线程   第2次运行...
      48线程   第2次运行...
      64线程   第2次运行...
      96线程   第2次运行...
      128线程  第2次运行...
      160线程  第2次运行...
      196线程  第2次运行...
      256线程  第2次运行...
      8线程    第3次运行...
      24线程   第3次运行...
      48线程   第3次运行...
      64线程   第3次运行...
      96线程   第3次运行...
      128线程  第3次运行...
      160线程  第3次运行...
      196线程  第3次运行...
      256线程  第3次运行...
    You have new mail in /var/spool/mail/root
    
    • 命令测试

      • 启动了256个线程,16个测试表,每个表中灌入500W条数据,测试时间1个小时

    sysbench /software/sysbench/tests/include/oltp_legacy/oltp.lua --mysql-host=localhost --mysql-port=3306 --mysql-user=root --mysql-password=yourpassword --oltp_tables_count=10 --oltp-table-size=5000000 --num-threads=256 --oltp-read-only=off --report-interval=10 --rand-type=uniform --max-time=3600 --max-requests=1000000 --percentile=99 run >> /tmp/sysbench_oltpX_256_20171026.log

    • 选项说明:

      • --num-threads=8 表示发起 8个并发连接

      • --oltp-read-only=off 表示不要进行只读测试,也就是会采用读写混合模式测试

      • --report-interval=10 表示每10秒输出一次测试进度报告

      • --rand-type=uniform 表示随机类型为固定模式,其他几个可选随机模式:uniform(固定),gaussian(高斯),special(特定的),pareto(帕累托)

      • --max-time=3600 表示最大执行时长为 3600秒

      • --max-requests=0 表示总请求数为 0,因为上面已经定义了总执行时长,所以总请求数可以设定为 0;也可以只设定总请求数,不设定最大执行时长

      • --percentile=99 表示设定采样比例,默认是 95%,即丢弃1%的长请求,在剩余的99%里取最大值;即模拟 对16个表并发OLTP测试,每个表500万行记录,持续压测时间为 1小时。

    真实测试场景中,建议持续压测时长不小于30分钟,否则测试数据可能不具参考意义。

    三、测试结果解读:

    • 测试结果解读如下:
    sysbench 1.1.0-76eeace (using bundled LuaJIT 2.1.0-beta3)
    
    Running the test with following options:
    Number of threads: 160
    Report intermediate results every 10 second(s)
    Initializing random number generator from current time
    
    
    Initializing worker threads...
    
    Threads started!
    
    [ 10s ] thds: 160 tps: 191.06 qps: 3607.09 (r/w/o: 2836.47/770.61/0.00) lat (ms,95%): 1648.20 err/s: 0.00 reconn/s: 0.00
    [ 20s ] thds: 160 tps: 201.81 qps: 3676.37 (r/w/o: 2858.73/817.64/0.00) lat (ms,95%): 1938.16 err/s: 0.00 reconn/s: 0.00
    [ 30s ] thds: 160 tps: 205.60 qps: 3676.07 (r/w/o: 2854.66/821.42/0.00) lat (ms,95%): 2120.76 err/s: 0.00 reconn/s: 0.00
    [ 40s ] thds: 160 tps: 203.50 qps: 3652.39 (r/w/o: 2844.99/807.40/0.00) lat (ms,95%): 1618.78 err/s: 0.00 reconn/s: 0.00
    [ 50s ] thds: 160 tps: 200.30 qps: 3619.58 (r/w/o: 2811.88/807.70/0.00) lat (ms,95%): 2120.76 err/s: 0.00 reconn/s: 0.00
    [ 60s ] thds: 160 tps: 194.80 qps: 3465.32 (r/w/o: 2695.62/769.70/0.00) lat (ms,95%): 2320.55 err/s: 0.00 reconn/s: 0.00
    [ 70s ] thds: 160 tps: 192.20 qps: 3476.24 (r/w/o: 2704.65/771.59/0.00) lat (ms,95%): 2279.14 err/s: 0.00 reconn/s: 0.00
    [ 80s ] thds: 160 tps: 185.59 qps: 3364.28 (r/w/o: 2613.91/750.37/0.00) lat (ms,95%): 2009.23 err/s: 0.00 reconn/s: 0.00
    [ 90s ] thds: 160 tps: 207.51 qps: 3688.34 (r/w/o: 2870.09/818.25/0.00) lat (ms,95%): 2082.91 err/s: 0.00 reconn/s: 0.00
    [ 100s ] thds: 160 tps: 194.10 qps: 3569.30 (r/w/o: 2781.90/787.40/0.00) lat (ms,95%): 2405.65 err/s: 0.00 reconn/s: 0.00
    [ 110s ] thds: 160 tps: 205.59 qps: 3637.99 (r/w/o: 2825.92/812.08/0.00) lat (ms,95%): 1803.47 err/s: 0.00 reconn/s: 0.00
    [ 120s ] thds: 160 tps: 195.60 qps: 3511.58 (r/w/o: 2723.56/788.02/0.00) lat (ms,95%): 2159.29 err/s: 0.00 reconn/s: 0.00
    [ 130s ] thds: 160 tps: 194.20 qps: 3524.78 (r/w/o: 2749.78/774.99/0.00) lat (ms,95%): 1973.38 err/s: 0.00 reconn/s: 0.00
    [ 140s ] thds: 160 tps: 205.10 qps: 3676.31 (r/w/o: 2857.41/818.90/0.00) lat (ms,95%): 2585.31 err/s: 0.00 reconn/s: 0.00
    [ 150s ] thds: 160 tps: 196.50 qps: 3542.90 (r/w/o: 2755.30/787.60/0.00) lat (ms,95%): 2880.27 err/s: 0.00 reconn/s: 0.00
    [ 160s ] thds: 160 tps: 193.90 qps: 3471.78 (r/w/o: 2700.59/771.20/0.00) lat (ms,95%): 2449.36 err/s: 0.00 reconn/s: 0.00
    [ 170s ] thds: 160 tps: 187.70 qps: 3372.65 (r/w/o: 2621.96/750.69/0.00) lat (ms,95%): 2493.86 err/s: 0.00 reconn/s: 0.00
    [ 180s ] thds: 160 tps: 197.50 qps: 3556.89 (r/w/o: 2767.29/789.60/0.00) lat (ms,95%): 2539.17 err/s: 0.00 reconn/s: 0.00
    [ 190s ] thds: 160 tps: 197.19 qps: 3608.89 (r/w/o: 2814.81/794.08/0.00) lat (ms,95%): 2238.47 err/s: 0.00 reconn/s: 0.00
    [ 200s ] thds: 160 tps: 192.31 qps: 3479.14 (r/w/o: 2706.11/773.03/0.00) lat (ms,95%): 2279.14 err/s: 0.00 reconn/s: 0.00
    [ 210s ] thds: 160 tps: 184.60 qps: 3331.36 (r/w/o: 2593.75/737.61/0.00) lat (ms,95%): 2320.55 err/s: 0.00 reconn/s: 0.00
    [ 220s ] thds: 160 tps: 202.56 qps: 3540.41 (r/w/o: 2740.76/799.64/0.00) lat (ms,95%): 2009.23 err/s: 0.00 reconn/s: 0.00
    [ 230s ] thds: 160 tps: 187.04 qps: 3465.88 (r/w/o: 2704.73/761.15/0.00) lat (ms,95%): 2362.72 err/s: 0.00 reconn/s: 0.00
    [ 240s ] thds: 160 tps: 194.70 qps: 3461.31 (r/w/o: 2687.71/773.60/0.00) lat (ms,95%): 1836.24 err/s: 0.00 reconn/s: 0.00
    [ 250s ] thds: 160 tps: 207.70 qps: 3708.63 (r/w/o: 2883.95/824.68/0.00) lat (ms,95%): 2320.55 err/s: 0.00 reconn/s: 0.00
    [ 260s ] thds: 160 tps: 184.60 qps: 3291.36 (r/w/o: 2553.87/737.49/0.00) lat (ms,95%): 2680.11 err/s: 0.00 reconn/s: 0.00
    [ 270s ] thds: 160 tps: 187.51 qps: 3392.20 (r/w/o: 2639.58/752.62/0.00) lat (ms,95%): 2585.31 err/s: 0.00 reconn/s: 0.00
    [ 280s ] thds: 160 tps: 180.58 qps: 3255.19 (r/w/o: 2529.88/725.31/0.00) lat (ms,95%): 2279.14 err/s: 0.00 reconn/s: 0.00
    [ 290s ] thds: 160 tps: 190.32 qps: 3438.59 (r/w/o: 2682.00/756.59/0.00) lat (ms,95%): 2449.36 err/s: 0.00 reconn/s: 0.00
    [ 300s ] thds: 160 tps: 172.00 qps: 3083.44 (r/w/o: 2397.83/685.61/0.00) lat (ms,95%): 2880.27 err/s: 0.00 reconn/s: 0.00
    [ 310s ] thds: 160 tps: 183.50 qps: 3334.89 (r/w/o: 2589.09/745.80/0.00) lat (ms,95%): 2009.23 err/s: 0.00 reconn/s: 0.00
    [ 320s ] thds: 160 tps: 182.20 qps: 3331.83 (r/w/o: 2593.85/737.99/0.00) lat (ms,95%): 2680.11 err/s: 0.00 reconn/s: 0.00
    [ 330s ] thds: 160 tps: 189.80 qps: 3384.82 (r/w/o: 2641.81/743.00/0.00) lat (ms,95%): 2279.14 err/s: 0.00 reconn/s: 0.00
    [ 340s ] thds: 160 tps: 190.10 qps: 3407.64 (r/w/o: 2642.15/765.49/0.00) lat (ms,95%): 2159.29 err/s: 0.00 reconn/s: 0.00
    [ 350s ] thds: 160 tps: 199.91 qps: 3588.45 (r/w/o: 2794.32/794.13/0.00) lat (ms,95%): 2238.47 err/s: 0.00 reconn/s: 0.00
    [ 360s ] thds: 160 tps: 208.20 qps: 3696.66 (r/w/o: 2867.77/828.89/0.00) lat (ms,95%): 2405.65 err/s: 0.00 reconn/s: 0.00
    [ 370s ] thds: 160 tps: 191.90 qps: 3550.07 (r/w/o: 2769.95/780.11/0.00) lat (ms,95%): 2449.36 err/s: 0.00 reconn/s: 0.00
    [ 380s ] thds: 160 tps: 205.80 qps: 3629.06 (r/w/o: 2816.37/812.69/0.00) lat (ms,95%): 1869.60 err/s: 0.00 reconn/s: 0.00
    [ 390s ] thds: 160 tps: 199.70 qps: 3576.92 (r/w/o: 2777.54/799.38/0.00) lat (ms,95%): 2279.14 err/s: 0.00 reconn/s: 0.00
    [ 400s ] thds: 160 tps: 198.70 qps: 3592.75 (r/w/o: 2798.64/794.11/0.00) lat (ms,95%): 2362.72 err/s: 0.00 reconn/s: 0.00
    [ 410s ] thds: 160 tps: 192.40 qps: 3501.76 (r/w/o: 2728.27/773.49/0.00) lat (ms,95%): 2320.55 err/s: 0.00 reconn/s: 0.00
    [ 420s ] thds: 160 tps: 196.70 qps: 3543.42 (r/w/o: 2756.12/787.31/0.00) lat (ms,95%): 2449.36 err/s: 0.00 reconn/s: 0.00
    [ 430s ] thds: 160 tps: 187.70 qps: 3436.94 (r/w/o: 2670.06/766.89/0.00) lat (ms,95%): 3095.38 err/s: 0.00 reconn/s: 0.00
    [ 440s ] thds: 160 tps: 202.50 qps: 3583.69 (r/w/o: 2787.97/795.72/0.00) lat (ms,95%): 2362.72 err/s: 0.00 reconn/s: 0.00
    [ 450s ] thds: 160 tps: 195.50 qps: 3458.02 (r/w/o: 2685.04/772.98/0.00) lat (ms,95%): 2880.27 err/s: 0.00 reconn/s: 0.00
    [ 460s ] thds: 160 tps: 191.00 qps: 3474.64 (r/w/o: 2707.93/766.71/0.00) lat (ms,95%): 2632.28 err/s: 0.00 reconn/s: 0.00
    [ 470s ] thds: 160 tps: 186.80 qps: 3435.28 (r/w/o: 2678.19/757.10/0.00) lat (ms,95%): 2632.28 err/s: 0.00 reconn/s: 0.00
    [ 480s ] thds: 160 tps: 190.80 qps: 3393.26 (r/w/o: 2633.24/760.01/0.00) lat (ms,95%): 2159.29 err/s: 0.00 reconn/s: 0.00
    [ 490s ] thds: 160 tps: 197.10 qps: 3532.41 (r/w/o: 2749.51/782.90/0.00) lat (ms,95%): 2585.31 err/s: 0.00 reconn/s: 0.00
    [ 500s ] thds: 160 tps: 192.00 qps: 3474.78 (r/w/o: 2708.19/766.60/0.00) lat (ms,95%): 2828.87 err/s: 0.00 reconn/s: 0.00
    [ 510s ] thds: 160 tps: 204.50 qps: 3636.19 (r/w/o: 2819.19/817.00/0.00) lat (ms,95%): 1836.24 err/s: 0.00 reconn/s: 0.00
    [ 520s ] thds: 160 tps: 196.90 qps: 3554.60 (r/w/o: 2764.70/789.90/0.00) lat (ms,95%): 2159.29 err/s: 0.00 reconn/s: 0.00
    [ 530s ] thds: 160 tps: 198.90 qps: 3658.99 (r/w/o: 2845.19/813.80/0.00) lat (ms,95%): 2320.55 err/s: 0.00 reconn/s: 0.00
    [ 540s ] thds: 160 tps: 199.40 qps: 3557.45 (r/w/o: 2771.24/786.21/0.00) lat (ms,95%): 2045.74 err/s: 0.00 reconn/s: 0.00
    [ 550s ] thds: 160 tps: 197.68 qps: 3497.51 (r/w/o: 2714.77/782.74/0.00) lat (ms,95%): 2362.72 err/s: 0.00 reconn/s: 0.00
    [ 560s ] thds: 160 tps: 194.62 qps: 3547.18 (r/w/o: 2763.52/783.66/0.00) lat (ms,95%): 2539.17 err/s: 0.00 reconn/s: 0.00
    [ 570s ] thds: 160 tps: 199.69 qps: 3613.70 (r/w/o: 2813.32/800.38/0.00) lat (ms,95%): 2405.65 err/s: 0.00 reconn/s: 0.00
    [ 580s ] thds: 160 tps: 192.40 qps: 3464.96 (r/w/o: 2696.55/768.41/0.00) lat (ms,95%): 2045.74 err/s: 0.00 reconn/s: 0.00
    [ 590s ] thds: 160 tps: 203.20 qps: 3576.71 (r/w/o: 2770.71/806.00/0.00) lat (ms,95%): 2120.76 err/s: 0.00 reconn/s: 0.00
    [ 600s ] thds: 160 tps: 183.70 qps: 3391.76 (r/w/o: 2640.25/751.51/0.00) lat (ms,95%): 2985.89 err/s: 0.00 reconn/s: 0.00
    [ 610s ] thds: 160 tps: 194.10 qps: 3529.10 (r/w/o: 2749.90/779.20/0.00) lat (ms,95%): 2585.31 err/s: 0.00 reconn/s: 0.00
    [ 620s ] thds: 160 tps: 199.10 qps: 3517.24 (r/w/o: 2736.15/781.09/0.00) lat (ms,95%): 2082.91 err/s: 0.00 reconn/s: 0.00
    [ 630s ] thds: 160 tps: 193.29 qps: 3486.63 (r/w/o: 2711.07/775.56/0.00) lat (ms,95%): 3151.62 err/s: 0.00 reconn/s: 0.00
    [ 640s ] thds: 160 tps: 194.91 qps: 3466.81 (r/w/o: 2693.06/773.75/0.00) lat (ms,95%): 2778.39 err/s: 0.00 reconn/s: 0.00
    [ 650s ] thds: 160 tps: 183.90 qps: 3358.45 (r/w/o: 2614.24/744.21/0.00) lat (ms,95%): 2828.87 err/s: 0.00 reconn/s: 0.00
    [ 660s ] thds: 160 tps: 212.10 qps: 3791.26 (r/w/o: 2943.27/847.99/0.00) lat (ms,95%): 2009.23 err/s: 0.00 reconn/s: 0.00
    [ 670s ] thds: 160 tps: 200.70 qps: 3661.95 (r/w/o: 2866.46/795.49/0.00) lat (ms,95%): 2279.14 err/s: 0.00 reconn/s: 0.00
    [ 680s ] thds: 160 tps: 210.21 qps: 3700.10 (r/w/o: 2859.97/840.12/0.00) lat (ms,95%): 2120.76 err/s: 0.00 reconn/s: 0.00
    [ 690s ] thds: 160 tps: 204.90 qps: 3721.54 (r/w/o: 2892.85/828.69/0.00) lat (ms,95%): 2405.65 err/s: 0.00 reconn/s: 0.00
    [ 700s ] thds: 160 tps: 208.89 qps: 3764.38 (r/w/o: 2936.01/828.37/0.00) lat (ms,95%): 2120.76 err/s: 0.00 reconn/s: 0.00
    [ 710s ] thds: 160 tps: 198.01 qps: 3523.37 (r/w/o: 2734.83/788.54/0.00) lat (ms,95%): 2198.52 err/s: 0.00 reconn/s: 0.00
    [ 720s ] thds: 160 tps: 195.50 qps: 3560.69 (r/w/o: 2771.69/789.00/0.00) lat (ms,95%): 2159.29 err/s: 0.00 reconn/s: 0.00
    [ 730s ] thds: 160 tps: 204.60 qps: 3710.72 (r/w/o: 2889.64/821.08/0.00) lat (ms,95%): 2680.11 err/s: 0.00 reconn/s: 0.00
    [ 740s ] thds: 160 tps: 198.40 qps: 3616.78 (r/w/o: 2815.09/801.70/0.00) lat (ms,95%): 2320.55 err/s: 0.00 reconn/s: 0.00
    [ 750s ] thds: 160 tps: 206.71 qps: 3681.49 (r/w/o: 2862.77/818.72/0.00) lat (ms,95%): 2198.52 err/s: 0.00 reconn/s: 0.00
    [ 760s ] thds: 160 tps: 204.28 qps: 3662.01 (r/w/o: 2843.08/818.94/0.00) lat (ms,95%): 2362.72 err/s: 0.00 reconn/s: 0.00
    [ 770s ] thds: 160 tps: 202.51 qps: 3694.72 (r/w/o: 2873.77/820.95/0.00) lat (ms,95%): 2279.14 err/s: 0.00 reconn/s: 0.00
    [ 780s ] thds: 160 tps: 207.50 qps: 3648.42 (r/w/o: 2837.42/811.00/0.00) lat (ms,95%): 2585.31 err/s: 0.00 reconn/s: 0.00
    [ 790s ] thds: 160 tps: 204.90 qps: 3658.86 (r/w/o: 2841.54/817.31/0.00) lat (ms,95%): 2198.52 err/s: 0.00 reconn/s: 0.00
    [ 800s ] thds: 160 tps: 206.70 qps: 3801.66 (r/w/o: 2962.17/839.49/0.00) lat (ms,95%): 2159.29 err/s: 0.00 reconn/s: 0.00
    [ 810s ] thds: 160 tps: 211.20 qps: 3764.16 (r/w/o: 2927.57/836.59/0.00) lat (ms,95%): 2728.81 err/s: 0.00 reconn/s: 0.00
    [ 820s ] thds: 160 tps: 187.40 qps: 3447.51 (r/w/o: 2686.51/761.00/0.00) lat (ms,95%): 2585.31 err/s: 0.00 reconn/s: 0.00
    [ 830s ] thds: 160 tps: 219.10 qps: 3842.56 (r/w/o: 2980.57/861.99/0.00) lat (ms,95%): 1938.16 err/s: 0.00 reconn/s: 0.00
    [ 840s ] thds: 160 tps: 208.51 qps: 3748.90 (r/w/o: 2914.48/834.42/0.00) lat (ms,95%): 2632.28 err/s: 0.00 reconn/s: 0.00
    [ 850s ] thds: 160 tps: 204.90 qps: 3690.12 (r/w/o: 2869.64/820.48/0.00) lat (ms,95%): 2493.86 err/s: 0.00 reconn/s: 0.00
    [ 860s ] thds: 160 tps: 196.70 qps: 3582.75 (r/w/o: 2794.64/788.11/0.00) lat (ms,95%): 2362.72 err/s: 0.00 reconn/s: 0.00
    [ 870s ] thds: 160 tps: 195.07 qps: 3511.24 (r/w/o: 2730.54/780.70/0.00) lat (ms,95%): 2680.11 err/s: 0.00 reconn/s: 0.00
    [ 880s ] thds: 160 tps: 192.62 qps: 3462.84 (r/w/o: 2692.84/770.00/0.00) lat (ms,95%): 2493.86 err/s: 0.00 reconn/s: 0.00
    [ 890s ] thds: 160 tps: 193.90 qps: 3463.73 (r/w/o: 2691.12/772.61/0.00) lat (ms,95%): 2728.81 err/s: 0.00 reconn/s: 0.00
    [ 900s ] thds: 160 tps: 190.90 qps: 3419.62 (r/w/o: 2657.14/762.48/0.00) lat (ms,95%): 2279.14 err/s: 0.00 reconn/s: 0.00
    [ 910s ] thds: 160 tps: 190.90 qps: 3480.76 (r/w/o: 2711.05/769.71/0.00) lat (ms,95%): 2279.14 err/s: 0.00 reconn/s: 0.00
    [ 920s ] thds: 160 tps: 199.90 qps: 3558.55 (r/w/o: 2763.56/794.99/0.00) lat (ms,95%): 2493.86 err/s: 0.00 reconn/s: 0.00
    [ 930s ] thds: 160 tps: 186.20 qps: 3409.94 (r/w/o: 2659.35/750.59/0.00) lat (ms,95%): 2880.27 err/s: 0.00 reconn/s: 0.00
    [ 940s ] thds: 160 tps: 186.20 qps: 3333.26 (r/w/o: 2589.44/743.81/0.00) lat (ms,95%): 2238.47 err/s: 0.00 reconn/s: 0.00
    [ 950s ] thds: 160 tps: 199.50 qps: 3593.41 (r/w/o: 2798.61/794.80/0.00) lat (ms,95%): 2585.31 err/s: 0.00 reconn/s: 0.00
    [ 960s ] thds: 160 tps: 198.60 qps: 3640.78 (r/w/o: 2827.28/813.49/0.00) lat (ms,95%): 2728.81 err/s: 0.00 reconn/s: 0.00
    [ 970s ] thds: 160 tps: 197.39 qps: 3442.37 (r/w/o: 2675.62/766.75/0.00) lat (ms,95%): 2238.47 err/s: 0.00 reconn/s: 0.00
    [ 980s ] thds: 160 tps: 196.42 qps: 3554.09 (r/w/o: 2763.90/790.19/0.00) lat (ms,95%): 2238.47 err/s: 0.00 reconn/s: 0.00
    [ 990s ] thds: 160 tps: 195.60 qps: 3511.54 (r/w/o: 2730.05/781.49/0.00) lat (ms,95%): 2493.86 err/s: 0.00 reconn/s: 0.00
    [ 1000s ] thds: 160 tps: 175.90 qps: 3246.12 (r/w/o: 2544.64/701.48/0.00) lat (ms,95%): 2320.55 err/s: 0.00 reconn/s: 0.00
    [ 1010s ] thds: 160 tps: 207.20 qps: 3695.93 (r/w/o: 2860.02/835.91/0.00) lat (ms,95%): 2045.74 err/s: 0.00 reconn/s: 0.00
    [ 1020s ] thds: 160 tps: 186.00 qps: 3300.41 (r/w/o: 2563.71/736.70/0.00) lat (ms,95%): 2320.55 err/s: 0.00 reconn/s: 0.00
    [ 1030s ] thds: 160 tps: 192.10 qps: 3443.46 (r/w/o: 2676.37/767.09/0.00) lat (ms,95%): 2680.11 err/s: 0.00 reconn/s: 0.00
    [ 1040s ] thds: 160 tps: 187.60 qps: 3419.36 (r/w/o: 2662.85/756.51/0.00) lat (ms,95%): 2279.14 err/s: 0.00 reconn/s: 0.00
    [ 1050s ] thds: 160 tps: 200.60 qps: 3606.84 (r/w/o: 2807.46/799.39/0.00) lat (ms,95%): 2680.11 err/s: 0.00 reconn/s: 0.00
    [ 1060s ] thds: 160 tps: 188.90 qps: 3430.83 (r/w/o: 2665.84/764.98/0.00) lat (ms,95%): 2828.87 err/s: 0.00 reconn/s: 0.00
    [ 1070s ] thds: 160 tps: 198.00 qps: 3519.85 (r/w/o: 2736.54/783.31/0.00) lat (ms,95%): 2680.11 err/s: 0.00 reconn/s: 0.00
    [ 1080s ] thds: 160 tps: 194.01 qps: 3471.30 (r/w/o: 2698.58/772.72/0.00) lat (ms,95%): 2585.31 err/s: 0.00 reconn/s: 0.00
    [ 1090s ] thds: 160 tps: 181.20 qps: 3289.85 (r/w/o: 2562.36/727.49/0.00) lat (ms,95%): 2985.89 err/s: 0.00 reconn/s: 0.00
    [ 1100s ] thds: 160 tps: 201.79 qps: 3650.76 (r/w/o: 2840.09/810.67/0.00) lat (ms,95%): 2585.31 err/s: 0.00 reconn/s: 0.00
    [ 1110s ] thds: 160 tps: 192.59 qps: 3512.49 (r/w/o: 2741.41/771.08/0.00) lat (ms,95%): 2932.60 err/s: 0.00 reconn/s: 0.00
    [ 1120s ] thds: 160 tps: 201.61 qps: 3568.76 (r/w/o: 2764.40/804.36/0.00) lat (ms,95%): 2405.65 err/s: 0.00 reconn/s: 0.00
    [ 1130s ] thds: 160 tps: 184.51 qps: 3334.10 (r/w/o: 2597.68/736.42/0.00) lat (ms,95%): 2932.60 err/s: 0.00 reconn/s: 0.00
    [ 1140s ] thds: 160 tps: 193.30 qps: 3443.23 (r/w/o: 2673.54/769.68/0.00) lat (ms,95%): 2778.39 err/s: 0.00 reconn/s: 0.00
    [ 1150s ] thds: 160 tps: 196.80 qps: 3578.58 (r/w/o: 2789.49/789.10/0.00) lat (ms,95%): 2828.87 err/s: 0.00 reconn/s: 0.00
    [ 1160s ] thds: 160 tps: 198.60 qps: 3552.13 (r/w/o: 2757.42/794.71/0.00) lat (ms,95%): 2828.87 err/s: 0.00 reconn/s: 0.00
    [ 1170s ] thds: 160 tps: 205.50 qps: 3679.06 (r/w/o: 2857.57/821.49/0.00) lat (ms,95%): 2632.28 err/s: 0.00 reconn/s: 0.00
    [ 1180s ] thds: 160 tps: 191.90 qps: 3497.02 (r/w/o: 2726.22/770.81/0.00) lat (ms,95%): 2880.27 err/s: 0.00 reconn/s: 0.00
    [ 1190s ] thds: 160 tps: 200.40 qps: 3545.44 (r/w/o: 2749.06/796.39/0.00) lat (ms,95%): 2279.14 err/s: 0.00 reconn/s: 0.00
    [ 1200s ] thds: 160 tps: 195.61 qps: 3547.92 (r/w/o: 2764.90/783.03/0.00) lat (ms,95%): 2493.86 err/s: 0.00 reconn/s: 0.00
    [ 1210s ] thds: 160 tps: 184.00 qps: 3358.26 (r/w/o: 2614.57/743.69/0.00) lat (ms,95%): 2045.74 err/s: 0.00 reconn/s: 0.00
    [ 1220s ] thds: 160 tps: 206.99 qps: 3737.11 (r/w/o: 2902.63/834.48/0.00) lat (ms,95%): 2493.86 err/s: 0.00 reconn/s: 0.00
    [ 1230s ] thds: 160 tps: 199.20 qps: 3522.66 (r/w/o: 2738.45/784.21/0.00) lat (ms,95%): 2728.81 err/s: 0.00 reconn/s: 0.00
    [ 1240s ] thds: 160 tps: 193.70 qps: 3517.27 (r/w/o: 2741.07/776.19/0.00) lat (ms,95%): 2632.28 err/s: 0.00 reconn/s: 0.00
    [ 1250s ] thds: 160 tps: 201.60 qps: 3624.48 (r/w/o: 2817.89/806.60/0.00) lat (ms,95%): 2828.87 err/s: 0.00 reconn/s: 0.00
    [ 1260s ] thds: 160 tps: 201.30 qps: 3593.84 (r/w/o: 2791.23/802.61/0.00) lat (ms,95%): 2585.31 err/s: 0.00 reconn/s: 0.00
    [ 1270s ] thds: 160 tps: 188.70 qps: 3480.99 (r/w/o: 2711.19/769.80/0.00) lat (ms,95%): 2728.81 err/s: 0.00 reconn/s: 0.00
    [ 1280s ] thds: 160 tps: 212.20 qps: 3731.42 (r/w/o: 2897.62/833.80/0.00) lat (ms,95%): 2320.55 err/s: 0.00 reconn/s: 0.00
    [ 1290s ] thds: 160 tps: 195.89 qps: 3565.53 (r/w/o: 2776.79/788.74/0.00) lat (ms,95%): 3151.62 err/s: 0.00 reconn/s: 0.00
    [ 1300s ] thds: 160 tps: 181.82 qps: 3257.29 (r/w/o: 2534.93/722.36/0.00) lat (ms,95%): 2985.89 err/s: 0.00 reconn/s: 0.00
    [ 1310s ] thds: 160 tps: 187.30 qps: 3350.23 (r/w/o: 2601.34/748.88/0.00) lat (ms,95%): 2680.11 err/s: 0.00 reconn/s: 0.00
    [ 1320s ] thds: 160 tps: 191.70 qps: 3478.15 (r/w/o: 2708.66/769.49/0.00) lat (ms,95%): 2449.36 err/s: 0.00 reconn/s: 0.00
    [ 1330s ] thds: 160 tps: 188.20 qps: 3411.48 (r/w/o: 2651.56/759.92/0.00) lat (ms,95%): 2632.28 err/s: 0.00 reconn/s: 0.00
    [ 1340s ] thds: 160 tps: 200.79 qps: 3574.84 (r/w/o: 2780.28/794.56/0.00) lat (ms,95%): 2585.31 err/s: 0.00 reconn/s: 0.00
    [ 1350s ] thds: 160 tps: 209.50 qps: 3751.76 (r/w/o: 2916.07/835.69/0.00) lat (ms,95%): 2828.87 err/s: 0.00 reconn/s: 0.00
    [ 1360s ] thds: 160 tps: 200.01 qps: 3628.39 (r/w/o: 2827.45/800.94/0.00) lat (ms,95%): 2680.11 err/s: 0.00 reconn/s: 0.00
    [ 1370s ] thds: 160 tps: 194.60 qps: 3553.29 (r/w/o: 2763.99/789.30/0.00) lat (ms,95%): 2405.65 err/s: 0.00 reconn/s: 0.00
    [ 1380s ] thds: 160 tps: 208.00 qps: 3710.18 (r/w/o: 2882.38/827.80/0.00) lat (ms,95%): 2728.81 err/s: 0.00 reconn/s: 0.00
    [ 1390s ] thds: 160 tps: 202.80 qps: 3620.72 (r/w/o: 2815.22/805.50/0.00) lat (ms,95%): 2828.87 err/s: 0.00 reconn/s: 0.00
    [ 1400s ] thds: 160 tps: 191.90 qps: 3510.13 (r/w/o: 2742.34/767.78/0.00) lat (ms,95%): 2632.28 err/s: 0.00 reconn/s: 0.00
    [ 1410s ] thds: 160 tps: 208.30 qps: 3712.07 (r/w/o: 2878.25/833.82/0.00) lat (ms,95%): 2632.28 err/s: 0.00 reconn/s: 0.00
    [ 1420s ] thds: 160 tps: 197.50 qps: 3563.93 (r/w/o: 2770.43/793.51/0.00) lat (ms,95%): 2585.31 err/s: 0.00 reconn/s: 0.00
    [ 1430s ] thds: 160 tps: 202.80 qps: 3623.10 (r/w/o: 2817.40/805.70/0.00) lat (ms,95%): 2493.86 err/s: 0.00 reconn/s: 0.00
    [ 1440s ] thds: 160 tps: 192.10 qps: 3448.14 (r/w/o: 2677.43/770.71/0.00) lat (ms,95%): 2728.81 err/s: 0.00 reconn/s: 0.00
    [ 1450s ] thds: 160 tps: 198.79 qps: 3577.86 (r/w/o: 2784.79/793.07/0.00) lat (ms,95%): 2539.17 err/s: 0.00 reconn/s: 0.00
    [ 1460s ] thds: 160 tps: 194.10 qps: 3513.33 (r/w/o: 2735.82/777.51/0.00) lat (ms,95%): 2449.36 err/s: 0.00 reconn/s: 0.00
    [ 1470s ] thds: 160 tps: 209.40 qps: 3785.55 (r/w/o: 2942.24/843.31/0.00) lat (ms,95%): 2539.17 err/s: 0.00 reconn/s: 0.00
    [ 1480s ] thds: 160 tps: 185.60 qps: 3392.19 (r/w/o: 2637.89/754.30/0.00) lat (ms,95%): 2632.28 err/s: 0.00 reconn/s: 0.00
    [ 1490s ] thds: 160 tps: 196.90 qps: 3515.97 (r/w/o: 2738.26/777.72/0.00) lat (ms,95%): 2320.55 err/s: 0.00 reconn/s: 0.00
    [ 1500s ] thds: 160 tps: 200.60 qps: 3562.34 (r/w/o: 2768.96/793.39/0.00) lat (ms,95%): 2279.14 err/s: 0.00 reconn/s: 0.00
    SQL statistics:
        queries performed:
            read:                            4127984    -- 读总数 
            write:                           1179424    -- 写总数
            other:                           0  --其他其他操作总数(SELECT、INSERT、UPDATE、DELETE之外的操作,例如COMMIT等) 
            total:                           5307408    -- 全部总数  
        transactions:                        294856 (196.45 per sec.)    -- 总事务数(每秒事务数)  
        queries:                             5307408 (3536.05 per sec.)  -- 读写总数(每秒读写次数)  
        ignored errors:                      0      (0.00 per sec.)  --忽略的错误数
        reconnects:                          0      (0.00 per sec.)  --重新连接的次数
    
    Throughput:                                          --吞吐量
        events/s (eps):                      196.4475    --每秒事务数
        time elapsed:                        1500.9407s  --耗时时间
        total number of events:              294856      -- 共发生多少事务数  
    
    Latency (ms):                                           --响应时间
             min:                                 29.52     -- 最小耗时
             avg:                                814.24     -- 平均耗时 
             max:                               8835.75     -- 最长耗时  
             95th percentile:                   2449.36     -- 超过95%平均耗时
             sum:                            240082588.49   --总的响应时间
    
    Threads fairness:   -- 线程的稳定性
        events (avg/stddev):           1842.8500/43.17    -- 事件(平均值/偏差)  
        execution time (avg/stddev):   1500.5162/0.29     -- 执行时间(平均值/偏差)  
    
    • 查询分析sysbench_test表
    
    (root@localhost) 18:56:10 [dbtest]> select
        ->         scenario,
        ->         server_name,
        ->         test_type,
        ->         sb_threads,
        ->         convert(avg(server_load),decimal(12,2)) as server_load,
        ->         convert(avg(request_read),decimal(12,0)) as request_read,
        ->         convert(avg(request_write),decimal(12,0)) as request_write,
        ->         convert(avg(transactions_per_second),decimal(12,2)) as transactions_per_second,
        ->         convert(avg(request_per_second),decimal(12,2)) as request_per_second,
        ->         convert(avg(95_pct_time),decimal(12,2)) as 95_pct_time
        ->         from dbtest.sysbench_test
        -> where scenario in ('deadlock_check_on','deadlock_check_off')
        -> group by scenario,server_name,test_type,sb_threads ;
    +--------------------+-------------+------------+------------+-------------+--------------+---------------+-------------------------+--------------------+-------------+
    | scenario           | server_name | test_type  | sb_threads | server_load | request_read | request_write | transactions_per_second | request_per_second | 95_pct_time |
    +--------------------+-------------+------------+------------+-------------+--------------+---------------+-------------------------+--------------------+-------------+
    | deadlock_check_off | localhost   | read-write |          8 |        9.14 |      2795590 |        798740 |                  133.12 |            2396.16 |      116.81 |
    | deadlock_check_off | localhost   | read-write |         24 |       21.28 |      3641904 |       1040544 |                  173.41 |            3121.41 |      239.72 |
    | deadlock_check_off | localhost   | read-write |         48 |       22.81 |      3899495 |       1114141 |                  185.67 |            3342.08 |      563.94 |
    | deadlock_check_off | localhost   | read-write |         64 |       26.81 |      4081868 |       1166248 |                  194.35 |            3498.30 |      778.54 |
    | deadlock_check_off | localhost   | read-write |         96 |       28.72 |      4180283 |       1194367 |                  199.02 |            3582.31 |     1288.69 |
    | deadlock_check_off | localhost   | read-write |        128 |       27.62 |      4186014 |       1196004 |                  199.27 |            3586.83 |     1858.48 |
    | deadlock_check_off | localhost   | read-write |        160 |       29.94 |      4177073 |       1193449 |                  198.80 |            3578.38 |     2434.79 |
    | deadlock_check_off | localhost   | read-write |        196 |       35.16 |      4137840 |       1182240 |                  196.95 |            3545.13 |     3076.97 |
    | deadlock_check_off | localhost   | read-write |        256 |       34.17 |      4036863 |       1153388 |                  192.06 |            3457.10 |     4154.36 |
    | deadlock_check_on  | localhost   | read-write |          8 |        8.97 |      2736836 |        781953 |                  130.32 |            2345.76 |      118.98 |
    | deadlock_check_on  | localhost   | read-write |         24 |       19.94 |      3384808 |        967088 |                  161.17 |            2901.02 |      261.94 |
    | deadlock_check_on  | localhost   | read-write |         48 |       24.19 |      3344198 |        955485 |                  159.23 |            2866.13 |      673.60 |
    | deadlock_check_on  | localhost   | read-write |         64 |       24.08 |      3835865 |       1095961 |                  182.62 |            3287.25 |      823.10 |
    | deadlock_check_on  | localhost   | read-write |         96 |       28.00 |      3902785 |       1115081 |                  185.79 |            3344.27 |     1379.47 |
    | deadlock_check_on  | localhost   | read-write |        128 |       31.52 |      4089064 |       1168304 |                  194.67 |            3504.02 |     1881.95 |
    | deadlock_check_on  | localhost   | read-write |        160 |       32.99 |      3991106 |       1140316 |                  189.98 |            3419.56 |     2527.94 |
    | deadlock_check_on  | localhost   | read-write |        196 |       29.47 |      4007155 |       1144901 |                  190.72 |            3433.05 |     3152.65 |
    | deadlock_check_on  | localhost   | read-write |        256 |       30.71 |      3546382 |       1013251 |                  168.80 |            3038.38 |     4783.57 |
    +--------------------+-------------+------------+------------+-------------+--------------+---------------+-------------------------+--------------------+-------------+
    18 rows in set (0.00 sec)
    
    (root@localhost) 18:56:12 [dbtest]>
    
    
    • 生成图片
    #sh mysql_oltp_sysbench.sh chart deadlock_check_on deadlock_check_off
    
    1. 竖轴为查询数,横轴为线程数

    1. 竖轴为事务数,横轴为线程数

    1. 竖轴为95%平均响应时间,横轴为线程数

    四、关于测试后的结论:

    innodb_deadlock_detect 开启都性能有5%左右的损耗,性能满是的情况下,尽量还是开启有助于优化

    五、关于测试后的性能优化:

    1、善后清理工作:

    • 进mysql删除dbtest库
    mysql> drop database dbtest;
    

    可以运行./mysqltuner.pl程序,根据建议对数据库进行调优后,再使用sysbench对OLTP进行测试,看看TPS是不是会有所提高。

    注意:sysbench的测试只是基准测试,并不能代表实际企业环境下的性能指标。

    2、运行优化建议工具

    
    # ./mysqltuner.pl
    建议:
    [plain] view plain copy
    General recommendations:  
        Run OPTIMIZE TABLE to defragment tables for better performance  
        MySQL started within last 24 hours - recommendations may be inaccurate  
        Reduce your overall MySQL memory footprint for system stability  
        Enable the slow query log to troubleshoot bad queries  
        Reduce or eliminate unclosed connections and network issues  
    Variables to adjust:  
      *** MySQL's maximum memory usage is dangerously high ***  
      *** Add RAM before increasing MySQL buffer variables ***  
        query_cache_type (=0) 
        innodb_buffer_pool_size (>= 1G) if possible.  
        innodb_buffer_pool_instances(=8)  
    

    3、修改mysql参数

    # vim /etc/my.cnf
    
    [plain] view plain copy
    innodb_buffer_pool_instances = 8  
    innodb_buffer_pool_size = 1024M 
    
    每轮测试完成后,都重启mysqld实例,并且用下面的方法删除系统cache,释放swap(如果用到了swap的话),甚至可以重启整个OS。
    
    # sync  -- 将脏数据刷新到磁盘
    # sync  -- 将脏数据刷新到磁盘
    # echo 3 > /proc/sys/vm/drop_caches  -- 清除OS Cache
    # swapoff -a && swapon -a     -- 清除OS swap缓存
    

    4、再次启动测试流程

  • 相关阅读:
    bzoj3996
    bzoj3157 3516
    bzoj1937
    bzoj1532
    bzoj3572
    bzoj1453
    bzoj3205
    bzoj2595
    关于高斯消元解决xor问题的总结
    linux查找和替换命令
  • 原文地址:https://www.cnblogs.com/gczheng/p/9051950.html
Copyright © 2020-2023  润新知