Kafka吞吐量测试案例
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
领英公司参考连接：https://www.slideshare.net/JiangjieQin/producer-performance-tuning-for-apache-kafka-63147600?qid=84026ff8-243f-49a7-a4d0-69976cf317b7&v=&b=&from_search=9(温馨提示，需要VPN哟～)。
一.kafka-producer-perf-test.sh 使用案例
1>.测试kafka写入的吞吐量
我用的kafka版本是0.10.2.1，领英用的貌似是0.10.0，我使用我的版本按照它的方式敲击，发现不太好事，这就有点小尴尬了！于是它表红线的参数我就没有使用。
[root@node105 ~]# nohup kafka-producer-perf-test.sh --num-records 100000000 --record-size 1000 --topic yinzhengjie-kafka --throughput 1000000 --producer-props bootstrap.servers=1 0.1.2.101:9092,10.1.2.102:9092,10.1.2.103:9092,10.1.2.104:9092,10.1.2.105:9092 > yinzhengjie-kafka.log & [root@node105 ~]# 相关参数说明： —topic 指定Kafka集群的topic名称，本例为yinzhengjie-kafka
--num-records
总共需要发送的消息数，本例为100000000
--record-size 每个记录的字节数，本例为1000
--throughput 每秒钟发送的记录数，本例为20000
--producer-props bootstrap.servers=10.1.2.101:9092,10.1.2.102:9092,10.1.2.103:9092,10.1.2.104:9092,10.1.2.105:9092 很显然是kafka集群的broker地址
2>.计算生产者的吞吐量
[root@node105 ~]# cat yinzhengjie-kafka.log 485204 records sent, 96673.4 records/sec (92.19 MB/sec), 284.4 ms avg latency, 648.0 max latency. 591703 records sent, 118340.6 records/sec (112.86 MB/sec), 260.7 ms avg latency, 795.0 max latency. 568061 records sent, 113612.2 records/sec (108.35 MB/sec), 268.2 ms avg latency, 870.0 max latency. 576896 records sent, 115034.1 records/sec (109.71 MB/sec), 268.7 ms avg latency, 732.0 max latency. 548140 records sent, 106912.4 records/sec (101.96 MB/sec), 257.9 ms avg latency, 913.0 max latency. 568845 records sent, 112575.7 records/sec (107.36 MB/sec), 294.4 ms avg latency, 1283.0 max latency. 557990 records sent, 111598.0 records/sec (106.43 MB/sec), 279.4 ms avg latency, 828.0 max latency. 464493 records sent, 92898.6 records/sec (88.60 MB/sec), 329.3 ms avg latency, 1213.0 max latency. 488281 records sent, 97656.2 records/sec (93.13 MB/sec), 312.9 ms avg latency, 1020.0 max latency. 517560 records sent, 103512.0 records/sec (98.72 MB/sec), 294.8 ms avg latency, 1062.0 max latency. 209745 records sent, 41949.0 records/sec (40.01 MB/sec), 740.2 ms avg latency, 3722.0 max latency. 394980 records sent, 78980.2 records/sec (75.32 MB/sec), 388.7 ms avg latency, 2079.0 max latency. 445178 records sent, 89035.6 records/sec (84.91 MB/sec), 342.4 ms avg latency, 1274.0 max latency. 693946 records sent, 138789.2 records/sec (132.36 MB/sec), 223.2 ms avg latency, 469.0 max latency. 367182 records sent, 62616.3 records/sec (59.72 MB/sec), 247.3 ms avg latency, 3303.0 max latency. 631412 records sent, 126282.4 records/sec (120.43 MB/sec), 381.9 ms avg latency, 3304.0 max latency. 393343 records sent, 78668.6 records/sec (75.02 MB/sec), 386.0 ms avg latency, 1857.0 max latency. 441035 records sent, 88207.0 records/sec (84.12 MB/sec), 347.2 ms avg latency, 1018.0 max latency. 521952 records sent, 104223.6 records/sec (99.40 MB/sec), 296.4 ms avg latency, 698.0 max latency. 641069 records sent, 127779.4 records/sec (121.86 MB/sec), 238.2 ms avg latency, 560.0 max latency. 514232 records sent, 101747.5 records/sec (97.03 MB/sec), 296.9 ms avg latency, 855.0 max latency. 582718 records sent, 116543.6 records/sec (111.14 MB/sec), 269.8 ms avg latency, 633.0 max latency. 15689 records sent, 2589.8 records/sec (2.47 MB/sec), 262.5 ms avg latency, 6291.0 max latency. 300150 records sent, 34393.3 records/sec (32.80 MB/sec), 954.0 ms avg latency, 6731.0 max latency. 528932 records sent, 105786.4 records/sec (100.89 MB/sec), 595.4 ms avg latency, 5857.0 max latency. 460320 records sent, 91770.3 records/sec (87.52 MB/sec), 335.8 ms avg latency, 1688.0 max latency. 613573 records sent, 119815.1 records/sec (114.26 MB/sec), 245.9 ms avg latency, 931.0 max latency. 316608 records sent, 63321.6 records/sec (60.39 MB/sec), 502.0 ms avg latency, 2900.0 max latency. 581817 records sent, 116084.8 records/sec (110.71 MB/sec), 262.5 ms avg latency, 540.0 max latency. 636679 records sent, 125801.0 records/sec (119.97 MB/sec), 244.7 ms avg latency, 517.0 max latency. 501821 records sent, 97554.6 records/sec (93.04 MB/sec), 298.4 ms avg latency, 1026.0 max latency. 92700 records sent, 18197.9 records/sec (17.35 MB/sec), 1709.4 ms avg latency, 4843.0 max latency. 430113 records sent, 86022.6 records/sec (82.04 MB/sec), 370.9 ms avg latency, 1119.0 max latency. 318603 records sent, 63720.6 records/sec (60.77 MB/sec), 478.7 ms avg latency, 2001.0 max latency. 389034 records sent, 76732.5 records/sec (73.18 MB/sec), 379.7 ms avg latency, 1345.0 max latency. 413297 records sent, 82659.4 records/sec (78.83 MB/sec), 391.5 ms avg latency, 1016.0 max latency. 258853 records sent, 51729.2 records/sec (49.33 MB/sec), 528.6 ms avg latency, 2228.0 max latency. 437625 records sent, 87490.0 records/sec (83.44 MB/sec), 377.5 ms avg latency, 1178.0 max latency. 352050 records sent, 70227.4 records/sec (66.97 MB/sec), 424.4 ms avg latency, 1456.0 max latency. 402187 records sent, 80437.4 records/sec (76.71 MB/sec), 394.4 ms avg latency, 1452.0 max latency. 406540 records sent, 81178.1 records/sec (77.42 MB/sec), 383.9 ms avg latency, 1344.0 max latency. 352573 records sent, 66965.4 records/sec (63.86 MB/sec), 426.7 ms avg latency, 2243.0 max latency. 405849 records sent, 81169.8 records/sec (77.41 MB/sec), 406.3 ms avg latency, 1204.0 max latency. 394551 records sent, 78674.2 records/sec (75.03 MB/sec), 369.5 ms avg latency, 980.0 max latency. 371400 records sent, 74235.5 records/sec (70.80 MB/sec), 376.5 ms avg latency, 1539.0 max latency. 224325 records sent, 44141.1 records/sec (42.10 MB/sec), 730.9 ms avg latency, 2348.0 max latency. 294300 records sent, 56142.7 records/sec (53.54 MB/sec), 537.1 ms avg latency, 1441.0 max latency. 350175 records sent, 69909.2 records/sec (66.67 MB/sec), 480.5 ms avg latency, 1660.0 max latency. 350175 records sent, 66674.6 records/sec (63.59 MB/sec), 437.3 ms avg latency, 1390.0 max latency. 364683 records sent, 72936.6 records/sec (69.56 MB/sec), 440.6 ms avg latency, 1651.0 max latency. 385692 records sent, 76708.8 records/sec (73.16 MB/sec), 326.2 ms avg latency, 1466.0 max latency. 388067 records sent, 76875.4 records/sec (73.31 MB/sec), 468.7 ms avg latency, 1773.0 max latency. 312283 records sent, 62170.6 records/sec (59.29 MB/sec), 492.0 ms avg latency, 1382.0 max latency. 350175 records sent, 65983.6 records/sec (62.93 MB/sec), 432.4 ms avg latency, 1238.0 max latency. 400200 records sent, 79912.1 records/sec (76.21 MB/sec), 420.5 ms avg latency, 1244.0 max latency. 335917 records sent, 66623.8 records/sec (63.54 MB/sec), 445.0 ms avg latency, 1054.0 max latency. 364433 records sent, 70929.0 records/sec (67.64 MB/sec), 413.4 ms avg latency, 1165.0 max latency. 305884 records sent, 61176.8 records/sec (58.34 MB/sec), 536.8 ms avg latency, 1517.0 max latency. 430627 records sent, 85953.5 records/sec (81.97 MB/sec), 354.6 ms avg latency, 1919.0 max latency. 414064 records sent, 80229.4 records/sec (76.51 MB/sec), 363.1 ms avg latency, 1194.0 max latency. 304470 records sent, 60894.0 records/sec (58.07 MB/sec), 509.9 ms avg latency, 1365.0 max latency. 385418 records sent, 77083.6 records/sec (73.51 MB/sec), 417.5 ms avg latency, 1468.0 max latency. 402698 records sent, 80059.2 records/sec (76.35 MB/sec), 343.1 ms avg latency, 1147.0 max latency. 419702 records sent, 83940.4 records/sec (80.05 MB/sec), 402.7 ms avg latency, 1136.0 max latency. 272698 records sent, 54539.6 records/sec (52.01 MB/sec), 554.7 ms avg latency, 2612.0 max latency. 327574 records sent, 65514.8 records/sec (62.48 MB/sec), 452.0 ms avg latency, 1112.0 max latency. 444357 records sent, 88871.4 records/sec (84.75 MB/sec), 304.7 ms avg latency, 1523.0 max latency. 325969 records sent, 65193.8 records/sec (62.17 MB/sec), 546.3 ms avg latency, 2538.0 max latency. 391147 records sent, 78229.4 records/sec (74.61 MB/sec), 380.9 ms avg latency, 1704.0 max latency. 418355 records sent, 83671.0 records/sec (79.79 MB/sec), 380.5 ms avg latency, 1402.0 max latency. 318282 records sent, 63656.4 records/sec (60.71 MB/sec), 465.3 ms avg latency, 2996.0 max latency. 431130 records sent, 85916.7 records/sec (81.94 MB/sec), 366.3 ms avg latency, 1606.0 max latency. 398883 records sent, 79776.6 records/sec (76.08 MB/sec), 380.9 ms avg latency, 1387.0 max latency. 551592 records sent, 109791.4 records/sec (104.71 MB/sec), 270.6 ms avg latency, 1512.0 max latency. 483225 records sent, 96645.0 records/sec (92.17 MB/sec), 317.9 ms avg latency, 988.0 max latency. 317175 records sent, 58519.4 records/sec (55.81 MB/sec), 473.6 ms avg latency, 2136.0 max latency. 339750 records sent, 67665.8 records/sec (64.53 MB/sec), 491.5 ms avg latency, 2083.0 max latency. 421923 records sent, 84384.6 records/sec (80.48 MB/sec), 384.9 ms avg latency, 1260.0 max latency. 365130 records sent, 73026.0 records/sec (69.64 MB/sec), 410.0 ms avg latency, 1084.0 max latency. 346304 records sent, 69260.8 records/sec (66.05 MB/sec), 456.2 ms avg latency, 1526.0 max latency. 347720 records sent, 69544.0 records/sec (66.32 MB/sec), 420.3 ms avg latency, 1192.0 max latency. 374846 records sent, 74969.2 records/sec (71.50 MB/sec), 421.8 ms avg latency, 1478.0 max latency. 332024 records sent, 66404.8 records/sec (63.33 MB/sec), 451.3 ms avg latency, 1840.0 max latency. 413353 records sent, 82079.6 records/sec (78.28 MB/sec), 355.0 ms avg latency, 1328.0 max latency. 456013 records sent, 91202.6 records/sec (86.98 MB/sec), 362.7 ms avg latency, 1106.0 max latency. 443987 records sent, 85744.9 records/sec (81.77 MB/sec), 302.0 ms avg latency, 1187.0 max latency. 401025 records sent, 78694.1 records/sec (75.05 MB/sec), 431.8 ms avg latency, 1545.0 max latency. 339375 records sent, 67820.7 records/sec (64.68 MB/sec), 443.9 ms avg latency, 1530.0 max latency. 460950 records sent, 84655.6 records/sec (80.73 MB/sec), 304.3 ms avg latency, 1762.0 max latency. 339450 records sent, 66952.7 records/sec (63.85 MB/sec), 546.6 ms avg latency, 2097.0 max latency. 394914 records sent, 78982.8 records/sec (75.32 MB/sec), 402.1 ms avg latency, 1448.0 max latency. 291396 records sent, 57679.3 records/sec (55.01 MB/sec), 517.6 ms avg latency, 2158.0 max latency. 372347 records sent, 74469.4 records/sec (71.02 MB/sec), 397.6 ms avg latency, 1819.0 max latency. 408480 records sent, 81696.0 records/sec (77.91 MB/sec), 399.9 ms avg latency, 1226.0 max latency. 367899 records sent, 73579.8 records/sec (70.17 MB/sec), 423.3 ms avg latency, 1575.0 max latency. 270450 records sent, 54090.0 records/sec (51.58 MB/sec), 513.7 ms avg latency, 1724.0 max latency. 395764 records sent, 78276.1 records/sec (74.65 MB/sec), 387.1 ms avg latency, 1762.0 max latency. 437056 records sent, 87411.2 records/sec (83.36 MB/sec), 380.6 ms avg latency, 1763.0 max latency. 363344 records sent, 69182.0 records/sec (65.98 MB/sec), 421.1 ms avg latency, 1361.0 max latency. 450225 records sent, 87202.2 records/sec (83.16 MB/sec), 359.5 ms avg latency, 1487.0 max latency. 400200 records sent, 75623.6 records/sec (72.12 MB/sec), 265.6 ms avg latency, 2355.0 max latency. 335026 records sent, 67005.2 records/sec (63.90 MB/sec), 625.9 ms avg latency, 2568.0 max latency. 376506 records sent, 75301.2 records/sec (71.81 MB/sec), 389.1 ms avg latency, 1071.0 max latency. 281738 records sent, 56347.6 records/sec (53.74 MB/sec), 541.7 ms avg latency, 2047.0 max latency. 346980 records sent, 66255.5 records/sec (63.19 MB/sec), 461.9 ms avg latency, 1747.0 max latency. 351900 records sent, 69244.4 records/sec (66.04 MB/sec), 408.8 ms avg latency, 1318.0 max latency. 265397 records sent, 53079.4 records/sec (50.62 MB/sec), 630.5 ms avg latency, 1979.0 max latency. 424290 records sent, 84722.4 records/sec (80.80 MB/sec), 387.0 ms avg latency, 1130.0 max latency. 302624 records sent, 60500.6 records/sec (57.70 MB/sec), 504.8 ms avg latency, 1415.0 max latency. 356639 records sent, 71213.9 records/sec (67.91 MB/sec), 397.4 ms avg latency, 1366.0 max latency. 400200 records sent, 76887.6 records/sec (73.33 MB/sec), 405.1 ms avg latency, 1269.0 max latency. 435842 records sent, 87168.4 records/sec (83.13 MB/sec), 378.4 ms avg latency, 1388.0 max latency. 364558 records sent, 70801.7 records/sec (67.52 MB/sec), 411.4 ms avg latency, 1323.0 max latency. 449387 records sent, 89877.4 records/sec (85.71 MB/sec), 355.8 ms avg latency, 1039.0 max latency. 403288 records sent, 77376.8 records/sec (73.79 MB/sec), 340.2 ms avg latency, 1702.0 max latency. 417825 records sent, 80242.9 records/sec (76.53 MB/sec), 421.3 ms avg latency, 1226.0 max latency. 369900 records sent, 72686.2 records/sec (69.32 MB/sec), 417.1 ms avg latency, 1447.0 max latency. 437842 records sent, 87568.4 records/sec (83.51 MB/sec), 359.2 ms avg latency, 2027.0 max latency. 450833 records sent, 89986.6 records/sec (85.82 MB/sec), 343.0 ms avg latency, 1238.0 max latency. 468195 records sent, 93639.0 records/sec (89.30 MB/sec), 318.6 ms avg latency, 887.0 max latency. 381300 records sent, 76260.0 records/sec (72.73 MB/sec), 416.3 ms avg latency, 1537.0 max latency. 386025 records sent, 77205.0 records/sec (73.63 MB/sec), 398.5 ms avg latency, 1199.0 max latency. 355139 records sent, 71027.8 records/sec (67.74 MB/sec), 431.3 ms avg latency, 1298.0 max latency. 374749 records sent, 74770.4 records/sec (71.31 MB/sec), 353.8 ms avg latency, 1290.0 max latency. 397407 records sent, 79481.4 records/sec (75.80 MB/sec), 438.1 ms avg latency, 1648.0 max latency. 360720 records sent, 72144.0 records/sec (68.80 MB/sec), 410.4 ms avg latency, 1585.0 max latency. 364249 records sent, 72849.8 records/sec (69.47 MB/sec), 413.8 ms avg latency, 1505.0 max latency. 401363 records sent, 80272.6 records/sec (76.55 MB/sec), 388.3 ms avg latency, 1054.0 max latency. 314103 records sent, 59220.0 records/sec (56.48 MB/sec), 495.4 ms avg latency, 1338.0 max latency. 298134 records sent, 59626.8 records/sec (56.86 MB/sec), 551.7 ms avg latency, 1610.0 max latency. 295673 records sent, 59134.6 records/sec (56.40 MB/sec), 522.3 ms avg latency, 1689.0 max latency. 386488 records sent, 77297.6 records/sec (73.72 MB/sec), 400.0 ms avg latency, 1311.0 max latency. 422477 records sent, 84495.4 records/sec (80.58 MB/sec), 359.9 ms avg latency, 1357.0 max latency. 364903 records sent, 72980.6 records/sec (69.60 MB/sec), 405.4 ms avg latency, 1113.0 max latency. 333375 records sent, 63415.4 records/sec (60.48 MB/sec), 451.5 ms avg latency, 1377.0 max latency. 407003 records sent, 81400.6 records/sec (77.63 MB/sec), 390.1 ms avg latency, 1321.0 max latency. 393397 records sent, 77076.2 records/sec (73.51 MB/sec), 408.1 ms avg latency, 1285.0 max latency. 388801 records sent, 77760.2 records/sec (74.16 MB/sec), 396.4 ms avg latency, 1162.0 max latency. 383069 records sent, 76613.8 records/sec (73.06 MB/sec), 417.4 ms avg latency, 1343.0 max latency. 378705 records sent, 73735.4 records/sec (70.32 MB/sec), 391.5 ms avg latency, 2222.0 max latency. 385890 records sent, 77178.0 records/sec (73.60 MB/sec), 413.3 ms avg latency, 1166.0 max latency. 444551 records sent, 88910.2 records/sec (84.79 MB/sec), 353.1 ms avg latency, 1147.0 max latency. 347314 records sent, 69296.5 records/sec (66.09 MB/sec), 438.3 ms avg latency, 1684.0 max latency. 316545 records sent, 62508.9 records/sec (59.61 MB/sec), 446.3 ms avg latency, 1352.0 max latency. 433528 records sent, 86085.8 records/sec (82.10 MB/sec), 387.1 ms avg latency, 1173.0 max latency. 374452 records sent, 74890.4 records/sec (71.42 MB/sec), 385.6 ms avg latency, 1550.0 max latency. 338620 records sent, 67133.2 records/sec (64.02 MB/sec), 453.3 ms avg latency, 1589.0 max latency. 374775 records sent, 73993.1 records/sec (70.57 MB/sec), 412.4 ms avg latency, 1358.0 max latency. 357450 records sent, 70586.5 records/sec (67.32 MB/sec), 456.5 ms avg latency, 1546.0 max latency. 275232 records sent, 55046.4 records/sec (52.50 MB/sec), 512.0 ms avg latency, 1688.0 max latency. 442490 records sent, 88498.0 records/sec (84.40 MB/sec), 370.3 ms avg latency, 1130.0 max latency. 313110 records sent, 62622.0 records/sec (59.72 MB/sec), 498.3 ms avg latency, 1636.0 max latency. 400308 records sent, 80061.6 records/sec (76.35 MB/sec), 395.6 ms avg latency, 1373.0 max latency. 337885 records sent, 66512.8 records/sec (63.43 MB/sec), 395.0 ms avg latency, 1690.0 max latency. 446967 records sent, 89393.4 records/sec (85.25 MB/sec), 386.0 ms avg latency, 1827.0 max latency. 439940 records sent, 87988.0 records/sec (83.91 MB/sec), 355.3 ms avg latency, 1251.0 max latency. 262322 records sent, 52464.4 records/sec (50.03 MB/sec), 516.0 ms avg latency, 1680.0 max latency. 314593 records sent, 62918.6 records/sec (60.00 MB/sec), 544.6 ms avg latency, 1431.0 max latency. 312654 records sent, 61985.3 records/sec (59.11 MB/sec), 481.0 ms avg latency, 1430.0 max latency. 399757 records sent, 79951.4 records/sec (76.25 MB/sec), 386.0 ms avg latency, 1160.0 max latency. 400965 records sent, 80193.0 records/sec (76.48 MB/sec), 381.1 ms avg latency, 1262.0 max latency. 363852 records sent, 69998.5 records/sec (66.76 MB/sec), 398.8 ms avg latency, 1203.0 max latency. 536223 records sent, 107223.2 records/sec (102.26 MB/sec), 317.9 ms avg latency, 1063.0 max latency. 314202 records sent, 60962.7 records/sec (58.14 MB/sec), 425.6 ms avg latency, 2061.0 max latency. 240900 records sent, 48151.1 records/sec (45.92 MB/sec), 674.7 ms avg latency, 1933.0 max latency. 209325 records sent, 40023.9 records/sec (38.17 MB/sec), 752.9 ms avg latency, 1802.0 max latency. 256875 records sent, 51344.2 records/sec (48.97 MB/sec), 622.6 ms avg latency, 1677.0 max latency. 193350 records sent, 37986.2 records/sec (36.23 MB/sec), 762.4 ms avg latency, 1840.0 max latency. 483000 records sent, 96311.1 records/sec (91.85 MB/sec), 308.9 ms avg latency, 1841.0 max latency. 382314 records sent, 76401.7 records/sec (72.86 MB/sec), 443.2 ms avg latency, 1543.0 max latency. 435336 records sent, 80602.9 records/sec (76.87 MB/sec), 356.3 ms avg latency, 1044.0 max latency. 373638 records sent, 74727.6 records/sec (71.27 MB/sec), 430.1 ms avg latency, 1650.0 max latency. 336477 records sent, 67295.4 records/sec (64.18 MB/sec), 438.0 ms avg latency, 1648.0 max latency. 350867 records sent, 70173.4 records/sec (66.92 MB/sec), 458.6 ms avg latency, 1661.0 max latency. 499218 records sent, 99843.6 records/sec (95.22 MB/sec), 312.3 ms avg latency, 1054.0 max latency. 362143 records sent, 72428.6 records/sec (69.07 MB/sec), 433.5 ms avg latency, 1483.0 max latency. 308457 records sent, 60269.1 records/sec (57.48 MB/sec), 491.1 ms avg latency, 1619.0 max latency. 375692 records sent, 75138.4 records/sec (71.66 MB/sec), 422.2 ms avg latency, 1286.0 max latency. 387466 records sent, 76908.7 records/sec (73.35 MB/sec), 374.8 ms avg latency, 1611.0 max latency. 347292 records sent, 62428.9 records/sec (59.54 MB/sec), 464.3 ms avg latency, 1787.0 max latency. 354210 records sent, 70842.0 records/sec (67.56 MB/sec), 484.1 ms avg latency, 2107.0 max latency. 496215 records sent, 87147.0 records/sec (83.11 MB/sec), 306.1 ms avg latency, 1225.0 max latency. 400200 records sent, 75395.6 records/sec (71.90 MB/sec), 433.9 ms avg latency, 1668.0 max latency. 361743 records sent, 72348.6 records/sec (69.00 MB/sec), 453.9 ms avg latency, 1441.0 max latency. 388632 records sent, 68590.2 records/sec (65.41 MB/sec), 397.4 ms avg latency, 1394.0 max latency. 282975 records sent, 56012.5 records/sec (53.42 MB/sec), 593.2 ms avg latency, 1667.0 max latency. 340549 records sent, 68109.8 records/sec (64.95 MB/sec), 480.5 ms avg latency, 1633.0 max latency. 320276 records sent, 60750.4 records/sec (57.94 MB/sec), 451.4 ms avg latency, 1519.0 max latency. 382376 records sent, 75778.0 records/sec (72.27 MB/sec), 423.8 ms avg latency, 1378.0 max latency. 375394 records sent, 75078.8 records/sec (71.60 MB/sec), 403.5 ms avg latency, 1199.0 max latency. 442290 records sent, 88334.3 records/sec (84.24 MB/sec), 367.1 ms avg latency, 935.0 max latency. 435002 records sent, 87000.4 records/sec (82.97 MB/sec), 360.3 ms avg latency, 1660.0 max latency. 348105 records sent, 69621.0 records/sec (66.40 MB/sec), 429.2 ms avg latency, 1201.0 max latency. 378901 records sent, 75780.2 records/sec (72.27 MB/sec), 403.9 ms avg latency, 1311.0 max latency. 242099 records sent, 48419.8 records/sec (46.18 MB/sec), 587.5 ms avg latency, 2074.0 max latency. 380623 records sent, 75475.5 records/sec (71.98 MB/sec), 441.9 ms avg latency, 1329.0 max latency. 299522 records sent, 59701.4 records/sec (56.94 MB/sec), 494.5 ms avg latency, 1573.0 max latency. 374171 records sent, 74834.2 records/sec (71.37 MB/sec), 375.8 ms avg latency, 1409.0 max latency. 439817 records sent, 84547.7 records/sec (80.63 MB/sec), 371.5 ms avg latency, 1183.0 max latency. 364202 records sent, 72840.4 records/sec (69.47 MB/sec), 456.6 ms avg latency, 1347.0 max latency. 408650 records sent, 81730.0 records/sec (77.94 MB/sec), 348.7 ms avg latency, 1212.0 max latency. 386220 records sent, 77244.0 records/sec (73.67 MB/sec), 430.5 ms avg latency, 1499.0 max latency. 590650 records sent, 118130.0 records/sec (112.66 MB/sec), 257.1 ms avg latency, 1053.0 max latency. 329128 records sent, 63208.8 records/sec (60.28 MB/sec), 349.1 ms avg latency, 1879.0 max latency. 408542 records sent, 81708.4 records/sec (77.92 MB/sec), 433.5 ms avg latency, 2531.0 max latency. 337366 records sent, 67405.8 records/sec (64.28 MB/sec), 505.4 ms avg latency, 1556.0 max latency. 416442 records sent, 73265.7 records/sec (69.87 MB/sec), 378.6 ms avg latency, 1670.0 max latency. 360097 records sent, 72019.4 records/sec (68.68 MB/sec), 428.8 ms avg latency, 1715.0 max latency. 403744 records sent, 80748.8 records/sec (77.01 MB/sec), 388.2 ms avg latency, 1085.0 max latency. 394909 records sent, 76266.7 records/sec (72.73 MB/sec), 387.5 ms avg latency, 1689.0 max latency. 392025 records sent, 78154.9 records/sec (74.53 MB/sec), 448.3 ms avg latency, 1543.0 max latency. 386595 records sent, 77319.0 records/sec (73.74 MB/sec), 400.5 ms avg latency, 1478.0 max latency. 424468 records sent, 84893.6 records/sec (80.96 MB/sec), 360.7 ms avg latency, 1372.0 max latency. 329087 records sent, 63604.0 records/sec (60.66 MB/sec), 431.9 ms avg latency, 1677.0 max latency. 334860 records sent, 66125.6 records/sec (63.06 MB/sec), 467.7 ms avg latency, 1670.0 max latency. 215415 records sent, 41101.9 records/sec (39.20 MB/sec), 710.9 ms avg latency, 2273.0 max latency. 243600 records sent, 48237.6 records/sec (46.00 MB/sec), 688.9 ms avg latency, 1916.0 max latency. 443820 records sent, 88764.0 records/sec (84.65 MB/sec), 364.9 ms avg latency, 1812.0 max latency. 711347 records sent, 142269.4 records/sec (135.68 MB/sec), 216.4 ms avg latency, 453.0 max latency. 577303 records sent, 115460.6 records/sec (110.11 MB/sec), 263.4 ms avg latency, 1072.0 max latency. 296205 records sent, 55510.7 records/sec (52.94 MB/sec), 212.9 ms avg latency, 3721.0 max latency. 329025 records sent, 60705.7 records/sec (57.89 MB/sec), 761.2 ms avg latency, 3900.0 max latency. 287524 records sent, 57390.0 records/sec (54.73 MB/sec), 592.4 ms avg latency, 2092.0 max latency. 504626 records sent, 100423.1 records/sec (95.77 MB/sec), 305.8 ms avg latency, 1828.0 max latency. 452748 records sent, 90549.6 records/sec (86.35 MB/sec), 340.5 ms avg latency, 1327.0 max latency. 305877 records sent, 57789.0 records/sec (55.11 MB/sec), 258.8 ms avg latency, 3117.0 max latency. 423903 records sent, 84780.6 records/sec (80.85 MB/sec), 556.3 ms avg latency, 3368.0 max latency. 451797 records sent, 84988.1 records/sec (81.05 MB/sec), 315.4 ms avg latency, 1309.0 max latency. 475265 records sent, 95053.0 records/sec (90.65 MB/sec), 363.2 ms avg latency, 1313.0 max latency. 577183 records sent, 114770.9 records/sec (109.45 MB/sec), 264.1 ms avg latency, 861.0 max latency. 673152 records sent, 112661.4 records/sec (107.44 MB/sec), 230.4 ms avg latency, 1321.0 max latency. 689925 records sent, 137957.4 records/sec (131.57 MB/sec), 265.3 ms avg latency, 1400.0 max latency. 400200 records sent, 79928.1 records/sec (76.23 MB/sec), 384.9 ms avg latency, 2438.0 max latency. 369566 records sent, 73913.2 records/sec (70.49 MB/sec), 410.9 ms avg latency, 2295.0 max latency. 466609 records sent, 86908.0 records/sec (82.88 MB/sec), 318.5 ms avg latency, 1511.0 max latency. 642825 records sent, 117196.9 records/sec (111.77 MB/sec), 253.9 ms avg latency, 1139.0 max latency. 365312 records sent, 73062.4 records/sec (69.68 MB/sec), 438.2 ms avg latency, 1618.0 max latency. 546463 records sent, 108792.2 records/sec (103.75 MB/sec), 309.4 ms avg latency, 975.0 max latency. 449057 records sent, 89811.4 records/sec (85.65 MB/sec), 338.8 ms avg latency, 1284.0 max latency. 461461 records sent, 92292.2 records/sec (88.02 MB/sec), 336.2 ms avg latency, 1115.0 max latency. 425741 records sent, 84438.9 records/sec (80.53 MB/sec), 360.2 ms avg latency, 1078.0 max latency. 527314 records sent, 105462.8 records/sec (100.58 MB/sec), 288.8 ms avg latency, 1230.0 max latency. 516075 records sent, 103215.0 records/sec (98.43 MB/sec), 300.2 ms avg latency, 1021.0 max latency. 513447 records sent, 102607.3 records/sec (97.85 MB/sec), 299.7 ms avg latency, 1108.0 max latency. 392184 records sent, 78436.8 records/sec (74.80 MB/sec), 377.8 ms avg latency, 1352.0 max latency. 516621 records sent, 97715.3 records/sec (93.19 MB/sec), 261.6 ms avg latency, 1416.0 max latency. 512059 records sent, 102411.8 records/sec (97.67 MB/sec), 325.1 ms avg latency, 1509.0 max latency. 377966 records sent, 73980.4 records/sec (70.55 MB/sec), 422.2 ms avg latency, 1893.0 max latency. 503959 records sent, 100791.8 records/sec (96.12 MB/sec), 332.4 ms avg latency, 1506.0 max latency. 474116 records sent, 94407.8 records/sec (90.03 MB/sec), 321.2 ms avg latency, 1079.0 max latency. 100000000 records sent, 78434.325371 records/sec (74.80 MB/sec), 390.67 ms avg latency, 6731.00 ms max latency, 2 ms 50th, 87 ms 95th, 158 ms 99th, 275 ms 99.9th. [root@node105 ~]#
100000000 records sent, 78434.325371 records/sec (74.80 MB/sec), 390.67 ms avg latency, 6731.00 ms max latency, 2 ms 50th, 87 ms 95th, 158 ms 99th, 275 ms 99.9th. 当你执行完kafka-producer-perf-test.sh 脚本时，会输出类似上面的一行，从上面一行我们可以得出一些消息 1>.每秒平均向kafka写入了74.80 MB的数据(大概是：78433条消息，计算方式：74.8*1024*1024/1000); 2>.每次写入的平均延迟为390.67毫秒； 3>.最大延迟为6731.00毫秒； 4>.87毫秒内占95%； 5>.158毫秒内占99%； 6>.275毫秒内占99.9%；
二.kafka-consumer-perf-test.sh 使用案例
1>. 测试Kafka消费的吞吐量
[root@node105 ~]# nohup kafka-consumer-perf-test.sh --broker-list 10.1.2.102:9092 --topic yinzhengjie-kafka --messages 1000000 --fetch-size 1048576 --threads 10 > yinzhengjie-kafka-consumer.log & [root@node105 ~]# 相关参数说明： --topic 指定topic的名称，本例为 yinzhengjie-kafka --fetch-size
指定每次fetch的数据的大小，本例为1048576，也就是1M --messages
总共要消费的消息个数，本例为1000000，100w
--topic
指定消费的topic
--threads
指定消费的线程数为10
2>.计算消费者的吞吐量
[root@node105 ~]# cat yinzhengjie-kafka-consumer.log start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec 2018-11-13 19:41:31:507, 2018-11-13 19:41:34:169, 953.9165, 358.3458, 1000411, 375811.7956 [root@node105 ~]# 当你执行完 kafka-consumer-perf-test.sh 脚本时，会输出类似于上面的一行，从上面一行我们可以得出一些消息 1>.共消费了953.9165M的数据； 2>.消费速度为358.3458M/s； 3>.总共消费了1000411 条消息； 4>.每秒消费375811.7956条消息；
三.编写脚本案例
1>.执行脚本
[root@node105 ~]# cat yinzhengjie-kafka.sh nohup kafka-consumer-perf-test.sh --broker-list 10.1.3.116:9092,10.1.3.117:9092,10.1.3.118:9092,10.1.3.119:9092,10.1.3.120:9092 --topic yinzhengjie-kafka --messages 100000000 --fetch-size 1048576 --threads 10 > /root/yinzhengjie-kafka-consumer.log & nohup kafka-producer-perf-test.sh --num-records 100000000 --record-size 1000 --topic yinzhengjie-kafka --throughput 1000000 --producer-props bootstrap.servers=10.1.3.116:9092,10.1.3.117:9092,10.1.3.118:9092,10.1.3.119:9092,10.1.3.120:9092 > /root/yinzhengjie-kafka-producer.log & [root@node105 ~]#
2>.运行脚本时查看Kafka manager的监控
3>.运行结束后查看Kafka manager的监控
4>.查看我的Kafka性能测试结果
生产端： 总共需要发送的消息数：1亿条，每个记录的字节数是：1000字节，每秒钟发送的记录数是：1000000 消费端： 模拟了10个消费者：进行消费 指定每次fetch的数据的大小：1M 总共要消费的消息个数：1亿条 生产端结论： 1>.每秒平均向kafka写入了225.07 MB的数据； 2>.每次写入的平均延迟为 129.81毫秒； 3>.最大延迟为2422.00毫秒； 消费端结论： 1>. 共消费了95367.4316M的数据； 2>.消费速度为225.2047M/s； 3>.总共消费了1亿条消息； 4>.每秒消费236144.2369条消息；