KK程序:
1. 两个线程,第一个从DPDK收包,通过一个ring数据传递给第二个线程。第二个线程将数据写入共享内存。
2. 第二个内存在发现共享内存已满时,会直接丢弃数据。
3. 线程二有个选项debug,用于每一次ring_dequeue之后,都将数据写入内存。
当这个选项为on时,内存未满,也不会丢包。
现象:当内存已满的时候,可以千兆线速收包。当内存未满时,丢包率为20%。
分别做三次gprof:
1. gmon-empty-off.txt
0.08 0.42 471955/471955 kk_assemble_pool_packet_process [21] [23] 1.8 0.08 0.42 471955 tcp_packet_process [23] 0.02 0.10 739629/739629 _assemble_session_find [36] 0.01 0.08 512494/512494 kk_tcp_session_request_find [43] 0.01 0.07 665134/818327 kk_table_entries_timeout_free [39] 0.04 0.04 330145/330145 _three_way_handshake_process [45] 0.00 0.06 456472/461571 _tcp_data_assemble_process [48]
2. gmon-full-off.txt
0.08 0.48 3746819/3746819 kk_assemble_pool_packet_process [15] [16] 2.4 0.08 0.48 3746819 tcp_packet_process [16] 0.09 0.08 3754332/3782242 kk_table_entries_timeout_free [26] 0.06 0.10 3746592/3746592 _assemble_session_find [27] 0.01 0.09 3747382/3747382 kk_tcp_session_request_find [32] 0.03 0.01 3702531/3702531 _three_way_handshake_process [45] 0.00 0.00 56275/71247 _tcp_data_assemble_process [4295]
3. gmon-mid-on.txt
0.10 0.55 3742005/3742005 kk_assemble_pool_packet_process [15] [16] 2.3 0.10 0.55 3742005 tcp_packet_process [16] 0.10 0.11 3745003/3745003 _assemble_session_find [21] 0.06 0.09 3753439/3777518 kk_table_entries_timeout_free [25] 0.02 0.11 3743276/3743276 kk_tcp_session_request_find [30] 0.02 0.04 3689792/3689792 _three_way_handshake_process [45] 0.00 0.00 64598/81267 _tcp_data_assemble_process [4295]
根据以上内容,对比一个关键步骤里的函数执行站比。可以发现。1中find查询的占比明确比其他两种情况更高。 而现象上也是情况1会有丢包,情况2,3不丢包。
再次测试,查看这三次的会话数。
1. gmon-empty-off.txt
name: tcp_assemble_task_1, size: 0, free: 1048575, pkts: 559700, session: 11688, hit: 0, drop: 0 name: udp_assemble_task_1, size: 0, free: 1048575, pkts: 0, session: 0, hit: 0, drop: 0 queue: 0, ipacket: 2595943, imissed: 1204057 self_counter: 2595943 queue: 0, total_tsc: 8825007304, tsc/pkt: 3399.538166
2. gmon-full-off.txt
name: tcp_assemble_task_1, size: 0, free: 1048575, pkts: 48176, session: 16740, hit: 0, drop: 33152 name: udp_assemble_task_1, size: 0, free: 1048575, pkts: 0, session: 0, hit: 0, drop: 0 queue: 0, ipacket: 3800000, imissed: 0 self_counter: 3800000 queue: 0, total_tsc: 8785588132, tsc/pkt: 2311.996877
3. gmon-full-on.txt
name: tcp_assemble_task_1, size: 0, free: 1048575, pkts: 50746, session: 16982, hit: 0, drop: 33600 name: udp_assemble_task_1, size: 0, free: 1048575, pkts: 0, session: 0, hit: 0, drop: 0 queue: 0, ipacket: 3800000, imissed: 0 self_counter: 3800000 queue: 0, total_tsc: 8868949684, tsc/pkt: 2333.934127
并未发现规律。
使用新数据再次做次测试:
每15个包1个http会话。共270000个会话,按顺序组装,4050000个包。
1. empty_on
name: tcp_assemble_task_1, size: 0, free: 1048575, pkts: 1754014, session: 270001, hit: 0, drop: 528988 name: udp_assemble_task_1, size: 0, free: 1048575, pkts: 0, session: 0, hit: 0, drop: 0 queue: 0, ipacket: 4050000, imissed: 0 self_counter: 4050000 queue: 0, total_tsc: 17489586080, tsc/pkt: 4318.416316
2. empty_off
name: tcp_assemble_task_1, size: 0, free: 1048575, pkts: 2429992, session: 269999, hit: 0, drop: 0 name: udp_assemble_task_1, size: 0, free: 1048575, pkts: 0, session: 0, hit: 0, drop: 0 queue: 0, ipacket: 4050000, imissed: 0 self_counter: 4050000 queue: 0, total_tsc: 19613438800, tsc/pkt: 4842.824395
与上一组同样的测试数据,但是每5000个作为一组并发。
1. empty_on
name: tcp_assemble_task_1, size: 0, free: 1048575, pkts: 285000, session: 270000, hit: 0, drop: 540000 name: udp_assemble_task_1, size: 0, free: 1048575, pkts: 0, session: 0, hit: 0, drop: 0 queue: 0, ipacket: 4015852, imissed: 34148 self_counter: 4015852 queue: 0, total_tsc: 11696532776, tsc/pkt: 2912.590597
2. empty_off
name: tcp_assemble_task_1, size: 0, free: 1048575, pkts: 2068418, session: 235000, hit: 0, drop: 0 name: udp_assemble_task_1, size: 0, free: 1048575, pkts: 0, session: 0, hit: 0, drop: 0 queue: 0, ipacket: 3756940, imissed: 293060 self_counter: 3756940 queue: 0, total_tsc: 17565322544, tsc/pkt: 4675.433343
最后,是并发数的问题:
KK程序的最大并发数,只能处理到4000.
name: tcp_assemble_task_1, size: 0, free: 1048575, pkts: 2430000, session: 270000, hit: 0, drop: 0 tcp_session: 1 max_concurrent: 5000 name: udp_assemble_task_1, size: 0, free: 1048575, pkts: 0, session: 0, hit: 0, drop: 0 tcp_session: 1 max_concurrent: 5000 queue: 0, max_concurrent: 5000 queue: 0, ipacket: 4020940, imissed: 29060 self_counter: 4020940 queue: 0, total_tsc: 19906132788, tsc/pkt: 4950.616718