tcpdump cksum incorrect

tcpdump在启用了Checksum Offloading的网卡上抓包提示cksum incorrect

https://huataihuang.gitbooks.io/cloud-atlas/content/network/packet_analysis/tcpdump/udp_tcp_checksum_errors_from_tcpdump_nic_hardware_offloading.html

大多数现代操作系统都支持网络卸载（network offloading）功能，即部分网络处理由网卡完成而不是由CPU处理。这样可以释放系统资源以便能够处理更多的连接。不过对于数据包捕捉分析会带来一些较为奇怪的结果或者丢失一些流量。

Checksum Offloading

在支持checksum offloading的系统中，IP,TCP和UDP checksum可以在传输到网线之前由网卡NIC来完成。此时在Wirshark中会提示数据包错误[incorrect, should be xxxx (maybe caused by "TCP checksum offload"?)].（tcpdump也有同样提示cksum xxxx incorrect）。抓包工具Wireshark/Tcpdump是在数据包被发送给网卡之前捕捉数据包的，此时它不会看到正确的checksum，因为此时尚未进行计算（因为checksum已经卸载到网卡，此时这个checksum字段会被填写为0）。这也就导致了抓包工具提示checksum错误的原因。

[root@bogon ~]# tcpdump -i enahisic2i3 tcp and host 10.10.103.229 -eenv 
tcpdump: listening on enahisic2i3, link-type EN10MB (Ethernet), capture size 262144 bytes
13:38:42.292481 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 58746, offset 0, flags [DF], proto TCP (6), length 67)
    10.10.103.81.58320 > 10.10.103.229.http: Flags [P.], cksum 0xe37f (incorrect -> 0x7afd), seq 3472498899:3472498926, ack 2821728936, win 29200, length 27: HTTP
13:39:40.532473 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 58747, offset 0, flags [DF], proto TCP (6), length 67)
    10.10.103.81.58320 > 10.10.103.229.http: Flags [P.], cksum 0xe37f (incorrect -> 0x7afd), seq 0:27, ack 1, win 29200, length 27: HTTP
13:41:33.172487 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 58748, offset 0, flags [DF], proto TCP (6), length 67)
    10.10.103.81.58320 > 10.10.103.229.http: Flags [P.], cksum 0xe37f (incorrect -> 0x7afd), seq 0:27, ack 1, win 29200, length 27: HTTP
13:43:36.052487 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 58749, offset 0, flags [DF], proto TCP (6), length 67)
    10.10.103.81.58320 > 10.10.103.229.http: Flags [P.], cksum 0xe37f (incorrect -> 0x7afd), seq 0:27, ack 1, win 29200, length 27: HTTP
13:45:38.932495 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 58750, offset 0, flags [DF], proto TCP (6), length 67)
    10.10.103.81.58320 > 10.10.103.229.http: Flags [P.], cksum 0xe37f (incorrect -> 0x7afd), seq 0:27, ack 1, win 29200, length 27: HTTP

[root@bogon ~]# ethtool --show-offload  enahisic2i3 | grep checksum
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
[root@bogon ~]# 

[root@bogon ~]# ethtool --offload enahisic2i3 rx off tx off
Actual changes:
rx-checksumming: off
tx-checksumming: off
        tx-checksum-ipv4: off
        tx-checksum-ipv6: off
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [requested on]
        tx-tcp6-segmentation: off [requested on]
[root@bogon ~]# 

[root@bogon ~]# tcpdump -i enahisic2i3 tcp and host 10.10.103.229 -eenv 
tcpdump: listening on enahisic2i3, link-type EN10MB (Ethernet), capture size 262144 bytes
13:50:00.491266 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 1837, offset 0, flags [DF], proto TCP (6), length 60)
    10.10.103.81.58740 > 10.10.103.229.http: Flags [S], cksum 0xaa31 (correct), seq 3170362118, win 29200, options [mss 1460,sackOK,TS val 2467508125 ecr 0,nop,wscale 7], length 0
13:50:00.491345 44:a1:91:a4:9b:eb > 48:57:02:64:ea:1e, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 64, id 1837, offset 0, flags [DF], proto TCP (6), length 40)
    10.10.103.229.http > 10.10.103.81.58740: Flags [S.], cksum 0x5dda (correct), seq 2821705385, ack 3170362119, win 29200, length 0
13:50:00.491385 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 64, id 1838, offset 0, flags [DF], proto TCP (6), length 40)
    10.10.103.81.58740 > 10.10.103.229.http: Flags [.], cksum 0x5ddb (correct), ack 1, win 29200, length 0
13:50:00.491497 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 1839, offset 0, flags [DF], proto TCP (6), length 67)
    10.10.103.81.58740 > 10.10.103.229.http: Flags [P.], cksum 0x2526 (correct), seq 1:28, ack 1, win 29200, length 27: HTTP
13:50:00.702452 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 1840, offset 0, flags [DF], proto TCP (6), length 67)
    10.10.103.81.58740 > 10.10.103.229.http: Flags [P.], cksum 0x2526 (correct), seq 1:28, ack 1, win 29200, length 27: HTTP
13:50:01.132447 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 1841, offset 0, flags [DF], proto TCP (6), length 67)
    10.10.103.81.58740 > 10.10.103.229.http: Flags [P.], cksum 0x2526 (correct), seq 1:28, ack 1, win 29200, length 27: HTTP
13:50:02.052451 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 1842, offset 0, flags [DF], proto TCP (6), length 67)
    10.10.103.81.58740 > 10.10.103.229.http: Flags [P.], cksum 0x2526 (correct), seq 1:28, ack 1, win 29200, length 27: HTTP
13:50:03.812450 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 1843, offset 0, flags [DF], proto TCP (6), length 67)
    10.10.103.81.58740 > 10.10.103.229.http: Flags [P.], cksum 0x2526 (correct), seq 1:28, ack 1, win 29200, length 27: HTTP
13:50:07.252454 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 1844, offset 0, flags [DF], proto TCP (6), length 67)
    10.10.103.81.58740 > 10.10.103.229.http: Flags [P.], cksum 0x2526 (correct), seq 1:28, ack 1, win 29200, length 27: HTTP

[root@bogon ~]# ethtool --offload enahisic2i3 rx on tx on
Actual changes:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ipv6: on
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp6-segmentation: on

[root@bogon ~]# tcpdump -i enahisic2i3 tcp and host 10.10.103.229 -eenv 
tcpdump: listening on enahisic2i3, link-type EN10MB (Ethernet), capture size 262144 bytes
13:50:58.966457 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 22335, offset 0, flags [DF], proto TCP (6), length 60)
    10.10.103.81.58770 > 10.10.103.229.http: Flags [S], cksum 0xe378 (incorrect -> 0xb938), seq 3320892, win 29200, options [mss 1460,sackOK,TS val 2467566599 ecr 0,nop,wscale 7], length 0
13:50:58.966538 44:a1:91:a4:9b:eb > 48:57:02:64:ea:1e, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 64, id 22335, offset 0, flags [DF], proto TCP (6), length 40)
    10.10.103.229.http > 10.10.103.81.58770: Flags [S.], cksum 0x334c (correct), seq 2821713065, ack 3320893, win 29200, length 0
13:50:58.966559 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 64, id 22336, offset 0, flags [DF], proto TCP (6), length 40)
    10.10.103.81.58770 > 10.10.103.229.http: Flags [.], cksum 0xe364 (incorrect -> 0x334d), ack 1, win 29200, length 0
13:50:58.966609 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 22337, offset 0, flags [DF], proto TCP (6), length 67)
    10.10.103.81.58770 > 10.10.103.229.http: Flags [P.], cksum 0xe37f (incorrect -> 0xfa97), seq 1:28, ack 1, win 29200, length 27: HTTP
13:50:59.182449 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 22338, offset 0, flags [DF], proto TCP (6), length 67)
    10.10.103.81.58770 > 10.10.103.229.http: Flags [P.], cksum 0xe37f (incorrect -> 0xfa97), seq 1:28, ack 1, win 29200, length 27: HTTP
13:50:59.612451 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 22339, offset 0, flags [DF], proto TCP (6), length 67)
    10.10.103.81.58770 > 10.10.103.229.http: Flags [P.], cksum 0xe37f (incorrect -> 0xfa97), seq 1:28, ack 1, win 29200, length 27: HTTP
13:51:00.532458 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 22340, offset 0, flags [DF], proto TCP (6), length 67)
    10.10.103.81.58770 > 10.10.103.229.http: Flags [P.], cksum 0xe37f (incorrect -> 0xfa97), seq 1:28, ack 1, win 29200, length 27: HTTP
13:51:02.292467 48:57:02:64:ea:1e > 44:a1:91:a4:9b:eb, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 22341, offset 0, flags [DF], proto TCP (6), length 67)
    10.10.103.81.58770 > 10.10.103.229.http: Flags [P.], cksum 0xe37f (incorrect -> 0xfa97), seq 1:28, ack 1, win 29200, length 27: HTTP