使用tc模拟网络延迟和丢包

Question

see https://unix.stackexchange.com/questions/288959/how-is-the-ifb-device-positioned-in-the-packet-flow-of-the-linux-kernel

I would like to know the exact position of the following device in the packet flow for ingress traffic shaping:

IFB: Intermediate Functional Block

I would like to better understand how packets are flowing to this device and exactly when this happens to understand what methods for filtering / classification can be used of the following:

tc filter ... u32 ...
iptables ... -j MARK --set-mark ...
iptables ... -j CLASSIFY --set-class ...

It seems hard to find documentation on this topic, any help where to find official documentation would be greatly appreciated as well.

Documentation as far as I know:

tc: tldp.org HOWTO, lartc.org HOWTO
ifb: linuxfoundation.org, tc-mirred manpage, wiki.gentoo.org
netfilter packet flow: kernel_flow, docum.org kptd

From the known documentation I interpret the following:

Basic traffic control

figure 1
+-------+                 +------+
|ingress|   +---------+   |egress|
|qdisc  +--->netfilter+--->qdisc |
|eth0   |   +---------+   |eth0  |
+-------+                 +------+

IFB?

tc filter add dev eth0 parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0 will result in?

figure 2
+-------+   +-------+   +------+                 +------+
|ingress|   |ingress|   |egress|   +---------+   |egress|
|qdisc  +--->qdisc  +--->qdisc +--->netfilter+--->qdisc |
|eth0   |   |ifb0   |   |ifb0  |   +---------+   |eth0  |
+-------+   +-------+   +------+                 +------+

Accepted Answer

I think I finally understood how redirecting ingress to IFB is working:

+-------+   +------+                 +------+
|ingress|   |egress|   +---------+   |egress|
|qdisc  +--->qdisc +--->netfilter+--->qdisc |
|eth1   |   |ifb1  |   +---------+   |eth1  |
+-------+   +------+                 +------+

My initial assumption in figure 2, that the ifb device is inserted between ingress eth1 and netfilter and that packets first enter the ingress ifb1 and then exit through egress ifb1 was wrong.

In fact redirecting traffic from an interface's ingress or egress to the ifb's egress is done directly by redirecting ("stealing") the packet and directly placing it in the egress of the ifb device.

Mirroring/redirecting traffic to the ifb's ingress is currently not supported as also stated in the documentation, at least on my version:

root@deb8:~# tc -V
tc utility, iproute2-ss140804
root@deb8:~# dpkg -l | grep iproute
ii  iproute2                       3.16.0-2
root@deb8:~# uname -a
Linux deb8 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-1 x86_64 GNU/Linux

Documentation

I was able to get this information thanks to the following documentation:

linux-ip.net Intermediate Functional Block
dev.laptop.org ifb-README
people.netfilter.org Linux Traffic Control Classifier-Action Subsystem Architecture Paper

Debugging

And some debugging using iptables -j LOG and tc filter action simple, which I used to print out messages to syslog when an icmp packet is flowing through the netdevs.

The result is as follows:

Jun 14 13:02:12 deb8 kernel: [ 4273.341087] simple: tc[eth1]ingress_1
Jun 14 13:02:12 deb8 kernel: [ 4273.341114] simple: tc[ifb1]egress_1
Jun 14 13:02:12 deb8 kernel: [ 4273.341229] ipt[PREROUTING]raw IN=eth1 OUT= MAC=08:00:27:ee:8f:15:08:00:27:89:16:5b:08:00 SRC=10.1.1.3 DST=10.1.1.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53979 DF PROTO=ICMP TYPE=8 CODE=0 ID=1382 SEQ=1
Jun 14 13:02:12 deb8 kernel: [ 4273.341238] ipt[PREROUTING]mangle IN=eth1 OUT= MAC=08:00:27:ee:8f:15:08:00:27:89:16:5b:08:00 SRC=10.1.1.3 DST=10.1.1.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53979 DF PROTO=ICMP TYPE=8 CODE=0 ID=1382 SEQ=1
Jun 14 13:02:12 deb8 kernel: [ 4273.341242] ipt[PREROUTING]nat IN=eth1 OUT= MAC=08:00:27:ee:8f:15:08:00:27:89:16:5b:08:00 SRC=10.1.1.3 DST=10.1.1.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53979 DF PROTO=ICMP TYPE=8 CODE=0 ID=1382 SEQ=1
Jun 14 13:02:12 deb8 kernel: [ 4273.341249] ipt[INPUT]mangle IN=eth1 OUT= MAC=08:00:27:ee:8f:15:08:00:27:89:16:5b:08:00 SRC=10.1.1.3 DST=10.1.1.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53979 DF PROTO=ICMP TYPE=8 CODE=0 ID=1382 SEQ=1
Jun 14 13:02:12 deb8 kernel: [ 4273.341252] ipt[INPUT]filter IN=eth1 OUT= MAC=08:00:27:ee:8f:15:08:00:27:89:16:5b:08:00 SRC=10.1.1.3 DST=10.1.1.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53979 DF PROTO=ICMP TYPE=8 CODE=0 ID=1382 SEQ=1
Jun 14 13:02:12 deb8 kernel: [ 4273.341255] ipt[INPUT]nat IN=eth1 OUT= MAC=08:00:27:ee:8f:15:08:00:27:89:16:5b:08:00 SRC=10.1.1.3 DST=10.1.1.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53979 DF PROTO=ICMP TYPE=8 CODE=0 ID=1382 SEQ=1
Jun 14 13:02:12 deb8 kernel: [ 4273.341267] ipt[OUTPUT]raw IN= OUT=eth1 SRC=10.1.1.2 DST=10.1.1.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=37735 PROTO=ICMP TYPE=0 CODE=0 ID=1382 SEQ=1
Jun 14 13:02:12 deb8 kernel: [ 4273.341270] ipt[OUTPUT]mangle IN= OUT=eth1 SRC=10.1.1.2 DST=10.1.1.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=37735 PROTO=ICMP TYPE=0 CODE=0 ID=1382 SEQ=1
Jun 14 13:02:12 deb8 kernel: [ 4273.341272] ipt[OUTPUT]filter IN= OUT=eth1 SRC=10.1.1.2 DST=10.1.1.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=37735 PROTO=ICMP TYPE=0 CODE=0 ID=1382 SEQ=1
Jun 14 13:02:12 deb8 kernel: [ 4273.341274] ipt[POSTROUTING]mangle IN= OUT=eth1 SRC=10.1.1.2 DST=10.1.1.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=37735 PROTO=ICMP TYPE=0 CODE=0 ID=1382 SEQ=1
Jun 14 13:02:12 deb8 kernel: [ 4273.341278] simple: tc[eth1]egress_1
Jun 14 13:02:12 deb8 kernel: [ 4273.341280] simple: tc[ifb0]egress_1

The debugging was done using the following settings:

iptables -F -t filter
iptables -F -t nat
iptables -F -t mangle
iptables -F -t raw
iptables -A PREROUTING -t raw -p icmp --icmp-type 8 -j LOG --log-level 7 --log-prefix 'ipt[PREROUTING]raw '
iptables -A PREROUTING -t mangle -p icmp --icmp-type 8 -j LOG --log-level 7 --log-prefix 'ipt[PREROUTING]mangle '
iptables -A PREROUTING -t nat -p icmp --icmp-type 8 -j LOG --log-level 7 --log-prefix 'ipt[PREROUTING]nat '
iptables -A INPUT -t mangle -p icmp --icmp-type 8 -j LOG --log-level 7 --log-prefix 'ipt[INPUT]mangle '
iptables -A INPUT -t filter -p icmp --icmp-type 8 -j LOG --log-level 7 --log-prefix 'ipt[INPUT]filter '
iptables -A INPUT -t nat -p icmp --icmp-type 8 -j LOG --log-level 7 --log-prefix 'ipt[INPUT]nat '
iptables -A FORWARD -t mangle -p icmp --icmp-type 8 -j LOG --log-level 7 --log-prefix 'ipt[FORWARD]mangle '
iptables -A FORWARD -t filter -p icmp --icmp-type 8 -j LOG --log-level 7 --log-prefix 'ipt[FORWARD]filter '
iptables -A OUTPUT -t raw -p icmp --icmp-type 8 -j LOG --log-level 7 --log-prefix 'ipt[OUTPUT]raw '
iptables -A OUTPUT -t mangle -p icmp --icmp-type 8 -j LOG --log-level 7 --log-prefix 'ipt[OUTPUT]mangle '
iptables -A OUTPUT -t nat -p icmp --icmp-type 8 -j LOG --log-level 7 --log-prefix 'ipt[OUTPUT]nat '
iptables -A OUTPUT -t filter -p icmp --icmp-type 8 -j LOG --log-level 7 --log-prefix 'ipt[OUTPUT]filter '
iptables -A POSTROUTING -t mangle -p icmp --icmp-type 8 -j LOG --log-level 7 --log-prefix 'ipt[POSTROUTING]mangle '
iptables -A POSTROUTING -t nat -p icmp --icmp-type 8 -j LOG --log-level 7 --log-prefix 'ipt[POSTROUTING]nat '
iptables -A PREROUTING -t raw -p icmp --icmp-type 0 -j LOG --log-level 7 --log-prefix 'ipt[PREROUTING]raw '
iptables -A PREROUTING -t mangle -p icmp --icmp-type 0 -j LOG --log-level 7 --log-prefix 'ipt[PREROUTING]mangle '
iptables -A PREROUTING -t nat -p icmp --icmp-type 0 -j LOG --log-level 7 --log-prefix 'ipt[PREROUTING]nat '
iptables -A INPUT -t mangle -p icmp --icmp-type 0 -j LOG --log-level 7 --log-prefix 'ipt[INPUT]mangle '
iptables -A INPUT -t filter -p icmp --icmp-type 0 -j LOG --log-level 7 --log-prefix 'ipt[INPUT]filter '
iptables -A INPUT -t nat -p icmp --icmp-type 0 -j LOG --log-level 7 --log-prefix 'ipt[INPUT]nat '
iptables -A FORWARD -t mangle -p icmp --icmp-type 0 -j LOG --log-level 7 --log-prefix 'ipt[FORWARD]mangle '
iptables -A FORWARD -t filter -p icmp --icmp-type 0 -j LOG --log-level 7 --log-prefix 'ipt[FORWARD]filter '
iptables -A OUTPUT -t raw -p icmp --icmp-type 0 -j LOG --log-level 7 --log-prefix 'ipt[OUTPUT]raw '
iptables -A OUTPUT -t mangle -p icmp --icmp-type 0 -j LOG --log-level 7 --log-prefix 'ipt[OUTPUT]mangle '
iptables -A OUTPUT -t nat -p icmp --icmp-type 0 -j LOG --log-level 7 --log-prefix 'ipt[OUTPUT]nat '
iptables -A OUTPUT -t filter -p icmp --icmp-type 0 -j LOG --log-level 7 --log-prefix 'ipt[OUTPUT]filter '
iptables -A POSTROUTING -t mangle -p icmp --icmp-type 0 -j LOG --log-level 7 --log-prefix 'ipt[POSTROUTING]mangle '
iptables -A POSTROUTING -t nat -p icmp --icmp-type 0 -j LOG --log-level 7 --log-prefix 'ipt[POSTROUTING]nat '

export TC="/sbin/tc"

$TC qdisc del dev eth1 root
$TC qdisc del dev eth1 ingress
ip link set dev ifb0 down
ip link set dev ifb1 down
$TC qdisc del dev ifb0 root
$TC qdisc del dev ifb1 root
rmmod ifb

modprobe ifb numifbs=2

$TC qdisc add dev ifb0 root handle 1: htb default 2
$TC class add dev ifb0 parent 1: classid 1:1 htb rate 2Mbit
$TC class add dev ifb0 parent 1: classid 1:2 htb rate 10Mbit
$TC filter add dev ifb0 parent 1: protocol ip prio 1 u32 \
  match ip protocol 1 0xff flowid 1:1 \
  action simple "tc[ifb0]egress"
$TC qdisc add dev ifb0 ingress
$TC filter add dev ifb0 parent ffff: protocol ip prio 1 u32 \
  match ip protocol 1 0xff \
  action simple "tc[ifb0]ingress"

$TC qdisc add dev ifb1 root handle 1: htb default 2
$TC class add dev ifb1 parent 1: classid 1:1 htb rate 2Mbit
$TC class add dev ifb1 parent 1: classid 1:2 htb rate 10Mbit
$TC filter add dev ifb1 parent 1: protocol ip prio 1 u32 \
  match ip protocol 1 0xff flowid 1:1 \
  action simple "tc[ifb1]egress"
$TC qdisc add dev ifb1 ingress
$TC filter add dev ifb1 parent ffff: protocol ip prio 1 u32 \
  match ip protocol 1 0xff \
  action simple "tc[ifb1]ingress"

ip link set dev ifb0 up
ip link set dev ifb1 up

$TC qdisc add dev eth1 root handle 1: htb default 2
$TC class add dev eth1 parent 1: classid 1:1 htb rate 2Mbit
$TC class add dev eth1 parent 1: classid 1:2 htb rate 10Mbit
$TC filter add dev eth1 parent 1: protocol ip prio 1 u32 \
  match ip protocol 1 0xff flowid 1:1 \
  action simple "tc[eth1]egress" pipe \
  action mirred egress redirect dev ifb0
$TC qdisc add dev eth1 ingress
$TC filter add dev eth1 parent ffff: protocol ip prio 1 u32 \
  match ip protocol 1 0xff \
  action simple "tc[eth1]ingress" pipe \
  action mirred egress redirect dev ifb1

使用tc模拟网络延迟和丢包

使用tc模拟网络延迟和丢包

netem简介

netem延迟设置

netem模拟丢包设置

netem模拟报文重复\损坏设置

netem模拟包乱序

其他工具

参考：

0）查看网卡配置

1）模拟网络延迟传输

2）模拟网络延迟波动

3）模拟网络延迟波动随机性

4）模拟网络丢包：

5）模拟网络丢包成功率

6）模拟包重复

7）模拟包损坏

8）模拟包乱序

9）删除相关配置

10）查看丢包率

Documentation

Debugging

一、流量控制方式

二、流量控制处理对象