UNDERSTANDING OPENFLOW RULES
OpenVswitch (OVS) is a virtual switch that connects virtual machines together using virtual links and ports. Traditionally this would be done by a physical switch over physical links and network cards and switch ports. In OpenStack, OVS also plays an important role which provides virtualised network services and both the Neutron node, and the compute node are running OpenVSwitches.
But what is important about OVS is its role in manipulating and directing the coming in and out. In this article we intend to describe the flow rules installed on OVS inside OpenStack Mitaka.
Login to Mitaka node using the following:
ssh root@Mitaka’s IP address
For example:
ssh root@192.168.127.101
Login to the compute node:
[root@mitaka ~]# ssh compute
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-135-generic x86_64)
* Documentation: https://help.ubuntu.com/
Last login: Wed Sep 26 06:40:57 2018 from 10.20.0.2
root@node-4:~#
Print the information of the br-tun of OpenStack as it provides communication inside and outside of the OpenStack:
root@node-4:~# ovs-ofctl dump-flows br-tun NXST_FLOW reply (xid=0x4): 1- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=0, n_packets=183, n_bytes=28498, idle_age=4, priority=1,in_port=1 actions=resubmit(,2) 2- cookie=0xbb7b3cdd52626a01, duration=9917.985s, table=0, n_packets=198, n_bytes=36045, idle_age=4, priority=1,in_port=2 actions=resubmit(,4) 3- cookie=0xbb7b3cdd52626a01, duration=13003.030s, table=0, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop 4- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=2, n_packets=1, n_bytes=42, idle_age=9913, priority=1,arp,dl_dst=ff:ff:ff:ff:ff:ff actions=resubmit(,21) --------广播报文 5- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=2, n_packets=168, n_bytes=26780, idle_age=4, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20) 6- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=2, n_packets=14, n_bytes=1676, idle_age=9904, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22) 7- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=3, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop 8- cookie=0xbb7b3cdd52626a01, duration=9921.166s, table=4, n_packets=198, n_bytes=36045, idle_age=4, priority=1,tun_id=0x2 actions=mod_vlan_vid:1,resubmit(,10) 9- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=4, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop 10- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=6, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop 11- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=10, n_packets=198, n_bytes=36045, idle_age=4, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xbb7b3cdd52626a01,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1 12- cookie=0xbb7b3cdd52626a01, duration=9917.984s, table=20, n_packets=102, n_bytes=14108, idle_age=9435, priority=2,dl_vlan=1,dl_dst=fa:16:3e:0b:cf:10 actions=strip_vlan,set_tunnel:0x2,output:2 13- cookie=0xbb7b3cdd52626a01, duration=9917.984s, table=20, n_packets=66, n_bytes=12672, idle_age=4, priority=2,dl_vlan=1,dl_dst=fa:16:3e:4a:10:2b actions=strip_vlan,set_tunnel:0x2,output:2 14- cookie=0xbb7b3cdd52626a01, duration=9913.613s, table=20, n_packets=0, n_bytes=0, hard_timeout=300, idle_age=9913, hard_age=4, priority=1,vlan_tci=0x0001/0x0fff,dl_dst=fa:16:3e:4a:10:2b actions=load:0->NXM_OF_VLAN_TCI[],load:0x2->NXM_NX_TUN_ID[],output:2 15- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=20, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=resubmit(,22) 16- cookie=0xbb7b3cdd52626a01, duration=9917.985s, table=21, n_packets=1, n_bytes=42, idle_age=9913, priority=1,arp,dl_vlan=1,arp_tpa=192.168.111.1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:0b:cf:10,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e0bcf10->NXM_NX_ARP_SHA[],load:0xc0a86f01->NXM_OF_ARP_SPA[],IN_PORT 17- cookie=0xbb7b3cdd52626a01, duration=9917.984s, table=21, n_packets=0, n_bytes=0, idle_age=9917, priority=1,arp,dl_vlan=1,arp_tpa=192.168.111.2 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:4a:10:2b,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e4a102b->NXM_NX_ARP_SHA[],load:0xc0a86f02->NXM_OF_ARP_SPA[],IN_PORT 18- cookie=0xbb7b3cdd52626a01, duration=13003.028s, table=21, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=resubmit(,22) 19- cookie=0xbb7b3cdd52626a01, duration=9917.956s, table=22, n_packets=10, n_bytes=1336, idle_age=9904, dl_vlan=1 actions=strip_vlan,set_tunnel:0x2,output:2 20- cookie=0xbb7b3cdd52626a01, duration=13003.002s, table=22, n_packets=4, n_bytes=340, idle_age=9920, priority=0 actions=drop
EXPLANATION OF THE RULES:
Table 0:
1- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=0, n_packets=183, n_bytes=28498, idle_age=4, priority=1,in_port=1 actions=resubmit(,2)
2- cookie=0xbb7b3cdd52626a01, duration=9917.985s, table=0, n_packets=198, n_bytes=36045, idle_age=4, priority=1,in_port=2 actions=resubmit(,4)
3- cookie=0xbb7b3cdd52626a01, duration=13003.030s, table=0, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop
| Rule 1 | Has priority=1 and checks if the packets coming on port in_port=”patch-int” then the action is: go to table 2 |
| Rule 2 | Checks if the packets coming on port in_port=vxlan-c0a80202 then the action is: go to table 4 |
| Rule 3 | Has priority=0 (lowest priority) and drop the packets that don’t match rule 1 and rule 2 |
Table 2:
4- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=2, n_packets=1, n_bytes=42, idle_age=9913, priority=1,arp,dl_dst=ff:ff:ff:ff:ff:ff actions=resubmit(,21)
5- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=2, n_packets=168, n_bytes=26780, idle_age=4, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
6- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=2, n_packets=14, n_bytes=1676, idle_age=9904, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
| Rule 4 |
Has priority=1 and checks if the packets are ARP packet with destination MAC address set to broadcast then the action is: go to table 21 |
| Rule 5 |
Has priority=0 and checks if the packets has dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 (match all unicast Ethernet packets) then the action is: go to table 20 |
| Rule 6 |
Has priority=0 and checks if the packets has dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 (match all multicast(including broadcast Ethernet packets) then the action is: go to table 22 |
Table 3:
7- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=3, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop
| Rule 7 | drop the packets |
Table 4:
8- cookie=0xbb7b3cdd52626a01, duration=9921.166s, table=4, n_packets=198, n_bytes=36045, idle_age=4, priority=1,tun_id=0x2 actions=mod_vlan_vid:1,resubmit(,10)
9- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=4, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop
| Rule 8 | Has priority=1 and checks if the packets tun_id=0x20 hen the action is to add the vlan_vid:1 and go to table 10 |
| Rule 9 | Has priority=0 (lower priority) and drop the packets that don’t match rule 8 |
Table 6:
10- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=6, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=drop
| Rule 10 | drop the packets |
Table 10:
11- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=10, n_packets=198, n_bytes=36045, idle_age=4, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xbb7b3cdd52626a01,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1
| Rule 11 |
Has priority=1 and the action has two parts: Part one: Is to install a rule in table 20. This table (20) will be a MAC learning table. The “learn” action modifies a flow table based on the content of the flow currently being processed by table 4. Here’s how you can interpret each part of the “learn” action above: table=20 Modify flow table 20. This will be the MAC learning table.
hard_timeout=300 Causes the flow to expire after the 300 seconds, regardless of activity.
priority=1 The priority at which a wildcarded entry will match in comparison to others
cookie=0x407518fa3ccd67d2 NXM_OF_VLAN_TCI[0..11] Make the flow that we add to flow table 20 match the same VLAN ID that the packet we’re currently processing contains. This effectively scopes the MAC learning entry to a single VLAN, which is the ordinary behavior for a VLAN-aware switch. NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[] Make the flow that we add to flow table 20 match, as Ethernet destination, the Ethernet source address of the packet we’re currently processing.
load:0->NXM_OF_VLAN_TCI[],
Strip off the VLAN ID by loading 0 as a VLAN ID
load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],
Load the tunnel ID of the proceesing packet as a tunnel id of the packet
output:OXM_OF_IN_PORT[]),
Send the packet out via input port
Part Two: output:”patch-int” sends the packet out via port patch-int |
Table 20:
12- cookie=0xbb7b3cdd52626a01, duration=9917.984s, table=20, n_packets=102, n_bytes=14108, idle_age=9435, priority=2,dl_vlan=1,dl_dst=fa:16:3e:0b:cf:10 actions=strip_vlan,set_tunnel:0x2,output:2
13- cookie=0xbb7b3cdd52626a01, duration=9917.984s, table=20, n_packets=66, n_bytes=12672, idle_age=4, priority=2,dl_vlan=1,dl_dst=fa:16:3e:4a:10:2b actions=strip_vlan,set_tunnel:0x2,output:2
14- cookie=0xbb7b3cdd52626a01, duration=9913.613s, table=20, n_packets=0, n_bytes=0, hard_timeout=300, idle_age=9913, hard_age=4, priority=1,vlan_tci=0x0001/0x0fff,dl_dst=fa:16:3e:4a:10:2b actions=load:0->NXM_OF_VLAN_TCI[],load:0x2->NXM_NX_TUN_ID[],output:2
15- cookie=0xbb7b3cdd52626a01, duration=13003.029s, table=20, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=resubmit(,22)
| Rule 12,13 |
Have priority=2 and check if the packets has VLAN id = 1 and a certain dl_dst addresses then the action is: strip the VLAN id and load the tunnel id of 0x2 and send the packets out via output:vxlan-c0a80202 |
| Rule 14 |
These rule are installed via the learn action of table 10: Has priority=1 and checks if the packets has vlan_tci=0x0001/0x0fff (VLAN id = 1) and ,dl_dst=fa:16:3e:4a:10:2b then the action is: strip the VLAN id and load the tunnel id of 0x2 and send the packets out via output:vxlan-c0a80202 |
| Rule 15 | Has priority=0 (lower priority) and the action is: go to table 22 |
Table 21:
16- cookie=0xbb7b3cdd52626a01, duration=9917.985s, table=21, n_packets=1, n_bytes=42, idle_age=9913, priority=1,arp,dl_vlan=1,arp_tpa=192.168.111.1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:0b:cf:10,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e0bcf10->NXM_NX_ARP_SHA[],load:0xc0a86f01->NXM_OF_ARP_SPA[],IN_PORT
17- cookie=0xbb7b3cdd52626a01, duration=9917.984s, table=21, n_packets=0, n_bytes=0, idle_age=9917, priority=1,arp,dl_vlan=1,arp_tpa=192.168.111.2 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:4a:10:2b,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e4a102b->NXM_NX_ARP_SHA[],load:0xc0a86f02->NXM_OF_ARP_SPA[],IN_PORT
18- cookie=0xbb7b3cdd52626a01, duration=13003.028s, table=21, n_packets=0, n_bytes=0, idle_age=13003, priority=0 actions=resubmit(,22)
| Rule 16, 17 |
Has priority=1 and checks if the packets are ARP packet and have certain VLAN ID (e.g. dl_vlan=1) and a certain destination IP address (e.g. arp_tpa=192.168.111.1) then the action of the flow is:
Note: the above flow indicate that the switch which is close to the host replies to arp MAC address |
| Rule 18 | Has priority=0 (lower priority) and the action is: go to table 22 |
Table 22:
19- cookie=0xbb7b3cdd52626a01, duration=9917.956s, table=22, n_packets=10, n_bytes=1336, idle_age=9904, dl_vlan=1 actions=strip_vlan,set_tunnel:0x2,output:2
20- cookie=0xbb7b3cdd52626a01, duration=13003.002s, table=22, n_packets=4, n_bytes=340, idle_age=9920, priority=0 actions=drop
| Rule 19 | Checks if the packet has VLAN ID=1 then the action is: strip the VLAN id and load the tunnel id of 0x2 and send the packets out via output:vxlan-c0a80202 |
| Rule 20 | Has priority=0 (lower priority) and drop the packets that don’t match rule 19 |
Having a good understanding of these rules will help us troubleshooting network traffic. If there are any connectivity issues in the network (internal/external) which result in the packet loss, we can easily follow the trail of packets within the engaged flow rules to find the leakage in the network.
For example, if we run ping between two OpenStack endpoints, first we need to understand which flow rules are being hit by the ping packets and then observe if there are any incremental changes in the “n_packets” count of the rule. The “n_packet” feature inform us if the packets are begin forwarded to another endpoint or being dropped in the network.