A bridge-domain can be associated with only one BVI. However a vrf can contain routes to multiple BVIs. To configure a BVI interface, create it using a loopback interface, assign a mac address, and set its vrf and bridge-domain. This example uses vrf 0 and bridge-domain 5. Learning should be disabled on the BVI interface. It is harmless but useless to enable it because the BVI l2fib entry is static: vpp# loop create 0000.0B51.0001 vpp# set int l2 bridge loop0 5 bvi vpp# set int ip table loop0 0 vpp# set int l2 learn loop0 disable vpp# set int state loop0 up Add L3 connectivity by creating a route and adjacency for the BVI interface. The static ARP is not strictly necessary -- routing to an IP with an unknown mac will cause an ARP request to be generated. vpp# ip route table 0 8.0.1.0/24 via loop0 vpp# set ip arp loop0 8.0.1.1 00:02:04:06:08:0a
loop create set int state loop0 up set int ip address loop0 10.0.0.1/24 set ip neighbor loop0 10.0.0.2 00:00:11:aa:bb:cc mpls tunnel add via 10.0.0.2 loop0 out-label 33 out-label 34 out-label 35 out-label 36 set int state mpls-tunnel0 up set int ip addr mpls-tunnel0 192.168.0.1/32 ip route add 2.0.0.2/32 via 192.168.0.2 mpls-tunnel0 mpls tunnel add via 10.0.0.2 out-label 33 set int state mpls-tunnel1 up set int ip addr mpls-tunnel1 192.168.1.1/32 ip route add 2.0.1.2/32 via 192.168.1.2 mpls-tunnel1 out-label 99 mpls tunnel add via 10.0.0.2 loop0 out-label 3 set int state mpls-tunnel2 up set int ip addr mpls-tunnel2 192.168.2.1/32 ip route add 2.0.2.2/32 via 192.168.2.2 mpls-tunnel2 mpls tunnel add l2-only via 10.0.0.2 loop0 out-label 234 out-label 0 set int state mpls-tunnel3 up set int l2 bridge mpls-tunnel3 1 loop create set int ip addr loop1 6.0.1.44/24 set int l2 bridge loop1 1 bvi set int l2 learn loop1 disable set int state loop1 up ip route add 2.0.3.2/32 via 6.0.1.45 loop1
Learning VPP: IPsec GRE over VxLAN
Overview
The goal is to create a layer-2 encrypted tunnel and hide inner network IP addresses.
To achieve this goal, the traffic will be encapsulated in GRE, protected with IPsec and encapsulated into VxLAN.
GRE is a tunneling protocol developed by Cisco. The GRE frame looks as follows.
VXLAN tunnel is an L2 overlay on top of an L3 network underlay. It uses the UDP protocol to traverse the network. The VXLAN frame looks as follows.
IPsec supports tunnel and transport modes. As far as our tunnel is based on GRE, the transport mode will be used. In this mode, only a payload of the IP packet is encrypted and/or authenticated and the IP header is not touched. The resulting frame looks as follows.
Setup
Two Ubuntu VMs with VPP ver. 19.01 and two Ubuntu VMs representing hosts.
VPP configuration
In terms of VPP we need to create two loopbacks. One loopback will be bridged with GRE-IPsec tunnel, while another will be bridged with VxLAN tunnel. And using routing we will direct traffic into the first loopback where it will be encapsulated into GRE header and encrypted with IPsec. Then the traffic will be routed into a second loopback where it will receive VxLAN header.
Router1
ipsec sa add 10 spi 1001 esp crypto-alg aes-cbc-128 crypto-key 4a506a794f574265564551694d653768 integ-alg sha1-96 integ-key 4339314b55523947594d6d3547666b45764e6a58 ipsec sa add 20 spi 1000 esp crypto-alg aes-cbc-128 crypto-key 4a506a794f574265564551694d653768 integ-alg sha1-96 integ-key 4339314b55523947594d6d3547666b45764e6a58 loopback create mac 1a:ab:3c:4d:5e:7f set interface ip address loop0 10.100.0.7/31 set int mtu 1360 loop0 set int l2 learn loop0 disable create ipsec gre tunnel src 10.101.0.7 dst 10.101.0.6 local-sa 10 remote-sa 20 set int state ipsec-gre0 up create bridge-domain 12 learn 0 forward 1 uu-flood 1 flood 1 arp-term 1 set bridge-domain arp entry 12 10.100.0.7 1a:ab:3c:4d:5e:7f set int l2 bridge loop0 12 bvi set int l2 bridge ipsec-gre0 12 1 loopback create mac 1a:2b:3c:4d:5e:7f set interface ip address loop1 10.101.0.7/31 create vxlan tunnel src 192.168.31.76 dst 192.168.31.47 vni 13 create bridge-domain 13 learn 0 forward 1 uu-flood 1 flood 1 arp-term 1 set bridge-domain arp entry 13 10.101.0.7 1a:2b:3c:4d:5e:7f set interface l2 bridge vxlan_tunnel0 13 1 set interface l2 bridge loop1 13 bvi ip route add 10.10.10.0/24 via 10.100.0.6
Router2
ipsec sa add 10 spi 1000 esp crypto-alg aes-cbc-128 crypto-key 4a506a794f574265564551694d653768 integ-alg sha1-96 integ-key 4339314b55523947594d6d3547666b45764e6a58 ipsec sa add 20 spi 1001 esp crypto-alg aes-cbc-128 crypto-key 4a506a794f574265564551694d653768 integ-alg sha1-96 integ-key 4339314b55523947594d6d3547666b45764e6a58 loopback create mac 1a:ab:3c:4d:5e:6f set interface ip address loop0 10.100.0.6/31 set int mtu 1360 loop0 set int l2 learn loop0 disable create ipsec gre tunnel src 10.101.0.6 dst 10.101.0.7 local-sa 10 remote-sa 20 set int state ipsec-gre0 up create bridge-domain 12 learn 0 forward 1 uu-flood 1 flood 1 arp-term 1 set bridge-domain arp entry 12 10.100.0.6 1a:ab:3c:4d:5e:6f set int l2 bridge loop0 12 bvi set int l2 bridge ipsec-gre0 12 1 loopback create mac 1a:2b:3c:4d:5e:6f set interface ip address loop1 10.101.0.6/31 create vxlan tunnel src 192.168.31.47 dst 192.168.31.76 vni 13 create bridge-domain 13 learn 0 forward 1 uu-flood 1 flood 1 arp-term 1 set bridge-domain arp entry 13 10.101.0.6 1a:2b:3c:4d:5e:6f set interface l2 bridge vxlan_tunnel0 13 1 set interface l2 bridge loop1 13 bvi ip route add 20.20.20.0/24 via 10.100.0.7
Results
Encap trace
00:04:26:418264: dpdk-input
GigabitEthernet0/8/0 rx queue 0
buffer 0xddb4: current data 0, length 98, free-list 0, clone-count 0, totlen-nifb 0, trace 0x0
ext-hdr-valid
l4-cksum-computed l4-cksum-correct
PKT MBUF: port 1, nb_segs 1, pkt_len 98
buf_len 2176, data_len 98, ol_flags 0x0, data_off 128, phys_addr 0x8e376d80
packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x0 fdir.hi 0x0 fdir.lo 0x0
IP4: 08:00:27:54:67:a2 -> 08:00:27:88:33:fd
ICMP: 20.20.20.2 -> 10.10.10.2
tos 0x00, ttl 64, length 84, checksum 0x66a0
fragment id 0x97e7, flags DONT_FRAGMENT
ICMP echo_request checksum 0x2ea5
00:04:26:418338: ethernet-input
frame: flags 0x3, hw-if-index 2, sw-if-index 2
IP4: 08:00:27:54:67:a2 -> 08:00:27:88:33:fd
00:04:26:418355: ip4-input-no-checksum
ICMP: 20.20.20.2 -> 10.10.10.2
tos 0x00, ttl 64, length 84, checksum 0x66a0
fragment id 0x97e7, flags DONT_FRAGMENT
ICMP echo_request checksum 0x2ea5
00:04:26:418369: ip4-lookup
fib 0 dpo-idx 14 flow hash: 0x00000000
ICMP: 20.20.20.2 -> 10.10.10.2
tos 0x00, ttl 64, length 84, checksum 0x66a0
fragment id 0x97e7, flags DONT_FRAGMENT
ICMP echo_request checksum 0x2ea5
00:04:26:418385: ip4-rewrite
tx_sw_if_index 3 dpo-idx 14 : ipv4 via 10.100.0.6 loop0: mtu:1360 1aab3c4d5e6f1aab3c4d5e7f0800 flow hash: 0x00000000
00000000: 1aab3c4d5e6f1aab3c4d5e7f08004500005497e740003f0167a0141414020a0a
00000020: 0a0208002ea555ec004d8f3afb5d0000000025b60400000000001011
00:04:26:418393: loop0-output
loop0
IP4: 1a:ab:3c:4d:5e:7f -> 1a:ab:3c:4d:5e:6f
ICMP: 20.20.20.2 -> 10.10.10.2
tos 0x00, ttl 63, length 84, checksum 0x67a0
fragment id 0x97e7, flags DONT_FRAGMENT
ICMP echo_request checksum 0x2ea5
00:04:26:418417: l2-input
l2-input: sw_if_index 3 dst 1a:ab:3c:4d:5e:6f src 1a:ab:3c:4d:5e:7f
00:04:26:418423: l2-fwd
l2-fwd: sw_if_index 3 dst 1a:ab:3c:4d:5e:6f src 1a:ab:3c:4d:5e:7f bd_index 1 result [0x1020000000004, 4] none
00:04:26:418428: l2-output
l2-output: sw_if_index 4 dst 1a:ab:3c:4d:5e:6f src 1a:ab:3c:4d:5e:7f data 08 00 45 00 00 54 97 e7 40 00 3f 01
00:04:26:418434: ipsec-gre0-output
ipsec-gre0
00000000: 1aab3c4d5e6f1aab3c4d5e7f08004500005497e740003f0167a0141414020a0a
00000020: 0a0208002ea555ec004d8f3afb5d0000000025b6040000000000101112131415
00000040: 161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435
00000060: 36370000000000000000000000000000000000000000000000000000
00:04:26:418436: ipsec-gre0-tx
GRE: tunnel 0 len 122 src 10.101.0.7 dst 10.101.0.6 sa-id 10
00:04:26:418440: esp4-encrypt
esp: spi 1001 seq 134 crypto aes-cbc-128 integrity sha1-96
00:04:26:418468: ip4-lookup
fib 0 dpo-idx 19 flow hash: 0x00000000
IPSEC_ESP: 10.101.0.7 -> 10.101.0.6
tos 0x00, ttl 254, length 168, checksum 0xa74d
fragment id 0x0000
00:04:26:418471: ip4-rewrite
tx_sw_if_index 5 dpo-idx 19 : ipv4 via 10.101.0.6 loop1: mtu:9000 1a2b3c4d5e6f1a2b3c4d5e7f0800 flow hash: 0x00000000
00000000: 1a2b3c4d5e6f1a2b3c4d5e7f0800450000a800000000fd32a84d0a6500070a65
00000020: 0006000003e9000000870573ff85554266537fd108913fe1aba4e3fc
00:04:26:418485: loop1-output
loop1
IP4: 1a:2b:3c:4d:5e:7f -> 1a:2b:3c:4d:5e:6f
IPSEC_ESP: 10.101.0.7 -> 10.101.0.6
tos 0x00, ttl 253, length 168, checksum 0xa84d
fragment id 0x0000
00:04:26:418488: l2-input
l2-input: sw_if_index 5 dst 1a:2b:3c:4d:5e:6f src 1a:2b:3c:4d:5e:7f
00:04:26:418489: l2-fwd
l2-fwd: sw_if_index 5 dst 1a:2b:3c:4d:5e:6f src 1a:2b:3c:4d:5e:7f bd_index 2 result [0x1020000000006, 6] none
00:04:26:418491: l2-output
l2-output: sw_if_index 6 dst 1a:2b:3c:4d:5e:6f src 1a:2b:3c:4d:5e:7f data 08 00 45 00 00 a8 00 00 00 00 fd 32
00:04:26:418493: vxlan4-encap
VXLAN encap to vxlan_tunnel0 vni 13
00:04:26:418499: ip4-rewrite
tx_sw_if_index 1 dpo-idx 18 : ipv4 via 192.168.31.47 GigabitEthernet0/3/0: mtu:9000 08002768d11e0800275a18a50800 flow hash: 0x00000001
00000000: 08002768d11e0800275a18a50800450000da00000000fd11fd46c0a81f4cc0a8
00000020: 1f2f12b512b500c600000800000000000d001a2b3c4d5e6f1a2b3c4d
00:04:26:418500: GigabitEthernet0/3/0-output
GigabitEthernet0/3/0
IP4: 08:00:27:5a:18:a5 -> 08:00:27:68:d1:1e
UDP: 192.168.31.76 -> 192.168.31.47
tos 0x00, ttl 253, length 218, checksum 0xfd46
fragment id 0x0000
UDP: 4789 -> 4789
length 198, checksum 0x0000
00:04:26:418502: GigabitEthernet0/3/0-tx
GigabitEthernet0/3/0 tx queue 0
buffer 0x1c332: current data -50, length 232, free-list 0, clone-count 0, totlen-nifb 0, trace 0x0
PKT MBUF: port 65535, nb_segs 1, pkt_len 232
buf_len 2176, data_len 232, ol_flags 0x0, data_off 78, phys_addr 0x8e70cd00
packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x0 fdir.hi 0x0 fdir.lo 0x0
IP4: 08:00:27:5a:18:a5 -> 08:00:27:68:d1:1e
UDP: 192.168.31.76 -> 192.168.31.47
tos 0x00, ttl 253, length 218, checksum 0xfd46
fragment id 0x0000
UDP: 4789 -> 4789
length 198, checksum 0x0000
Decap trace
00:04:26:419224: dpdk-input
GigabitEthernet0/3/0 rx queue 0
buffer 0x1afa: current data 0, length 232, free-list 0, clone-count 0, totlen-nifb 0, trace 0x1
ext-hdr-valid
l4-cksum-computed l4-cksum-correct
PKT MBUF: port 0, nb_segs 1, pkt_len 232
buf_len 2176, data_len 232, ol_flags 0x0, data_off 128, phys_addr 0x8dc6bf00
packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x0 fdir.hi 0x0 fdir.lo 0x0
IP4: 08:00:27:68:d1:1e -> 08:00:27:5a:18:a5
UDP: 192.168.31.47 -> 192.168.31.76
tos 0x00, ttl 253, length 218, checksum 0xfd46
fragment id 0x0000
UDP: 4789 -> 4789
length 198, checksum 0x0000
00:04:26:419299: ethernet-input
frame: flags 0x3, hw-if-index 1, sw-if-index 1
IP4: 08:00:27:68:d1:1e -> 08:00:27:5a:18:a5
00:04:26:419313: ip4-input-no-checksum
UDP: 192.168.31.47 -> 192.168.31.76
tos 0x00, ttl 253, length 218, checksum 0xfd46
fragment id 0x0000
UDP: 4789 -> 4789
length 198, checksum 0x0000
00:04:26:419320: ip4-lookup
fib 0 dpo-idx 5 flow hash: 0x00000000
UDP: 192.168.31.47 -> 192.168.31.76
tos 0x00, ttl 253, length 218, checksum 0xfd46
fragment id 0x0000
UDP: 4789 -> 4789
length 198, checksum 0x0000
00:04:26:419328: ip4-local
UDP: 192.168.31.47 -> 192.168.31.76
tos 0x00, ttl 253, length 218, checksum 0xfd46
fragment id 0x0000
UDP: 4789 -> 4789
length 198, checksum 0x0000
00:04:26:419334: ip4-udp-lookup
UDP: src-port 4789 dst-port 4789
00:04:26:419345: vxlan4-input
VXLAN decap from vxlan_tunnel0 vni 13 next 1 error 0
00:04:26:419358: l2-input
l2-input: sw_if_index 6 dst 1a:2b:3c:4d:5e:7f src 1a:2b:3c:4d:5e:6f
00:04:26:419365: l2-learn
l2-learn: sw_if_index 6 dst 1a:2b:3c:4d:5e:7f src 1a:2b:3c:4d:5e:6f bd_index 2
00:04:26:419375: l2-fwd
l2-fwd: sw_if_index 6 dst 1a:2b:3c:4d:5e:7f src 1a:2b:3c:4d:5e:6f bd_index 2 result [0x700000005, 5] static age-not bvi
00:04:26:419381: ip4-input
IPSEC_ESP: 10.101.0.6 -> 10.101.0.7
tos 0x00, ttl 253, length 168, checksum 0xa84d
fragment id 0x0000
00:04:26:419385: ip4-lookup
fib 0 dpo-idx 8 flow hash: 0x00000000
IPSEC_ESP: 10.101.0.6 -> 10.101.0.7
tos 0x00, ttl 253, length 168, checksum 0xa84d
fragment id 0x0000
00:04:26:419387: ip4-local
IPSEC_ESP: 10.101.0.6 -> 10.101.0.7
tos 0x00, ttl 253, length 168, checksum 0xa84d
fragment id 0x0000
00:04:26:419390: ipsec-if-input
IPSec: spi 1000 seq 93
00:04:26:419399: esp4-decrypt
esp: crypto aes-cbc-128 integrity sha1-96
00:04:26:419421: ipsec-gre-input
GRE: tunnel -1 len 122 src 10.101.0.6 dst 10.101.0.7
00:04:26:419427: l2-input
l2-input: sw_if_index 4 dst 1a:ab:3c:4d:5e:7f src 1a:ab:3c:4d:5e:6f
00:04:26:419429: l2-learn
l2-learn: sw_if_index 4 dst 1a:ab:3c:4d:5e:7f src 1a:ab:3c:4d:5e:6f bd_index 1
00:04:26:419435: l2-fwd
l2-fwd: sw_if_index 4 dst 1a:ab:3c:4d:5e:7f src 1a:ab:3c:4d:5e:6f bd_index 1 result [0x700000003, 3] static age-not bvi
00:04:26:419438: ip4-input
ICMP: 10.10.10.2 -> 20.20.20.2
tos 0x00, ttl 63, length 84, checksum 0x82f4
fragment id 0xbc93
ICMP echo_reply checksum 0x36a5
00:04:26:419441: ip4-lookup
fib 0 dpo-idx 26 flow hash: 0x00000000
ICMP: 10.10.10.2 -> 20.20.20.2
tos 0x00, ttl 63, length 84, checksum 0x82f4
fragment id 0xbc93
ICMP echo_reply checksum 0x36a5
00:04:26:419445: ip4-rewrite
tx_sw_if_index 2 dpo-idx 26 : ipv4 via 20.20.20.2 GigabitEthernet0/8/0: mtu:9000 0800275467a20800278833fd0800 flow hash: 0x00000000
00000000: 0800275467a20800278833fd080045000054bc9300003e0183f40a0a0a021414
00000020: 1402000036a555ec004d8f3afb5d0000000025b60400000000001011
00:04:26:419449: GigabitEthernet0/8/0-output
GigabitEthernet0/8/0
IP4: 08:00:27:88:33:fd -> 08:00:27:54:67:a2
ICMP: 10.10.10.2 -> 20.20.20.2
tos 0x00, ttl 62, length 84, checksum 0x83f4
fragment id 0xbc93
ICMP echo_reply checksum 0x36a5
00:04:26:419455: GigabitEthernet0/8/0-tx
GigabitEthernet0/8/0 tx queue 0
buffer 0x1c359: current data 38, length 98, free-list 0, clone-count 0, totlen-nifb 0, trace 0x1
PKT MBUF: port 65535, nb_segs 1, pkt_len 98
buf_len 2176, data_len 98, ol_flags 0x0, data_off 166, phys_addr 0x8e70d6c0
packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x0 fdir.hi 0x0 fdir.lo 0x0
IP4: 08:00:27:88:33:fd -> 08:00:27:54:67:a2
ICMP: 10.10.10.2 -> 20.20.20.2
tos 0x00, ttl 62, length 84, checksum 0x83f4
fragment id 0xbc93
ICMP echo_reply checksum 0x36a5