Openstack (Pike) L2GW setup on Ubuntu 16.04

What is L2GW

Layer 2 Gateway (L2GW) is an Openstack plugin which allows you to extend communication from virtual machines on your Openstack deployment, to physical or virtual machines on a network outside of the Openstack deployment. More specifically, it allows you to map VXLAN's on your Openstack deployment, to VLANS on your data centre network. This mapping function is performed by a VXLAN Tunnel End Point (VTEP) server.

Before we begin...

 The assumption at this point is that you already have a basic operational Openstack Deployment and at least have the following services installed, configured and working:

• Keystone - Identity Service
• Glance - Image Service
• Nova - Compute Service
• Neutron - Networking Service
• Horizon - Dashboard

If you don't have these working yet, follow the respective installation guides found here for your operating system of choice.

Architecture

This deployment is made up of a few key components. An Openstack Controller, Compute node hosting two VM's on a VXLAN (74), a Cumulus VX virtual switch to act as the VTEP and a KVM virtual machine hosted outside of the Openstack environment, directly connected to the VXSwitch on a Vlan (114). All of these are hosted on a single physical server running Ubuntu 16.04.

VXSwitch installation

Sign up at Cumulus Networks for a free trial of their Cumulus VX virtual switch, then download the image that applies to the virtualisation technology you're using.

Spin up a VM with the Cumulus VX image as the disk. Put one interface into the bridge that the Openstack nodes are in, in this case virbr0 and the other interface into the bridge that the External Hosts will connect to, virbr2 in the diagram.

virt-install --connect qemu:///system -n vxswitch --vcpus=1 -r 512 --network=bridge:virbr0,model=virtio --network=bridge:virbr2,model=virtio 
 -f /var/lib/libvirt/images/vxswitch.qcow2 
--vnc --noautoconsole --boot hd

Open the switches console and login with the Username cumulus and Password CumulusLinux!

Add a management ip address to eth0

net add interface eth0 ip address 192.168.122.20/24
net pending
net commit

Reboot the switch and you should be able to connect to it from your management network.

Edit the following file to force the VTEP to start with the switch boots. Modify "START=no" to "START=yes" and save the file

/etc/default/openvswitch-vtep

Now we start the openvswitch-vtep service and then bootstrap the database with some information.

sudo service openvswitch-vtep start 
sudo vtep-bootstrap VXSWITCH 192.168.122.20 192.168.122.20 --no_encryption

The vtep-boostrap command does the following:

• Creates the VTEP OVSDB database schema. This is where the VXLAN to VLAN mappings are stored.
• within that schema a physical switch called "VXSWITCH" is created
• sets the vtep ip address to 192.168.122.20
• starts listening to incoming OVSDB connections on 192.168.122.20

At this point your VXSwitch is set up and we can proceed with the installation of the L2GW Plugin on the Openstack environment.

Openstack L2GW Plugin installation

If you've been working with Openstack, you'll know that sometimes it's hard to find documentation specific to your deployment and when you do find it, it doesn't always work as expected. Or you'll encounter a problem and see that other people have encountered the same problem, but there are no solutions. I encountered a lot of this with my L2GW installation, so I will share some of the common problems people have encountered and what I did to resolve them. I'm hoping the sharing of my journey will help someone else expedite their installation. No point in the greater community coming across the same problems without solving them together and sharing the answers.

First up, install the neutron-l2gateway-agent and python-networking-l2gw packages from the Xenial repositories.

apt install neutron-l2gateway-agent python-networking-l2gw

 Next up, edit your neutron.conf file to add the service plugin "networking_l2gw.services.l2gateway.plugin.L2GatewayPlugin"

vi /etc/neutron/neutron.conf 
...
service_plugins = router,networking_l2gw.services.l2gateway.plugin.L2GatewayPlugin

 Then you need to edit your l2gateway_agent.ini file to add the ip address of your OVSDB. This is the ip address you assigned to the VXSwitch in the previous section.

vi /etc/neutron/l2gateway_agent.ini
...
ovsdb_hosts = 'ovsdb1:192.168.122.20:6640'
# Example: ovsdb_hosts = 'ovsdb1:16.95.16.1:6632,ovsdb2:16.95.16.2:6632'

WARNING! You'll notice the the example provided in the config file, they use port 6632 to connect to the VTEP OVSDB. But if you do a "netstat-a" on the VXSwitch, you'll notice that the switch is listening on port 6640. This tripped me up in the beginning and I wasted a fair amount of time chasing "connection refused messages" all over the Openstack deployment.

 Now we need to stop the Neutron service, update the database with the information in the l2gw_plugin.ini file and then start Neutron again.

service neutron-server stop
neutron-db-manage --config-file /etc/neutron/neutron.conf 
--config-file /etc/neutron/l2gw_plugin.ini  upgrade head
service neutron-server start

The next step in the installation gave me a lot of problems. Most installation instructions say that you need to configure the neutron-server.service file to load the /etc/neutron/l2gw_plugin.ini config file on start. This is supposed to feed Neutron the Service Provider information it needs to load the plugin. However, this doesn't work and you get the following errors in your /etc/var/neutron/neutron-server.log file

INFO neutron.manager	Loading Plugin: networking_l2gw.services.l2gateway.plugin.L2GatewayPlugin
ERROR neutron.services.service_base	No providers specified for 'L2GW' service, exiting

To resolve this you need to explicitly define the service providers in your neutron.conf file.

[service_providers]
service_provider=L2GW:l2gw:networking_l2gw.services.l2gateway.service_drivers.rpc_l2gw.L2gwRpcDriver:default

Once you've done this, you'll see the following in your neutron-server.log

...
INFO neutron.api.extensions Loaded extension: l2-gateway
INFO neutron.api.extensions Loaded extension: l2-gateway-connection
...

 Next, you can start restart the Neutron-Server service and start the Neutron-L2gateway-Agent service

systemctl daemon-reload
service neutron-server restart
service neutron-l2gateway-agent start

At this point, source the saved admin credentials and check to see that the Neutron-L2gateway-Agent is running.

openstack network agent list -c "Agent Type" -c Host -c Alive -c State
+--------------------+------------+-------+-------+
| Agent Type         | Host       | Alive | State |
+--------------------+------------+-------+-------+
| Metadata agent     | controller | :-)   | UP    |
| Open vSwitch agent | compute    | :-)   | UP    |
| L3 agent           | controller | :-)   | UP    |
| L2 Gateway agent   | controller | :-)   | UP    |
| DHCP agent         | controller | :-)   | UP    |
| Open vSwitch agent | controller | :-)   | UP    |
+--------------------+------------+-------+-------+

Now it's just a case of creating a L2 Gateway device and creating the connection between the Openstack tenant network and the external vlan network.

To create the L2 Gateway device, enter into the neutron configuration mode and enter the following command:

neutron

l2-gateway-create --device name="vxswitch",interface_names="swp1" CUMULUS-L2GW

It is important to note what interface on the VXSwitch connects to the host you're trying to connect to. In this instance its switchport "swp1".

 Lastly let's set up the connection between the tenant network and the VLAN network. To do this, you'll need to know the name of the Openstack tenant network that you want to bridge to the outside. In this case we want to bridge the "selfservice3" network.

openstack network list -c Name -c Subnets
+--------------+--------------------------------------+
| Name         | Subnets                              |
+--------------+--------------------------------------+
| external-net | 7942d261-c15b-4aa6-b5e7-e9895f15d070 |
| selfservice3 | a7d5a449-a097-40d5-a85d-011afdb35a83 |
| selfservice2 | a32c53ae-f219-478f-b3ac-73e174228600 |
| selfservice1 | 34bdfd01-c9cf-4096-9f78-23bb6b4fb3a6 |
+--------------+--------------------------------------+

openstack network show selfservice3
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        | nova                                 |
| created_at                | 2018-01-15T17:00:06Z                 |
| description               |                                      |
| dns_domain                | None                                 |
| id                        | 71b65a58-4635-470c-ba1d-66776e666602 |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | None                                 |
| is_vlan_transparent       | None                                 |
| mtu                       | 1450                                 |
| name                      | selfservice3                         |
| port_security_enabled     | True                                 |
| project_id                | f2fd567fb52c4da5ac8b806a2fb8b89d     |
| provider:network_type     | vxlan                                |
| provider:physical_network | None                                 |
| provider:segmentation_id  | 74                                   |
| qos_policy_id             | None                                 |
| revision_number           | 3                                    |
| router:external           | Internal                             |
| segments                  | None                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | a7d5a449-a097-40d5-a85d-011afdb35a83 |
| tags                      |                                      |
| updated_at                | 2018-01-15T17:00:07Z                 |
+---------------------------+--------------------------------------+<br><br>

We also need to know what the Vlan number is on the data centre network that the host is connected to. In this case we're using Vlan 114.

neutron
l2-gateway-connection-create --default-segmentation-id 114 CUMULUS-L2GW selfservice3

Created a new l2_gateway_connection:
+-----------------+--------------------------------------+
| Field           | Value                                |
+-----------------+--------------------------------------+
| id              | 2268f9d9-8617-4a02-9eea-47fa5aa7c482 |
| l2_gateway_id   | 1adf7124-da1b-4265-96bb-1fbfe371e322 |
| network_id      | 71b65a58-4635-470c-ba1d-66776e666602 |
| segmentation_id | 114                                  |
| tenant_id       | 5326fb7f7a844bff931ae9205b20f799     |
+-----------------+--------------------------------------+

If you go back to your VXswitch you will see that a new virtual port has been created called vxln74 along with a bridge called br-vxln74.

brctl show
bridge name	bridge id		STP enabled	interfaces
br-vxln74		8000.829608997b08	no		vxln74

But, there is something missing from the bridge. We need the swp1 to be included in the bridge. But looking a little closer, the port is in a DOWN state.

ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:57:69:0f brd ff:ff:ff:ff:ff:ff
3: swp1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:0d:3f:90 brd ff:ff:ff:ff:ff:ff
4: br-vxln74: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 82:96:08:99:7b:08 brd ff:ff:ff:ff:ff:ff
5: vxln74: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-vxln74 state UNKNOWN mode DEFAULT group default 
    link/ether 82:96:08:99:7b:08 brd ff:ff:ff:ff:ff:ff
6: swp1.114@swp1: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default 
    link/ether 52:54:00:0d:3f:90 brd ff:ff:ff:ff:ff:ff

You'll also notice that a new subinterface has been created for us "swp1.114". This is the vlan interface on swp1.

Let's bring up swp1

sudo ifconfig swp1 up

And now we have a switchport in br-vxln74

brctl show
bridge name	bridge id		STP enabled	interfaces
br-vxln74		8000.5254000d3f90	no		swp1.114
							vxln74

 For completeness sake here is the /etc/network/interfaces config for the external VM host

auto ens6
iface ens6 inet static

auto ens6.114
iface ens6.114 inet static
        address 192.168.3.114
        netmask 255.255.255.0

Testing

So, you're really excited, you hop onto your Openstack VM's and you can ping the external VM from VM1. Then you ping from the external VM to the Openstack VM1. Life has never been better. And then you decide to ping Openstack VM2 from the external VM and it doesn't work. You open the console of Openstack VM2 and you ping the external VM AND IT WORKS!?!?!

Something isn't quite right. But we'll get into that in the next post.

References

There are some really great articles out there explaining how to build an Openstack deployment and the various other components I've used in this article. Here are the links to sources of a lot of my information.

• networkop.co.uk - OpenStack SDN - Interconnecting VMs and Physical Devices With Cumulus VX L2 Gateway
• Red Hat Openstack Networking Guide
• Cumulus VX Documentation