Now its time to start work on the data plane and we start by preparing my hosts and network. Preparing the hosts consists of deploying additional vibs to the ESXi hosts, configuring a VXLAN Tunnel Endpoint (VTEP) kernel interface on each host, configuring a range of segment IDs for new VXLANs and finally to create a Transport Zone.
For reference my lab looks like the following nested in Workstation 12
1 x ESXi 6.0U1 (Management Cluster)
2 x ESXi 6.0U1 (Compute Cluster)
1 x vCenter 6.0
My Workstation networking looks like this
- VMnet0 | Bridged | ESXi Managment, vCenter Managment
- VMnet1 | Host-only | vMotion
- VMnet2 | Host-only | NFS
- VMnet3 | Host-only | VSAN
- VMnet10 | Host-only | VXLAN Transport
I have a vDS with 2 uplinks - Uplink 1 assigned to VMnet10 for VXLAN transport network and Uplink 2 assigned to VMnet0 for management and internet access. When you create your VTEP kernel ports you need to specify a vDS - once created I make sure the uplink port is using Uplink 1 my VMnet10 assigned for VXLAN. VXLAN and VTEP will be described further down. As I am using Workstation this is my way to assign a specific VMnet interface. In the real world you can use VLAN tags. I am using a white box PC with a standard Virgin Media router so I am limited but I wanted so see how far I can go with just using Workstation and nested NSX.
The first part is to install the necessary vib file onto the host, this can be done automatically using the NSX manager or it can be done using Update Manager. I have a post on how to do this here. Also if you are running vSphere 6 and have changed the default certificates you may see a failure in the automatic deployment. See my post here
Open the Web Client and browse to Network & Security - Installation - Host Preperation
Select the cluster you want and the options that will appear next to Not Ready and select to Install. This will install the vibs for vsip and vxlan. Again see my previous post above to install manually and if you have changed the default vCenter certs
Once the vibs have been deployed, you must now configure the VXLAN interfaces. VXLAN is a L2 over L3 encapsulation technology, this encapsulation allows for layer 2 traffic to stretch across the network using VXLAN endpoints (VTEP), our ESXi hosts.
Select Not Configured. Add the relevent vDS and VLAN. MTU must be 1600 for VXLAN encapsulation. Make sure you configure any physical interface for MTU 1600 that will be used for VXLAN
You need to assign an IP address to each interface on the host by using either DHCP of an IP Pool. I choose to use an IP Pool as below
The Installation Status should now report a nice green tick with the correct version. Expand each cluster and you will see the kernel address along with the MTU setting and VTEP number.
You will now see a port group on the selected vDS that have an interface for each host connected to it. As described above I make sure it is using Uplink 1 for my lab
Make sure you can ping the VTEP interfaces from each host, an excellent troubleshooting guide can be found at this blog
To test a VTEP interface along with the correct MTU setting, SSH onto one of the confgirued hosts and run the below - replace the IP address and kernel interface for your envionment
ping ++netstack=vlan -d -s 1572 -I vmk3 192.168.10.13
If that fails try a standard ping
ping ++netstack=vlan -I vmk3 192.168.10.13
If the first one fails but the standard ping works check MTU 1600 has been enabled on the physical interface. My setup for instance I cant change the MTU as my router is a bog standard Virgin Media router so the first test fails.
Now we must create a Segment ID pool that is unique to your NSX Manager. New VXLANs created (Logical Switches) will be assigned one of these IDs. If you are planning cross-VC connectivity keep these numbers unique.
Open the Web Client browse to Network & Security - Installation - Logical Network Preparation - Segment ID. Select to add. Choose a range of numbers between 5000-16777215. This will be the amount of VXLAN segments (Logical Switches) you can create
Now create a Transport Zone, open Network & Security - Installation - Logical Network Preparation - Transport Zones. A Transport Zone allows you to select a collection of ESXi hosts, which can be in different clusters, that can communicate with each other across the physical network. This happens over the configured VTEP interfaces. A Transport Zone is required for Logical Switches.
Choose which clusters will participate in the Transport Zone and choose the replication type. I choose Unicast - for more information on replication type see NSX Reference Design