This is the first VXLAN lesson, and we’ll work with the Cisco Nexus (NX-OS) 9000v switches.
In this lesson, we’ll use static ingress replication, which means manually configuring the remote VXLAN Tunnel Endpoint (VTEP)s on each VTEP. Each VTEP replicates incoming broadcast, unknown-unicast, and multicast (BUM) traffic to all configured VTEPs. It’s the most simple method to configure VXLAN, because you don’t have to configure multicast or BGP EVPN. It uses a flood and learn mechanism on the data plane.
Static ingress replication is suitable for smaller networks. It’s easy to configure, and you don’t have a complex control plane like when you use VXLAN with multicast or MP-BGP EVPN. You also have greater control because you manually configure all VTEPs.
The biggest issues with static ingress replication are related to scalability. You need to manually configure all VTEPs so there is administrative overhead. When you remove a VTEP, it has to be configured on all VTEPs. Also, BUM traffic is replicated to all other VTEPs.
This is a great way to get started with VXLAN and familiarize yourself with the different configurations and show commands. We’ll add more complexity throughout the series of VXLAN lessons.
Configuration
Let’s get started. This is the topology we’ll use:
We have two switches connected to each other with a L3 link. Our two servers are connected to L2 interfaces and use IP addresses in the same subnet. They won’t be able to communicate with each other until we configure VXLAN.
For this topology, I’m using two device types:
Usually, I like to use routers as hosts, but for these VXLAN labs, I decided to use Ubuntu containers. These hosts only need to send some ICMP traffic between each other, and each Ubuntu container only requires a couple of MB of RAM.
Cisco Nexus 9000v switches require 8 GB of RAM each. This is a small topology, but with two spine switches, four leaf switches, and many hosts, the memory requirements add up quickly. This lab requires about ~17GB of RAM to virtualize.
Having said that, let’s start with the configuration.
Underlay Network
First, we have to set up the underlay network. Each leaf switch requires a loopback interface with an IP address that we’ll use for the VTEP addresses. We’ll have to configure some IP addresses, and we’ll need an IGP to advertise the loopback interfaces.
Leaf Switches
We’ll start with the interfaces:
LEAF1(config)# interface Ethernet 1/1
LEAF1(config-if)# no switchport
LEAF1(config-if)# mac-address 0050.c253.1001
LEAF1(config-if)# ip address 192.168.12.1/24
LEAF1(config)# interface Loopback 0
LEAF1(config-if)# ip address 1.1.1.1/32
LEAF2(config)# interface Ethernet 1/1
LEAF2(config-if)# no switchport
LEAF2(config-if)# mac-address 0050.c253.2001
LEAF2(config-if)# ip address 192.168.12.2/24
LEAF2(config)# interface Loopback 0
LEAF2(config-if)# ip address 2.2.2.2/32
I’m using static MAC addresses so that it’s easier to recognize our devices in a packet capture. It also makes it easier to lab this in the future because otherwise, the switches could use random MAC addresses each time we boot the topology.
VTEPs need loopback interfaces, so we need an IGP to advertise them. I’ll keep it simple, so we’ll use OSPF:
LEAF1 & LEAF2
(config)# feature ospf
(config)# router ospf 1
(config)# interface Ethernet 1/1
(config-if)# ip ospf network point-to-point
(config-if)# ip router ospf 1 area 0.0.0.0
(config)# interface Loopback 0
(config-if)# ip router ospf 1 area 0.0.0.0
That’s all we need.
There are different options for the underlay network. For now, we’ll keep it simple and stick to something you know: an IP address on each interface and OSPF to advertise those IP addresses. We’ll cover different underlay network options in another lesson.
Hosts
I’ll also configure a static MAC address and IP address on our hosts:
root@S1:/# ip link set dev eth1 address 00:50:c2:53:30:01
root@S1:/# ip addr add 172.16.12.1/24 dev eth1
root@S2:/# ip link set dev eth1 address 00:50:c2:53:40:01
root@S2:/# ip addr add 172.16.12.2/24 dev eth1
This completes the configuration of the underlay network.
Overlay
For the overlay network, we need to configure the following:
- VXLAN Network Identifier (VNI)
- Network Virtual Interface (NVE)
Let me walk you through this.
VNI
We must configure our VNI and map a VLAN to a new VNI. Before we can do this, we have to enable the VN-Segment feature:
LEAF1#(config)# feature vn-segment-vlan-based
Now, we can use the vn-segment
command:
LEAF1(config)# vlan 10
LEAF1(config-vlan)# vn-segment 10010
This assigns VLAN 10 to VNI 10010. You can pick whatever values you like. The interfaces on the leaf switches that connect to the hosts are normal access interfaces and are assigned to VLAN 10:
LEAF1(config)# interface Ethernet 1/2
LEAF1(config-if)# switchport access vlan 10
We’ll configure the same thing on LEAF2:
LEAF2#(config)# feature vn-segment-vlan-based
LEAF2(config)# vlan 10
LEAF2(config-vlan)# vn-segment 10010
LEAF2(config)# interface Ethernet 1/2
LEAF2(config-if)# switchport access vlan 10
That’s all we need to configure for the VNI.
NVE Interface
Before we can create the NVE interface, we need to enable the NV overlay feature:
LEAF1(config)# feature nv overlay
Now, we can create an NVE interface:
LEAF1(config)# interface nve 1
LEAF1(config-if-nve)# no shutdown
LEAF1(config-if-nve)# source-interface Loopback 0
LEAF1(config-if-nve)# member vni 10010
LEAF1(config-if-nve-vni)# ingress-replication protocol static
LEAF1(config-if-nve-vni-ingr-rep)# peer-ip 2.2.2.2
The NVE interface uses the loopback 0 interface as the source. We configure VNI 10010 to use static ingress replication, and 2.2.2.2 (LEAF2) is a peer.
We’ll configure the same thing on LEAF2:
LEAF2(config)# feature nv overlay
LEAF2(config)# interface nve 1
LEAF2(config-if-nve)# no shutdown
LEAF2(config-if-nve)# source-interface Loopback 0
LEAF2(config-if-nve)# member vni 10010
LEAF2(config-if-nve-vni)# ingress-replication protocol static
LEAF2(config-if-nve-vni-ingr-rep)# peer-ip 1.1.1.1
This completes our overlay network configuration.
Verification
Everything is in place. Let’s verify our work.
Underlay Network
Let’s make sure the underlay network is working. We have an OSPF neighbor adjacency:
LEAF1# show ip ospf neighbors
OSPF Process ID 1 VRF default
Total number of neighbors: 1
Neighbor ID Pri State Up Time Address Interface
2.2.2.2 1 FULL/ - 00:00:05 192.168.12.2 Eth1/1
And we can ping between the two loopback interfaces:
LEAF1# ping 2.2.2.2 source 1.1.1.1
PING 2.2.2.2 (2.2.2.2) from 1.1.1.1: 56 data bytes
64 bytes from 2.2.2.2: icmp_seq=0 ttl=254 time=2.319 ms
64 bytes from 2.2.2.2: icmp_seq=1 ttl=254 time=1.456 ms
64 bytes from 2.2.2.2: icmp_seq=2 ttl=254 time=1.312 ms
64 bytes from 2.2.2.2: icmp_seq=3 ttl=254 time=1.481 ms
64 bytes from 2.2.2.2: icmp_seq=4 ttl=254 time=1.373 ms
That’s all we need to know.
Overlay Network
For the overlay network, there are a couple of things to check. First of all, let’s check the VLAN to VNI mapping:
LEAF1# show vxlan
Vlan VN-Segment
==== ==========
10 10010
Our VLAN is mapped to a VNI. The other thing to check is the NVE interface. There are a couple of show commands that give similar outputs. Let’s check if our two switches see each other as peers:
LEAF1# show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- -------------------------------------- ----- --------- -------- -----------------
nve1 2.2.2.2 Up DP 00:10:12 n/a
The state is “up” so this is looking good. The learn type shows “DP,” which means data plane. This means we use data plane learning (instead of the control plane). We can take a closer look with the detail
parameter:
LEAF1# show nve peers detail
Details of nve Peers:
----------------------------------------
Peer-Ip: 2.2.2.2
NVE Interface : nve1
Peer State : Up
Peer Uptime : 00:10:33
Router-Mac : n/a
Peer First VNI : 10010
Time since Create : 00:10:33
Configured VNIs : 10010
Provision State : peer-add-complete
Learnt CP VNIs : 10010
vni assignment mode : SYMMETRIC
Peer Location : N/A
This output also shows the uptime, first VNI, etc. Here’s one more command that gives a similar output:
LEAF1# show nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
SU - Suppress Unknown Unicast
Xconn - Crossconnect
MS-IR - Multisite Ingress Replication
HYB - Hybrid IRB mode
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 10010 UnicastStatic Up DP L2 [10]
This tells us that the NVE1 interface uses static ingress replication for VNI 10010, that the state is up, and that we use data plane learning.
We can also check the NVE interface:
LEAF1# show interface nve 1
nve1 is up
admin state is up, Hardware: NVE
MTU 9216 bytes
Encapsulation VXLAN
Auto-mdix is turned off
RX
ucast: 0 pkts, 0 bytes - mcast: 0 pkts, 0 bytes
TX
ucast: 1 pkts, 120 bytes - mcast: 0 pkts, 0 bytes
The NVE interface is up, and the encapsulation is VXLAN. You can also see the number of packets sent and received. Here is a similar command:
LEAF1# show nve interface
Interface: nve1, State: Up, encapsulation: VXLAN
VPC Capability: VPC-VIP-Only [not-notified]
Local Router MAC: 0cc6.9700.1b08
Host Learning Mode: Data-Plane
Source-Interface: loopback0 (primary: 1.1.1.1, secondary: 0.0.0.0)
This gives us a bit more information. You can see even more if you add the detail
parameter:
LEAF1# show nve interface nve 1 detail
Interface: nve1, State: Up, encapsulation: VXLAN
VPC Capability: VPC-VIP-Only [not-notified]
Local Router MAC: 0cc6.9700.1b08
Host Learning Mode: Data-Plane
Source-Interface: loopback0 (primary: 1.1.1.1, secondary: 0.0.0.0)
Source Interface State: Up
Virtual RMAC Advertisement: No
NVE Flags:
Interface Handle: 0x49000001
Source Interface hold-down-time: 180
Source Interface hold-up-time: 30
Remaining hold-down time: 0 seconds
Virtual Router MAC: N/A
Interface state: nve-intf-add-complete
Fabric convergence time: 135 seconds
Fabric convergence time left: 0 seconds
Last but not least, let’s check the MAC address table of one of the leaf switches:
LEAF1(config)# show mac address-table dynamic
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
(NA)- Not Applicable
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
It’s currently empty because our hosts haven’t sent anything yet. Let’s send a ping from S1 to S2:
root@S1:/# ping 172.16.12.2 -c 5
PING 172.16.12.2 (172.16.12.2) 56(84) bytes of data.
64 bytes from 172.16.12.2: icmp_seq=1 ttl=64 time=6.51 ms
64 bytes from 172.16.12.2: icmp_seq=2 ttl=64 time=3.49 ms
64 bytes from 172.16.12.2: icmp_seq=3 ttl=64 time=4.94 ms
64 bytes from 172.16.12.2: icmp_seq=4 ttl=64 time=4.08 ms
64 bytes from 172.16.12.2: icmp_seq=5 ttl=64 time=3.41 ms
--- 172.16.12.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 3.414/4.486/6.506/1.148 ms
Our ping works. That tells us that VXLAN encapsulation works because these two hosts are in the same subnet, and we have an L3 link between the leaf switches.
Packet Capture
If you want to see VXLAN packets in action, we’ll have to capture some traffic. Let’s look at a Wireshark capture of this traffic. The first frame is an ARP:
Frame 7: 110 bytes on wire (880 bits), 110 bytes captured (880 bits) on interface eth1, id 0
Ethernet II, Src: NVE_53:10:01 (00:50:c2:53:10:01), Dst: NVE_53:20:01 (00:50:c2:53:20:01)
Destination: NVE_53:20:01 (00:50:c2:53:20:01)
Source: NVE_53:10:01 (00:50:c2:53:10:01)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 1.1.1.1, Dst: 2.2.2.2
User Datagram Protocol, Src Port: 64901, Dst Port: 4789
Virtual eXtensible Local Area Network
Flags: 0x0800, VXLAN Network ID (VNI)
0... .... .... .... = GBP Extension: Not defined
.... 1... .... .... = VXLAN Network ID (VNI): True
.... .... .0.. .... = Don't Learn: False
.... .... .... 0... = Policy Applied: False
.000 .000 0.00 .000 = Reserved(R): 0x0000
Group Policy ID: 0
VXLAN Network Identifier (VNI): 10010
Reserved: 0
Ethernet II, Src: NVE_53:30:01 (00:50:c2:53:30:01), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
Destination: Broadcast (ff:ff:ff:ff:ff:ff)
Source: NVE_53:30:01 (00:50:c2:53:30:01)
Type: ARP (0x0806)
Trailer: 000000000000000000000000000000000000
Address Resolution Protocol (request)
Hardware type: Ethernet (1)
Protocol type: IPv4 (0x0800)
Hardware size: 6
Protocol size: 4
Opcode: request (1)
Sender MAC address: NVE_53:30:01 (00:50:c2:53:30:01)
Sender IP address: 172.16.12.1
Target MAC address: Xerox_00:00:00 (00:00:00:00:00:00)
Target IP address: 172.16.12.2
Above, you see a VXLAN encapsulated packet. Let me explain:
- We have an IP packet from LEAF1 (1.1.1.1) to LEAF2 (2.2.2.2).
- The UDP source port is random. This is helpful when you use ECMP because it allows you to do load balancing.
- The UDP destination port is 4789 for VXLAN.
- The VNI is 10010.
- The inner packet is an ARP request from S1 (00:50:c2:53:30:01) with destination broadcast.
This is an example of BUM traffic, which is replicated and sent to all configured VTEPs.
Here is the ARP reply from S2 to S1:
Frame 8: 110 bytes on wire (880 bits), 110 bytes captured (880 bits) on interface eth1, id 0
Ethernet II, Src: NVE_53:20:01 (00:50:c2:53:20:01), Dst: NVE_53:10:01 (00:50:c2:53:10:01)
Destination: NVE_53:10:01 (00:50:c2:53:10:01)
Source: NVE_53:20:01 (00:50:c2:53:20:01)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 2.2.2.2, Dst: 1.1.1.1
User Datagram Protocol, Src Port: 52742, Dst Port: 4789
Virtual eXtensible Local Area Network
Flags: 0x0800, VXLAN Network ID (VNI)
0... .... .... .... = GBP Extension: Not defined
.... 1... .... .... = VXLAN Network ID (VNI): True
.... .... .0.. .... = Don't Learn: False
.... .... .... 0... = Policy Applied: False
.000 .000 0.00 .000 = Reserved(R): 0x0000
Group Policy ID: 0
VXLAN Network Identifier (VNI): 10010
Reserved: 0
Ethernet II, Src: NVE_53:40:01 (00:50:c2:53:40:01), Dst: NVE_53:30:01 (00:50:c2:53:30:01)
Destination: NVE_53:30:01 (00:50:c2:53:30:01)
Source: NVE_53:40:01 (00:50:c2:53:40:01)
Type: ARP (0x0806)
Trailer: 000000000000000000000000000000000000
Address Resolution Protocol (reply)
Hardware type: Ethernet (1)
Protocol type: IPv4 (0x0800)
Hardware size: 6
Protocol size: 4
Opcode: reply (2)
Sender MAC address: NVE_53:40:01 (00:50:c2:53:40:01)
Sender IP address: 172.16.12.2
Target MAC address: NVE_53:30:01 (00:50:c2:53:30:01)
Target IP address: 172.16.12.1
Above, you can see that this packet is transmitted from LEAF2 to LEAF1. The inner packet contains the ARP reply from S2 to S1.
Here is the ICMP echo request from S1 to S2:
Frame 9: 148 bytes on wire (1184 bits), 148 bytes captured (1184 bits) on interface eth1, id 0
Ethernet II, Src: NVE_53:10:01 (00:50:c2:53:10:01), Dst: NVE_53:20:01 (00:50:c2:53:20:01)
Destination: NVE_53:20:01 (00:50:c2:53:20:01)
Source: NVE_53:10:01 (00:50:c2:53:10:01)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 1.1.1.1, Dst: 2.2.2.2
User Datagram Protocol, Src Port: 57060, Dst Port: 4789
Virtual eXtensible Local Area Network
Flags: 0x0800, VXLAN Network ID (VNI)
0... .... .... .... = GBP Extension: Not defined
.... 1... .... .... = VXLAN Network ID (VNI): True
.... .... .0.. .... = Don't Learn: False
.... .... .... 0... = Policy Applied: False
.000 .000 0.00 .000 = Reserved(R): 0x0000
Group Policy ID: 0
VXLAN Network Identifier (VNI): 10010
Reserved: 0
Ethernet II, Src: NVE_53:30:01 (00:50:c2:53:30:01), Dst: NVE_53:40:01 (00:50:c2:53:40:01)
Destination: NVE_53:40:01 (00:50:c2:53:40:01)
Source: NVE_53:30:01 (00:50:c2:53:30:01)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 172.16.12.1, Dst: 172.16.12.2
Internet Control Message Protocol
And here is the ICMP echo reply from S2 to S1:
Frame 10: 148 bytes on wire (1184 bits), 148 bytes captured (1184 bits) on interface eth1, id 0
Ethernet II, Src: NVE_53:20:01 (00:50:c2:53:20:01), Dst: NVE_53:10:01 (00:50:c2:53:10:01)
Destination: NVE_53:10:01 (00:50:c2:53:10:01)
Source: NVE_53:20:01 (00:50:c2:53:20:01)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 2.2.2.2, Dst: 1.1.1.1
User Datagram Protocol, Src Port: 53590, Dst Port: 4789
Virtual eXtensible Local Area Network
Flags: 0x0800, VXLAN Network ID (VNI)
0... .... .... .... = GBP Extension: Not defined
.... 1... .... .... = VXLAN Network ID (VNI): True
.... .... .0.. .... = Don't Learn: False
.... .... .... 0... = Policy Applied: False
.000 .000 0.00 .000 = Reserved(R): 0x0000
Group Policy ID: 0
VXLAN Network Identifier (VNI): 10010
Reserved: 0
Ethernet II, Src: NVE_53:40:01 (00:50:c2:53:40:01), Dst: NVE_53:30:01 (00:50:c2:53:30:01)
Destination: NVE_53:30:01 (00:50:c2:53:30:01)
Source: NVE_53:40:01 (00:50:c2:53:40:01)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 172.16.12.2, Dst: 172.16.12.1
Internet Control Message Protocol
This shows nicely how the switches deal with BUM traffic and how unicast traffic is forwarded. If you want to take a look at these packets yourself:
VXLAN Static Ingress Replication Two Leafs Two Hosts
Let’s take another look at the MAC address tables of our leaf switches. Here’s LEAF1:
LEAF1# show mac address-table dynamic
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
(NA)- Not Applicable
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 10 0050.c253.3001 dynamic NA F F Eth1/2
* 10 0050.c253.4001 dynamic NA F F nve1(2.2.2.2)
Above, you can see it learned the MAC address of S2 on its NVE1 interface. Here’s LEAF2:
LEAF2# show mac address-table dynamic
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
(NA)- Not Applicable
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 10 0050.c253.3001 dynamic NA F F nve1(1.1.1.1)
* 10 0050.c253.4001 dynamic NA F F Eth1/2
Here, we see the MAC address of S1 on the NVE1 interface of LEAF2. That’s all there is to it.
Configurations
Want to take a look for yourself? Here, you will find the final configuration of each device.
LEAF1
hostname LEAF1
feature ospf
feature vn-segment-vlan-based
feature nv overlay
vlan 10
vn-segment 10010
interface nve1
no shutdown
source-interface loopback0
member vni 10010
ingress-replication protocol static
peer-ip 2.2.2.2
interface Ethernet1/1
no switchport
mac-address 0050.c253.1001
ip address 192.168.12.1/24
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
no shutdown
interface Ethernet1/2
switchport access vlan 10
interface loopback0
ip address 1.1.1.1/32
ip router ospf 1 area 0.0.0.0
router ospf 1
LEAF2
hostname LEAF2
feature ospf
feature vn-segment-vlan-based
feature nv overlay
vlan 10
vn-segment 10010
interface nve1
no shutdown
source-interface loopback0
member vni 10010
ingress-replication protocol static
peer-ip 1.1.1.1
interface Ethernet1/1
no switchport
mac-address 0050.c253.2001
ip address 192.168.12.2/24
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
no shutdown
interface Ethernet1/2
switchport access vlan 10
interface loopback0
ip address 2.2.2.2/32
ip router ospf 1 area 0.0.0.0
router ospf 1
S1
ip link set dev eth1 address 00:50:c2:53:30:01
ip addr add 172.16.12.1/24 dev eth1
S2
ip link set dev eth1 address 00:50:c2:53:40:01
ip addr add 172.16.12.2/24 dev eth1
Conclusion
You have now learned how to configure VXLAN static ingress replication on Cisco NX-OS 9000v switches:
- The underlay network uses IP addresses on all L3 interfaces.
- OSPF advertises the loopback interfaces.
- The VTEPs are configured statically.
- We sent some traffic between the two servers.
- We used some show commands to verify our work.
- We captured some packets to see what VXLAN encapsulation looks like.
Configuring remote VTEPs manually works in a small environment like this, but with many leaf switches, scalability is an issue. In other lessons, we’ll look at multicast and MP-BGP EVPN L2 VNI, which are easier to scale.
I hope you enjoyed this lesson. If you have any questions, feel free to leave a comment!