Tag: NVGRE

Updated: Hybrid Cloud with NVGRE (WSSC 2012 R2) Whitepaper

Hybrid Cloud with NVGRE (WSSC 2012 R2) Whitepaper is updated here to reflect R2 RTM. Hope you will like it. Authors are Kristian Nese and Flemming Riis. Technical reviewers are Daniel Neumann and me.

Hybrid Cloud with NVGRE Whitepaper

I would like to introduce a new whitepaper titled “Hybrid Cloud with NVGRE” that is authored by my mates Kristian Nese (MVP) and Flemming Riis (MVP). Daniel Neumann and me are technical reviewers and helpers with some knowledge. Flemming also provided us with the hardware for the lab and this wouldn’t be possible without him.

Please note that this white paper is in its first release.
Once we reach RTM, additional content will be added.

  • Site-2-Site VPN
  • Self-service and tenants with Windows Azure Pack

You can download the whitepaper from here.

You can leave any comments or questions on Kristian’s blogpost.

TechEd Europe 2013 Day 2

Today for me was all about Networking and NVGRE Gateways. Here are some of my highlights from day 2:

  • There are no changes in the NVGRE packet which means WS 2012 and WS 2012 R2 can co exist.
  • You have to have separate hosts/clusters for NVGRE Gateways to reside.
  • Microsoft NVGRE Gateway will have automatic failover.
  • For high availability you will have two gateways in cluster and one will be active the other standby.
  • In the Preview build of VMM 2012 R2 there are some limitations like you can have only one gateway that resides only on one Hyper-v Host
  • I was on the Pete Zerger and Anders Bengtsson session for Orchestrator – very funny session with a lot to learn.
  • Met Mark Russinovich – great guy, very friendly.
  • Azure is also constructed in stamps. Every stamp is controlled by Azure Fabric Controller. On top of the Fabric Controllers there is API and on top of the API is the Azure portal.
  • There will be Service Templates for all System Center components even for VMM.
  • Microsoft Gateways and the hosts where they reside must be Windows Server 2012 R2.
  • In preview the Hyper-V host that will host the gateways is marked in VMM trough PowerShell. In RTM there will be checkbox.
  • Demo was shown of automatic failover of NVGRE Gateways without downtime of the network.
  • The logic for Multitenant dividing of the traffic happens on host level.
  • Each Tenant has its own Public NAT address.
  • GWs can be domain joined or not domain joined but for failover you will need domain joined.

Tomorrow will be my last day at TechEd 2013 as I have to leave early on Friday morning.

Test Lab Guide: Windows Server 2012 R2 Hyper-V Network Virtualization with System Center 2012 R2 VMM

Even though SC 2012 R2 and WS 2012 R2 are not released yet Microsoft published a guide to setup lab for Hyper-V Network Virtualization. There are no screenshot in the guide but you can have a sneak peak preview of the architecture of the new Multi-Tenant NVGRE Gateway. Check it out here.

Network Virtualization (NVGRE) in Windows Server 2012 May Not work if You Do Not Have Update KB2779768 Installed

Two of my colleagues were testing Network Virtualization these days and stumbled on some strange issue. They’ve setup two standalone Hyper-V hosts, installed System Center 2012 Virtual Machine Manager SP1 and added the two hosts to be managed by VMM. They’ve created two networks with network virtualization in VMM. The first network was for customer A the second was for custom B. You can see a demo here how to setup Network Virtualization in VMM. After this 4 virtual machines were installed:

  • 2 for customer A. One (VMA1) located on Host1 and second (VMA2) located on Host 2.
  • 2 for customer B. One (VMB1) located on Host 1 and second (VMB2) located on Host 2.

Then they’ve started to ping VMA2 from VMA1 an vice versa but no replies were received.

It was time for some hard troubleshooting on network level with Wireshark. They’ve started to analyze the packets and compare them also to the RFC of the NVGRE. They were seeing this icmpType3Code10（Destination host administratively prohibited）upon inspection. They’ve searched over Internet and found this presentation Satoshi GOTO who is Microsoft MVP. The presentation is written in Japanese and provides information for our issue and why it was happening.

When Microsoft release Windows Server 2012 RTM Network Virtualization was included but it was based on the initial version of NVGRE RFC. So let’s compare the initial and the latest version of  NVGRE’s RFC:

In the initial version we have the following diagram for GRE Encapsulation Frame Format:

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   Outer Ethernet Header:             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                (Outer) Destination MAC Address                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |(Outer)Destination MAC Address |  (Outer)Source MAC Address    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  (Outer) Source MAC Address                   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Optional Ethertype=C-Tag 802.1Q| Outer VLAN Tag Information    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Ethertype 0x0800        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   Outer IPv4 Header:
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Version|  IHL  |Type of Service|          Total Length         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         Identification        |Flags|      Fragment Offset    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Time to Live | Protocol 0x2F |         Header Checksum       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      (Outer) Source Address                   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  (Outer) Destination Address                  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   GRE Header:
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| |1|0| Reserved0       | Ver |   Protocol Type 0x6558        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Tenant Network ID (TNI)|   Reserved    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   Inner Ethernet Header
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                (Inner) Destination MAC Address                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |(Inner)Destination MAC Address |  (Inner)Source MAC Address    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  (Inner) Source MAC Address                   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Optional Ethertype=C-Tag 802.1Q| PCP |0| VID set to 0          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Ethertype 0x0800        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   Inner IPv4 Header:

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Version|  IHL  |Type of Service|          Total Length         |


Sridharan et al             Informational                      [Page 7]
 
Internet-Draft                  NVGRE                    September 2011


   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         Identification        |Flags|      Fragment Offset    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Time to Live |    Protocol   |         Header Checksum       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Source Address                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Destination Address                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Options                    |    Padding    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      Original IP Payload                      |
   |                                                               |
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

In the latest version we have the following diagram for GRE Encapsulation Frame Format:

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   Outer Ethernet Header:             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                (Outer) Destination MAC Address                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |(Outer)Destination MAC Address |  (Outer)Source MAC Address    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  (Outer) Source MAC Address                   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Optional Ethertype=C-Tag 802.1Q| Outer VLAN Tag Information    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Ethertype 0x0800        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   Outer IPv4 Header:
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Version|  IHL  |Type of Service|          Total Length         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         Identification        |Flags|      Fragment Offset    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Time to Live | Protocol 0x2F |         Header Checksum       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      (Outer) Source Address                   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  (Outer) Destination Address                  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   GRE Header:
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| |1|0| Reserved0       | Ver |   Protocol Type 0x6558        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |               Virtual Subnet ID (VSID)        |   Reserved    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   Inner Ethernet Header
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                (Inner) Destination MAC Address                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |(Inner)Destination MAC Address |  (Inner)Source MAC Address    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  (Inner) Source MAC Address                   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Optional Ethertype=C-Tag 802.1Q| PCP |0| VID set to 0          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Ethertype 0x0800        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   (Continued on the next page)




Sridharan et al             Informational                      [Page 6]
 
Internet-Draft                  NVGRE                     February 2013


   Inner IPv4 Header:
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Version|  IHL  |Type of Service|          Total Length         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         Identification        |Flags|      Fragment Offset    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Time to Live |    Protocol   |         Header Checksum       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Source Address                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Destination Address                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Options                    |    Padding    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      Original IP Payload                      |
   |                                                               |
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

On first sight there are no significant differences. I’ve highlighted with yellow the area where you need to focus your attention. You can see that the only difference is that the term Tenant Network ID (TNI) was replaced with Virtual Subnet ID (VSID). but let’s look closer at the explanation in the RFC.

Initial version:

  • The Key field in the GRE header is used to carry the Tenant Network Identifier. Key field is 32 bits long of which the lower 24 bits are used for TNI. The Key Present (bit 2 in the GRE header) is always set to 1.
  • The upper 8 bits of the Key field are reserved for use by NVGRE endpoints and are not part of the TNI space. NVGRE endpoints MUST set this value to zero.

 

Latest version:

  • Virtual Subnet ID (VSID): The first 24 bits are used for VSID as shown in Figure 1.
  • FlowID: The last 8 bits of the Key field are (optional) FlowID, which can be used to add per-flow entropy within the same VSID, where the entire Key field (32-bit) is used for ECMP purposes by switches or routers in the physical network infrastructure. If a FlowID is not generated, the FlowID field MUST be set to all zero.

 

So if you are reading right in initial version VSID 32 bits were looking like this:

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-

| 8 bits FlowID |                          32 bits VSID                         |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-

and in the latest version we have this:

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-

|                   32 bits VSID                                 | 8 bits FlowID |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-

Format was changed.

Even it was not stated nowhere update KB2779768 was changing this format from the initial version to the format of the latest version. If you see in the description of the update you will find that only these issues are fixed in it:

  • Improves the speed of text input in some Windows Store apps.
  • Resolves an issue that prevents some users who have Windows Display Driver Model (WDDM) 1.0-based devices from using mirror drivers.
  • Resolves an issue that may prevent some games from being updated.
  • Resolves an issue that may cause blips in audio playback when a computer is running in "Connected Standby" mode.
  • Resolves an issue in which Windows Store apps do not obtain data when they connect to Windows Store through a cellular network.

But this update has another web page where are listed which files will be changed after the update is applied. An there you can see these files are changed:

  • Wnv.sys
  • Wnvapi.dll

Both files define Network Virtualization in Windows Server 2012.

In our case Host1 was having update KB2779768 and Host2 didn’t had it. This is like the two hosts are speaking in different languages for Network Virtualization.

You should install KB2779768 on all your Hyper-V hosts if you plan to implement Network Virtualization because every network device that will be NVGRE aware will follow the latest NVGRE format.

This issue is widely spread over Internet but only in Japanese web sites.

All credits for these findings go to my colleagues Yordan Dimov and Kiril Tsvetanov for tracing and resolving this issue and to Satoshi GOTO who detailed it in his presentation.