Nuage Networks

Virtualised
Services Platform

Packet Pushers
White Paper
About the Author
Greg Ferro is a Network Engineer/Architect, mostly focussed on Data Centre, Security Infrastructure, and recently Virtualisation. He has
over 20 years in IT, in a wide range of employers working as a freelance consultant including Finance, Service Provid-
ers and Online Companies. He is CCIE#6920 (Emeritus) and has a few ideas about the world, but not enough to really
count. Also a host of the Packet Pushers Podcast and writes on his well known blog at http://etherealmind.com.

Packet Pushers Profile

Packet Pushers is a podcast and community website where real engineers get together to talk about their experiences.
Its deliberately nerdy, passionately technical and led by experienced network engineers. Some of the best and bright-
est technical people share their knowledge and experience talking about the latest topics. We strive to engage across all areas of Networking.

Packet Pushers Perspective

Packet Pushers is focussed on technology and engineers. We provide independent views and opinions. There were very few independent
voices telling their stories and experiences in 2009; Packet Pushers was started to discuss data networking and form a community. Today,
Packet Pushers is a thriving group of independent voices sharing knowledge, experiences and insight into Network Engineering with over
100 contributors, widespread recognition and growing vendor & industry support.

The Packet Pushers Podcast is deliberately technical, nerdy and lighthearted. There is no technical pandering to the audience. We discuss in-
dustry, products, technology, heavy on tech and find humour where we can. The pod-
cast has over 12,000 listeners per show.

About White Papers

Most vendor white papers are aimed at the CIO and other managers, tending to be
heavy on marketing and long on business value. However, a senior engineer will only
consider these aspects once the technology has a viable purpose or use case. Packet
Pushers delivers content that focusses on the technology.

2
N UA G E NE T W OR KS
Today's virtual networking is about multiple logical devices
from a single physical network device, but the future of net-
working is software network devices that are hosted on ge-
neric hypervisors. The Nuage Networks strategy delivers a
Software Defined Networking (SDN) product that controls &
manages the virtual access networks at three tiers: in the
WAN, throughout the data centre, and between data centres.

The Nuage Networks Virtualised Service Platform (VSP) can

be described in three parts. The Virtualised Services Direc-
tory VSD) policy management and analytics application
drives the Virtualised Services Controller (VSC) which con-
figures distributed virtual routing & switching (dVRS) soft-
ware agents in the server hypervisor.

First, we look at how the data and control planes merge into
your existing network with the dVRS flow forwarding agent.
Then, we look at how the Nuage Networks VSP management
plane uses a policy engine, derived from service provider net-
working, that not only manages & configures the dVRS de-
vices but also provides deep visibility for operations & control
of configuration.

3
Introduction The Server as the Network
In networking, the physical Ethernet connection to the server has been Nuage Networks is integrating software networking with the physical
the edge of the network. The growing use of hypervisors means that network by extending the network edge inside the hypervisor. Existing
networking now happens inside the server platform in software as well virtual switches allow programming of the network connectivity by
as extending the network reach beyond the physical uplink. How can creating connections between the physical NIC and virtual NIC. In ef-
you engineer a reliable and trustworthy network unless you can man- fect, todays virtual switches are virtual patch panels that simply con-
age the true network edge ? nect virtual NICs to the physical NIC installed into the server. Net-
working needs to move away from static placement like that. Today's
One of the most common network design principles of the last decade is so-called "virtual networks" are still dependent on physical devices in
to move complexity to the network edge and simplify the network core. the network the physical appliance has a virtual instance of itself, but
For example, in the mid 2000s, MPLS changed the WAN with label the network services are still delivered from the hardware appliance.
forwarding in the core while performing trac classification at the
edge. In virtual networking, we are seeing the same sort of transforma- If we upgrade the software switch from a simple physical network con-
tion, where the software switching in the hypervisor can perform com- nection that is shared among virtual machines to a highly functional
plex trac manipulation, and then forward across a simple network network device, and then add path forwarding to the software so that
core. To properly manage this functionality, software networking in the the server can switch frames and route packets in the server kernel, we
server hypervisor must become part of the overall network. end up with an active network device in the server that can make com-
plex forwarding decisions.
Consider how your network will change if the software network inside
the server becomes the new network edge and moves the physical net- For example, deploying firewalls in virtual contexts does not place the
work devices to the network core. Moving the access layer into the hy- firewall service close to the application itself. Designers still have to
pervisor gives control and visibility to the operator and the security build secure LAN and WAN connections to and from the virtual fire-
team. wall. Nuage Networks has designed a solution that can extend the net-
work services INSIDE the server.
This whitepaper is a sponsored introduction to Nuage Networks and
their unique approach to Software Defined Networking that bring to-
gether a complex software edge and simple core into a unified network
Controller Based Networking
infrastructure for both Enterprise and Service Provider environments. We need a new approach to operation and control of networking. Today,
it is possible for an engineer to manage a few hundred network devices
with an SSH client, a good diagram, an SNMP monitoring tool and ex-
perience. But what about tomorrow? Consider an Ethernet switch of
today connecting as many as 48 servers - one network device to man-

4
age. But what if each of those servers houses a network service for rout- While Network Management Systems (NMS) help with complexity,
ing, switching and firewalling that must be managed? Adding a network todays tools area based on aging protocols that lack feature richness
device to the server operating system results in an explosion of net- and flexibility. Certainly SNMP has been a successful tool over time,
work devices to be managed, meaning that CLI administration of those but the protocol has serious limitations. The data is not well defined,
devices becomes unrealistic. and SNMP MIBs are too frequently poorly designed and badly docu-
mented, making their use non-trivial. For these reasons, SNMP is often
To add to operational complexity, virtual machines can move between deployed as a read only" tool, its usefulness limited to statistics gath-
hypervisors at any time. The Network team cannot easily identify the ering and status monitoring.
sources and destination point of servers within the network when
those points are moving targets. To overcome these management challenges, the central component of
the Nuage Networks SDN solution is the VSD application. VSD is a
web-based, graphical console that connects to all of the dVRS nodes in
the network to manage their deployment and configuration. The VSD
module distributes the policies through a number of Nuage Virtual
Services Controllers (more on this later) to all of the dVRS nodes in the
network to manage their deployment and configuration.

For an example of VSDs management interface, lets examine the

screenshot on the next page. First, the Nuage Networks VSD Dash-
board shows the network adapter of the server as part of the overall
network configuration. Second, the domain is separated into policy
zones that each server is assigned to. Finally, because the controller
performed the configuration on all the dVRS agents and is notified
when changes occur, VSD can display the actual network configuration
of the virtual machine.

Theres more power to dVRS than just the tight integration with
VSD, though. Lets take a look.

Virtual Machines Move Between Hypervisors

Change Network Dynamic

5
 The Nuage
Networks VSD
displays
information about
virtual machines
The Value of dVRS
The dVRS approach oers a range of network services. SIMPLE
Uses standards-based protocols such as OpenFlow between the VSC controller and dVRS agents. CO R E ,
SMART EDGE
Uses flow routing to manage trac flows in the server.
Can perform trac load balancing through flow path management.
Based on popular & proven Open vSwitch software.
Performs routing locally in the hypervisor.
Performs packet filtering in the server. The data centre network has always focused
on using L2 Switching at the edge and L3
While development of dVRS continues, with features such as stateful firewalling & load balancing on
routing in the core because high speed rout-
the roadmap, the highlight feature is Distributed Virtualised Routing. This concept allows net-
work data to be routed at the edge of the network instead of being routed via the core through large ing was expensive & complex. But around the
hardware switches. To demonstrate how this works, we need to introduce the concept of a Tunnel mid-2000's, service providers network de-
Fabric. ployed MPLS to perform edge routing on PE
routers & perform label switching in the Core.
The Tunnel Fabric The advantages were to distribute complexity
In a traditional network design that connects virtual machines to the physical LAN, the network edge and simplify the core network so that stability
(usually a top-of-rack L2 switch) is connected to each of the hypervisors; hypervisors then connect to and performance was improved.
virtual machines using virtual NICs. This approach is in common use today and shown in the diagram
below. So why do Data Centre Networks have smart
cores & dumb edges ?

7
 the most ecient path between two hypervisors while maintaining se-
cure multi-tenant separation and eliminating the requirement for a
complex separation protocol.

Basic Physical Network Connectivity

Distributed Routing with a Tunnel Fabric from dVRS
While satisfactory for simple installations, there are a number of tech-
dVRS endpoints can map directly to each other in this way because the
nical limitations with this approach. Multi-tenant trac flows require
VSC models a complete network topology, including knowledge about
complex technologies to maintain separation, e.g. MPLS, VRF-Lite or
all endpoints in the network. An algorithm determines the outbound
PVLAN. Another issue is that security gaps in the virtual switch require
paths for each dVRS in the network and encapsulate the flows into a
expensive audits to ensure policy is being enforced. And of course,
tunnelling protocol such as the industry-standard VXLAN which is op-
theres no visibility of virtual machine trac that is switched inside the
timised for high performance in multipath LANs. MPLS over GRE is
hypervisor.
used for connections to WAN edge devices.
We can update the traditional model and overcome these limitations by
using a tunnelling protocol to connect the dVRS switches together.
Thus, both L2 and L3 flows moving through the dVRS device can be en-
capsulated from one dVRS device to another or sent directly to the
physical network if required. In this way, virtual machine trac follows

8
 Distributed Routing
Each dVRS routes trac into the network according to its flow table.
Therefore, the entire dVRS system performs routing at the edge of the
network. Distributed routing is like an ultimate "trac engineering"
setup where routing CPU load is distributed to a large number of de-
vices, the routing complexity is managed by a single controller, and the
entire data plane routes the shortest path across the underlay network.

The Tunnel Fabric

Replacing an existing physical network is enormously challenging for

most organisations, due to the capital costs and business risk of a net-
work refresh. Using tunnels between the dVRS means that the existing
IP/Ethernet Fabric in your data centre needs no upgrades or changes.
In the modern ITIL compliant organisation, network change remains
enormously dicult since existing network protocols are not designed
for rapid change or capable of reliable reconfiguration.

It should also be noted that tunnelling protocols use minimal resources

in the Data Centre LAN. To the physical network, tunnel packets are Distributed Virtual Routing
simply UDP streams that consume a handful of TCAM entries in core
A dVRS cant make a forwarding decision in a vacuum, as events in the
switches. The dVRS implementation of VXLAN does not require an IP
underlying physical network must be considered. Nuage Networks has
Multicast PIM-BiDir or PIM-SM configuration like VMwares vCloud.
extensively considered how to provide the VSC controller with all the
Features like anycast replication and local acknowledgement to cope
information required to have a complete model of the network. The
with broadcast, unknown unicast, and multicast (BUM) trac handle
VSC implements an OSPF, IS-IS or BGP listener to monitor the state of
the issues addressed by multicast in other VXLAN implementations.
the physical network. Therefore, if routes starts flapping, the VSC is
able to incorporate those events into the decision tree.

9
Multiple Controllers, Multiple Data Nuage Networks delivers an site to site SDN strategy by integrating
with the existing equipment in your network and utilises existing
Centers and Multivendor MPLS WAN services with little change to your existing network.

One of SDNs fundamental questions is that of scale. In how large of a

network can a unified SDN environment function? The Nuage Net-
works solution is designed with the capability of growing to the mas-
sive scale required by cloud providers and service providers.

Within a single data centre, scale is handled by multiple VSC control-

lers, each handling a group of dVRS devices. To scale between multiple
data centres, VSC controllers build on the single data centre scale
model by horizontally connecting controllers at the top of the hierar-
chy.

As shown in the diagram on the left, VSC controllers are synchronised

using MP-BGP. A BGP connection peers with PE routers at the WAN
edge, and then the VSC controller uses MP-BGP to synchronise con-
troller state & configuration with VSCs in other data centres. This is
vital for end-to-end network stability. The VSC controllers must run
at the same critical level as BGP routing updates to ensure coherence
between the data centres - controller synchronisation is as vital as
BGP peering updates. BGP is well-designed to transfer a large volume
of structured data - that's what Internet routing tables are.

When dVRS devices are communicating to non-local dVRS devices,

data is tunnelled in MPLS-over-GRE to the PE router. Modern routers
from most vendors support GRE termination in hardware, and this
ensures low latency and immediate compatibility with existing net-
work equipment in the WAN. The VSC controller will be communicat-
ing with the PE using BGP to establish the MPLS path and monitor
the circuit establishment.

10
The VSD Policy Controller moves from one dVRS to another
dVRS as part of the server move, the
We have covered the data and control plane of the Nuage Network
VSP platform where the dVRS agent creates a network edge that de-
policy template is inherited at the new
location with attributes like MAC Ad- P O L I CY
S E R VE R S
livers new services in the hypervisor and the VSD controller provides dress, QoS, VLAN membership, moni-
operational control of the elements. But there is more to uncover in toring data, etc. This design is derived
the VSD Controller. from existing software handling large
3G and DSL networks for a proven ap- In 3G & DSL networks, an end-
The "VSD Controller" consists of the Virtualised Services Directory proach to scaling and flexibility.
point connects to a central server
(VSD) application and Virtualised Service Controller (VSC) SDN
controller. A second benefit of a policy driven for login, authentication, authori-
configuration is that the local node sation and network parameters. A
only contains configuration that is di- customer inherits the final con-
rectly relevant. A BGP-enabled figuration after the server has
Internet-facing router must house
analysed policies such as physical
enough compute resources be able to
hold 400K route entries in memory location, account standing, device
and in the TCAM table, even if only type, network status. These poli-
forwarding trac for 1,000 routes. In a cies combine to build the final
VSD system, the exact flow rules to configuration in a completely
handle the trac that will reach the
flexible way.
dVRS are calculated and distributed.
As a result, the CPU / Memory con-
summating is much reduced. There
are less mature vSwitch solutions that
require up to 2 Gigabytes of memory to
hold a very large and mostly unused flow table.

The policy derived configuration allows for mobility, reduced re-

The Virtualised Services Directory is a policy engine; configuration of source consumption and improved application control. Its a key dif-
the dVRS is derived from a series of templates. For example, the vir- ferentiator from other SDN applications.
tual port is a policy element with a unique identifier that applies to the
server template AND the dVRS template. When the virtual port

11
Seven Pillars of Cloud Network Auto- The software network also scales since the state/configuration in the
dVRS is minimised by the VSD controller using policy templates to lo-
mation calise the flow tables.

The premise of cloud networking is to achieve a network that is pro-

L2 Isolation
grammable, flexible and valuable. Nuage Networks promotes this vi-
sion with "Seven Pillars of Network Automation" that are instantly Securing a Layer 2 network is complex. Extending multiple VLANs
recognisable as key elements that solve challenges in Software Defined across an Ethernet trunk to a virtual host is an unmanaged security
Networking. The seven pillars are: risk with loss of control and possible VLAN bypass attacks. With dVRS
agents installed to the hypervisor, trac flows are managed end to end
L3 Scalability - right up to the server network adapter. This complete isolation of
L2 Isolation Layer 2 is inherent in a flow managed network and dramatically re-
L2-L4 services duces the operational cost of the network.
VPN Compatibility
SDN Programmability
Policy Management L2-L4 Services
Performance Monitoring
We described flow forwarding at the network edge using a extended
definition. Today, network forwarding uses only the destination ad-
L3 Scalability dress. The VSD will define flows on source and destination IP address,
source/destination TCP port, VLAN tags and more. The result is appli-
L3 Scalability refers to the state held in the physical network of the
cation aware forwarding. This is further enhanced with the VSD end to
Data Centre and the WAN. Each IP Route and MAC Address consumes
end view of the network and the servers within its own ecosystem.
CPU & Memory resources in physical routers and switches. Creating a
Tunnel Fabric means that only the IP and MAC Address of tunnel end-
VSD has the ability to "know more" about payloads and application in-
points are visible to the physical network. Because the forwarding oc-
tention and allow the operator to build new network services. The pol-
curs at the edge of the network, network paths are optimal from the
icy derived configuration provides granular control and configuration
edge, through the core, and even over the WAN.
of L4 services that move with the VM.
The physical network does not hold state regarding the endpoints.
Thus, additional tenants do not consume MPLS VRFs, BGP routes or VPN Compatibility
TCAM resources across the backbone.
A VSD can operates directly with MPLS VPN and allows interoperabil-
ity with existing WAN services. This provides for Data Centre Inter-
connection at Layer 2 and Layer 3 between customer owned facilities,

12
but also provides for Hybrid Cloud connectivity to external cloud serv- that are vital in meeting the diverse requirements of a multi-tenant
ices. Nuage dVRS agents in an external cloud can be integrated into data centre. Deriving profit from a cloud often means reliable support
single coherent network that spans an MPLS backbone. Consider an for a diverse of range of dierent requirements.
MPLS path between multiple data centres that can support VM migra-
tion and recovery from a single application platform. When the server Performance Monitoring
team relocates a VM, the network operator will have a visual display of
the network configuration from the VSD web interface. The VSC Controller will poll performance and status information
from the dVRS agent and show the statistics and graphs of the current
Combined with the flow monitoring capabilities, you can gain real status that is roughly equivalent to physical switch port monitoring.
visibility of end-to-end performance because the dVRS has visibility of The diagram below show the utilisation of an entire subnet. In a tradi-
the entire flow at the server ingress. tional network, this would require a significant investment in sFlow
collectors and analysers to crunch this type of data.
SDN Programmability
Remember that the performance data is collected by a unique object in
Nuage Networks has developed an extensive set of the REST APIs for the VSD architecture. Even though a server or VLAN moves within a
the VSC controller to support SDN programmability to external re- single data centre, or between data centres, the data is still presented
sources. An SDN platform is not self contained and must connect to from a single interface.
other orchestration platforms and services. The VSC controller is ar-
chitected to add new APIs as SDN Networking
develops interoperability standards over the
coming years. OpenStack is already supported.

Policy Management
A VSD uses a number of administrative abstrac-
tions to, perhaps for the first time, apply policy
to network in virtualised platforms. This VSD
policy engine allows for flexible configuration
tools in the web configuration. For example,
master templates allow for baseline tenant setup
and then creation of per-tenant policies. Con-
structs like Domains, Shared Domains and
Zones allows for flexible configuration options

13
SDN is arriving
The Packet Pushers have been discussing the possibilities of SDN for the last two years. Weve speculated, discussed, wondered and dreamed about
what we would need in an SDN solution while knowing what programmatic networking could do. When you run down the capabilities of the Nuage
Network VSP product, its hard to find anything missing. Lets start at the top.

The VSD policy & analytics engine presents a unified web interface where configuration and monitoring data is presented. The VSD is API-
enabled for integration with other orchestration tools. Alternatively, you can develop your apps. Either way, the VSD is based on tools from the
service provider world, and therefore scaling potential looks very good. It integrates multiple data centre networks by linking VSDs together and
exchanging policy data (not configuration data).

The VSC also addresses scaling - you can have multiple VSC controllers per data centre
to meet your performance requirements and uses the same operating system used in
Alcatel-Lucent Service Routers today. Nuage Networks has chosen to use standards-
based protocols like OpenFlow and MP-BGP where practical, and gave us a verbal com-
mitment to use open standards where possible and practical in the future.

The dVRS network agent addresses many of the known issues when using software
switching and tunnels. Unique configuration per agent means better performance while
consuming less CPU/Memory in the hypervisor. dVRS avoids the IP Multicast require-
ment in the network core with smart features (that will certainly need some proving).
Finally, the dVRS agent uses the VSD policy configuration to deliver real network serv-
ices to applications - security through edge filtering, flow balancing, Layer 3 routing at
network ingress and even plain L2 Switching by simple path selection.

Finally, you can get started on your existing network by simply installing three virtual
machines for the Nuage Networks VSP and few more to be hypervisors. Your existing
network needs zero configuration changes to get started in a single data centre.

This whitepaper is sponsored by Nuage Networks, but we still say that they have delivered on much of SDNs promise. Its hard not to be excited
about the positive changes in networking that SDN is making. Nuage Networks has a product that you should add to the very short SDN list of solu-
tions available today for your network strategy in the years to come.

14
 Packet Pushers Interactive LLC (US)
Thropos Ltd (UK)

http://packetpushers.net
All Rights Reserved 2013

15