Cisco Webex Teams

Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session

How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

cs.co/ciscolivebot#BRKCRS-2117

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
 BRKCRS-2117

Cisco SDWAN Design &

Deployment

Steven Wood - Principal Engineer – Enterprise Networks

David Prall – Principal Systems Engineer – Enterprise Networks
Agenda
• Introduction
• Network Architecture
• Controller Design
• Routing & Site Design
• Policy Design
• Resiliency Considerations
• Summary

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
About the jargon…

vEdge – vEdge Router

i.e. an SDWAN router
cEdge – ISR/ASR Router

vSmart - controller vBond - orchestrator

vManage – Management Application

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Cisco SD-WAN Solution Overview
Applying SDN Principles To The Wide Area Network
vManage

Orchestration Plane vBond

vSmart

MANAGEMENT
vBond

Management Plane API vEdge

(Multi-tenant or Dedicated)

ORCHESTRATION ANALYTICS

Control Plane
(Containers or VMs)
CONTROL

Secure DTLS Control Channel

Secure IPSEC Data Channel INET MPLS 4G

Data Plane
(Physical or Virtual)

Data Center Campus Branch Home Office

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
 Orchestration Plane
vBond Orchestrator
vBond
Main
MANAGEMENT Characteristics
API • Orchestrates control and
ORCHESTRATION ANALYTICS management plane
• First point of authentication
• Distributes list of vSmarts/
vManage to all vEdge routers
CONTROL
• Facilitates NAT traversal
• Requires public IP Address
Secure IPSEC Data Channel INET MPLS 4G
Secure DTLS Control Channel
[could sit behind 1:1 NAT]
• Highly resilient
• Multitenant or single tenant

Data Center Campus Branch Home Office

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
 Management Plane
vManage
vBond Main
Characteristics
MANAGEMENT
• Single pane of glass for
API Day0, Day1 and Day2
ANALYTICS
operations
ORCHESTRATION
• Centralized provisioning
• Multitenant or single tenant
• Policies and Templates
CONTROL
• Troubleshooting and
Monitoring
Software upgrades
Secure DTLS Control Channel
Secure IPSEC Data Channel INET MPLS 4G •
• GUI with RBAC
• Programmatic interfaces
(REST, NETCONF)
Data Center Campus Branch Home Office • Highly resilient

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
 Control Plane
vSmart Controller
vBond
Main
MANAGEMENT Characteristics
API • Facilitates fabric discovery
ORCHESTRATION ANALYTICS • Disseminates control plane
information between vEdges
• Distributes data plane and app-
aware routing policies to the
CONTROL vEdge routers
• Implements control plane policies
Secure IPSEC Data Channel INET MPLS 4G
Secure DTLS Control Channel
• Dramatically reduces control
plane complexity
• Highly resilient

Data Center Campus Branch Home Office

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
 Data Plane
vEdge Router Main
Characteristics
vBond
• WAN edge router
MANAGEMENT • Provides secure data plane with
remote vEdge routers
API
• Establishes secure control plane
ORCHESTRATION ANALYTICS with vSmart controllers (OMP)
• Implements data plane and
application aware routing
policies
CONTROL
• Exports performance statistics
• Leverages traditional routing
protocols like OSPF, BGP and
Secure DTLS Control Channel
Secure IPSEC Data Channel INET MPLS 4G
VRRP
• Support Zero Touch
Deployment
Data Center Campus Branch Home Office • Physical or Virtual form factor
(100Mb, 1Gb, 10Gb, 20Gb+)

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Network Architecture
Typical SDWAN Deployment Architecture
Private Cloud Site Enterprise Controllers Virtual Private Cloud SaaS

App
Servers

SDWAN Servers
VPC VPC
Headend
VPC VPC
Distro
Switch

V V
CE
Routers

MPLS1 INET

V = Virtual Router

Single
Legacy Dual Router
Router
Branch Branch
Branch

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Controller Deployment
Cloud-Delivered Control
Flexible Deployment Options
Cisco Cloud Ops MSP Ops Team Enterprise IT

Deploy Deploy Deploy

vManage vManage vManage

Recommended

DTLS DTLS DTLS

Or TLS Or TLS Or TLS
Connections Connections Connections

vSmart vBond vSmart vBond vSmart vBond

Cisco MSP Private
Cloud Cloud Cloud

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Controller Deployment
NIC0 NIC1

 Cloud or on premise
deployment
 Separate interfaces for control
VPN0 VPN512 and management
vBond
 Separate VPNs for control and
management
Control Management
Interface Interface
- Zone-based security
 Minimal configuration for bring-
up
ESXi, KVM, AWS, MS Azure - Connectivity, System IP, Site
ID, Org-Name, vBond IP (local)

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Controller Deployment
NIC0 NIC1

 Cloud or on premise
deployment
 Separate interfaces for control
VPN0 VPN512
vSmart
and management
 Separate VPNs for control and
management
Control Management
Interface Interface - Zone-based security
 Minimal configuration for bring-
up
ESXi, KVM, AWS, MS Azure - Connectivity, System IP, Site
ID, Org-Name, vBond IP

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Typical Controller Deployment
NIC0 NIC1 NIC2

 Cloud or on premise deployment

 Separate interfaces for control and
management
VPN0 VPN512  Separate VPNs for control and
vManage
management
Control Mgmt
Interface Interface - Zone-based security
 Internal Cluster I/F for vManage
Cluster
Interface instance clustering
(vManage
Only)
 Minimal configuration for bring-up
ESXi, KVM, AWS, MS Azure - Connectivity, System IP, Site ID,
Org-Name, vBond IP (local)

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Controller Communication Principles
• The vBond is a special control element because it acts as a STUN server
for the network to allow vEdges to sit behind NAT devices
• To work properly, the other control elements (vManage, vSmart) need to
communicate to vBond through NAT as well.
• vSmarts and vManages can communicate with each other either through
NAT or un-NATed connections
• NAT must be 1:1 with no PAT
• The choice of deployment model is dependent on security posture vs
network complexity tradeoff.

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Significance of Interface (TLOC) Color
Private Colors
• Color is an abstraction used to identify individual WAN Metro-ethernet
transport as PUBLIC or PRIVATE mpls
private1
private2
• Colors are KEYWORDS not free form LABELS private3
private4
• Used for automation and policy writing private5
private6
• “Color” dictates the use of private-ip vs public-ip (dest) for
Public Colors
Tunnel Establishment when there is NAT present
• Example: 3g
lte
• If two ends have a private color: private IP address/port used for biz-internet
DTLS/TLS or IPSec public-internet
blue
• If endpoint has public color: Public IP is used for DTLS/TLS or IPSec green
red
gold
silver
bronze

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Controllers Public Cloud Deployment
Controllers Communication
• 1 (2)
(1) 2 vSmart and vManage point
to the vBond IP address
1 - NATed public IP address
• 3 vBond learns interface private
(3)
2 4
Public and NATed public IP address of
1:1 1:1 1:1
Cloud NAT NAT NAT vSmart and vManage
- Private is pre-NAT, public is post-
NAT
• 4 vSmart and vManage use NATed
(4)
public IP addresses for
3 communication
- vSmart and vManage use public color
vBond vSmart vManage (default)
- Public color to public color uses
Public IP address (post-NAT) Private IP address (pre-NAT) Public Color public IP address

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
 Controllers Public Cloud Deployment
1 vEdge 1 vEdge points to the vBond NATed
• (1)
public IP
- vBond NATed public IP address is
MPLS
2 INET reachable through MPLS (direct or via
Data Center) and Internet transports

Data Center
2 vEdge communicates with vSmart
• (2)
and vManage using NATed public IP
1:1 1:1 1:1
NAT NAT NAT address
Public Cloud
- Private color to public color uses public
IP address, public color to public color
uses public IP address
- vSmart and vManage NATed public IP
Public Color
addresses are reachable through MPLS
Private Color vBond vSmart vManage (direct or via Data Center) and Internet
Public IP address (post-NAT) Private IP address (pre-NAT)
transports
BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
On-Prem Controllers Hybrid Deployment
Controllers Communication • 2 vSmart and vManage point to
1 (2)
(1)
the vBond IP address
- NATed public IP address
1 • 3 vBond learns interface private
(3)
2 and NATed public IP address of
DMZ 1:1 1:1 1:1 vSmart and vManage
Firewall NAT NAT NAT
- Private is pre-NAT, public is post-
NAT
4
• 4 vSmart and vManage use
(4)
interface private IP addresses for
3 communication
- vSmart and vManage use private
color (non-default)
vBond vSmart vManage
- Private color to private color uses
private IP address
Public IP address (post-NAT) Private IP address (pre-NAT) Private Color

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
On-Prem Controllers Hybrid Deployment
1 • 1 vEdge points to the vBond FQDN that
(1)
vEdge resolves to both public and private IP
MPLS
addresses
2 Private IP Public IP
Data MPLS
Center INET
Core Internet
Switch

1:1 1:1 1:1 • 2 vEdge communicates with vSmart and

(2)
DMZ
NAT NAT NAT
vManage NATed public IP addresses over
Firewall Internet and interface private IP addresses
over MPLS
- Private color to private color uses private IP
address, private color to public color uses
public IP address
vBond vSmart vManage
Public IP address (post-NAT) Private IP address (pre-NAT) Private Color
Public Color

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
 Firewall Rules for On-Prem Controllers

UDP/12346-12445 UDP/12346-12445
UDP/12346 UDP/12346 UDP/12346-13065 TCP/23456-23555 TCP/ Ephemeral UDP/12346-13065

vBond vSmart vManage

UDP/12346 UDP/12346-13065 UDP/12346-13065

TCP/23456-24175 TCP/23456-24175

UDP/12346-12445
TCP/Ephemeral UDP/12346-12445
UDP/12346-12445 TCP/Ephemeral

Firewall DMZ
WAN Edge
BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Example Controller System Deployment
Service Side: Mgmt subnet OOB
VPN 512 Management Access Mgmt
VPN512 VPN512 VPN512
CSP5216
3 VM Redundant vM vS vB
DMZ
FW
(3x)
Cluster VPN0 VPN0 VPN0
Public IP Address
Transport Side: Tunnel
Transport subnet Private IP Address
I/Fs on VPN0

Redundant NAT/FW NAT NAT

1:1 1:1 vM vS vB Controller
Public IPs

DMZ SDWAN SDWAN

SDWAN Headend
MPLS Hub A SR1002X
Internet Hub
A SR1002X

Transport CE Termination
A SR1002X A SR1002 A SR1002 A SR1002 A SR1002 A SR!002X

or
vM vS 0/0
Public IPs or Default Route vB vM vS

MPLS INET
Same Design
in backup DCs

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Controllers Connectivity and Scale
5400
vBond vSmart Con* vManage
1500 1500 1500 2000 2000 2000
Con Con Con 5400 5400 Dev Dev Dev
Con* Con*
x6 x6
x20

FQDN Networked Cluster

1 permanent connection
per-transport per vSmart
DNS Up to max-control-connections
Hash

1 transient connection 1 permanent connection

WAN Edge * 8 Core with minimum 17.1 code

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Onboarding
 Certificate-Based Trust
Administrator • Bi-directional certificate-based trust between all
Signed
Defined
vEdge List elements
Controllers
- Public or Enterprise PKI
vManage • White-list of valid vEdges and controllers
- Certificate serial number as unique identification

vBond vSmart

vEdge

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Controllers Identity and Trust
Device Certificate • Controller identity is provided by the digicert
signed certificate (1-2 yr lifetime)
Root
Identity - Alternatively can use Enterprise CA. Requires
Enterprise Root cert on all other controllers
and vEdge routers

• Avnet Root chain to authenticate vEdge

Root Chain Root Chain
routers
Root
- Embedded into Viptela software
• Viptela Root chain to authenticate vEdge
Root Chain Trust Cloud routers
- Provided by vManage. Multiple if cluster.
• digicert Root chain to authenticate other
controllers
- Embedded into Viptela software
- Alternatively can use Enterprise Root chain
BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
vEdge Router Identity and Trust
• Each physical vEdge router is uniquely
TPM
Chip identified by the chassis ID and certificate
serial number
Identity • Certificate is stored in on-board Tamper Proof
Module (TPM)
- Installed during manufacturing (25 year lifetime)
Device Certificate
• Certificate is signed by Avnet root CA

Root Chain • digicert root chain of trust is used to validate

Control Plane elements
Root - Embedded into Viptela software
Trust
• Alternatively, Enterprise root chain of trust can
be used to validate Control Plane elements
- Can be automatically installed during ZTP

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
 vEdge Cloud ⟺ vBond, vSmart, vManage

Validate: Root trust, Validate: Root trust, Validate: Root trust, • vManage root cert is distributed to controllers
certificate serial certificate serial certificate serial
org-name org-name org-name • vManage issues certificate identity
• vBond, vSmart and vManage validate:
vBond vSmart vManage
- Trust for vEdge certificate root CA
Root Root Root
- Certificate serial numbers against authorized white-
list (from vManage)
- Organization name (received certificate OU) against
locally configured one
Signed
Signed
Signed
• vEdge validates:
- Trust for vBond, vSmart and vManage certificate
root CA
- Organization name (received certificate OU) against
vEdge
locally configured one
Root Signed • Persistent DTLS/TLS connection comes up
between vEdge and vSmart/vManage
DTLS - vEdge is a client
Validate: Root trust,
org-name DTLS/TLS

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
 Zero Touch Provisioning – SDWAN Router
Control and Policy
Zero Touch Provisioning
Elements
Server

2 3
5
1 Full Registration and
Configuration
4
Assumption:
• DHCP on Transport Side (WAN)
• DNS to resolve ztp.viptela.com/
devicehelper.cisco.com*
SDWAN Appliance
* Factory default config

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 Control Plane Sessions
• Secure Channel to SD-WAN Controllers. DTLS only
• Viptela Primitives
• Operates over DTLS/TLS authenticated and • Permanent
vManage
secured tunnels • Multiple Sessions

• OMP - between vEdge routers and vSmart vBond

controllers and between the vSmart controllers
• NETCONF – Provisioning from vManage. vSmart vSmart
Access via admin credentials over
authenticated tunnel.
• Internal control connections do not carry DTLS or TLS
HTTP/HTTPS primitives DTLS or TLS • Viptela Primitives
• Viptela Primitives • OMP
• No need for reverse proxy protection. SDN • NETCONF • Permanent
• Permanent • 1 session / vSmart / TLOC
Controllers are not webservers • Single Session
• vManage is the exception, but HTTP access
can be restricted to private access under DTLS Only
• Viptela Primitives
RBAC • Temporary

vEdge

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Routing Design
 Understanding SDWAN Routing
vEdge vSmart vEdge
Registration Update Registration Update
A T1 T2 T3 B T1 T2 T3

System IP System IP
10.0.0.1 B T1 T2 T3 A T1 T2 T3 10.0.0.10

A B

SITE 1 vpn1 vpn1 SITE 10

PREFIXES Overlay Network (OMP) PREFIXES

TLOCs SP Peer or
GW Router
SP Peer or
GW Router
TLOCs
Default Default
T1 T1
Default Default
INET MPLS 4G
T2 T2
Def Def
vpn0 T3 T3 vpn0
Underlay Network (e.g. BGP)
Site ID 1 Site ID 10

Underlay Peering Underlay Peering

(BGP or Def GW) (BGP or Def GW)
BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
SDWAN Fabric Terminology
• Overlay Management Protocol – Control plane protocol distributing reachability,
security and policies throughout the fabric
• Transport Locator (TLOC) – Transport attachment point and next hop route attribute
• Color – Control plane tag used for IPSec tunnel establishment logic
• Site ID – Unique per-site numeric identifier used in policy application
• System IP – Unique per-device (vEdge and controllers) IPv4 notation identifier. Also
used as Router ID for BGP and OSPF.
• Organization Name – Overlay identifier common to all elements of the fabric
• VPN – Device-level and network-level segmentation

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
 Overlay Management Protocol Overview
• TCP based extensible control plane protocol
vSmart2
• Runs between vEdge routers and vSmart
controllers and between the vSmart
controllers
- Inside permanent TLS/DTLS connections
- Automatically enabled on bringup
vSmart1 vSmart3
• vSmarts create full mesh of OMP peers
• Distribution of data-plane security parameters
and policies
• Implementation of control (routing) and VPN
vEdge vEdge membership policies
OMP Peers

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
 Overlay Routing: Service Routes
• Service routes are routes
originating from outside the
Dynamic (OSPF/BGP)
Dynamic (OSPF/BGP) SDWAN
Static
Static
Connected
Connected • i.e. “LAN” routes
Site2 • Redistributed to/from OMP
Site1 vSmart
Overlay • OMP learns and translates
Management routing information across the
Protocol overlay
Site3
- OMP routes, TLOC routes,
Site4
network service routes
Connected
Connected - Unicast and multicast address
Static
Static families
Dynamic (OSPF/BGP)
Dynamic (OSPF/BGP) - IPv4 and IPv6 (March 2019)

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Overlay Routing: OMP Routes
• Routes learnt from local service
side
vSmart
• Advertised to vSmart
controllers
MPLS INET • Most prominent attributes:
OMP Update - TLOC
- Site-ID
vEdge - Label
- Tag
- Preference
Connected
Service - Originator System IP
Static Side - Origin Protocol
- Origin Metric
Dynamic (OSPF/BGP)
- AS PATH

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Overlay Routing: TLOC Routes
• Routes connecting locations to
physical networks
vSmart • Advertised to vSmart
controllers
OMP Update

MPLS INET • Most prominent attributes:

- Site-ID
- Encap-SPI
TLOCs - Encap-Authentication
vEdge
- Encap-Encryption
- Public IP
Connected - Public Port
Static
- Private IP
- Private Port
Dynamic (OSPF/BGP) - BFD-Status
- Tag
- Weight
BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
 Transport Locators (TLOCs)
TLOCs vSmart
vSmarts advertise TLOCs to
vEdges in TLOC routes

SD-WAN Fabric TLOCs advertised to

with TLOCs as vSmarts in TLOC routes
tunnel endpoints vEdge

IPsec
IPsec Local TLOCs
IPsec (System IP, Color, Encap)
MPLS INET

vEdge vEdge

vEdge vEdge

Transport Locator (TLOC) OMP IPSec Tunnel

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
 Transport Colors
T3 T4 T1 T2
T3 T4
Internet1 T1 T2 Internet
T1 T3
T3 vEdge vEdge
T1
vEdge vEdge T2 T4
T2 T4
MPLS

T1, T3 – Internet Color T2, T4 – MPLS Color

Internet2
T1, T3 – Internet1 Color T2, T4 – Internet2 Color
T1 T3 T2 T4
T1 T3 T2 T4
T1 T4 T2 T3
T1 T4 T2 T3
Color restrict will prevent attempt to establish IPSec tunnel to
TLOCs with different color

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
 SysIP: 10.0.0.1
TLOC Color: Internet

Significance of TLOC Color

Encap: IPSec

IPSEC Public
G0/0 WAN

• Color is an abstraction used to identify individual WAN

transport G0/1 Private
IPSEC WAN

• Colors are KEYWORDS not just LABELS System IP

10.0.0.1
• Policy is written based on these SysIP: 10.0.0.1
TLOC Color: MPLS
Encap: IPSec
• TLOC maps to a physical WAN interfaces
Private Colors Public Colors
• “Color” dictates the use of private-ip vs public-ip (dest) for Metro-ethernet 3g
Tunnel Establishment when there is NAT present mpls lte
private1 biz-internet
• Example: private2 public-internet
private3 blue
• If two ends have a private color: private IP address/port used for private4 green
DTLS/TLS or IPSec private5 red
• If endpoint has public color: Public IP is used for DTLS/TLS or IPSec private6 gold
silver
bronze

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Site Design &
Deployment
Deployment Sequence

Controllers Datacenters Branches

vManage

Data Center SOHO

A

Data Center
vSmart vBond B Branch Campus

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Deploying SDWAN – Routing Overview
Data Center Core Cloud Controllers
DC Routes DC Routes
0/0 0/0

OSPF/BGP/EIGRP*
• Transition hubs
• Overlay Site Routing
• Legacy Site Routing
SD-WAN • DC Access

SD-WAN
Tunnel

Tunnel
WAN • Hub & Spoke Migration
Approach
• Regionalization
Non-SDWAN
Networks SD-WAN *EIGRP support available March 2019
OSPF/BGP/EIGRP* Networks

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Data Center Routing

DC prefixes, 0/0, MPLS DC prefixes, 0/0 INET DC prefixes, 0/0

SD-WAN prefixes non-SDWAN prefixes non-SDWAN prefixes
CE
BGP OMP
Non-SDWAN DMZ
OMP SD-WAN prefixes
prefixes Firewall
SD-WAN prefixes
OMP-to-BGP
CE BGP-to-OMP
DC prefixes, 0/0,
SD-WAN prefixes DC prefixes, 0/0 WAN Edge
SD-WAN non-SDWAN
BGP prefixes prefixes
BGP
Non-SDWAN COLOR KEY:
prefixes Underlay Domain Advertisement
Overlay Domain Advertisement
Data Center Core

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Branch Migration – In place
Traditional SD-WAN

Local
MPLS MPLS INET prefixes
OMP
Remote
prefixes*
OMP-to-BGP/OSPF
Router WAN Edge
BGP/OSPF-to-OMP

L2 L2 Local
OSPF/BGP OSPF/BGP
prefixes
OSFP/BGP
Remote
L3 Switch L3 Switch prefixes*

* SD-WAN and non-SDWAN SD-WAN Tunnel

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
 In Place Migration - Redundant
Traditional SD-WAN

Local
MPLS INET MPLS INET prefixes
OMP
TLOC Remote
Extension prefixes*
OMP-to-BGP/OSPF
Router Router WAN Edge
BGP/OSPF-to-OMP

L2 HSRP L2 VRRP Local

prefixes
OSPF/BGP OSPF/BGP
OSFP/BGP
Remote
L3 Switch L3 Switch prefixes*

* SD-WAN and non-SDWAN SD-WAN Tunnel

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Topology – Regional Mesh
Region 1 Region 2 • M x Regions, N x Edges
1 1
2 2 • (N-1)¹ tunnel scale for intra-region
N N WAN Edge
T1
T1 T1
• 2*(M-1)² tunnel scale for border
T2 T2
5 5 WAN Edge
3 T2 3
4 4
• Doubled tunnel scale in case of
dual transports
1 2
• Deny regional TLOCs in WAN core
N 3
T1 T2
• Permit regional edge routes
through borders.
5 4
• Set next-hop to Border TLOC

Border WAN Edge

Region M ¹ Assumes single WAN Edge per-site
² Assumes dual border WAN Edges per-region

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
Device Template Structure
Site Router Configuration
• Specific to one device-type.
(e.g. vEdge100, ISR4321)

• Basic Information

• Transport & Management (WAN side)

• Transport VPN & Interfaces
• Management VPN & Interface
• Underlay BGP & OSPF configurations

• Service Side Configuration (LAN side)

• Routing – BGP, OSPF, PIM, NAT, DHCP
• Encaps – GRE, IPSec
• VPNs (i.e. VRFs)

• Additional Templates
• Housekeeping, logging servers, localized policy

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Centralized Device Configuration via Templates

• Centralized Feature Templates

• Configuration with variables
• Self-recover on misconfiguration

https://www.youtube.com/watch?v=4hMMfM8OsoY

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Frequently-used Feature Templates
For Your
Reference

• System - Configure basic system information, site ID, system IP, time zone, • OSPF (optional) - Configure router ID, distance, areas, OSPF interfaces,
hostname, device groups, GPS coordinates, port hopping, and port offset. reference bandwidth, default information originate, metrics, metric type, and SPF
timers.
• Logging - Configure logging to disk and/or to a remote logging server.
• VPN Interface configuration - Configure interface name and status, static or
dynamic IPv4 and v6 address, DHCP helper, NAT, VRRP, shaping, QoS,
• AAA - Specify authentication method, order. Configure Radius, TACACs, or local
authentication, including local user groups with different read/write permissions. ingress/egress Access Control List (ACL) for IPv4 and 6, policing, static Address
Resolution Protocol (ARP), 802.1x, duplex, MAC address, IP Maximum
Transmission Unit (MTU), Transmission Control Protocol Maximum Segment Size
• BFD - Specify BFD app-route multiplier and poll interval and specify the hello and (TCP MSS), TLOC extension, and more. In the case of the transport VPN, configure
BFD multiplier for each transport. tunnel, transport color, allowed protocols for the interface, encapsulation,
preference, weight, and more.
• OMP - Change graceful restart timers, advertisement and hold timers; change
number of paths advertised; configure AS overlay number; set local redistribution • VPN interface bridge (optional) - Configure layer 3 characteristics of a bridge
into OMP; and change the number of equal-cost paths installed in the router. interface, including IPv4 address, DHCP helper, ACLs, VRRP, MTU, and TCP MSS.

• Security - Change the rekey time, anti-replay window, and authentication types • DHCP server (optional) - Configure DHCP server characteristics, such as
for IPSec. address pool, lease time, static leases, domain name, default gateway, DNS
servers, and TFTP servers.
• Archive (optional) - Archive the full running configuration onto a file server within
a time period specified. • Banner (optional) - Configure the login banner or message-of-the-day banner.

• NTP (optional) - Configure NTP servers and authentication if required. • Policy (optional) - Attach a localized policy.

• VPN - Change ECMP hash, add DNS servers, advertise protocols (BGP, static, • SNMP (optional) - Configure SNMP parameters, including SNMP device name
connected, OSPF external) from the VPN into OMP, add IPv4 or v6 static routes, and location, SNMP version, views, and communities, and trap groups.
service routes, and GRE routes.
• Bridge (optional) - Define layer 2 characteristics of a bridge, including the VLAN
• BGP (optional) - Configure the AS number, router ID, distance, maximum paths, ID, MAC address aging, maximum MAC addresses, and physical interfaces for the
neighbors, redistribution of protocols into BGP, hold time, and keepalive timers. bridge

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Deployment Planning
Port Numbering, System IP

• Have a consistent Port Numbering scheme throughout the network

• Factory config specifies certain ports in VPN 0 for DHCP to drive ZTP
• Be sure this port has reachability to DHCP and DNS servers
• System IP – Persistent IPv4 Address
• Uniquely identifies the device independently of interface addresses
• Choose a hierarchy that follows Site ID or other logical scheme
• Does not need to routable or advertised but…
• Used as a marker for control policy
• Best practice: advertise System IP as a source IP for SNMP/Logging correlation

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Deployment Planning
Site ID
• Site ID is a unique identifier for the
SITE location
• Value from 1 – 4294967295.
• Carried in OMP updates.
• Same ID for all vEdges residing in the
same site
• Site could be a branch, campus, DC
etc.
Site Types Definition
• Required attribute to join the overlay
Service-sites – eg. Firewall Type 1
• Choose Site ID allocation schema Sites with Direct Internet Access Type 2
carefully
Lower BW Sites Type 3
• Critical attribute use to scope Policy
application (site, range of sites, etc) Higher BW Sites Type 4

Hub sites Type 5

• No wild card support!

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Example Design For Your
Reference

HUB1-MPLS- HUB1-INET-
ASR1002HX-1 ASR1002HX-2
Site ID: 100 SiteID:
Org-name: 100
mycorp-61205 Org-name:
System IP:10.100.0.1 mycorp-
VPN 1 VPN 0
VPN 0 VPN 1 61205
SystemIP:
WAN IF: Int GigE 0/0/1 WAN IF: Int GigE 0/0/3:
10.100.0.2
WAN IP: 192.168.6.1 WAN IP: 192.168.146.10
Bandwidth Up: 1000 Mbps Bandwidth Up: 300 Mbps
Bandwidth Dn: 1000 Mbps INTERNET Bandwidth Dn: 300 Mbps
VRF default

MPLS VRF Internet Inside: 192.168.146.1

default Edge Public I/F: 172.16.146.1
Static NAT DMZ Default Gateway: 172.16.146.252
MPLS PE: 192.168.6.2

MPLS PE: 192.168.6.6

MPLS RS10-MPLS-4431-1 RS10-INET-4431-2 Internet Default Gateway: 172.16.102.8
MPLS VRF Site ID: 10 Site ID: 10
Org-name: Org-name: INTERNET
default
mycorp-61205 mycorp-61205 VRF default
WAN IF Int GigE 0/0 System IP:10.10.0.1 System IP:10.10.0.2
WAN IP: 192.168.6.5 VPN 1 VPN 0 WAN IF Int GigE 0/1
VPN 0 VPN 1 TLOC Extension
Color: mpls WAN IP: 172.16.102.7 (dhcp)
Bandwidth Up: 150 Mbps IF: ge3 Color: biz-internet
Bandwidth Dn: 150 Mbps tunnel interface ge3 Bandwidth Up: 150 Mbps
Bandwidth Dn: 150 Mbps

ge4 TLOC Extension

IF: ge4
tunnel interface
BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Policy Design
Cisco SDWAN Policy Framework
Centralized Authoring – Globally or Locally Significant

Network Policy

Control Data
Affects Control Plane Affects Data Plane

Centralized Localized Centralized Localized

Affects Routing Route policy in Site- Affects Data traffic Affects a Single Interface
Network-Wide Local Network Network-wide on a Single Router

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Centralized Policy
Centralized Policy refers to policy provisioned on vSmart controllers. There are two
types:
— Control Policy: Affects the routing in the overlay network-wide
Filter or modify information that is stored in the vSmart controller routing table
Filter advertisements made by the vSmart controller to the dataplane elements.
Control Policy is always on the vSmart controller. Never pushed to the vEdge routers

— Data Policy: Affects the flow of traffic throughout the VPN segments in the network
Apply to the flow of data traffic
Permit or restrict access based either on a 6-tuple match or on VPN membership
These policies are pushed to the affected vEdge routers

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Localized Policy
Localized Policy refers to policy provisioned on dataplane routers. There are two types:
— Local Control Policy: underlay route policies on the service or transport networks
Implement traditional BGP or OSPF routing behaviours required to interface to the service or
transport networks at the local site.

— Local Data Policy: Affects the flow of traffic throughout the VPN segments in the
network
Access lists applied to a specific interface(s) on the router.
Simple access lists permit and restrict access based on a 6-tuple match.
Access lists for class of service (CoS) marking, queuing, policing, route-mapping and SPANning,
Control how data traffic flows out of and in to the router's interfaces and interface queues

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Policy Distribution
vManage

Centralized Control Policy

(Fabric Routing)
Centralized Data Policy Local Control Policy
(Fabric Data Plane) (OSPF/BGP)
Centralized Localized
Centralized App-Aware Policy Policies Policies Local Data Policy
(Application SLA)
(QoS/Mirror/ACL)
VPN Membership
(Fabric Routing+Segmentation)

Centralized Data Policy Centralized App-Aware Policy

(Fabric Data Plane) (Application SLA)
vSmart
vEdge

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
vSmart Policy Construction
Lists Policy Definition Policy Application
• data-prefix-list – list of • App-route-policy is used together • Apply-policy is used in
prefixes for use with a data- with sla-classes for application- conjunction with a site-list to
aware-routing determine where policies are
policy
• Cflowd-template configures the applied
• Prefix-list – list of prefixes for cflowd agents on the vEdge nodes
use with any other policy
• Control-policy expresses OMP
• Site-list – list of site-ids for routing control
use in policy and apply-policy
• Data-policy provides vpn-wide
• Tloc-list – list of tlocs for use policy-based routing
in policy • Vpn-membership-policy controls
• Vpn-list – list of vpns for use in vpn membership across nodes
policy

Complete policy definition configured on vSmart and

enforced either on vSmart or on vEdge

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Policy Creation Workflow in vManage

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
High Availability and
Redundancy
 Site Redundancy - Routed
 Redundant pair of vEdge routers operate in
active/active mode
SD-WAN
Fabric  vEdge routers are one or more Layer 3 hops away
from the hosts
 Standard OSPF/BGP/EIGRP* routing protocols are
running between the redundant pair vEdge routers
vEdge A vEdge B and the site router
 Bi-directional redistribution between OMP and
OSPF/BGP and vice-versa on the vEdge routers
Site
Router
- OSPF DN bit, BGP SoO community

 Site router performs equal cost multipathing for

remote destinations across SD-WAN Fabric
*EIGRP support - Can manipulate OSPF/BGP to prefer one vEdge
Host March 2019 router over the other
BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
 Site Redundancy - Bridged
 vEdge routers are Layer 2 adjacent to the hosts
SD-WAN - Default gateway for the hosts
Fabric  Virtual Router Redundancy Protocol (VRRP) runs
between the two redundant vEdge routers
- Active/active when using multi-group (per-
VLAN)
vEdge A vEdge B
VRRP Active VRRP Standby  VRRP Active vEdge responds to ARP requests
VRRP
for the virtual IP with its physical interface MAC
address
- No virtual MAC

 In case of failover, new VRRP Active vEdge

router sends out gratuitous ARP to update ARP
table on the hosts and mac address table on the
Host intermediate L2 switches
BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
Transport Redundancy - Meshed
 vEdge routers are directly connected to all the
transports
- No need for L2 switches front-ending the vEdge
MPLS Internet routers
 When transport goes down, vEdge routers
detect the condition and bring down the tunnels
built across the failed transport
vEdge vEdge - BFD times out across tunnels
 Both vEdge routers still draw the traffic for the
prefixes available through the SD-WAN fabric
 If one of the vEdge routers fails (dual failure),
second vEdge router takes over forwarding the
traffic in and out of site
- Both transport are still available

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Transport Redundancy – TLOC Extension
 vEdge routers are connected only to their
respective transports
MPLS Internet  vEdge routers build IPSec tunnels across
directly connected transports and across the
transports connected to the neighboring vEdge
router
vEdge vEdge - Neighboring vEdge router acts as an underlay
router for tunnels initiated from the other vEdge

 If one of the vEdge routers fails (dual failure),

second vEdge router takes over forwarding the
traffic in and out of site
- Only transport connected to the remaining
vEdge router can be used

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Internet and MPLS TLOC-Extension (Static)

ISP A MPLS

Static
Route to T2
NAT: tloc-extension ge0/0
T4->T1 0/0 0/0
ge0/0 ge0/0
T1 T3
VPN0 ge0/1 0/0 ge0/1 VPN0

VPN0 T2 VPN0

VPN0 T4 VPN0
ge0/2 0/0 ge0/2

vEdge 1 vEdge 2
tloc-extension ge0/0

Note: vEdge router connected to ISP will perform NAT

on the traffic from the TLOC-extended interface Color biz-internet Color mpls

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
 Controllers Redundant Deployment
• Controllers are distributed across multiple
Public Cloud private data centers
Region A Region B • Active-active, vManage is cold-standby

Data Center A Data Center B

Export Import

• Controllers are distributed across multiple

public cloud regions Export Import
• Active-active, vManage is cold-standby

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
 Control Redundancy - vBond
Cisco Region A Region B
Cloud  vBond orchestrators have a whole view of the
devices allowed on the network
 Multiple vBonds can be deployed for Function
and Geographic redundancy

vBonds vBonds
 vEdges pick vBond via DNS

Control Plane
 DNS round-robin is utilized
Data Plane  vEdge will try first entry, then second, and so on
 vEdge to vBond connection
MPLS 4G
INET DNS  Temporary
Server
 Stateless
 vBonds maintain view of load on vSmarts
 Delivers list of vSmart’s to use
Branch Campus Data Center
BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
 Control Redundancy - vSmart
vSmart  vSmart controllers exchange OMP
Controllers messages and have identical view of the
Control Plane SD-WAN fabric
Data Plane
 vEdge routers connect to up to three vSmart
controllers for redundancy
Cloud
Data Center  Single vSmart controller failure has no
impact, other registered vSmart still available
 If all vSmart controllers fail or become
MPLS 4G
Data Center unreachable, vEdge routers will continue
INET operating on a last known good state for a
configurable amount of time (min of re-key
Small Office timer and GR timer)
Home Office
- No updates to reachability
Campus - No IPSec rekey
Branch - No policy changes propagation
BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
Controllers Regional Affinity (Optional)
AMER EMEA
vManage vBond Group2
Group1 vSmart

vSmart APAC
vBond vBond
Group3
Group 2,1
vSmart
FQDN AMER Group 1,2 FQDN EMEA

WAN Edge Group 3,2

WAN Edge FQDN APAC

WAN Edge

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
 Control Redundancy - vManage
DC1 DC2
vManage
Cluster
 vManage servers form a cluster for
Export Import
redundancy and high availability
Management Plane
 All servers in the cluster act as active/active
Data Plane
nodes
- All members of the cluster must be in the
Cloud same DC / metro area
Data Center
 For geo-redundancy, vManage servers
operate in active/standby mode
4G
Data Center - Not clustered
MPLS
- Database replication between sites is needed
INET

 Loss of all vManage servers has no impact

Small Office
Home Office on fabric operation
Campus
- No administrative changes
- No statistics collection
Branch

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
vManage Snapshots and Recovery
Best Practices
DC1
Snapshots
• vManage data volume periodically backed-up in
active DC. Period is based on change frequency.
• vSmart and vBond are stateless (state can be re-
created). No need to snapshot.
Recovery
• New volumes created from backed-up data
volume
• New vManage instances created and attached to
new volume (can be in cold standby – loaded and DC2
ready to start)

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
Closing
 Cisco SD-WAN Platform Options

SD-WAN + Branch Services SD-WAN

ISR 1000 ISR 4000 ASR 1000 vEdge 100 vEdge 1000 vEdge 2000

• 100 Mbps • Up to 1 Gbps • 10 Gbps

• 200 Mbps • Up to 2 Gbps • 2.5-200Gbps • 4G LTE & Wireless • Fixed • Modular
• Next-gen • Modular • High-performance
connectivity service w/hardware
• Integrated service
• Performance containers assist
flexibility
• Compute with UCS E • Hardware & software
redundancy

Virtualization Public Cloud

ENCS 5100 ENCS 5400

• Up to 250Mbps • 250Mbps – 2GB

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
Summary

Plan, Prepare, Educate

Leverage Cloud Controllers

Follow Migration Best Practice

Simplify the routing design

Embrace the flexible architecture

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
 Your SD-WAN learning map at CLEUR
Monday Tuesday Wednesday Thursday Friday
TECCRS-2014 BRKRST-2560
Deep Dive BRKRST-2559
Analytics / ML
On-prem
deployment BRKCRS-2117
BRKCRS-2112
Design
Serviceability
Deployment
TECCRS-2191 BRKCRS-2114
Deployment / BCP BRKCRS-2111 Security
Migration

TECSEC-2355
Security BRKRST-2558 BRKCRS-2113
BRKCRS-2110 SD-WAN as a Cloud on Ramp
The foundation Managed Service

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
 Cisco Webex Teams

Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session

How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

cs.co/ciscolivebot#BRKCRS-2117

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
Complete your online
session survey
• Please complete your Online Session
Survey after each session
• Complete 4 Session Surveys & the Overall
Conference Survey (available from
Thursday) to receive your Cisco Live T-
shirt
• All surveys can be completed via the Cisco
Events Mobile App or the Communication
Stations

Don’t forget: Cisco Live sessions will be available for viewing

on demand after the event at ciscolive.cisco.com

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
Continue Your Education

Demos in Meet the Related

Walk-in
the Cisco engineer sessions
self-paced
Showcase labs 1:1
meetings

BRKCRS-2117 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
Thank you